
making gpu's that go brrrrr @AgraniLabs
Joined December 2024
- Tweets969
- Following505
- Followers69
- Likes28K
from making gpu's go brrr to making gpu's that go brrr
I am joining @AgraniLabs as an Performance Architect Intern
1
4
213
Wow
The only thing faster than GB300 Dynamo + vLLM is Vera Rubin Dynamo + vLLM 🏎️💨.
The first DeepSeek V4.1 Flash results on @NVIDIA Vera Rubin NVL72 are here, powered by Dynamo + vLLM on @SemiAnalysis_'s AgentX benchmark.
Compared with GB300 NVL72 running Dynamo + vLLM, InferenceX's perf ruler shows:
• 63.4× total token throughput per chip at 400 tok/s/user
• 93.3× at 600 tok/s/user
1
106
Replying to @maharshii
You can look into sgl-nvidia-megakerles repo
Its a nice learning materal for PDL, they have got good gains
1
2
179
Replying to @AryaTschand
"and the insights should generalize to any hybrid transformer + MoE model architecture" -- why is that? can't think about it from first principles
20
claude --remote-control is the best thing this world has every produced; i can't switch to codex just because of this feature!
1
2
189
I can't believe the things happening to this world after corona
Still feels like a dream
NVIDIA announces CEO transition. Read the release: nvidianews.nvidia.com/news/n…
31
Replying to @cudagdb
Isn't that just a modern day megakernel
The thing which nvida-sglang-megakernel repo is doing
44
Replying to @TrijalSrivasta3 @cudagdb
Literally me after reading sglang PR which adds megakernel support for some chinese model to improve performance on inferncex
1
41
Replying to @gaunernst
Or you can ask agent to convert your nsys trace to json so you can seet in in perfetto
67
Replying to @AlpinDale
Noo.. they are underrated
These guys are starting to go from host side scheduling to device side scheduling, which is giving way too good results than it should
2
100
Replying to @gauravisnotme
I think thinking 1-2 years from now is a good starting point
Most of the abstractions will no longer be in the picture
144
This is not really the case when you inspect the execution of inference engines, things like tensor core, hbm util,... are mostly the bottlenecks
In FP8 and FP4, attention is no longer bottlenecked by matrix multiplication. It is bottlenecked by the transcendental and vector math of softmax and the on-chip memory traffic required to shuttle intermediate scores between execution domains:
(Article 🧵) vincentzed.com/writing/softm…
That playbook just hit a physical wall of its own making. By scaling matrix throughput 2× to 4× per generation while leaving Special Function Units (SFUs), vector ALUs, and shared-memory bandwidth behind, chipmakers inverted the attention roofline.
In FP8 and FP4, attention is no longer bottlenecked by matrix multiplication. It is bottlenecked by the transcendental and vector math of softmax and the on-chip memory traffic required to shuttle intermediate scores between execution domains:
When the non-matmul work outruns the matmuls, low-precision attention stops being a GEMM problem.
The industry already voted with production code: drop tokens across the sequence (DSA/QSA via a cheap ReLU indexer with 0 SFU calls), then run the surviving tokens as 100% dense tiles at peak low-bit Tensor Core speed.
1
103







