Darshan (@neuronfitting) header
Darshan (@neuronfitting) profile picture

making gpu's that go brrrrr @AgraniLabs

Joined December 2024
from making gpu's go brrr to making gpu's that go brrr I am joining @AgraniLabs as an Performance Architect Intern
1
4
213
Replying to @maharshii
Nvidia doesn't have fp6 cores.... Where is the speedup comming from?
34
Wow
The only thing faster than GB300 Dynamo + vLLM is Vera Rubin Dynamo + vLLM 🏎️💨. The first DeepSeek V4.1 Flash results on @NVIDIA Vera Rubin NVL72 are here, powered by Dynamo + vLLM on @SemiAnalysis_'s AgentX benchmark. Compared with GB300 NVL72 running Dynamo + vLLM, InferenceX's perf ruler shows: • 63.4× total token throughput per chip at 400 tok/s/user • 93.3× at 600 tok/s/user
1
106
Replying to @zephyr_z9
Nvlink is the moat
260
Replying to @maharshii
You can look into sgl-nvidia-megakerles repo Its a nice learning materal for PDL, they have got good gains
1
2
179
Replying to @AryaTschand
"and the insights should generalize to any hybrid transformer + MoE model architecture" -- why is that? can't think about it from first principles
20
claude --remote-control is the best thing this world has every produced; i can't switch to codex just because of this feature!
1
2
189
Darshan retweeted
A good reminder that just because you cuda doesn’t mean you shuda.
38
218
24
2,782
79,894
A man who pioneered parallel processing cannot be expected to run a single thread
261
2,779
459
31,790
1,058,151
Replying to @KheteshAkoliya
We'll mog chinese olympics
16
Public infra: Ahmedabad >>> Bangalore
1
2
99
Replying to @cudagdb
Isn't that just a modern day megakernel The thing which nvida-sglang-megakernel repo is doing
44
Literally me after reading sglang PR which adds megakernel support for some chinese model to improve performance on inferncex
1
41
Replying to @gaunernst
Or you can ask agent to convert your nsys trace to json so you can seet in in perfetto
67
Replying to @AlpinDale
Noo.. they are underrated These guys are starting to go from host side scheduling to device side scheduling, which is giving way too good results than it should
2
100
Replying to @gauravisnotme
I think thinking 1-2 years from now is a good starting point Most of the abstractions will no longer be in the picture
144
Had this saved, just in case
28
This is not really the case when you inspect the execution of inference engines, things like tensor core, hbm util,... are mostly the bottlenecks
In FP8 and FP4, attention is no longer bottlenecked by matrix multiplication. It is bottlenecked by the transcendental and vector math of softmax and the on-chip memory traffic required to shuttle intermediate scores between execution domains: (Article 🧵) vincentzed.com/writing/softm… That playbook just hit a physical wall of its own making. By scaling matrix throughput 2× to 4× per generation while leaving Special Function Units (SFUs), vector ALUs, and shared-memory bandwidth behind, chipmakers inverted the attention roofline. In FP8 and FP4, attention is no longer bottlenecked by matrix multiplication. It is bottlenecked by the transcendental and vector math of softmax and the on-chip memory traffic required to shuttle intermediate scores between execution domains: When the non-matmul work outruns the matmuls, low-precision attention stops being a GEMM problem. The industry already voted with production code: drop tokens across the sequence (DSA/QSA via a cheap ReLU indexer with 0 SFU calls), then run the surviving tokens as 100% dense tiles at peak low-bit Tensor Core speed.
1
103