Darshan (@neuronfitting) header
Darshan (@neuronfitting) profile picture

making gpu's that go brrrrr @AgraniLabs

Joined December 2024
Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)Photo by Darshan (@neuronfitting)
from making gpu's go brrr to making gpu's that go brrr I am joining @AgraniLabs as an Performance Architect Intern
1
4
213
Wow
The only thing faster than GB300 Dynamo + vLLM is Vera Rubin Dynamo + vLLM 🏎️💨. The first DeepSeek V4.1 Flash results on @NVIDIA Vera Rubin NVL72 are here, powered by Dynamo + vLLM on @SemiAnalysis_'s AgentX benchmark. Compared with GB300 NVL72 running Dynamo + vLLM, InferenceX's perf ruler shows: • 63.4× total token throughput per chip at 400 tok/s/user • 93.3× at 600 tok/s/user
1
106
claude --remote-control is the best thing this world has every produced; i can't switch to codex just because of this feature!
1
2
189
Darshan retweeted
A good reminder that just because you cuda doesn’t mean you shuda.
38
218
24
2,782
79,894
A man who pioneered parallel processing cannot be expected to run a single thread
261
2,779
459
31,790
1,058,142
Public infra: Ahmedabad >>> Bangalore
1
2
99
Had this saved, just in case
28
This is not really the case when you inspect the execution of inference engines, things like tensor core, hbm util,... are mostly the bottlenecks
In FP8 and FP4, attention is no longer bottlenecked by matrix multiplication. It is bottlenecked by the transcendental and vector math of softmax and the on-chip memory traffic required to shuttle intermediate scores between execution domains: (Article 🧵) vincentzed.com/writing/softm… That playbook just hit a physical wall of its own making. By scaling matrix throughput 2× to 4× per generation while leaving Special Function Units (SFUs), vector ALUs, and shared-memory bandwidth behind, chipmakers inverted the attention roofline. In FP8 and FP4, attention is no longer bottlenecked by matrix multiplication. It is bottlenecked by the transcendental and vector math of softmax and the on-chip memory traffic required to shuttle intermediate scores between execution domains: When the non-matmul work outruns the matmuls, low-precision attention stops being a GEMM problem. The industry already voted with production code: drop tokens across the sequence (DSA/QSA via a cheap ReLU indexer with 0 SFU calls), then run the surviving tokens as 100% dense tiles at peak low-bit Tensor Core speed.
1
103
The way these high interactivity point guys use software to manipulate hardware should be illegal My man.. you are using device side scheduling just to avoid PCIe transfer
ALERT🚨🚨: AMD MI355 is 1.42x FASTER THAN GB300 NVL72. THIS IS ACHIEVED USING THE TILERT SOFTWARE, WHICH GETS 1330 p90 tok/s/user on long context workloads! Thanks to AMD leadership for providing an MI350 cluster to TileRT maintainers, enabling them to achieve this great result!
30
We've got auto improving inference engines based on user traffic before GTA VI
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨ Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur. Together Compiler research highlights include: 1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems. 2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2. 3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation. 4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions. From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
31
I literally feel like a sorcerer when dealing with such tools. My imagination and understanding are the only bottleneck left. PS: dark arts is such a good reference, it literally feels like doing dark arts sometimes
Instead of fearing that AI will take your job, imagine and experiment with new kinds of sorceries and wild magic that is possible now. There will always be employers for those who know the dark arts.
1
48
*Performance Engineer
4
Agrani literally has the best hardware team in the country.
Replying to @kingofknowwhere
India is trying to build AI hardware for a billion people. Agrani is building custom GPUs, high-performance computing, AI software stack. This is what everyone will see as a frontend. (Similarly optoML has its specialized inference thing going on) Agrani has a bigger financial investment. Founded in 2024 by former Intel and AMD engineers, it wants to develop AI GPUs and the associated software stack for datacentre computing. An equivalent would be Huawei Ascend. Deepseek Flash was trained on this. Huawei Ascend 910 is basically what Agrani labs wants to get at. Which Huawei owns the entire stack including RISC-V called Kunpeng (india has Incore and Mindgrove trying to reproduce these too btw) Huawei CANN notes 4/n: xitter.gq/kingofknowwhere/status…
1
3
69
Word on the street: Agrani Labs has one of the absolute strongest teams in the Indian deep-tech scene right now
#MCExclusive 🚨 | Agrani Labs in talks to raise $100 million from Groww founders’ Stargazer fund, Google, Peak XV, others Agrani is being valued at $300-350 million, sources told Moneycontrol. Stargazer’s first bet, which is on Agrani Labs, is also the latest sign of consumer entrepreneurs turning their wealth and attention towards India’s deeptech opportunity. break by @Goenka_Tushar1 & @chandrarsrikant 👇 moneycontrol.com/news/busine…
3
59
This is lit!!!!!
Launching Junction: come speed-run the semiconductor tech tree this December in Bangalore. We’re flying in students from all over India 🇮🇳 to a 2-week fabrication retreat at @_lagrangepoint to build equipment, design chips and engineer processes, with our friends at @hackerfabindia. India still needs thousands of semicon engineers -- the ecosystem to produce them isn’t mature enough yet -- we're changing that. Current applicants have built vacuum chambers to study plasma physics, RISC-V processors from scratch, scanning tunneling microscopes, et cetera At junction you will: 1. Propose a project 2. Figure out a plan with the Junction team. 3. Iterate fast towards a working prototype, with help from the Junction team 4. Be mentored by top semiconductor hackers and specialists. 5. Demo to industry leaders, founders, investors, and your fellow craftspeople Flights, accommodation and project expenses? All covered. Apply on the link below
22
Darshan retweeted
FlashAttention was a mega kernel @marksaroufim @tri_dao @__tensorcore__ @vinodg – at San Jose, CA
19
5
1
143
34,951
You must study the simplest system you think has the properties you are interested in.
we do not realize the added power that the regular and explicit use of alternative hypotheses and sharp exclusions could give us at every step
26
the art of experiments is essentially the old question: How small and elegant an experiment can you perform?
18
we do not realize the added power that the regular and explicit use of alternative hypotheses and sharp exclusions could give us at every step
1) Devising alternative hypotheses; 2) Devising a crucial experiment (or several of them), with alternative possible outcomes, each of which will, as nearly as possible, exclude one or more of the hypotheses; 3) Carrying out the experiment so as to get a clean result;
1
43