zach (@swe_zach) header
zach (@swe_zach) profile picture

riding the tiger of accelerationism

SF → Seattle
Joined August 2016
Photo by zach (@swe_zach)Photo by zach (@swe_zach)Photo by zach (@swe_zach)Photo by zach (@swe_zach)Photo by zach (@swe_zach)Photo by zach (@swe_zach)Photo by zach (@swe_zach)Photo by zach (@swe_zach)Photo by zach (@swe_zach)
Sometimes I think I’m a good engineer then I see dudes like Karpathy and geohot cook and I’m like oh I suck
Day 24 of llm.c: we now do multi-GPU training, in bfloat16, with flash attention, directly in ~3000 lines of C/CUDA, and it is FAST! 🚀 We're running ~7% faster than PyTorch nightly, with no asterisks, i.e. this baseline includes all modern & standard bells-and-whistles: mixed precision training, torch compile and flash attention, and manually padding vocab. (Previous comparisons included asterisks like *only inference, or *only fp32 etc.) Compared to the current PyTorch stable release 2.3.0, llm.c is actually ~46% faster. My point in these comparisons is just to say "llm.c is fast", not to cast any shade on PyTorch. It's really amazing that PyTorch trains this fast in a fully generic way, with ability to cook up and run ~arbitrary neural networks and run them on a ton of platforms. I see the goals and pros and cons of these two projects as different, even complementary. Actually I started llm.c with my upcoming education videos in mind, to explain what PyTorch does for you under the hood. How we got here over the last ~1.5 weeks - added: ✅ mixed precision training (bfloat16) ✅ many kernel optimizations, including e.g. a FusedClassifier that (unlike current torch.compile) does not materialize the normalized logits. ✅ flash attention (right now from cudnn) ✅ Packed128 data structure that forces the A100 to utilize 128-bit load (LDG.128) and store (STS.128) instructions. It's now also possible to train multi-GPU - added: ✅ First version of multi-gpu training with MPI+NCCL ✅ Profiling the full training run for NVIDIA Nsight Compute ✅ PR for stage 1 of ZeRO (optimizer state sharding) merging imminently We're still at "only" 3,000 lines of code of C/CUDA. It's getting a bit less simple, but still bit better than ~3 million. We also split off the fp32 code base into its own file, which will be pure CUDA kernels only (no cublas or cudnn or etc), and which I think would make a really nice endpoint of a CUDA course. You start with the gpt2.c pure CPU implementation, and see how fast you can make it by the end of the course on GPU, with kernels only and no dependencies. Our goal now is to create a reliable, clean, tested, minimal, hardened and sufficiently optimized LLM stack that reproduces the GPT-2 miniseries of all model sizes, from 124M to 1.6B, directly in C/CUDA. A lot more detail on: "State of the Union [May 3, 2024]" github.com/karpathy/llm.c/di…
20
33
2
838
133,430
The zoomers are reappropriating the word goy? Fascinating development
Normies be like “lets go do something fun like skydiving” yea no thanks dumb goy.
5
395
Many people say that I have a striking resemblance to Danial Craig (no one has ever said that)
1
14
410
Would have chosen Cavill first cuz he’s so hot 💦💦💦 but I bet this will be fire 🔥🔥 🔥
Identity Confirmed
4
328
Retard alert
What SpaceX did today is the normal consequence of applying capital and engineering to any industry. You get better at it. Musk isn't a genius, he's just a guy who can get capital from Wall Street. Maybe that's unique right now, but it's not the skill his fanboys think it is.
4
43
743
How if feels to talk to my girlfriend
2
18
1,226
Unfortunately overheard some folks discussing AI while vacationing in Patagonia
4
1
56
1,884
Type of shit my girlfriend says to me
BREAKING: Former Nvidia CEO Jensen Huang reportedly told a top executive "I hope you die" during a meeting
1
1
17
398
They just like me fr
Happy birthday sweet little ones! Thank you for such a wonderful birthday experience daddy-da.
3
4
254
Nvidia isn’t Goldman Sachs. Tech companies are much more dependent on being founder run imo.
Sad about Jensen. Shakespearean. But as de Gaulle said, “The graveyards are filled with indispensable men.”
1
2
43
702
What do you think, he was whispering Nvidia’s latest R&D breakthroughs into her ear while giving her backshots?
Replying to @swe_zach
I think it’s more about her being a Chinese spy
1
6
229
Cheating is horrific and Jensen’s private life will suffer the consequences but how does this have anything to do with his ability to run the company? If the board is accountable to shareholders this is just wrong I’m just coping and seething idk
BREAKING: Jensen Huang to be fired as Nvidia CEO after he had affairs with Chinese spy and Nvidia executive Over $250,000,000,000.00 wiped from Nvidia perpetual futures
40
50
7,717
I am on the side of not cheating on your wife - of course this assumes she was deceived and not ok with it - we don’t know
1
4
500
Just used Muse to book a bus ticket in Patagonia. Wild.
8
417
My girlfriend gave me a haircut, what do you guys think?
20
42
1,911
If you can’t manage a woman with bpd can you really be trusted to manage anything at all?
my unsolicited advice to men is, don't have casual sex with high neuroticism/bpd-ish women. Stick to stable, reasonable, chill ladies. If the only women you can convince to have sex with you are the unstable ones, then this is a red flag about you, and you will be never truly innocent of the resulting dramas
1
2
330
Oh shit
If Texas turns blue I’m posting hog
3
27
624
If Texas turns blue I’m posting hog
1
1
11
1,156
zach retweeted
NPCs be like “vote!”
207
191
172
3,786
509,145
Bearish on the Seattle housing market in general obviously but I’m skeptical the Microsoft voluntary layoff would change things. Why would you volunteer to retire if it means you can no longer afford your mortgage.
A massive supply pile-up is happening in Seattle's housing market right now. Listings just broke 13,000 across the metro. And spiked 173% over the last 3 years. Supply is now at a 15-year high, and continues to climb, with many local sellers in panic mode. Microsoft just offered buy-outs to 60,000 workers, which could lead to an even bigger selloff. Demand is already down 50% from peak, and near the lowest levels since the end of the last crash. Indicating that Seattle's housing market is coming periously close to a major correction. Values are already down YoY, and could drop substantially in 2027. Track inventory trends for your area here: reventure.app/map/ovEcBABk8Z
5
13
828
Insane boomer brain flip attempt here 2x in one year after grayscaling the kitchen Curious to follow along, I bet this sells for like $2M ultimately
13
1
1
181
45,427