Ferdinand Mom (@FerdinandMom) header
Ferdinand Mom (@FerdinandMom) profile picture

Distributed & Decentralized training @HuggingFace

France
Joined October 2013
Photo by Ferdinand Mom (@FerdinandMom)Photo by Ferdinand Mom (@FerdinandMom)Photo by Ferdinand Mom (@FerdinandMom)Photo by Ferdinand Mom (@FerdinandMom)Photo by Ferdinand Mom (@FerdinandMom)Photo by Ferdinand Mom (@FerdinandMom)Photo by Ferdinand Mom (@FerdinandMom)Photo by Ferdinand Mom (@FerdinandMom)Photo by Ferdinand Mom (@FerdinandMom)
Interested in 4D parallelism but feeling overwhelmed by Megatron-LM codebase? We are currently cooking something with @Haojun_Zhao14 and @xariusrke 😉 In the meantime, here is a self-contained script that implements Pipeline Parallelism (AFAB + 1F1B) in 200 LOC 🧵👇
12
43
2
230
27,052
Ferdinand Mom retweeted
I want to thanks @SemiAnalysis_ for the motivation. In the Gemini RL scaling team, we printed & put that article on our desks to be reminded daily of what used to be true at the time. But the times they are changing :)
Gemini is Cooked but GCP is Cooking GCP YoY rev growth >100%, DeepMind's long term failure is Google Cloud's short term gain newsletter.semianalysis.com/…
149
277
93
5,011
547,281
Looking for a reference manual for building a frontier LLM? We wrote one 👇 The K2 Horizon technical report is now on arXiv: 134 pages on how we built six models, from architecture to inference. It may be the most detailed LLM training reference you can read today. • Architecture: we bring two new attention designs, MoVA and softplus attention gating, to strengthen reasoning. With them, our 36B model nearly matches a 32B dense one on one-seventh of the active parameters. • Pre-training: we trained four models, from 3.7B to 36B, on the same 22T tokens. Their normalized loss curves collapse onto nearly the same line, so a recipe that works on a small model carries over when scaling up. • Data: 60+ pages on the part most reports skip. How we generated ~10T synthetic tokens, and why we chose the data mix on predicted scores at scale, which reversed the small-run ranking. • Post-training: we document our full production-scale run for the 375B flagship. • Inference: Uno gives an autoregressive model diffusion speed with one small adapter. Up to 2.2× faster decoding, lossless by construction, no draft model. This is the last piece of the openness puzzle. K2 Horizon already shipped with 380 checkpoints, 13 TB of pre-training data, 58 training runs and 17 code repositories. The report explains the decisions behind them. More details: arxiv.org/abs/2611.21003
23
72
22
494
44,760
Ferdinand Mom retweeted
Announcing harness adapters for @Alibaba_Qwen Qwen3.8-27B on Pi harness! We're releasing a LoRA adapter that makes Qwen3.8-27B a much better and efficient agent behind Pi harness. SWE-Bench Pro (Hard 51 subset) pass@1: 29% -> 63% Terminal-Bench 2.1 pass@1: 40% -> 56% Much more efficient: 55-62% fewer tool calls per task 🤯 huggingface.co/harness-hub/Q… The era of harness-specific adapters is now on! More harness adapters to come in the next days 🧵
54
124
43
945
115,045
I was testing Huggingface Jobs by running AsyncGRPO + LoRA on Qwen3-30B-A3B: 3 vLLM Jobs + 2 trainer Jobs. Inside a trainer Job, GPUs talk over NVLink. Between Jobs, only TCP. Same GPUs, same model. One code change, and each training step took 40% less time. Just a simple trick 🧵
6
26
6
166
9,273
The results are 40% less time per step and 1.67x more tokens. Sometimes taking a look at the topology might help !
1
4
273
Ferdinand Mom retweeted
GET UP Jensen Huang. Get the hell up. Save your family, save your company, then save yourself. History judges us not on how we fell, but how we got back up You have far more people to inspire than to disappoint
329
1,548
97
16,841
243,438
Ferdinand Mom retweeted
I’ve decided to leave NVIDIA. There are a few things I want to say. As a robotics engineer at NVIDIA, I have seen Jensen as much more than the company’s CEO. I admire his willingness to invest in a future that was not yet obvious. He committed to CUDA long before the deep learning boom and positioned NVIDIA for AI before it became everyone’s priority. Those decisions helped build the infrastructure that powers today’s AI. I have seen that same vision in physical AI: investing in robotics, sharing open-source tools and projects, and building computing platforms that bring AI into the physical world. This is the work I do every day—and a future I believe in. But what matters most to me is the culture he helped build. Engineers are encouraged to explore, take risks, and learn from mistakes. Jensen reads employees’ Top 5 emails, follows their work, and engages directly. In a company this large, that connection matters. It makes an individual engineer feel that their ideas can still be heard. That is why this has been so disappointing. I do not have all the facts surrounding the allegations, and I cannot prejudge the investigation. My admiration for Jensen does not put him above scrutiny. What I find difficult to accept is having to come to terms with his departure before receiving an explanation that helps me understand it. What this means for our direction, our culture, and the people doing the work remains unclear to me. I have felt that engineers’ perspectives matter here. On this decision, I have not felt heard. That gap has been more painful than I expected. I understand that some information cannot be shared immediately. But that does not erase the feeling of being left outside a decision that affects the company we help build every day. A company’s greatness comes from its leadership, its governance, and the people who turn its ambitions into reality. Employees’ trust and sense of belonging are part of that, too. Jensen’s vision and the engineering culture he shaped were important reasons I wanted to stay. His departure led me to decide that my own chapter here should also come to a close. I leave with sadness and gratitude. The colleagues I have worked with, and the work we believed in together, will always matter to me. These are my personal views.
4,051
13,754
1,204
133,910
7,042,860
Ferdinand Mom retweeted
Replying to @yunta_tsai
A petition is underway to reinstate Jensen and I’m signing it too
7,502
20,242
3,133
228,914
10,914,066
Ferdinand Mom retweeted
說句實在話:如果黃仁勳最後被政治風暴拉下台,那不是輝達的損失,而是美國國運到頭的訊號。 一個移民之子,從丹尼餐廳的一張桌子起家,三十年把一家顯卡公司做成整個 AI 時代的地基。 這種人是美國夢最好的廣告,也是美國在科技競賽裡最難被複製的資產。美國的對手不必研發出更好的晶片,只要看著美國自己把最會做晶片的人推下台就夠了。 一個國家強盛的時候,會把人才捧在手上;開始衰落的時候,才會把人才當成政治祭品。
423
727
93
9,550
922,472
Ferdinand Mom retweeted
This whole thing sucks, and I wish none of it had happened. It’s a really sad day! But I want to make one thing clear: we’ll keep fighting for open-source AI, harder than ever. This is too important to slow down now. Let’s go!
447
940
161
10,561
614,637
Ferdinand Mom retweeted
Today we are announcing 🖐️MacroHands: our state-of-the-art model family for 3D hands and camera path from egocentric video. Our mono variant beats every existing baseline, while the stereo version further improves depth estimation by almost 3x.
4
17
5
101
11,631
Ferdinand Mom retweeted
📖 The Ultimate MoE Training Handbook 📖 We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training. huggingface.co/spaces/huggin…
33
244
20
1,594
112,630
A must read it !
📖 The Ultimate MoE Training Handbook 📖 We spent 1/2M GPUh to optimize MoE training and wrote the most comprehensive blog about it. With 30k words and 50+ interactive figures it covers everything from profiling to large cluster training. huggingface.co/spaces/huggin…
3
368
Ferdinand Mom retweeted
TRL has day-0 support to post-train Le Chonk on a single node 🤯 LoRA SFT on a 1T-parameter model, training adapters on top of FP8 weights: ⚡ 32.7k tokens per step in ~8 seconds 💾 142 GiB per GPU on 8× B300 🔀 Expert parallelism + FSDP Recipe: github.com/huggingface/trl/b… Open weights = open training after all ;)
Today, we release the weights of a new and improved Le Chonk, advancing the open frontier for agentic coding and cybersecurity. 💻 Built to build: Le Chonk ranks #1 on @cognition’s FrontierCode among open models, and sets a new benchmark for Pareto efficiency on @artificialanalysis. 🛡 Built to defend: State-of-the-art in cyberdefense, put through the paces by the world's leading cyber vendors, including @CrowdStrike, @cloudflare, @rubrik, @zscaler, and others. Built for sovereignty: Make it yours. Download the weights on @huggingface , customize it with Mistral Forge, and deploy it wherever you need it. Read more from our team and customers 👇:
8
5
1
51
2,468
Introducing Carbon-X: The best open DNA model. We midtrained a 14B model on DNA and text, and it beats every open language model and matches GPT-6.1 Sol at 1,000x lower cost. Ask it about a sequence, what a mutation does, or to generate new DNA. huggingface.co/HuggingFaceBi…
49
114
69
654
107,397
Muse Spark Open is out, and TRL supports it from day 0 🤗 Fine-tune the 816B MoE (42B active) with SFTTrainer: transformers shards the model and splits the experts across GPUs at load time, and our example trains a LoRA adapter on 8 nodes of 8 H100s.
2
6
1
21
1,278