Pinned Tweet
Training LLMs across multiple datacenters is hard. 🛑
Synchronization demands often cause massive slowdowns as we scale up. If you're at @NeurIPSConf, come see how we tackle this!
Our work, "Scaling Laws for DiLoCo," shows how DiLoCo relax synchronization without compromising model quality, allowing training to scale incredibly well.
Come chat with me and @NovaFallen8:
🗓️ Thu, Dec 4 ⏰ 11 AM – 2 PM PST 📍 Exhibit Hall C,D,E, #811
#NeurIPS2025 #LLMs #DistributedTraining #ScalingLaws
It is today!
⏰ 11 AM – 2 PM PST 📍 Exhibit Hall C,D,E, #811
Training LLMs across multiple datacenters is hard. 🛑
Synchronization demands often cause massive slowdowns as we scale up. If you're at @NeurIPSConf, come see how we tackle this!
Our work, "Scaling Laws for DiLoCo," shows how DiLoCo relax synchronization without compromising model quality, allowing training to scale incredibly well.
Come chat with me and @NovaFallen8:
🗓️ Thu, Dec 4 ⏰ 11 AM – 2 PM PST 📍 Exhibit Hall C,D,E, #811
#NeurIPS2025 #LLMs #DistributedTraining #ScalingLaws
49
Heading to @NeurIPSConf in San Diego.
I’ve got some DiLoCo stickers to give away! 👾 ❤️
Come check out our poster.
🗓️ Thu, Dec 4
⏰ 11 AM – 2 PM PST
📍 Exhibit Hall C,D,E, #811
#NeurIPS2025
gabriel teston retweeted
Attending @NeurIPSConf and interested in distributed, modular, and/or open AI?
Hadn't seen someone put together a list of poster presentations in this area so took it upon myself to thread out who I'm excited to talk to next week🧵
gabriel teston retweeted
We have TPUs in space.
I have a DiLoCo implem running on TPUs.
Cosmic Distributed Learning when?
Our TPUs are headed to space!
Inspired by our history of moonshots, from quantum computing to autonomous driving, Project Suncatcher is exploring how we could one day build scalable ML compute systems in space, harnessing more of the sun’s power (which emits more power than 100 trillion times humanity’s total electricity production).
Like any moonshot, it’s going to require us to solve a lot of complex engineering challenges. Early research shows our Trillium-generation TPUs (our tensor processing units, purpose-built for AI) survived without damage when tested in a particle accelerator to simulate low-earth orbit levels of radiation. However, significant challenges still remain like thermal management and on-orbit system reliability.
More testing and breakthroughs will be needed as we count down to launch two prototype satellites with @planet by early 2027, our next milestone of many. Excited for us to be a part of all the innovation happening in (this) space!
I would say probably get a full time role as RS/RE, maybe spend some time abroad and connect to more smart people 🤓
1
5
722
Want to learn how to train models across the world, with 400x less bits exchanged and a huge latency tolerance? 🌎
I’ll be presenting our work on how to efficiently scale distributed training at @COLM_conf.
🗓️ TODAY: Tuesday, 11:00 - 13:00
📍 Room 710
#COLM2025
Attending @COLM_conf next week?
Want to know how to scale distributed training across the globe with 400x less data transfer?
Come check out our poster.
I'll be there to discuss the future of efficient large-scale model training.
gabriel teston retweeted
This has now been accepted as a spotlight paper at NeurIPS!
Thank you to everyone for their hard work, excited to see what distributed learning this helps enable!
Scaling Laws for DiLoCo was accepted in #NeurIPS2025
Much thanks for all involved.
gabriel teston retweeted
Two DiLoCo-related papers accepted at NeurIPS.
My distributed learning team at GDM is thriving.
More bullish than ever on distributed learning.
gabriel teston retweeted
Streaming DiLoCo has been accepted to COLM 2025!
gabriel teston retweeted
DiLoCo continues to enable ML progress when training at massive scales!
DiLoCo reached @huggingface robotics 🤗
Massively distributed low-communication training will be key to scale robotics.
gabriel teston retweeted
and robots will be distributed everywhere, massively gathering data 🫡
DiLoCo reached @huggingface robotics 🤗
Massively distributed low-communication training will be key to scale robotics.
gabriel teston retweeted
and that DrJax (JAX for map-reduce style of distributed learning) has been used for the latest three DiLoCo releases:
* streaming: arxiv.org/abs/2501.18512
* async eager updates by @SatyenKale: arxiv.org/abs/2502.12996
* scaling laws by @MatharyCharles: arxiv.org/abs/2503.09799
The paper is kinda cool but their discussion of JAX is just wrong.
Case in point: We were able to implement new types of distributed training parallelism (a la MapReduce) in JAX in a few hundred lines of python code (github.com/google/drjax).
Could've done it in less but we wanted to make the underlying primitives differentiable.
gabriel teston retweeted
Want to ask questions about DiLoCo, scaling laws, and distributed training of LLMs using all the compute? Get your questions in!
Wondering how DiLoCo works at scale compared to Data parallel ? Or even how to tune DiLoCo for your model size ?
For the second edition of interviewing researchers with their paper, I'll have the pleasure to have as a guest @MatharyCharles, Research Scientist at Google Research to discuss his paper "Scaling Laws For Diloco" !
If you have any questions on the paper for the author, feel free to comment on thread below 🧵
gabriel teston retweeted
DiLoCo's Scaling Advantage: They analyze how distributed low communication training of transformers scales with increasing model size, finding it outperforms traditional data-parallel methods.
Under fixed compute budgets, they discover DiLoCo's performance advantage grows with model size and can even outperform data-parallel training at smaller scales when properly tuned.
DiLoCo enables
> larger optimal batch sizes
> improves downstream generalization as models scale
> achieves better evaluation loss for fixed token budgets
gabriel teston retweeted
i’m going to repeat myself:
DiLoCo just works





















