gabriel teston (@GabrielTeston) header
gabriel teston (@GabrielTeston) profile picture

Solving language @Google Search

Joined December 2023
Photo by gabriel teston (@GabrielTeston)Photo by gabriel teston (@GabrielTeston)Photo by gabriel teston (@GabrielTeston)
Training LLMs across multiple datacenters is hard. 🛑 Synchronization demands often cause massive slowdowns as we scale up. If you're at @NeurIPSConf, come see how we tackle this! Our work, "Scaling Laws for DiLoCo," shows how DiLoCo relax synchronization without compromising model quality, allowing training to scale incredibly well. Come chat with me and @NovaFallen8: 🗓️ Thu, Dec 4 ⏰ 11 AM – 2 PM PST 📍 Exhibit Hall C,D,E, #811 #NeurIPS2025 #LLMs #DistributedTraining #ScalingLaws
We just put out a key step for making distributed training work at larger and larger models: Scaling Laws for DiLoCo TL;DR: We can do LLM training across datacenters in a way that scales incredibly well to larger and larger models!
2
3
10
6,312
It is today! ⏰ 11 AM – 2 PM PST 📍 Exhibit Hall C,D,E, #811
Training LLMs across multiple datacenters is hard. 🛑 Synchronization demands often cause massive slowdowns as we scale up. If you're at @NeurIPSConf, come see how we tackle this! Our work, "Scaling Laws for DiLoCo," shows how DiLoCo relax synchronization without compromising model quality, allowing training to scale incredibly well. Come chat with me and @NovaFallen8: 🗓️ Thu, Dec 4 ⏰ 11 AM – 2 PM PST 📍 Exhibit Hall C,D,E, #811 #NeurIPS2025 #LLMs #DistributedTraining #ScalingLaws
49
Heading to @NeurIPSConf in San Diego. I’ve got some DiLoCo stickers to give away! 👾 ❤️ Come check out our poster. 🗓️ Thu, Dec 4 ⏰ 11 AM – 2 PM PST 📍 Exhibit Hall C,D,E, #811 #NeurIPS2025
1
5
607
gabriel teston retweeted
Attending @NeurIPSConf and interested in distributed, modular, and/or open AI? Hadn't seen someone put together a list of poster presentations in this area so took it upon myself to thread out who I'm excited to talk to next week🧵
5
5
48
4,672
We have TPUs in space. I have a DiLoCo implem running on TPUs. Cosmic Distributed Learning when?
Our TPUs are headed to space!  Inspired by our history of moonshots, from quantum computing to autonomous driving, Project Suncatcher is exploring how we could one day build scalable ML compute systems in space, harnessing more of the sun’s power (which emits more power than 100 trillion times humanity’s total electricity production). Like any moonshot, it’s going to require us to solve a lot of complex engineering challenges. Early research shows our Trillium-generation TPUs (our tensor processing units, purpose-built for AI) survived without damage when tested in a particle accelerator to simulate low-earth orbit levels of radiation. However, significant challenges still remain like thermal management and on-orbit system reliability.  More testing and breakthroughs will be needed as we count down to launch two prototype satellites with @planet by early 2027, our next milestone of many. Excited for us to be a part of all the innovation happening in (this) space!
9
3
2
57
8,052
I would say probably get a full time role as RS/RE, maybe spend some time abroad and connect to more smart people 🤓
what are you 5 years goal folks
1
5
722
Want to learn how to train models across the world, with 400x less bits exchanged and a huge latency tolerance? 🌎 I’ll be presenting our work on how to efficiently scale distributed training at @COLM_conf. 🗓️ TODAY: Tuesday, 11:00 - 13:00 📍 Room 710 #COLM2025
2
1
10
4,079
I though Canada was cooler 😎☀️
3
149
Attending @COLM_conf next week? Want to know how to scale distributed training across the globe with 400x less data transfer? Come check out our poster. I'll be there to discuss the future of efficient large-scale model training.
We release today the next step for distributed training: --> Streaming DiLoCo with Overlapping Communication. TL;DR: train data-parallel across the world with low-bandwidth for the same performance: 400x less bits exchanged & huge latency tolerance
1
4
2,981
This has now been accepted as a spotlight paper at NeurIPS! Thank you to everyone for their hard work, excited to see what distributed learning this helps enable!
We just put out a key step for making distributed training work at larger and larger models: Scaling Laws for DiLoCo TL;DR: We can do LLM training across datacenters in a way that scales incredibly well to larger and larger models!
1
6
3
46
8,082
Scaling Laws for DiLoCo was accepted in #NeurIPS2025 Much thanks for all involved.
We just put out a key step for making distributed training work at larger and larger models: Scaling Laws for DiLoCo TL;DR: We can do LLM training across datacenters in a way that scales incredibly well to larger and larger models!
6
1
36
5,039
Two DiLoCo-related papers accepted at NeurIPS. My distributed learning team at GDM is thriving. More bullish than ever on distributed learning.
4
10
4
142
17,616
Streaming DiLoCo has been accepted to COLM 2025!
We release today the next step for distributed training: --> Streaming DiLoCo with Overlapping Communication. TL;DR: train data-parallel across the world with low-bandwidth for the same performance: 400x less bits exchanged & huge latency tolerance
1
5
89
4,171
DiLoCo continues to enable ML progress when training at massive scales!
DiLoCo reached @huggingface robotics 🤗 Massively distributed low-communication training will be key to scale robotics.
1
2
14
1,260
and robots will be distributed everywhere, massively gathering data 🫡
DiLoCo reached @huggingface robotics 🤗 Massively distributed low-communication training will be key to scale robotics.
7
3
34
4,076
and that DrJax (JAX for map-reduce style of distributed learning) has been used for the latest three DiLoCo releases: * streaming: arxiv.org/abs/2501.18512 * async eager updates by @SatyenKale: arxiv.org/abs/2502.12996 * scaling laws by @MatharyCharles: arxiv.org/abs/2503.09799
The paper is kinda cool but their discussion of JAX is just wrong. Case in point: We were able to implement new types of distributed training parallelism (a la MapReduce) in JAX in a few hundred lines of python code (github.com/google/drjax). Could've done it in less but we wanted to make the underlying primitives differentiable.
1
7
1
43
3,827
Want to ask questions about DiLoCo, scaling laws, and distributed training of LLMs using all the compute? Get your questions in!
Wondering how DiLoCo works at scale compared to Data parallel ? Or even how to tune DiLoCo for your model size ? For the second edition of interviewing researchers with their paper, I'll have the pleasure to have as a guest @MatharyCharles, Research Scientist at Google Research to discuss his paper "Scaling Laws For Diloco" ! If you have any questions on the paper for the author, feel free to comment on thread below 🧵
1
3
1
26
2,879
gabriel teston retweeted
DiLoCo's Scaling Advantage: They analyze how distributed low communication training of transformers scales with increasing model size, finding it outperforms traditional data-parallel methods. Under fixed compute budgets, they discover DiLoCo's performance advantage grows with model size and can even outperform data-parallel training at smaller scales when properly tuned. DiLoCo enables > larger optimal batch sizes > improves downstream generalization as models scale > achieves better evaluation loss for fixed token budgets
1
2
11
896
i’m going to repeat myself: DiLoCo just works
We just put out a key step for making distributed training work at larger and larger models: Scaling Laws for DiLoCo TL;DR: We can do LLM training across datacenters in a way that scales incredibly well to larger and larger models!
3
7
82
7,942
We just put out a key step for making distributed training work at larger and larger models: Scaling Laws for DiLoCo TL;DR: We can do LLM training across datacenters in a way that scales incredibly well to larger and larger models!
12
78
28
375
143,543