training llms @ cohere. previously @ tum and inria.
Paris, France
Joined July 2020
- Tweets564
- Following779
- Followers1.4K
- Likes2K
Fabian Schaipp retweeted
Why would you want to do this?
1) With our law, you get the correct scaling for the optimal batch size with two/three batch sizes per sweep
2) You can also derive scaling laws for *suboptimal* batch size intervals (e.g. which batch sizes waste at most 5% of compute)
Fabian Schaipp retweeted
🎬 Field Notes: is more data always better?
Physical AI is booming, and so are the data providers feeding it. But what actually makes a robot smarter?
Awesome to chat with @fabinsch1, researcher at @inria who just presented his thesis in policy learning for robotics.
🧵
Fabian Schaipp retweeted
Judging optimizer gaps by looking only at language modeling with a fixed batch size is dangerous: one gets only 1/2 of the story. @orientino_ and @ruuustem_10 went beyond. Turns out that for every model, task, and data, there is always a setup where Adam > SGD. 🧵
Fabian Schaipp retweeted
Good to see SOAP and Muon being quite performant in another setting — training of Diffusion Models. Similarly to our benchmark, the authors find Prodigy a decent “proxy-optimizer” for tuning hyperparams of Adam-like methods
arxiv.org/pdf/2510.19376
Fabian Schaipp retweeted
If you’re scrambling a last-minute submission with an uncertain result, remember: putting it off is hard in the moment. It will sting for 10 minutes (because you care so deeply), but in 10 months you’ll be incredibly proud you made the scientifically rigorous call.
Fabian Schaipp retweeted
Die Berliner Verkehrssenatorin Uta Bonde (CDU) jetzt im Gespräch mit dem Tagesspiegel zum Thema Schulwegsicherheit: „Wir können nicht nach Gutdünken Tempo 30 einführen“
Autofreie Schulstraßen wie in Paris sieht sie skeptisch. Dafür ihr Rat an alle Kinder und Jugendlichen:
„Helm tragen und ganz, ganz vorsichtig sein“
Täter-Opfer Umkehr par Excellence. Die Kinder sollen sich schützen, damit Autofahrer freie spielen können.
Währenddessen hat Paris einfach in der ganzen Stadt Tempo 30 eingeführt und innerhalb von 10 Jahren über 300 Straßen vor Schulen in autofreie Fußgängerzonen umgewandelt.
Die Ergebnisse sehen einfach wunderschön aus❤️
Fabian Schaipp retweeted
Long in the making, finally released: Apertus-8B and Apertus-70B, trained on 15T tokens of open data from over 1800 languages. Unique opportunity in academia to work on and train LLMs across the full-stack. We managed to pull off a pretraining run with some fun innovations, ...
Fabian Schaipp retweeted
We uploaded V3 of our draft book "The Elements of Differentiable Programming". Lots of typo fixes, clarity improvements, new figures and a new section on Transformers! arxiv.org/abs/2403.14606
Fabian Schaipp retweeted
tuning is inseparable from the optimizer
when a new optimizer is released claiming to be 2x better than sota baseline, the baseline was probably under tuned.
Lecture 11: benchmarking optimizers
1. the problem: comparing optimizers (sgd, adam, etc.) in deep learning is tricky.
2. challenge 1: defining "speed". curves cross, so use time-to-result.
3. challenge 2: hyperparameter tuning trap. protocol matters more than algo? (choi et al. '19).
4. solution: rigorous benchmarking principles (algoperf framework: std hw, workloads, tuning rulesets).
5. results: algoperf competition findings (shampoo, schedulefreeadamw win), robustness is key (kasimbeg et al. '24).
6. takeaway: optimizer performance depends heavily on the tuning protocol used.
Fabian Schaipp retweeted
A really fun project to work on. Looking at these plots side-by-side still amazes me! How well can **convex optimization theory** match actual LLM runs?
My favorite points of our paper on the agreement for LR schedules in theory and practice: 1/n
Learning rate schedules seem mysterious?
Turns out that their behaviour can be described with a bound from *convex, nonsmooth* optimization.
Short thread on our latest paper 🚇
arxiv.org/abs/2501.18965
Fabian Schaipp retweeted
An inspirational talk by Michael Jordan: a refreshing, deep, and forward-looking vision for AI beyond LLMs.
youtube.com/live/W0QLq4qEmKg…
Fabian Schaipp retweeted
The sudden loss drop when annealing the learning rate at the end of a WSD (warmup-stable-decay) schedule can be explained without relying on non-convexity or even smoothness, a new paper shows that it can be precisely predicted by theory in the convex, non-smooth setting!
1/2
Fabian Schaipp retweeted
Im Salon-Teaser sagte ich Anfang Januar: "Wir erleben gerade, wie man an der einen Hand noch Margot Friedländer führt und mit der anderen Hand schon nach Alice Weidel greift."
youtube.com/watch?v=EuWN3TnN…
Fabian Schaipp retweeted
We did try a transformer model and got good results on a vision transformer on Imagenet, improving both stability with respect to learning rates, and the best validation score. But when we tried a language model, we didn't get such a good result, and we know exactly why ...(1/2)
Fabian Schaipp retweeted
Together with @phschiele1 , we wrote a package to solve constrained optimization problems, where all functions are arbitrary @PyTorch modules.
This is mainly intended for optimization with pre-trained NNs as objective/constraints.
github.com/fabian-sp/ncOPT
Fabian Schaipp retweeted
We're now at 400 consecutive days of record-breaking ocean temperatures in the North Atlantic. 🤯
Fabian Schaipp retweeted
Do sharper minima generalize better?
We explore this question in different settings in our new paper - check out this thread for details.
xitter.cf/maksym_andr/status/162…
🚨 Excited to share our new paper “A modern look at the relationship between sharpness and generalization” arxiv.org/abs/2302.07011!
❓ Do flatter minima really generalize better? Let’s find out!
🧵 1/n
Fabian Schaipp retweeted
We @BioDataSc have open positions for PhD/Postdocs at the interface of statistics, optimization, and microbiome data analysis #stats, #optimization, #microbiome, #datascience, #oceanmicrobiome #gutmicrobiome @CompHealthMuc @LMU_Muenchen; PM me or send an email for more info!!