Pinned Tweet
Batch size is one of the most important and most debated questions in large-scale training.
We still rely on heuristics:
linear scaling (Goyal et al)
critical batch size (McClandlish et al.)
hyperparameter transfer (Yang et al.)
We show that under a fixed token budget, there is a derived scaling law:
BS ≍ T²⁄³
Paper: arxiv.org/abs/2603.21191
🧵
I am happy to share that ARC, my ERC Advanced Grant project, starts at @EPFL_en on January 1, 2027.
I am recruiting PhD students and postdocs at LIONS, @EPFLEngineering.
epfl.ch/labs/lions/positions…
@ERC_Research #ERCAdG #PhD #Postdoc
🤖 Made with AI
ARC: Algorithms and Representations for Continual Learning.
How can models keep learning as data and objectives change, retaining useful capabilities and building on the computation already invested in training?
#ContinualLearning #MachineLearning
1
5
539
We will connect optimization, representation learning, reinforcement learning, and game theory with experiments on foundation models.
I welcome applicants with theoretical or experimental strengths.
cordis.europa.eu/project/id/…
#Optimization
Volkan Cevher retweeted
LLM training runs can look healthy for billions of tokens or more then suddenly diverge. We find that quantization error in the LM head drives these delayed time-bomb loss spikes. A simple optimizer change can prevent them. 🧵 1/n
Volkan Cevher retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data.
Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less?
Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
Volkan Cevher retweeted
A looped transformer has to know when to stop, and its state has to stay bounded while it loops.
We introduce Spherical Fixed-Point Reasoners (SFPRM) 🌐: loop on the unit sphere, where each step is an angle, and stop once the angle is small enough.
🧵 (1/9)
A little (batch-size) déjà vu 🙂 this Saturday.
Scion and BST Scaling Rule should be a big part of this story, but you would not know from reading these posts.
Let me show you why. 1/8
Please say clearly how this work builds on the batch picture from Scion, and give BST's fixed-budget theory its place in the scaling comparison.
That's the acknowledgment we are asking for, in the paper and public summary. 7/8
1
6
463
Further comments on BST: xitter.gq/CevherLIONS/status/203…
8/8
Batch size is one of the most important and most debated questions in large-scale training.
We still rely on heuristics:
linear scaling (Goyal et al)
critical batch size (McClandlish et al.)
hyperparameter transfer (Yang et al.)
We show that under a fixed token budget, there is a derived scaling law:
BS ≍ T²⁄³
Paper: arxiv.org/abs/2603.21191
🧵
5
465
Volkan Cevher retweeted
We had a similar feeling after this experiment with Scion, so we studied the difference between too few steps and large batch effects w/ opt. theory. If you look at the convergence rate, you see 3 regimes: one with too few steps, one too noisy, and a goldilocks zone.
All super impressive. Unique games conjecture: Full claim. This hits home.
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
1
26
1,903
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
Volkan Cevher retweeted
The Bernoullis were more than a family of mathematicians; they formed a remarkable mathematical dynasty.
Across several generations, Jacob, Johann, Daniel, and others made important contributions to mathematics and mathematical physics during the 17th and 18th centuries, alongside figures such as Newton, Leibniz, and Euler.
Their story also includes one of the most famous rivalries in mathematical history. Jacob and Johann initially collaborated in studying Leibniz’s calculus, but their relationship gradually deteriorated into a bitter and very public competition, with the brothers challenging and criticizing each other over mathematical problems and priority.
Whether this rivalry actually fueled their achievements is difficult to establish. What is clear is that their intense competition existed alongside work of lasting importance, and their family became one of the most influential mathematical lineages in early modern Europe.
Volkan Cevher retweeted
Delta-rule linear attention appears in LLMs as Gated DeltaNet (GDN; Qwen3.8) and Kimi Delta Attention (KDA; Kimi K3). KDA swaps GDN’s scalar gate for a channel-wise one, but does that improve recurrence expressivity? With 2 tweaks, KDA can do 2D rotations. GDN can’t. How? 🧵👇
Volkan Cevher retweeted
TMLR has faced a deluge of submissions, necessitating stricter desk rejection policies due to limited reviewer capacity
Co-EiC Nihar Shah reached out to authors of 10 papers slated for desk reject. Could they answer questions about their *own* submission?
medium.com/@TmlrOrg/asking-a…
















