Volkan Cevher (@CevherLIONS) header
Volkan Cevher (@CevherLIONS) profile picture

Professor of Electrical Engineering, EPFL. Amazon Scholar (AGI Foundations). IEEE Fellow. ELLIS Fellow.

Lausanne, Switzerland
Joined January 2013
Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)Photo by Volkan Cevher (@CevherLIONS)
Batch size is one of the most important and most debated questions in large-scale training. We still rely on heuristics: linear scaling (Goyal et al) critical batch size (McClandlish et al.) hyperparameter transfer (Yang et al.) We show that under a fixed token budget, there is a derived scaling law: BS ≍ T²⁄³ Paper: arxiv.org/abs/2603.21191 🧵
6
27
5
320
39,251
I am happy to share that ARC, my ERC Advanced Grant project, starts at @EPFL_en on January 1, 2027. I am recruiting PhD students and postdocs at LIONS, @EPFLEngineering. epfl.ch/labs/lions/positions… @ERC_Research #ERCAdG #PhD #Postdoc
🤖 Made with AI
9
11
142
4,709
ARC: Algorithms and Representations for Continual Learning. How can models keep learning as data and objectives change, retaining useful capabilities and building on the computation already invested in training? #ContinualLearning #MachineLearning
1
5
539
Volkan Cevher retweeted
LLM training runs can look healthy for billions of tokens or more then suddenly diverge. We find that quantization error in the LM head drives these delayed time-bomb loss spikes. A simple optimizer change can prevent them. 🧵 1/n
21
46
7
590
42,918
Volkan Cevher retweeted
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
261
826
288
7,379
1,048,208
Volkan Cevher retweeted
A looped transformer has to know when to stop, and its state has to stay bounded while it loops. We introduce Spherical Fixed-Point Reasoners (SFPRM) 🌐: loop on the unit sphere, where each step is an angle, and stop once the angle is small enough. 🧵 (1/9)
14
56
6
401
24,297
A little (batch-size) déjà vu 🙂 this Saturday. Scion and BST Scaling Rule should be a big part of this story, but you would not know from reading these posts. Let me show you why. 1/8
Which optimizer is best? It turns out the answer can change with batch size! We perform a comprehensive sweep and find the best BSZ scaling rule depends on settings and can't be derived from early theories like SDE or convex bound!
3
8
1
73
9,972
Please say clearly how this work builds on the batch picture from Scion, and give BST's fixed-budget theory its place in the scaling comparison. That's the acknowledgment we are asking for, in the paper and public summary. 7/8
1
6
463
Further comments on BST: xitter.gq/CevherLIONS/status/203… 8/8
Batch size is one of the most important and most debated questions in large-scale training. We still rely on heuristics: linear scaling (Goyal et al) critical batch size (McClandlish et al.) hyperparameter transfer (Yang et al.) We show that under a fixed token budget, there is a derived scaling law: BS ≍ T²⁄³ Paper: arxiv.org/abs/2603.21191 🧵
5
465
Volkan Cevher retweeted
We had a similar feeling after this experiment with Scion, so we studied the difference between too few steps and large batch effects w/ opt. theory. If you look at the convergence rate, you see 3 regimes: one with too few steps, one too noisy, and a goldilocks zone.
1
3
10
1,023
All super impressive. Unique games conjecture: Full claim. This hits home.
We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results. github.com/openai/math
1
26
1,903
Volkan Cevher retweeted
We’re releasing a broad range of new mathematical results produced by an internal frontier model. We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results. github.com/openai/math
1,896
6,272
4,764
40,545
22,845,731
Volkan Cevher retweeted
The Bernoullis were more than a family of mathematicians; they formed a remarkable mathematical dynasty. Across several generations, Jacob, Johann, Daniel, and others made important contributions to mathematics and mathematical physics during the 17th and 18th centuries, alongside figures such as Newton, Leibniz, and Euler. Their story also includes one of the most famous rivalries in mathematical history. Jacob and Johann initially collaborated in studying Leibniz’s calculus, but their relationship gradually deteriorated into a bitter and very public competition, with the brothers challenging and criticizing each other over mathematical problems and priority. Whether this rivalry actually fueled their achievements is difficult to establish. What is clear is that their intense competition existed alongside work of lasting importance, and their family became one of the most influential mathematical lineages in early modern Europe.
19
205
52
1,797
1,321,438
Volkan Cevher retweeted
Delta-rule linear attention appears in LLMs as Gated DeltaNet (GDN; Qwen3.8) and Kimi Delta Attention (KDA; Kimi K3). KDA swaps GDN’s scalar gate for a channel-wise one, but does that improve recurrence expressivity? With 2 tweaks, KDA can do 2D rotations. GDN can’t. How? 🧵👇
5
36
6
219
23,808
TMLR has faced a deluge of submissions, necessitating stricter desk rejection policies due to limited reviewer capacity Co-EiC Nihar Shah reached out to authors of 10 papers slated for desk reject. Could they answer questions about their *own* submission? medium.com/@TmlrOrg/asking-a…
26
216
95
1,065
709,730