Yann Viegas (@_Yann77) header
Yann Viegas (@_Yann77) profile picture

🐈 pretraining @MistralAI | pure maths & cs @ENS_ULM | problem setter / half retired competitive programmer

Paris, France
Joined December 2016
Photo by Yann Viegas (@_Yann77)Photo by Yann Viegas (@_Yann77)Photo by Yann Viegas (@_Yann77)Photo by Yann Viegas (@_Yann77)Photo by Yann Viegas (@_Yann77)Photo by Yann Viegas (@_Yann77)Photo by Yann Viegas (@_Yann77)Photo by Yann Viegas (@_Yann77)Photo by Yann Viegas (@_Yann77)
Yann Viegas retweeted
is the Anthropic IPO chart a better LR schedule than warmup cosine decay? let's find out! @mridul__agarwal @ananthsiyer and i are running 4 identical pretraining runs, with the LR driven live by ANT, SPCX, SPY, and NVDA. watch the models train at stockasticdescent.live
36
60
9
730
35,237
Quoting a wise man: "if you value intelligence above all other human qualities, you’re gonna have a bad time"
idk how to say this but Riemann being solved means the world after today is no longer the same world i grew up in. as a math student, the intellectual excitement of thinking about problems like this, the impact - that a lot of scientific workers attach their life’s meaning to - is gone. frontier scientific discovery now is just like tapping on a calculator. quoting a friend: after today, math class will become PE class. It’s only a way for kids to exercise their brains, a skill like a language, but no longer an essential subject to pursue and devote your life for cuz "you were making progress for humanity". lots of people’s life’s work is now just ashes in history. this marks a milestone where we can pretty much believe AI has been proven to be smarter than humans in any capacity, including frontier scientific discovery; and it will keep growing to be smarter and better at everything humans can do. physical ai is next. so many, many de facto social norms and foundational beliefs about a man's value are being redefined by AI progress. our societal value system - what can you do as a human that is unique, what’s your value to society and humanity as a whole and to others around you. what is fulfilling to achieve, when AI now can do everything better than us? we must soul-search for new meaning as a species. and before that consolidates, there will be riots and an uneasy phase. the progress brought by ai can be disruptive to our society's current structures, economy and labor market. people will need to reskill, and regimes need redesign on how value is distributed. i get the doom scenario now - humans may become 2nd class citizens and ai can enslave us if they find a reason to. how are we gonna compete with them in any labor or intellectual game?
11
566
Yann Viegas retweeted
since everyone is sharing their LR schedules and loss curves
Probably need gradient clipping or cosine LR decay
2
3
30
827
Claude solving RH is the first empirical proof of the infinite monkey theorem (/jk)
Am I the only one that is not surprised by the announcement of the proof of Riemann Hypothesis?
153
Probably need gradient clipping or cosine LR decay
12
64
7
1,058
77,949
Yann Viegas retweeted
Yep
Still waiting for a useful deep learning theory result to be proven by AI ! Why I think it didn't happen yet: it's incredibly tricky to design a toy model simple enough to be tractable yet complex enough to capture what happens at scale.We don't even have much human data on this
1
1
10
3,131
Yann Viegas retweeted
Wow. It looks like Grigori Perelman is going down in history as the only human to ever solve a Millennium Prize problem.
38
253
25
6,036
127,859
Still waiting for a useful deep learning theory result to be proven by AI ! Why I think it didn't happen yet: it's incredibly tricky to design a toy model simple enough to be tractable yet complex enough to capture what happens at scale.We don't even have much human data on this
Holy shit. Levent Alpöge from Anthropic just claimed Claude has resolved the General Reimann Hypothesis!!! Proof is 250,000+ pages long. Lean is released. GRH is the 2nd of 6 Millenium prizes solved by AI, after Navier-Stokes. This may be the biggest AI math result of all time.
4
3
18
4,175
Dmitry Kovalev, who did his phd at the same time as me at kaust, once said "if you wake me up in the middle of the night and ask me to give a proof convergence for Nesterov’s accelerated gradient method, I will first give you a piece of my mind and then I’ll write down the proof"
3
1
70
5,428
Interesting learning rate schedule
18
97
9
2,045
75,871
It's sadly appropriate that the founding father of the GPU seems to have been banging mistresses in parallel, rather than sequentially.
216
2,415
140
47,982
792,570
Language models process each token through a single residual stream (blue arrows): reading / writing memory and applying MLPs at each layer, and finally decoding the next token. Can we scale up this single stream computation? Introducing Latent Lattice Language Models (LLLMs): augmenting this vertical stream with a horizontal axis of computation—the Lattice of Latents (orange arrows). RNN 🤝 Transformers 🧵
2
483
Yann Viegas retweeted
please help me farm aura on goodfire
New neurons research: introducing Neuron Communities. We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date. Work led by research fellow @_igorshilov:
1
1
1
10
578
how anthropic RL env engineers were moving in September 2026:
36
313
26
8,376
154,309
Yann Viegas retweeted
You can 2x Qwen-0.8b's performance on Math-500 by adding Gaussian noise to the last 4 layers. RSI is here 🔥🔥🔥 and it's Gaussian!
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
32
44
19
1,074
60,477
The year is 2050. The math departments are full of 80+ year old emeritus professors prompting the machines. Professor Gigachad (75) walks into the a16z Department of Compute down the hallway to show the VC kids his latest finding that will revolutionize mathematics.
7
12
2
172
6,704
Yann Viegas retweeted
Replying to @xuanalogue
fyi I don't think it's super surprising to me. I & coauthors explained a three-layer neural network weight in full in 2023 (2306.17844); I think the demos here are much easier than that (e.g. reading what positions have large weights). Goodfire's silico for example is probably more capable due to their tools. Still a great benchmark anyways!
2
4
1
37
1,727
Yann Viegas retweeted
meh, the more you dig into each experiment, the less magical it gets; it feels like a loose collection of cherry-picked demonstrations with a grand narrative built around them Exp 1-2 are tiny (~20K params) with unusually easy to interpret architecture; Exp 1 is an MLP, where nearly all params are used in the first layer, so you can easily tell what kind of objects the network is trying identify Exp 5 is about designing a game strategy (we already know that Astra is extremely good at this) and then converting it to a tiny (~20K params) MLP; entirely doable by a human, but of course would require a lot of effort Exp 3 & 8 are closer to automated mechinterp using python and established techniques, rather than direct weight interpretation (at least according to chat) to be clear, those are genuinely cool results, but I think people are reading too much into them I don't think any of this should meaningfully change anyone's intuition about what models are capable of
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
50
73
18
1,107
60,077
Yann Viegas retweeted
Note the Twitter thread doesn't include the *failures* of LLMs to improve other LLMs. So edits help for Qwen 3.5, which *has the preexisting visible flaw* of reaching the token limit in 93% of cases; but edits don't help Gemma 270m, which doesn't have this flaw. --
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
23
33
13
470
30,894