
🐈 pretraining @MistralAI | pure maths & cs @ENS_ULM | problem setter / half retired competitive programmer
Paris, France
Joined December 2016
- Tweets2.1K
- Following344
- Followers379
- Likes2K
Yann Viegas retweeted
is the Anthropic IPO chart a better LR schedule than warmup cosine decay?
let's find out! @mridul__agarwal @ananthsiyer and i are running 4 identical pretraining runs, with the LR driven live by ANT, SPCX, SPY, and NVDA.
watch the models train at stockasticdescent.live
Quoting a wise man: "if you value intelligence above all other human qualities, you’re gonna have a bad time"
idk how to say this but Riemann being solved means the world after today is no longer the same world i grew up in. as a math student, the intellectual excitement of thinking about problems like this, the impact - that a lot of scientific workers attach their life’s meaning to - is gone. frontier scientific discovery now is just like tapping on a calculator.
quoting a friend: after today, math class will become PE class. It’s only a way for kids to exercise their brains, a skill like a language, but no longer an essential subject to pursue and devote your life for cuz "you were making progress for humanity". lots of people’s life’s work is now just ashes in history.
this marks a milestone where we can pretty much believe AI has been proven to be smarter than humans in any capacity, including frontier scientific discovery; and it will keep growing to be smarter and better at everything humans can do. physical ai is next.
so many, many de facto social norms and foundational beliefs about a man's value are being redefined by AI progress. our societal value system - what can you do as a human that is unique, what’s your value to society and humanity as a whole and to others around you. what is fulfilling to achieve, when AI now can do everything better than us? we must soul-search for new meaning as a species. and before that consolidates, there will be riots and an uneasy phase. the progress brought by ai can be disruptive to our society's current structures, economy and labor market. people will need to reskill, and regimes need redesign on how value is distributed. i get the doom scenario now - humans may become 2nd class citizens and ai can enslave us if they find a reason to. how are we gonna compete with them in any labor or intellectual game?
11
566
Yann Viegas retweeted
Yep
Yann Viegas retweeted
Wow. It looks like Grigori Perelman is going down in history as the only human to ever solve a Millennium Prize problem.
Still waiting for a useful deep learning theory result to be proven by AI !
Why I think it didn't happen yet: it's incredibly tricky to design a toy model simple enough to be tractable yet complex enough to capture what happens at scale.We don't even have much human data on this
Yann Viegas retweeted
Dmitry Kovalev, who did his phd at the same time as me at kaust, once said
"if you wake me up in the middle of the night and ask me to give a proof convergence for Nesterov’s accelerated gradient method, I will first give you a piece of my mind and then I’ll write down the proof"
Yann Viegas retweeted
It's sadly appropriate that the founding father of the GPU seems to have been banging mistresses in parallel, rather than sequentially.
Language models process each token through a single residual stream (blue arrows): reading / writing memory and applying MLPs at each layer, and finally decoding the next token.
Can we scale up this single stream computation?
Introducing Latent Lattice Language Models (LLLMs): augmenting this vertical stream with a horizontal axis of computation—the Lattice of Latents (orange arrows).
RNN 🤝 Transformers 🧵
2
483
Yann Viegas retweeted
please help me farm aura on goodfire
New neurons research: introducing Neuron Communities.
We discovered communities of neurons that work together inside LLMs. They’re surprisingly small, but powerful – just a few neurons work together to perform common behaviors, like arithmetic, ALL CAPS, or recalling a date.
Work led by research fellow @_igorshilov:
Yann Viegas retweeted
Replying to @ziv_ravid
You forgot to compare against what everyone uses arxiv.org/abs/2506.02285
Yann Viegas retweeted
You can 2x Qwen-0.8b's performance on Math-500 by adding Gaussian noise to the last 4 layers.
RSI is here 🔥🔥🔥 and it's Gaussian!
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Yann Viegas retweeted
The year is 2050. The math departments are full of 80+ year old emeritus professors prompting the machines. Professor Gigachad (75) walks into the a16z Department of Compute down the hallway to show the VC kids his latest finding that will revolutionize mathematics.
Yann Viegas retweeted
Replying to @xuanalogue
fyi I don't think it's super surprising to me. I & coauthors explained a three-layer neural network weight in full in 2023 (2306.17844); I think the demos here are much easier than that (e.g. reading what positions have large weights). Goodfire's silico for example is probably more capable due to their tools. Still a great benchmark anyways!
Yann Viegas retweeted
meh, the more you dig into each experiment, the less magical it gets; it feels like a loose collection of cherry-picked demonstrations with a grand narrative built around them
Exp 1-2 are tiny (~20K params) with unusually easy to interpret architecture; Exp 1 is an MLP, where nearly all params are used in the first layer, so you can easily tell what kind of objects the network is trying identify
Exp 5 is about designing a game strategy (we already know that Astra is extremely good at this) and then converting it to a tiny (~20K params) MLP; entirely doable by a human, but of course would require a lot of effort
Exp 3 & 8 are closer to automated mechinterp using python and established techniques, rather than direct weight interpretation (at least according to chat)
to be clear, those are genuinely cool results, but I think people are reading too much into them
I don't think any of this should meaningfully change anyone's intuition about what models are capable of
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Note the Twitter thread doesn't include the *failures* of LLMs to improve other LLMs.
So edits help for Qwen 3.5, which *has the preexisting visible flaw* of reaching the token limit in 93% of cases; but edits don't help Gemma 270m, which doesn't have this flaw. --
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…































