Mohammed Sabry (@M___Sabry) header
Mohammed Sabry (@M___Sabry) profile picture

Currently: Training Mellum models @jetbrains , Previously: PhD @dcucomputing, research @GoogleAI

∆
Joined May 2020
Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)Photo by Mohammed Sabry (@M___Sabry)
Mohammed Sabry retweeted
meh, the more you dig into each experiment, the less magical it gets; it feels like a loose collection of cherry-picked demonstrations with a grand narrative built around them Exp 1-2 are tiny (~20K params) with unusually easy to interpret architecture; Exp 1 is an MLP, where nearly all params are used in the first layer, so you can easily tell what kind of objects the network is trying identify Exp 5 is about designing a game strategy (we already know that Astra is extremely good at this) and then converting it to a tiny (~20K params) MLP; entirely doable by a human, but of course would require a lot of effort Exp 3 & 8 are closer to automated mechinterp using python and established techniques, rather than direct weight interpretation (at least according to chat) to be clear, those are genuinely cool results, but I think people are reading too much into them I don't think any of this should meaningfully change anyone's intuition about what models are capable of
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math. It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵 Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math. rikiyat.github.io/blog/weight-sp… github.com/RikiyaT/lm_und… huggingface.co/Qwen/Qwen3.5-0… huggingface.co/RikiyaT/Qwen3.…
50
73
18
1,107
60,055
One thing I hate about training frameworks (Megatron specifically but not limited to it) is it's annoying to run arch ablations on them. You have a very rigid structure that you need your model to fit in, like "Qwen3.5" or whatever else is implemented in it. If you come up with something that doesn't fit into anything, you need to implement it practically everywhere: the framework itself, transformers, vllm/sglang. If only there would be a nice way to express an arbitrary arch and then "compile" it into the actual framework code. I started writing my own "arch language" (and a training framework) for fun. Now, obviously, the hardest part is to actually compile it into the stuff I want and I'm seriously considering it shouldn't be hardcoded but be a fast agent that will quickly draft the model implementation from building blocks instead.
3
1
1
30
4,205
it's fast
Meet Mellum2.1 – JetBrains' open-source 12B MoE model built for coding agents and local execution. 🚀 140+ T/S in local testing scenarios 🇪🇺 Ethically trained European open model 💻 Efficient worker model inside IDE agent workflows Try it locally: huggingface.co/collections/J…
1
3
299
Mohammed Sabry retweeted
Long-term AI adoption depends on driving down token costs. Mellum2.1 is our contribution to that shift – bringing an open-source 12B MoE model into local PyCharm workflows to deliver 140+ T/S without cloud overhead. Learn more and see it in action:
Meet Mellum2.1 – JetBrains' open-source 12B MoE model built for coding agents and local execution. 🚀 140+ T/S in local testing scenarios 🇪🇺 Ethically trained European open model 💻 Efficient worker model inside IDE agent workflows Try it locally: huggingface.co/collections/J…
5
9
1
82
9,298
Meet Mellum2.1 – JetBrains' open-source 12B MoE model built for coding agents and local execution. 🚀 140+ T/S in local testing scenarios 🇪🇺 Ethically trained European open model 💻 Efficient worker model inside IDE agent workflows Try it locally: huggingface.co/collections/J…
4
26
3
184
18,065
We're doing a meetup during GTC Berlin at the JetBrains office in Berlin! It's a great chance to talk to people from Mellum and ML teams. We'll also have friends from NVIDIA joining. Everyone's welcome! Wednesday, 21 October.
1
2
1
254
Replying to @M___Sabry
We observed a lot of similar behaviours in Mellum2 pre-training run but it smoothed out in the lr decay stage
1
1
216
Today we're releasing Mellum2.1: a major update to the Mellum2 that we open-sourced in June. This release is the result of a significant scaling of our RL post-training to enable agentic coding workflows. Mellum2.1 achieves top scores on the agentic coding benchmarks relative to its speed. We're releasing the weights in both HF and GGUF formats, and MTP support is coming in the following days. 🤗 collection: huggingface.co/collections/J… Blog post: blog.jetbrains.com/ai/2026/1…
24
30
15
206
18,860
Mohammed Sabry retweeted
Yesterday I got two useful pieces of feedback on the recirculation paper: 1. the training-free effect might grow with model size, as @xidulu suggested; 2. @qubitium also observed gains on Llama. So I tested three ~30B models, together with my Dense and MoE models, on much larger MMLU/GSM8K test sets (14,042 / 1,319 examples) and added 95% bootstrap CIs to reduce evaluation noise. Unfortunately, the picture is much weaker than I expected: most MMLU gains are below 1 pp (Qwen 27B is an outlier, explained in the thread below), all GSM8K confidence intervals cross zero, and I see no clear trend with model size. The training-free recirculation story is looking less robust than I initially thought.
Recirculation is coool, but the evidence seems narrower. PPL looks great: −23.0% adaptive, −8.5% training-free. Yet across 8 standard single-token benchmarks, every training-free change is <0.6pp, with 2 regressions. This looks like benchmark noise to me. Table 2: arxiv.org/pdf/2608.17981#pag…
11
8
2
86
9,239
We will have an architecture that obsoletes the transformer within the next three years.
63
28
22
545
70,122
Mohammed Sabry retweeted
There is no pre-training, post-training, or test-time training. There are only priors, updates, constraints, and compute budgets. There is only TRAINING. Last several years we shipped the org chart to fundamental optimization science.
22
34
11
538
68,798
In an era where many frontier labs are converging towards conservative incremental approaches, it's heartening to see resources being directed into ambitious efforts and fresh ideas!
Unveiling our new startup Advanced Machine Intelligence (AMI Labs). We just completed our seed round: $1.03B / 890M€, one the largest seeds ever, probably the largest for a European company. We're hiring! [the background image is the Veil Nebula - a picture I took from my backyard, most appropriate for an unveiling] More details here: techcrunch.com/2026/03/09/ya…
6
140
13,351
How giraffes drink.
33
111
13
1,133
124,591
Continual learning has always been important, but people have not found effective ways. Tweaking loss objectives brings tons of ml papers a few year's back but cannot really achieve inf long context. Maybe we need more arch innovation like efficient attn and learnable memory.
Continual learning is the next big thing in AI. Today’s deep networks show catastrophic forgetting where new data points cause old knowledge to get erased because backpropagation updates all weights in a network, causing old useful info to be overwritten. Contrast this with brain - it doesn’t do backpropagation. no global updates, only local updates with each new data point. The day we have a credible alternative to backpropagation is the day we will solve continual learning. It’s an exciting time to be doing AI research!
6
12
2
157
20,080
Mohammed Sabry retweeted
most ppl don’t really think. they rearrange cached thoughts until they feel smart. true thinking is rare because it’s metabolically expensive (just like thinking models are computationally expensive).
218
566
81
6,618
347,909
Continual learning as a discipline seems to have catastrophic forgetting that it has been focused on catastrophic forgetting for a decade with virtually no progress. Time for some radically new ideas in that area.
25
23
2
360
29,209
Mohammed Sabry retweeted
New paper studies when spectral gradient methods (e.g., Muon) help in deep learning: 1. We identify a pervasive form of ill-conditioning in DL: post-activations matrices are low-stable rank. 2. We then explain why spectral methods can perform well despite this. Long thread
11
67
9
335
99,627
Mohammed Sabry retweeted
Replying to @giffmana
Something something cake something something cherry.
12
2
254
29,632
Mohammed Sabry retweeted
anyone who is convinced that scaling era is over and want to sell me their gpus, pls dm
The @ilyasut episode 0:00:00 – Explaining model jaggedness 0:09:39 - Emotions and value functions 0:18:49 – What are we scaling? 0:25:13 – Why humans generalize better than models 0:35:45 – Straight-shotting superintelligence 0:46:47 – SSI’s model will learn from deployment 0:55:07 – Alignment 1:18:13 – “We are squarely an age of research company” 1:29:23 – Self-play and multi-agent 1:32:42 – Research taste Look up Dwarkesh Podcast on YouTube, Apple Podcasts, or Spotify. Enjoy!
1
9
694
Mohammed Sabry retweeted
Even from NeurIPS experiments a decade ago, the bottom 25% of papers had a 5% chance of acceptance whereas the top 25% had a 50% chance. Review scores are noisy! With the recent flood of papers and the downfall of reviews, the correlation between merit and acceptance is going to 0 fast. Can someone estimate the convergence rate?
An LLM-generated paper is in the top 17% of ICLR submissions in terms of average reviewer score, having received two 8's. The paper has tons of BS jargon and hallucinated references. Fortunately, one reviewer actually looked at the paper and gave it a zero. 1/3
2
8
1
140
26,402