Working on Gemini @GoogleDeepMind | formerly: PhD @MIT_CSAIL @MITNLP. Opinions my own
New York, NY
Joined September 2014
- Tweets514
- Following831
- Followers1.9K
- Likes1.8K
Tal Schuster retweeted
We’re rolling out Gemini 4 Argon in Antigravity to paid Google AI Pro subscribers today. Demand has been incredible and we've scaled capacity deliberately to get here. We’ve rolled out a separate weekly-only Argon quota for Pro on top of the existing Gemini bucket for 3.8 Flash so you have the best of Gemini for all your tasks.
Tal Schuster retweeted
Gemini 4 Argon from @Google on ARC-AGI (Verified):
- ARC-AGI-3: 97.5%, $4.8K (provider adapter harness), 49.8%, $5.8K (standard harness)
- ARC-AGI-2: 91.3%, $0.32/task
- ARC-AGI-1: 98.5%, $0.16/task
Google DeepMind is the second frontier AI lab to score above 95% on ARC-AGI-3.
Tal Schuster retweeted
Introducing Gemini 4 Argon – our new frontier model.
It’s built for complex workflows across coding, enterprise knowledge work, and cybersecurity defense – rolling out today to a set of trusted testers through our Fairwind Program.
Tal Schuster retweeted
Banger paper from Google DeepMind and colleagues.
(bookmark it)
A model reads its entire KV cache on every generated token, even though it ends up attending to a tiny slice of it.
In other words, if you ask about one detail from a 1M-token conversation the global attention layers re-read all of it, per token.
The usual fix is to guess the relevant tokens first with cheap proxy scores, which still costs O(N) every step. Declarative Attention asks the model instead.
The model declares where it needs to look, inside its own chain-of-thought.
In this way, generation splits into three modes: global reads the full context, focus reads one specific region, and local reads only recent output.
The inference engine parses those declarations the same way it parses tool calls and skips most of the cache read.
On zero-shot on off-the-shelf weights across 15 long-context tasks, attended tokens during decoding drop 52.0% on Gemma-4-31B and 31.1% on Qwen-3.6-27B.
Paper: arxiv.org/abs/2609.02737
Chat with Paper: academy.dair.ai/papers/langu…
Tal Schuster retweeted
A lot of hype around OpenAI's Astra model here on my timeline today. Apparently, this goes back to a new article from The Information, which said Astra is a "recurrent depth or looped transformer".
It's always interesting to read about new or different approaches (including rumors about what the closed labs may be up to), but let's debunk this a bit.
About 2 months ago, I shared the architecture details of Nanbeige, for example, where "Nanbeige4.2-3B is pretrained from scratch on 28T tokens with a Looped Transformer that reuses the layer stack to increase capacity without adding parameters."
Yes, that's it. The looped transformer idea is just reusing layers in the transformer block.
In the case of Nanbeige, the main idea is to reuse the same 22-layer stack (=transformer block) twice instead of once. So, effectively it extends the 22-layer architecture to 44 layers, but without duplicating the weights.
In simple terms, this roughly doubles the size of the model (if we ignore the embedding and output layers for a second). But instead of requiring 2x the storage and RAM to host this model, it stays at the same size since we reuse the components. However, it's almost 2x as expensive in terms of compute, because we run the embedded text through almost 2x as many layers.
Why? In the Nanbeige 4.2 technical report, the researchers found that two passes gave the best trade-off and retained about 75% of the token efficiency of a standard architecture. (More passes gave barely any gains but made the training much slower and much more expensive.)
While, as far as I know, Nanbeige 4.2 is the first notable open-weight model that adopted this approach, the idea goes back to the NeurIPS paper "Mixture-of-recursions: Learning dynamic recursive depths for adaptive token-level computation". Actually, this paper proposes a mechanism that is a bit more sophisticated by adding a learned router that determines whether each token receives one, two, or more passes. So, easy tokens can exit early while harder tokens receive additional computation.
In sum, Astra may be a really good model, but this shouldn't be about this "looped transformer aspect," which is just a tiny architectural tweak.
Also, the statement "the new technique works in a way that obscures some or all of the AI's reasoning, otherwise known as 'chain-of-thought'" is not necessarily true with respect to the looped transformer method. It's possible that The Information journalist refers to some other technique or misunderstood the looped transformer method.
Reusing layers does not by itself suppress visible chain of thought. It adds computation in hidden states before the next token is emitted, just as ordinary transformer layers do.
But based on the information we have, the only plausible interpretation here is that if a model uses more of these recurrent passes, it may need to generate fewer intermediate reasoning tokens. So then more of its computation happens in latent activations that cannot be read as text. But we would get the same effect if we were scaling up the model size, like GPT 5.6 Luna -> GPT 5.6 Sol.
Tal Schuster retweeted
This is a big one. Full thread coming soon.
huggingface.co/papers/2609.0…
Tal Schuster retweeted
Nice to see this idea getting attention!
This is essentially what we called "depth-wise batching": overlapping requests at different recurrent depths (much higher utilization with adaptive compute).
Recursive Models: arxiv.org/abs/2410.20672
MoR: arxiv.org/abs/2507.10524
a few thoughts on "recurrent depth" transformers
the main question: why recurrent depth instead of just scaling depth? it's not faster at inference or training* since you still go through the full "effective depth", the advantage is storage (for kv cache storage btw you could do kv sharing) and a few other things i tried to list below
1) could be better in undertrained regimes (too few data or compute constraints) but i don't think any frontier model is in this regime? could also help convergence somehow, maybe it's just not possible to train a 10T effective param model (so maybe 2Tx5) without tricks like this, but i doubt it..
2) one positive aspect could be better adaptive depth/compute (which is a good direction imo): depending on the token, you skip some recurrent blocks. but you could also just skip layers (or skip moe routing like in longcat flash) and get something quite similar
you could also imagine co-training models this way, sol with 8 blocks, terra with 4, luna with 2 or something like that
3) another interesting thing (to verify?): you could simulate higher batch size at inference by overlapping the computation of depth_block=1 of request 1 and depth_block=t of request 2 on the same gpus, so effective batch size = batch size * number of recurrent blocks (and you could even overlap across models if you do the co-training i mentioned above!), see the picture attached for an illustration of what i mean
still quite skeptical overall, i think scaling depth is just more expressive when you are not compute/data bound but excited to see a potential new direction
*on "faster to train", could be if you only backprop through the last few loops or something like that
Tal Schuster retweeted
Recursive or Looped Transformers are powerful tools, especially when initializing and training them correctly, or when slightly relaxing the layer tying constraint with layer-wise LoRAs.
Check out @raymin0223 student research project @GoogleDeepMind which found the right recipe
🚀 Excited to share our latest research @GoogleDeepMind on ♻️Recursive Transformers!
We make smaller LMs by "sharing parameters" across layers. A novel serving paradigm, ✨Continuous Depth-wise Batching, with 🏃Early-Exiting could significantly boost their decoding speed!
🧵👇
Tal Schuster retweeted
Google has released Gemini 3.7 Flash, improving 4 points over Gemini 3.6 Flash and reaching the Intelligence vs. Time per Task Pareto frontier
@GoogleDeepMind has released its third new Gemini Flash model in three months. Gemini 3.7 Flash (high) scores 56 on the Artificial Analysis Intelligence Index, just behind GPT-5.6 Terra (max, 57) and Muse Spark 1.2 (xhigh, 57)
We benchmarked Gemini 3.7 Flash across all three reasoning levels (high, medium, low) ahead of release. With high reasoning, Gemini 3.7 Flash is a 4 point improvement over 3.6 Flash, while achieving an average Time per Task of 1.7, 40% faster than GPT-5.6 Terra (max). This places Gemini 3.7 Flash on the Intelligence vs. Time per Task Pareto frontier, reinforcing Google’s focus on speed across the Gemini Flash family of models
Key benchmarking results across Gemini 3.7 Flash’s three reasoning levels:
➤ 4 point Intelligence Index improvement: Gemini 3.7 Flash (high) scores 56 on the Artificial Analysis Intelligence Index, up 4 points from Gemini 3.6 Flash. The improvement is driven primarily by gains on agentic evaluations, including Tau3 Banking (+3 points), Terminal-Bench v2.1 (+8 points), and GDPval-AA v2 (+103 Elo). With medium reasoning, Gemini 3.7 Flash scores 53, matching DeepSeek V4 Pro 0813 (max, 53) and GLM-5.2 (max, 53). With low reasoning, it scores 51, just behind DeepSeek V4 Flash 0731 (max, 52)
➤ Pareto frontier on Intelligence vs. Time per Task: Gemini 3.7 Flash produces ~340 output tokens per second, nearly 3x the output speed of GPT-5.6 Terra and GLM-5.2. With high reasoning, this translates to an average Time per Task of 1.7 minutes, placing Gemini 3.7 Flash on the Intelligence vs. Time per Task Pareto frontier
➤ 30% lower Cost per Task than Gemini 3.6 Flash: Gemini 3.7 Flash retains Gemini 3.6 Flash’s standard pricing of $1.50/$7.50 per 1M input/output tokens, however Google is offering discounted pricing through the end of the year at $0.75/$3.75 per 1M tokens. At this discounted price, Gemini 3.7 Flash (high) costs $0.40 per Intelligence Index task, 30% less than Gemini 3.6 Flash and matching Muse Spark 1.2 (xhigh, $0.40). With medium reasoning, Cost per Task falls to $0.26, placing the model on the Intelligence vs. Cost per Task Pareto frontier
➤ Leading performance on AutomationBench-AA and AA-AnalystAgent: On AA-AnalystAgent, our recently released benchmark measuring models’ ability to answer complex questions about spreadsheets and documents, Gemini 3.7 Flash (high) achieves the highest pass^5 score at 60%, ahead of Claude Opus 5 (max, 54%) and Fable 5 (49%). Gemini 3.7 Flash also leads AutomationBench-AA, our benchmark of agentic capabilities in simulated SaaS environments, with a score of 62.7%, ahead of Kimi K3 (max, 53%) and GPT-5.6 Sol (max, 51.2%)
Key model details:
➤ Context Window: 1M tokens, unchanged from Gemini 3.6 Flash
➤ Multimodality: Text, image, video, and speech input, with text output
➤ Pricing: $1.50/$7.50 per 1M input/output tokens at standard pricing. Google is offering discounted pricing of $0.75/$3.75 per 1M tokens through the end of the year. Cached input tokens retain the same 90% discount
Tal Schuster retweeted
Introducing Gemini 3.7 Flash : )
- it is fast!
- 50% lower price than 3.6 flash (through end of year)
- strong intelligence increase in only ~3 weeks (thanks to some awesome algorithmic improvements)
- available in the API, AI Studio, Antigravity, and more!
Tal Schuster retweeted
Gemma 4 MTP just got officially merged into llama.cpp
This means you can use Gemma 4 QAT + MTP for a lightweight + super fast setup. Excited to see what the community builds with it
github.com/ggml-org/llama.cp…
Tal Schuster retweeted
Gemma 4 12B + MTP speculative decoding on mlx-vlm 🚀
We benchmarked MTP on Gemma 4 12B across all 4 modalities in mlx-vlm — and it speeds up everything:
text, image, audio, and combined audio+image, up to 1.72× and 80 tok/s on a single M3 Ultra.
Get started today:
> uv pip install -U mlx-vlm
github.com/Blaizzy/mlx-vlm
🚀 Gemma 4 12B is here!
We partnered with @GoogleDeepMind to bring and optimize their new dense and unifed multimodal model for Apple Silicon.
◈ 12B dense · 256K context
◈ Thinking mode (built-in reasoning)
◈ Vision: dynamic res, OCR, UI + charts
◈ Native audio: ASR + speech translation
◈ Function calling for agents
◈ Text + image + audio, interleaved
Runs local. Get started now ⚡
> uv pip install -U mlx-vlm
github.com/Blaizzy/mlx-vlm
Tal Schuster retweeted
Local AI at 400 tok/s, zero loss, Gemma 4 31B 🤯 Google's MTP approach is way better than Qwen's because it SCALES with MTP parameter but Qwen peaks at MTP=3. I can now have FULL BF16 Gemma 4 31B at +400 tok/s on RTX 6000 Pro using MTP=100, video coming soon!
staggering! 🚨 I ran Google's new MTP for Gemma 4 31B (full BF16) on vLLM + RTX 6000 Pro (600W). The results are humbling! Next: I'll run dFlash from @zhijianliu for comparison; local AI is shaping up good 😎
Tal Schuster retweeted
Gemma 4 up to 3x faster, directly in your phone! 🚀
Check out the difference Speculative Decoding makes! Multi-Token Prediction (MTP) is supercharging inference speeds for Gemma 4.
Tal Schuster retweeted
I benchmarked Google’s new MTP for Gemma 4 31B using vLLM with 4 speculative tokens, a fairly conservative setup.
Results:
- Much higher throughput than Qwen3.6’s MTP
- Lower latency too, helped by Gemma 4 generating fewer tokens
- For coding tasks with reasoning enabled, Gemma 4 is now at least 6x faster than Qwen3.6. So you can generate 5 outputs, run your tests to select the best one, and it would still be cheaper than a single output by Qwen3.6.
I’ve updated my full comparison with the new numbers:
kaitchup.substack.com/p/qwen…
I also confirmed what others have reported: Gemma 4’s MTP handles a high number of speculative tokens very well.
On simple text generation, I’m now testing values above 10 and reached 129 tok/s on an RTX Pro 6000, compared with 20 tok/s without MTP.
Next step: confirming how this translates to real tasks.
Tal Schuster retweeted
Google dropped MTP versions of Gemma4. Ran them on my DGX Spark.
The 31B dense model went from 3.94 → 8.91 tok/s. That's +126%.
Full results:
[26B A4B]
> 25.24 → 31.69 tok/s (+25.6%)
> TTFT 755 → 332ms (-56%)
[31B]
> 3.94 → 8.91 tok/s (+126%)
> TTFT 599 → 378ms (-37%)
If you're not running MTP, you're leaving free perf on the table.
Tal Schuster retweeted
🚨 Google just made Gemma 4 up to 3x faster with MTP ⚡
Same quality, way more speed. It predicts multiple tokens at once and verifies them in parallel, removing latency bottlenecks.
You can also run powerful models locally on mobile like me using Google AI Edge Gallery.
Tal Schuster retweeted
🚀 Day-0 MTP support for Gemma4 now available at vLLM with ready-to-use docker image!
⚡️Enjoy up to 3x faster decoding performance to supercharge your development with zero quality degradation!
Check out the full vLLM recipes for Gemma 4 model series👇
recipes.vllm.ai/Google/gemma…
