Marcos Treviso (@MarcosTreviso) profile picture

Assistant Professor at IST, Univ. of Lisbon

Lisbon, Portugal
Joined November 2019
Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)Photo by Marcos Treviso (@MarcosTreviso)
Marcos Treviso retweeted
📢 Join us for the next ELLIS & Sardine Lab Seminar on ML & NLP featuring @manuelmlmadeira from @EPFL, who will present his work "On Trajectory-Aware Training for Masked Diffusion Language Models". 📅 23rd November, 5pm WET 🔗 More info & meeting link: lnkd.in/diMpUD57
3
4
255
Marcos Treviso retweeted
On the zeroth order lm training stuff: It's cool that people try stuff instead of only reading and dooming. But i feel the need to curb the enthusiasm on this one a little. Two things: (1) methods in the genetic-algo and fwd-diff categories, which imo these fall in, is established that they are fine for smaller parameter counts but fall off hard as parameter space increases. It's not necessarily raw param count that matters here but "intrinsic" size. This is the whole point of backprop: few outputs many params. (2) loss 4.2 for LM is very high even for a 30M parameter model. Go a bit higher and you are at learning token unigram distribution. This means for this score not much interesting/difficult learning has to happen yet. Again it's fine to rediscover things like this from first principles. In fact it's the very best way to learn and i recommend it; i spent a lot of time learning like this myself! But now with social media hype amplification and crazy vcs and journalist attention that our field gets, i do feel the need to be a bit more cautious with communication and there to be "old pessimist" voice like mine in the room to. (But unlike some other old pessimist voices, if i turn out to be wrong I'll happily admit it and cheer on =)
WE DID IT! We can now pretrain transformers without backprop and even beat backprop+Adam on validation loss with insane speeds. Now the big question is scale. It’s Ironic, we have reached a breakthrough In AI capability due to inspiration from a safety blog
31
39
7
955
54,411
Marcos Treviso retweeted
Our book "The Elements of Differentiable Programming" can now be read directly in the browser: diffprog.github.io/
11
91
7
784
59,427
Hmmmmmmmmm. Which one is more efficient at prefill? And generalization to longer sequences? What's the impact of the positional encoding? What about the ratio of swa:full? And the placement? So many open questions...
I agree with one thing. Making linear attention work at scale takes talent, and not every lab has it. The rest doesn’t hold up. In order: (1) “Why are the Western labs choosing to drop context instead of compressing it?” The first rigorous, controlled, large-budget test of compression was what we did at NVIDIA’s in June 2024(arxiv.org/abs/2406.07887). We trained 8B Mamba, Mamba-2 and Transformer models on the same data for up to 3.5T tokens. Additionally we trained an 8B hybrid of 43% Mamba-2, 7% attention and 50% MLP and we showed it beats the 8B Transformer on all 12 standard tasks (+2.65 points on average). Interestingly this hybrid variant was up to 8× faster at generation, and matched or beat the Transformer across 23 additional long-context tasks. This is seven months before MiniMax-01, fifteen before Qwen3-Next, sixteen before Kimi Linear. And since then NVIDIA has been shipping a hybrid architecture, with modifications, repeatedly. Good examples are Nemotron-H, Nemotron Nano 2, and Nemotron 3 (Nano, Super, Ultra) which is a Mixture-of-Experts hybrid Mamba-Transformer family with context up to 1M tokens, post-trained with multi-environment RL. Nano keeps exactly 6 attention layers out of 52 and the other 46 are Mamba-2 and MoE blocks. Super is 120B total / 12B active, pre-trained on 25 trillion tokens. IBM’s Granite 4.0 (October 2025) interleaves Mamba-2 and attention layers at 9:1. To this list, you can also add Jamba (AI21, March 2024, a 52B production hybrid), RecurrentGemma (Google, April 2024), Codestral Mamba (Mistral, July 2024) and Phi-4-mini-flash (Microsoft, July 2025). (2) “SWA is a lossless local transformation, therefore superior.?” SWA is lossless inside the window and zero outside of it and it is exactly why it fails so badly. (arxiv.org/abs/2509.24552) for example shows that larger sliding windows hurts the long-context performance. This is because a precise local softmax lets the model avoid training the long-term memory it would otherwise rely on. They show that a hybrid with a 128-token window beat the same hybrid with a 2048 window by 16 NIAH points (88.9% higher recall), and at 131k the short-window models still retrieved about 30% of needles while the 2048-window model was near 0%. And the same degradation from larger training windows shows up in local-global stacks that alternate SWA with full attention ( e.g. the GPT-OSS/Gemma recipe). (3) “Compression is strictly harder to serve.” This was true in 2024. But now vLLM, FlashInfer, SGLang and TensorRT-LLM serve hybrid linear models extensively and without any issues. The most exciting part is the recent FlashInfer additions that let you use chunked gated-delta-rule prefill and gated-delta-rule decode kernels, with an SM100 (Blackwell) path. It even supports much newer models like Gated DeltaNet-2 and also lets you use efficient kernels for training. Also, let's not forget ReplaySSM (arxiv.org/abs/2610.13778) which enables a very optimized speculative decoding path for linear attention. (4) “If you write all the kernels anyway, why not roll your own attention in the meantime?” On NVIDIA hardware, linear attention is just as free as attention. The Mamba CUDA kernels exist. flash-linear-attention, the Triton library that the GDN and KDA layers in Qwen3-Next, Qwen3.5 and Kimi Linear use, was written for NVIDIA GPUs, and vLLM and SGLang use it directly instead of reimplementing it. Megatron has supported Mamba-2 training since 2024. A Western lab on H100s that wants a hybrid doesn’t need to write a kernel. It can just import it. So kernel cost can’t be what keeps them on using the SWA. Also consider that decode is memory-bandwidth-bound, so the cost of every generated token is dominated by reading the KV cache, which grows with context length. A constant-size state removes that growth. That’s why Nemotron is mostly Mamba-2, and why Nemotron 3 Super gets 2.2× the throughput of GPT-OSS-120B at 8K in / 16K out. (5) “For your first release, it’s a much safer option to ship SWA. Everyone knows it’s working just fine.”? Safe compared to what? “Safe” should mean proven in production at scale. By that standard the hybrids are the safe choice and SWA is the bet. Look at what has shipped on Gated DeltaNet alone. Qwen3-Next, then Qwen3.5 from 27B up to 397B-A17B, then Qwen3.6. Kimi Linear, then Kimi K3 which is a 2.8 trillion parameters, open weights, 1M-token context, a 3:1 hybrid of Kimi Delta Attention and gated MLA. GLM-5.3-Flash on the same KDA layout. KDA is Gated DeltaNet with channel-wise instead of scalar gating. Right now, three frontier labs run their model in production with 1M context, RL post-training, etc. all on the same layer. Meanwhile SWA is the architecture with the documented issue. It fails past the window and drags any hybrid it sits in down to pure-SWA recall. And when MiniMax reached for an SWA hybrid as the fallback in M2, it failed and they went to full attention instead. In my view, shipping SWA for a first release in late 2026 isn’t conservative. It’s picking the layer with a known failure mode because it needs no new code.
1
168
Model size: 10M Dataset: MNIST Related work: non-existent Sure. Revolutionary.
21
Marcos Treviso retweeted
Why do neural models fail to generalize to longer sequences? Not just an architectural problem. We find that transformers can lose extrapolation ability as training progresses. Weight decay accelerates this degradation, while properly placed dropout improves extrapolation. 1/
1
4
2
22
1,739
Highly recommend your with @PontiEdoardo! This is a great opportunity to do creative and impactul research.
I am looking for 1 postdoc and 2 fully funded PhDs at Imperial @ICComputing to join my @ERC_Research project AToM ⚛ We aim to build models that learn to compress what they perceive and remember, for permanent memory, longer horizons, and >10x speedups ducdauge.github.io/atom/
1
146
If you missed the AdaSplash-2 poster at @icmlconf, check out our short video below! +NunoGoncalves, @twigs0000, @PontiEdoardo, @lileics, @andre_t_martins
Glad to share that AdaSplash-2 was accepted to ICML 2026! 🇰🇷 Btw, we've released the Triton kernels for AdaSplash-2 in our GitHub repo. Nuno's histogram implementation with bit-packing to avoid atomics is, in my totally unbiased opinion, remarkable! github.com/deep-spin/adaspla…
5
2
16
3,608
DashAttention combines the best of the best: differentiable sparse attention for flexibility, online softmax for efficiency, and hierarchical modeling for long-context! Paper, code, and models are all available ;) 🚀🚀🚀
[1/n] Can a model learn *where* and *how much* information it should attend to, and do so efficiently? We introduce DashAttention: Differentiable and Adaptive Sparse Hierarchical Attention! This pushes the accuracy-efficiency frontier in LLMs.
4
408
Glad to share that AdaSplash-2 was accepted to ICML 2026! 🇰🇷 Btw, we've released the Triton kernels for AdaSplash-2 in our GitHub repo. Nuno's histogram implementation with bit-packing to avoid atomics is, in my totally unbiased opinion, remarkable! github.com/deep-spin/adaspla…
1/ We are excited to release AdaSplash-2 🚀 A big milestone from our lab on faster differentiable sparse attention. And honestly, one of my favorite examples of sparsity giving a real win-win: more efficiency + better downstream performance, especially for long-context tasks.
3
3
2
15
6,183
PS: I'm also thankful to the ICML organizers for the Gold Reviewer recognition, which will allow me to attend the conference in person 🥰
1
98
1/ We are excited to release AdaSplash-2 🚀 A big milestone from our lab on faster differentiable sparse attention. And honestly, one of my favorite examples of sparsity giving a real win-win: more efficiency + better downstream performance, especially for long-context tasks.
1
2
3
24
14,934
Huge kudos to my amazing co-authors: Nuno Gonçalves, @twigs0000 , Vlad Niculae, @PontiEdoardo , @lileics , and @andre_t_martins
1
2
210
PS: Andre Martins and I will be at ICLR in Rio 🇧🇷 discussing our paper with @pavlo_vasylenko on long-context generalization with entmax attention. Stop by our poster: Thu, Apr 23, 2026 11:15 AM – 1:45 PM Happy to chat about sparsity + long-context 🚀
2
224