PhD @ Inria/ENS; prev @SakanaAILabs, @UniofOxford, @Polytechnique. Building new approaches in Generative Models, Diffusion LLMs
Paris, France
Joined February 2025
- Tweets20
- Following85
- Followers108
- Likes44
Dario Shariatian retweeted
Continuous Diffusion LMs went from niche to contender this year, but the recipe is shaky: why self-conditioning? How to embed tokens? 🤨
Our new work takes your one-hot flow from 4% to 50% on TinyGSM without self-conditioning, and explores the design space of CDLMs. 🔥🚀 (1/6)
Under reverse KL discrepancy, the same student architecture gives two methods:
🔺 Simplex-DMD: uses token probabilities on the Simplex, with pathwise gradients
🎲 Reinforce-DMD: samples tokens, with REINFORCE and a learned density ratio
Both support multi-step generation
(4/n)
Co-led with the amazing @PaulLeVanKiem, with @UmutSimsekli and @AlainDurmus 🤩
📜 Paper: arxiv.org/abs/2609.40235
(6/n)
On a parallel note, great to see progress with Simplex Diffusion Models by @jdeschena, @ValentinDeBort1, @JamesTThorn, @ArnaudDoucet1 and colleagues!
Closely connected distribution-matching ideas across Gaussian and Dirichlet diffusion. Plenty to explore!
Tagging people who might be interested: @yjelid @samsongvch @iNikitaGushchin @TheOneKloud @sedielem @StefanoErmon @ArashVahdat @ssahoo_ @canaesseth @oussamazekri_ @osclsd @Viacheslav91112 @pidstrigach @PPotaptchik @FelixKoulischer @VickyKalogeiton @PatrickPyn35903 @LucaAmb
We unify distributional distillation through a local discrepancy between noised student and data distributions
The choice of discrepancy recovers existing objectives in prior works; the gradient decomposition reveals how the student's parameterization shapes its training
(3/n)
🔥 Accelerate your diffusion LMs with Distribution Matching Distillation
Introducing two methods for continuous diffusion LMs with gains at complementary sampling budgets:
🔺 Simplex-DMD at very few-step budgets
🎲 Reinforce-DMD at larger budgets
🔺 At 4 NFEs Simplex-DMD halves the strongest baseline’s generative perplexity at matched entropy
(1/n)
Dario Shariatian retweeted
📢 June 15 (Mon): Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
🤔 Discrete diffusion models are often trained through clean-data prediction, but the prediction can be used in different ways to define the reverse dynamics. In Masked Diffusion Models (MDM) these choices largely coincide, whereas in Uniform Diffusion Models (UDM) they do not.
💡 The authors show that the standard plug-in bridge parameterization for UDM is not optimized by the denoising posterior, but by a leave-one-out posterior that predicts each clean token without using its own noisy observation. This identifies a mismatch between the plug-in ELBO and the usual cross-entropy denoising objective.
🔧 The authors characterize the leave-one-out target and derive exact conversions between the denoiser, the leave-one-out posterior, and the score. These conversions allow them to disentangle parameterization and the training objective.
📈 Their results also lead to inference improvements without any additional training through an informed predictor-corrector sampler and improved temperature sampling based on the leave-one-out predictor.
🔧 The authors further introduce an absorbing-state reformulation of uniform diffusion that preserves the UDM joint law while decomposing it into masked-diffusion-like sampling operations, with simpler denoising posteriors, carry-over unmasking, and a natural remasking mechanism.
📈 On language modeling, leave-one-out parameterizations consistently improve UDM generation, while the absorbing construction matches or surpasses masked diffusion. These results suggest that the empirical gap between masked and uniform diffusion is driven less by the choice of marginals themselves than by parameterization and sampling design.
This Monday, Samson Gourevitch (@samsongvch, samsongourevitch.github.io/), Yazid Janati (@yjelid, yazidjanati.github.io/), and Dario Shariatian (@dario_sha, darioshar.github.io/) will present their paper "Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation".
Dario Shariatian retweeted
Happy to share our recent preprint:
Uniform Diffusion Models Revisited: Leave-One-Out Denoiser and Absorbing State Reformulation
Takeaways:
- standard implementations of uniform diffusion do not learn a denoiser, but rather a
"leave-one-out" denoiser
- uniform diffusion and masked diffusion are two sides of the same coin
paper: arxiv.org/abs/2605.22765
code: github.com/samsongourevitch/… (we release 9 models trained on owt)
with @samsongvch @dario_sha @umutsimsekli @EricMoulines @ericxing
@AlainDurmus
A short thread on our findings:
(1/5)
⚡️Latent-Augmented Discrete Diffusion (LADD)⚡️
Discrete diffusion updates tokens in parallel, but reverse steps factorize. In few-step sampling, these independent commitments combine incoherently.
We add an auxiliary latent channel that carries global structure. 1/4
The framework makes the core tradeoff explicit: informative latents can reduce the factorization gap, but then we must also learn to generate those latents.
We consider a two-stage procedure and other tricks to implement our latent-augmentation choices. 3/4
Gains show up where parallel generation matters most.
OWT @ 1600 GFLOPs:
MDLM Gen PPL: 298.4
Co-LADD-16: 138.7
At matched entropy. The framework is not text-specific and can transfer to any structured discrete data.
Paper: arxiv.org/abs/2510.18114
4/4
Dario Shariatian retweeted
As a bit of trivia, before flow matching / rectified flows etc in late 2022 the core idea existed about a year earlier in late 2021
Non-Denoising Forward-Time Diffusions by Peluchetti
openreview.net/forum?id=oVfI…
shows the role of marketing in the academic publishing game
Excited to present our work “Denoising Lévy Probabilistic Models” at #ICLR2025 (Thursday 10am)
We extend diffusion models with heavy-tailed noise for better coverage & faster sampling.
Let’s chat about heavy-tailed/discrete diffusion and novel methods!
arxiv.org/abs/2407.18609