
Member of Technical Staff, AGI Autonomy @amazon. Building artificial general intelligence. Opinions are my own.
San Francisco, CA
Joined April 2022
- Tweets353
- Following1.3K
- Followers453
- Likes1.3K
Wai Tong retweeted
The boundary between midtraining and SFT is increasingly meaningless. High quality, diverse data is always your best bet
Great paper on why SFT can generalize just as well as RL if you give it enough diverse training data.
Like everyone else since DeepSeek-R1 dropped, I’ve been treating "RL generalizes, SFT memorizes" as gospel.
Whenever SFT runs plateaued on out-of-domain reasoning, everyone just blame the algorithm.
Turns out, we might have been running an unfair comparison the entire time.
In standard setups, RL gets to explore a huge rollout budget per prompt while SFT gets trained on a tiny slice of data.
So the authors decided to actually level the playing field on Qwen2.5 and Qwen3 models by testing two things: mixing data sources instead of just distilling from one giant teacher, and scaling up the SFT data to match RL's rollout budget.
The results completely flip the usual narrative.
A carefully mixed dataset easily beats single-teacher distillation, and matching RL's rollout budget lets SFT generalize just as well as RL.
And when you combine both, SFT actually outperforms it.
To figure out why, they used LLM annotations to inspect the rollouts and proved (both empirically and theoretically on shallow transformers) that larger data budgets expose SFT to rare, "tail" solution methods.
Those uncommon reasoning paths are where the actual generalization signal lives.
Honestly, for everyone tired of debugging finicky online RL pipelines, this is a huge relief.
SFT wasn't broken, we were just starving it of diverse rollouts.
Full paper PDF: arxiv.org/pdf/2611.14912
Wai Tong retweeted
I’ve felt increasingly overwhelmed by the rapid progress of frontier models this year. I can’t help but wonder: what’s left for humans to do?
Building a benchmark feels like a great way to really understand how AI might automate my own day-to-day work.
So we built one!
Introducing Singularity-Bench 🕳️
🔄 We measure whether agents can autonomously conduct the full agent development lifecycle: six long-horizon, end-to-end tasks spanning pre-, mid-, and post-training, inference engines, cluster management, and harness development on production GB300 clusters!
We emphasize rigorous verification to guarantee correctness and execution integrity 👮♂️
🙌 Big shoutout to all authors contributing from academia and industry: @IronSteveZhou, @wonderingkrish, @chenzhuoming911, Zerui Wang, @KyriectionZhang, Wenshuang Li, Xinpeng Wei, Zhichen Zeng, @Jingyu227, @JueWANG26088228, @Wu_Sen, Ziheng Jiang, @zhendongucb, @hancai_hm, @Hades317, @ying11231, @ce_zhang, @uiuc_aisecure, and @BeidiChen!
How far are current agents from building their successors end to end? 🤔 We're here to find out!
🧵1/n
🌐 Website: infini-ai-lab.github.io/sing…
Wai Tong retweeted
The secrets of good data research are out!
The tech report behind our 6x pretraining compute multiplier is out today.
Training LLMs for data research takes more scientific control than training one great model does. Even small changes can confound results.
Some spoilers on how we did it:
datologyai.com/blog/text-cur…
Wai Tong retweeted
tl;dr of what Together Compiler does, from @realDanFu on @thecompute100:
1/ run the model end to end
2/ find the bottlenecks
3/ let the model fix them
Links below 👇
Wai Tong retweeted
Fun podcast on kernels, origins of @togethercompute, and how we’re using AI to build inference in an RSI inference loop with Together Compiler.
Thanks so much for having me @gabyllorenzi and @BSchech!
Today’s guest is @realDanFu, SVP of Research at @togethercompute.
A co-creator of FlashAttention, Dan now leads research at a company that provides the compute and software to train and serve AI models.
From GPU instructions to the economics of open models, we discuss:
- What CUDA, kernels and compilers each do
- Why FlashAttention rethought data movement
- How AI is changing kernel engineering
- How new open models get to market
- How inference revenue funds open models
- Owning versus leasing GPUs
Timestamps:
0:00 Introduction
1:04 From AI models to CUDA kernels
11:38 Launching new models at Together
22:36 How model architectures are changing
27:42 Comparing open and closed models
31:57 How AI is changing kernel engineering
38:01 From systems research to FlashAttention
47:11 Using AI to improve inference
48:16 Together’s origins and open models
51:17 Owning versus leasing compute
52:55 Building the research team and new chips
54:56 Frontier labs, open models and partnerships
57:16 Inference profits and funding the next model
1:00:13 Academic research and new talent
1:02:47 Robotics and AI in the physical world
Listen to the show, and follow along for new episodes every Tuesday.
Wai Tong retweeted
I left my PhD 16 months ago to figure out how to scale AI inference optimization. Today, I’m proud to announce the release of Together Compiler! It's the first continual inference optimization system that adapts to new-age architectures and diverse user behavior. The first system designed for the age of “polymorphic” inference.
Together Compiler generates optimizations specialized to a workload and posts best-in-class TPS numbers on highly-optimized open source benchmarks: 1100+ TPS on MiniMax M3.1 Flash and 1200+ TPS for DeepSeek v4.1 Flash off InferenceX. 24 hours and just 2 nodes per model. That’s it!
Paper: alphaxiv.org/abs/2611.togeth…
It was super exciting to build this from scratch :) more thoughts in thread below!! 1/
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
Wai Tong retweeted
We had the first inklings last summer that it might be time to let go of all these abstractions with the new strength of AI agents. We tasked @reyna_abhyankar (then a PhD student and intern) with - “what if we just do all these kernels automatically?” 3/
Wai Tong retweeted
What if your agent could switch harnesses mid-run?
Imagine starting with Claude Code, then switching to Pi or Codex. Would it approach the task differently? Catch mistakes it previously missed? Could different harnesses complement each other? 🧵
Wai Tong retweeted
check out our recent work on cascade harnesses! i think more so than designing any one harness, it's important to understand what structures will maximize performance at low cost. this was a great collab!
Wai Tong retweeted
As a researcher, I cannot understate how much easier this determinism makes my life. No more mystery "maybe it's just distributed computing?? 🤷♀️" results from my dataloader
Zephon’s TorchTitan integration preserves identical global batches when training resumes on a different GPU count.
In our latest blog’s 1B-parameter model test, training resumed from a 16-GPU checkpoint on 8, 32, and 64 GPUs, and the SHA-256 hash of every global batch matched the original run.
Zephon tokenizes, packs, shuffles, and mixes data inside the training job, avoiding a preprocessed copy of the corpus.
The data, tokenizer, seed, mixture, context length, and global batch size must stay unchanged when resuming. The lane count must divide evenly across data-parallel replicas, and each optimizer step’s batch must split evenly across lanes. Training losses can still differ because floating-point reduction order depends on GPU count.
✍️ @datologyai @josh_wills @MaxiBoether, Ties Robroek (@datologyai)
Read more: pytorch.org/blog/bringing-el…
Wai Tong retweeted
Spent the past three weeks on this idea.
A sampler on a different precision or accelerator is just off-policy, like stale samples in fully async RL, and off-policy correction doesn't care why the sampler differs from the trainer. Meanwhile, inference keeps diversifying across precisions(MXFP8, MXFP4, NVFP4) and accelerators outside RL. So why not use all of that compute for RL rollout?
The sampler and the trainer are two systems built for different purposes. Hope this opens up new possibilities for RL algorithms and systems, and leads to cleaner system designs. Enjoy the blog, and stay tuned for the next one on our infra :)
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Wai Tong retweeted
Amazing work from @modal and @nanjiangwill !
A refreshing glimpse into the future of large-scale RL!
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Training large models with RL typically runs on a fixed homogeneous cluster of the latest accelerators for numerical stability and faster throughput. This throws away inference optimization work done for other vendors and generations, limits the capacity you can utilize, and is unnecessarily expensive.
So we trained a model on agentic coding tasks using fully asynchronous RL and mixed-precision, mixed-accelerator samplers. With off-policy correction, training stayed stable through convergence, on par with homogeneous samplers, at a lower cost.
Wai Tong retweeted
This result is super impressive ESPECIALLY considering the "baseline" here is crazy sophisticated
Replying to @datologyai
The gains persisted after post-training: our 30B-A3B model scored 46.8% vs. 37.7% for the baseline, averaged across benchmarks.
A curated 12B-A1.4B model also beat the 30B-A3B baseline despite using roughly one-fifth of the pretraining compute.
Wai Tong retweeted
Today, we’re excited to launch the Datology Curation Studio, customizable frontier data curation for every team training its own models.
Data quality is the ultimate compute multiplier, and Curation Studio makes it easy for everyone.
Wai Tong retweeted
Can confirm this works really well - amazing work @MaxiBoether and team!
At @datologyai we run thousands of data experiments. Two things keep biting us:
- Same run on 16 vs 8 GPUs → different evals
- Every pipeline change → re-materialize the dataset
Introducing Zephon, the data loader we built to process data on the fly, deterministically. 🧵
Wai Tong retweeted
My lab today at COLM: @GeorgeMorgulis presenting Subliminal Steering, 11:00-1:00 in Franciscan A #78
If you’re studying subliminal learning in LMs, you should be studying George’s variant, which is induced by steering. It just works way more consistently.
Proud to have contributed to this RL repo via the open source
Today we're launching Miles v0.1, an open-source RL framework for LLMs and multimodal models.
RL training is easy to start and hard to debug. Miles helps you ensure your run is correct, use hardware efficiently, and keep RL running at scale.
Over the past 9 months, 72 contributors have landed 1,326 commits, 85 GPU E2E CI tests, battle-testing Miles on frontier open models like Kimi K3, DeepSeek V4, Qwen 3.8, GLM 5.2, Inkling, MiniMax H3, etc.
Miles powers frontier-model development and production RL workloads at @humansand, @periodiclabs, @modal, @DecagonAI, @Eigent_AI, @nebiusai, @IBM and more, on both @NVIDIAAI and @AIatAMD hardware.
Here is what we built, and why teams picked Miles🧵


























