Building agentic and tiled stuff @UofIllinois | prev @togethercompute @ZJU_China
San Francisco, CA
Joined November 2023
- Tweets84
- Following1K
- Followers425
- Likes206
Tagging folks who might be interested in FlashState! Who are also my heroes and much of the reason why i started looking into linear attention and efficient architectures since my second year of undergrad two years ago! @SonglinYang4 @simran_s_arora @tri_dao @_albertgu @realDanFu @Mlsys_HaoKang @wenhaocha1 @Xinyu2ML
7
895
Weili Xu retweeted
Is claude code usage nerfed for max 20x this week? Feeling like it depletes much faster...
We open source everything, and we can't wait for you to give FlashState a try! 👋
💻 Code: github.com/FutureMLS-lab/Fla…
📄 Paper: alphaxiv.org/abs/2611.flashs…
🌐 Project Page: flashstate.ai
It's been a lot of fun and learning to work it out, with @connorliml, @tianyi___zhang, @JueWANG26088228, @ZhongzhuZhou, and @Chenfeng_X!
#AgenticAI #MLsys #vLLM #LLMinference #DeltaNet #mamba #attention #gpu #inference #infra #nvidia #KVCache #AIinfra #SGLang #kimi #glm #qwen #LLM
3
16
1,928
Weili Xu retweeted
Introducing the Together Compiler: continual compilation for AI inference! I’m super proud to finally share this work from our research team! ✨
Today’s inference is “Polymorphic” or shape-shifting, it doesn’t sit still. On OpenRouter, 80% of traffic goes to models released in the last three months and the top ten most popular open models each use a different architecture. Our customers constantly change their usage behaviors. Together Compiler leverages agents to continually optimize our deployments immediately as these changes occur.
Together Compiler research highlights include:
1. 252x better throughput: TKCC lets agents try 252x more optimization experiments per hour compared to prior agent inference optimization systems.
2. State-of-the-art TPS across InferenceX workloads we tried: TKCC outperforms the best InferenceX engine across all tested settings in just 24 hours & 2 nodes per compile. Including 1111 TPS on MiniMax M3.1 B200 and 1205 TPS on DeepSeek v4.1 Flash GB300 at batch size 1. NVIDIA, AMD, Trainium2.
3. Continuous recompilation: TKCC currently serves many trillions of production tokens per day and adapts to traffic shifts within just a few hours of recompilation.
4. Bitter lessons: TKCC’s design builds on 2+ years of building frameworks to accelerate AI Inference, starting from ThunderKittens and Megakernels. Compilers have always used abstractions and IRs, but we show that agentic compilers now reach higher performance ceilings without abstractions.
From an incredible team @reyna_abhyankar @dylan__lim @realDanFu @togethercompute!!
Glad to see llm-d adopting ThunderAgent!
Stay calmed, we have something even better coming soon :)
Reduce your agent serving cost over 2x !!!!!
Thunderagent is merged into LLM-D and delivers a 1.5x–2x output throughput boost and drops median TTFT from ~50s to 2–4s on real multi-turn agent traces (Claude Code / Weka traces).
Meanwhile, for how long do you think PD-dissagreegation is gonna be killed for agentic serving/RL rollout?🤣
code: github.com/HaoKang-Timmy/Thu…
paper: github.com/HaoKang-Timmy/Thu…
1
11
832
Weili Xu retweeted
While exploring diffusion language models (DLMs), we kept asking ourselves: given speculative decoding, a lossless acceleration technique, why do we still need DLMs? 🤔 Our latest work, ORL, tries to answer this question: AR optimizes intelligence 👆, speculative decoding optimizes speed 👉, and DLMs optimize the Pareto frontier of speed and intelligence 🚀. Through co-design across the algorithms, the training and inference engine, and the kernels, ORL pushes out a new Pareto frontier of intelligence and speed. Similar performance, much faster than speculative decoding. You say lossy spec? ORL-DLM is better too! 😎
🌐 Project + live demos: introspective-diffusion.gith…
Weili Xu retweeted
While QAT helps improve model quality at 2-4 bits, it has a structural optimization problem. Indeed, at every step, the high-precision latent weights receive the gradient updates, but the loss and gradients are computed on their low-bit reconstructions, i.e. the weights the model will use at inference.
This mismatch results in suboptimal training dynamics and a higher loss floor. The main existing QAT methods (learning the scales alongside model parameters, soft-to-hard rounding, or better initialization) do not directly address this mismatch.
With QUASAR, we introduce a simple and computationally cheap way to address the mismatch. QUASAR consistently achieves lower loss than all QAT baselines and adds no inference overhead. Also, it can be combined with most other QAT methods, since it only changes the weight reconstruction map in the fwd pass.
Weili Xu retweeted
Real-time video assistants have to decide when to respond and when to stay silent. Offline scores do not capture this. InteractionBench evaluates this decision for the complete system of model, memory, and response controller, with 1,060 interactions over 812 videos.
Weili Xu retweeted
We’re teaching a new course at Penn: CIS 6280 · World Models 🌍
cis.upenn.edu/~cis6280/
Every day, LLMs amaze me with one more thing they can solve. But I’ve always felt there’s more to learn than what humans have written down—from observing the world, interacting with it, and experiencing what happens next. To me, that’s what world models are about—and why I see them as one of AI’s next big bets.
However, I struggled to find a systematic course on them—so we are building one!
Topics will include: representation learning, generative models, simulation, model-based RL, video and 3D generation, world models for robotics, reasoning, and code-based world models.
Hands-on work will include:
- Building an environment
- Training a world model
- Learning a policy
and a final research project with leaderboard.
We’re already six lectures in. Slides, demos, and readings are public, and we’ll keep adding materials throughout the fall.
Big thanks to our TA team— @TongMutianTMT @hagsaeng_bag @EnxinSong @KeelyAi04 —for helping bring this course to life!
Weili Xu retweeted
🎉Qwen3.8-27B 迎来最强 4-bit 量化版！
社区发布了基于量化感知训练（QAT）的：QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4
🔥核心亮点
1. 极致压缩
全网络 496 个线性层全面采用 NVFP4（W4A4）量化，模型体积从 55.6GB → 19.7GB，大幅降低显存占用
2. 近乎无损性能
通过蒸馏学习，保留原版几乎 100% 的推理与数学能力，表现远超传统 PTQ 量化方案
3. 即插即用
原生支持 vLLM / SGLang，无需额外转换，直接部署运行
4-bit 不再只是能跑就行，而是开始追求接近原版的性能了！
🎉Qwen 3.8 27B 变身 20GB！Blackwell 架构完美适配版！
社区又又又推出了 Huihui-Qwen3.8-27B-abliterated-NVFP4 最新量化版本！
🔥核心亮点：
1. 体积暴减： 顺应 8 月新发布的 Qwen 3.8 27B 基座，将 55.6 GB 降至 20.0 GB，双 16GB 显卡即可留足 KV 缓存轻松跑起来，无压力
2. 修复长思考 Bug： 完美解决上一版在 Qwen 3.8 标志性的 xhigh 深度思考模式下，因思维奔逸陷入万字复读（thinking-runaway）的致命缺陷
3. MTP 投机采样： 完好保留 bf16 格式的 MTP 团队草稿头，在 RTX PRO 2000 Blackwell 上单流 Prefill 飙到 3,250 tok/s
划重点 部署提示： 基于 vanilla vLLM v0.22.0，直接 vllm serve 自动识别 compressed-tensors，无需额外量化 flag
无论你是想本地调教还是用作生产力
这个版本都是性价比之王
Qwen 牛牪犇！
传送门👇🏻
