4th Machine Learning PhD student at @WisconsinCS | research scientist intern @MSFTResearch @AIatMeta FAIR @Adobe | BS 23' @WHU_1893
Joined November 2021
- Tweets83
- Following846
- Followers299
- Likes980
Leitian Tao retweeted
🧠 Thinking more ≠ telling users what they need to know. Introducing our #TMLR paper: "Thinking Is Not Telling: Information Disclosure in User-Service LLM Agents", w/ @Changdae_Oh @HyeonggyuC @jindong_wang92 @SharonYixuanLi
❓ Can stronger internal reasoning improve agent–user collaboration? The challenge: agents see policies, tool results, and environment states that users cannot. Users need this information before making consequential decisions—but private reasoning does not guarantee timely disclosure.
🔬 We introduce a five-category information-disclosure taxonomy, evidence-grounded probing, and turn-level counterfactual interventions to examine what agents communicate—and how missing information affects their actions.
🌟 Across five models and three service scenarios, enforced thinking does not consistently improve task success. More thinking does not reliably yield richer disclosure. State corrections and constraints are the most frequently missing categories in our matched failure analysis.
🎯 Injecting missing decision-relevant information improves the agent’s immediate downstream action correction beyond verbosity and think-content controls. A lightweight, training-free InfoDis prompt also mitigates enforced-thinking degradation: +5.26 percentage points for GPT-4o and +3.44 for DeepSeek-V3.1 in overall task success.
💬 Better user-service agents must communicate the right information before users decide.
📑 Paper: agent-infodis.github.io/pape…
🔗 Project Page: agent-infodis.github.io/
Leitian Tao retweeted
Introducing *HackBench* 🤡💀⏱️📊 ⏲️ 💪🔧 🔥
- Detecting & localizing reward hacking in autoresearch agents
- Frontier models fail
- Dataset on HuggingFace
- Recipe to collect natural & synthetic hacks
Blog post: facebookresearch.github.io/R…
🧵1/5
Leitian Tao retweeted
Your coding agent’s code can pass every test and still be exploitable, but users rarely know👾
🤔SFT, RL, or OPD: which teaches security best?
🔥SecureVibe helps vibe coding plan, code, and test for security, +6.9 pts BaxBench, +11.5 unseen CWEs SusVibes, +4.1 SWE-bench Verified
Leitian Tao retweeted
🔥 Excited to share Agent World Model in the Wild (WildAWM)! 🌍
Fully open-source synthesis 🔓: No proprietary models needed! We synthesized 341 high-quality environment–task pairs, addressing 3 challenges in task synthesis:
✅ Solvability: At least one agent can successfully complete each task.
🔥 Difficulty: Challenging even for frontier models, with GPT-6-Astra achieving only ~72.7 pass@8.
🎯 Verifier reliability: 95% agreement between task verifiers and agent-as-a-judge ensemble.
📈 Effective RL training: Trained exclusively in these synthetic worlds, Qwen3.6-35B-A3B achieves an average +10.58-point improvement across 8 OOD benchmarks!
Leitian Tao retweeted
🤩 Introducing Self-HPO: let the coding agent improve its own harness, then train a stronger policy model with the optimized harness, and repeat the process for recursive improvement!
💪 Self-evolving: no external model required
🎯 Harness search targets the policy's evolving failure modes
📈 Avg over SWE-Bench Verified/Multilingual/Pro: 57.8% → 64.8%
🧵[1/n]
Leitian Tao retweeted
🚀 What if AI agents could build the environments that train their next generation?
Excited to share our new work: AutoEnvScaling!
We turn environment design into a terminal task. A proposer agent studies the solver’s rollouts, builds and tests new RL environments, and adapts the curriculum as the solver improves. Build → train → learn → repeat. 🔄
✨ Up to 25× higher valid-environment rate than prompt-only generation, at up to 75% lower cost per valid environment.
📈 With Qwen3.6-35B-A3B playing BOTH proposer and solver, Terminal-Bench 2.1 improves from 41.1 to 53.3.
One model, two roles, an evolving curriculum—a step toward recursive self-improvement.
🌐 Project: microsoft.github.io/Orchard-…
📄 Paper: simonucl.github.io/assets/pd…
Leitian Tao retweeted
I’ll be at @COLM_conf this week (Oct 5–10), presenting work from my internship at ByteDance Seed:
“Towards Full Pipeline FP8 Reinforcement Learning for LLMs”
arxiv.org/abs/2609.22870
Happy to chat about RL infra, agentic RL, and continual learning. Come say hi if you’re around :)
Leitian Tao retweeted
🛑 Does post-training teach LLMs new agentic skills, or just sharpen what they already know?
[1/11] Introducing Sharpening Tax 🧾 to measure the hidden cost of post-training!
Leitian Tao retweeted
⏱️⚙️Introducing *AutoBenchmark* 📊🏁
- Creating benchmarks automatically
- Benchmarking benchmark creation
- Studying the role & impact of humans in the loop
Blog post: facebookresearch.github.io/R…
Key takeaways:
1) We find human-agent collaboration gives big wins over agents alone
- fine-grained feedback in ideation stage crucial
- autobench can be used to measure this in the future with stronger agents
2) We show that it's possible to make *autoresearch benchmarks* for AI research using this recipe – full recursive improvement loop!
3) Important Ingredients: Autobenchmark creation works best with feedback from two sources: benchmark solvers + external verifiers (human+AI).
🧵1/5
Leitian Tao retweeted
As hybrid computer use—the ability to dynamically interleave CLI and GUI actions for complex work—becomes essential for universal digital agents, we need tasks and environments to measure and advance it.
Introducing RecreationWorld: scalable, verifiable environments grounded in executable references across Ubuntu, macOS, Windows, Android and Web. 🧵
Website: recreation-bench.cc/
Code: github.com/QwenLM/Recreation…
Coding agents often submit patches without reliable executable verification—and poor tests can make repairs worse. EXECCRITIC learns both to test and improve, boosting Qwen3.5-35B-A3B from 61.2% to 72.6% on SWE-bench Verified, without stronger-model or Oracle feedback!
Excited to release our new work ExecCritic: Learn to Test, Test to Improve for Coding Agents.
We introduce a new test-verify-revise agentic scaffold and a role-specific RL recipe to train agents within it. On SWE-bench Verified, ExecCritic improves Qwen3.5-35B-A3B from 61.2% → 72.6%, without stronger-model or Oracle feedback at evaluation time.
🤔 The core problem:
Coding agents typically explore the repository, edit the source code, and submit once they believe the patch is ready. But this loop only works if both sides are reliable: the test must capture the right behavior, and the agent must know how to improve from its feedback.
When the same trajectory writes both the patch and the test, their errors can agree. A wrong patch may pass a wrong test and appear correct.
And even with a useful test, the agent still needs to learn how to interpret execution feedback and turn it into the right source-code revision.
Empirically, test quality determines whether feedback helps or hurts. Poor tests can actually make the agent worse, while better tests lead to better repairs.
⚙️ Our approach: EXECCRITIC
We separate test construction from source-code repair and train both roles specifically for this new agentic scaffold.
Learn to Test
A Test agent independently explores the repository and generates a task-specific executable regression test. The harness verifies that it runs and fails cleanly on the buggy repository, then freezes it.
issue + repo → generate test → verify on Base repo → freeze verifier
Test to Improve
A separate Repair agent generates a repair patch, runs it against the fixed verifier, receives execution feedback, and iteratively revises the source code.
generate repair patch → run fixed verifier → FAIL → revise repair patch → run verifier → … → PASS → submit
The Repair agent cannot modify the test to make its own patch pass.
📈 The results:
Learning to test substantially improves test reliability, measured by Base-to-Gold success:
Base Qwen-35B: 22.2%
+ SFT: 39.6%
+ RL: 62.2%
Composing the trained Test and Repair agents reaches 72.6% on SWE-bench Verified, an 11.4 percentage-point improvement over the original 61.2% baseline, without a stronger model or Oracle feedback at evaluation time.
🔍 The broader takeaway:
For coding agents, execution feedback alone is not enough.
We need agents that can both construct reliable feedback and learn to improve from it.
As coding agents become longer-horizon and more autonomous, learning how to test may become just as important as learning how to code.
📄 Paper: arxiv.org/abs/2609.09133
💻 Code: github.com/MSR-Orchard/execc…
Led by the amazing @LeitianT during his internship at @MSFTResearch, with huge thanks to our collaborators Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, and @JianfengGao0217.
Leitian Tao retweeted
😢RLVR is powerful but expensive
🤯Imagine using <20% RLVR training while achieving 100% performance?
Sounds surprising? We show that minimal RLVR training is enough to know where training is going, and predict future ckpts at no training cost!
📃tinyurl.com/minimal-rlvr
🧵[1/n]
Leitian Tao retweeted
In this recent paper, the scientists from @MSFTResearch and @UWMadison develop TRACE, a novel credit-assignment method using log-ratio state values and Temporal-Difference changes to significantly improve long-horizon agent tool-use in reinforcement learning, achieving substantial performance gains on complex search benchmarks without requiring extensive supervised pretraining or specialized critics.
Read with an AI tutor to understand everything: chapterpal.com/s/c8e5636e/tr…
Excited to share our new work with Microsoft! We dig into why outcome-based RL struggles with long-horizon agents: credit assignment. TRACE offers a simple way to assign turn-level rewards—without process labels, trained critics, Monte Carlo continuations, or trained LLM judges!
Scaling outcome-based RL won't solve long-horizon agentic tasks. Credit assignment is the bottleneck, and turn-level reward is inevitable.
The question is how to get it. Process labels are expensive; we all know that.
Excited to release TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents. No trained critic, no process labels, no Monte Carlo continuations, no strong LLM judge. 🧵
📄 arxiv.org/abs/2607.13988
🤔 The core problem:
Outcome rewards work well when a task contains a short reasoning trace and an easily verified answer. But agents may perform tens or hundreds of tool interactions before responding.
A single final reward tells us whether the agent succeeded, but not which tool calls uncovered decisive evidence, which were redundant, or which sent the agent down the wrong path. This makes RL increasingly sparse, noisy, and inefficient as trajectories grow longer.
⚙️ Our approach: TRACE
TRACE assigns credit at individual tool-call boundaries.
For each trajectory prefix, a frozen reference model measures how much more predictable the gold answer has become. TRACE converts this into a state value and uses temporal-difference changes to reward each interaction:
→ Positive credit when a tool call moves the agent closer to the answer
→ Near-zero credit when it adds redundant information
→ Negative credit when it moves the agent away from the answer
Probability to predict the final answer remains the global training objective; turn-level rewards simply reveal which actions contributed to it.
🔍 Why the formulation matters:
The turn-level rewards telescope across the trajectory. Repeating searches or reopening the same evidence cannot artificially accumulate additional one-step credit.
TRACE can also propagate delayed progress backward: if a search surfaces a promising page and a later `open` reveals the decisive evidence, both interactions can receive appropriately localized credit.
📈 The results:
Using pure RL (without cold-start SFT, agentic mid-training, or live-web training data), TRACE substantially improves long-horizon search:
→ Qwen3-4B: 7.2 → 35.6 on BrowseComp-Plus
→ Qwen3-30B-A3B: 8.4 → 42.6 which is on-par with the Tongyi-DeepResearch-30B-A3B.
Under controlled comparisons with the same models, data, tools, and outcome rewards, TRACE also outperforms GRPO, GSPO, and GiGPO across both model scales.
📄 Paper: arxiv.org/abs/2607.13988
Led by the amazing @LeitianT, and huge thanks to our collaborators at Microsoft Research: Baolin Peng, Wenlin Yao, Tao Ge, Hao Cheng, Mike Hang Wang, and @JianfengGao0217.
Leitian Tao retweeted
TRACE assigns dense credit at tool boundaries by asking a frozen reference model whether each new observation raises the gold answer's log-probability, converting those state-value changes into temporal-difference rewards without training a critic or process reward model.
TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents
Paper: arxiv.org/abs/2607.13988
Leitian Tao retweeted
Outcome reward models: cheap, but vulnerable to spurious shortcuts 😣
Process reward models (PRMs): robust, but too expensive to build from scratch 😫
What if you could get a ready-to-use PRM right after any RL post-training?
Introducing 'Progress Advantage' 🧵
Leitian Tao retweeted
🚨 New paper: "Self-Compacting Language Model Agents"
LM agents build up long traces of reasoning and tool calls. As the trace grows, old mistakes and stale info stick around and anchor everything that follows. We ask: can the model itself decide when to clean up?
Leitian Tao retweeted
Agent RL training can be fragile and far less stable than reasoning RL. In our latest work, we identify and explain a phenomenon called Cyclical Entropy Eruption: a recurring instability unique to agent RL where entropy erupts, recovers, and erupts again throughout training. 🧵
📄 arxiv.org/abs/2605.27954 (led by @Wendi_Li_ and @shawnim00)
We decompose it into three phases:
Phase 1 Entropy Descent. The model first learns the basics: how to call tools, satisfy schema constraints, and use the right format tokens. Probability mass shifts rapidly from invalid outputs to valid trajectories. Entropy drops fast.
Phase 2 Entropy Eruption. The core instability. Correct and incorrect agent trajectories end up extremely close in representation space, far more overlapping than in non-agent tasks. This causes gradient interference: when RL suppresses bad trajectories, it accidentally drags down the likelihood of good ones too. The policy flattens, entropy spikes, and degenerate patterns like sentence duplication and hallucination emerge.
Phase 3 Entropy Subsidence. The flatter distribution leads to more diverse sampling, which reduces representation similarity and eases interference. Training recovers... but then reconverges, similarity rises again, and the next eruption begins. A self-perpetuating cycle.
The damage compounds. Degenerate patterns acquired during eruption persist and accumulate across cycles. In the worst case (Llama3.2-1B on WebShop), a single eruption triggers complete training collapse.
Motivated by our analysis, we propose SEAL (Separation-Enhanced Agent Learning)--a lightweight auxiliary loss that pushes correct and incorrect trajectories apart in representation space, directly targeting the root cause.
SEAL stabilizes training and improves performance across AlfWorld, WebShop, and search-augmented QA, on both Qwen and Llama backbones, with GRPO and GIGPO. On a Llama run where vanilla GRPO completely collapsed (0% success), adding SEAL recovered performance to ~80%.
💡The broader takeaway: agent RL is a fundamentally different optimization problem than reasoning RL. The multi-turn structure, tool interactions, and validity constraints create training dynamics that deserve their own analysis. We hope this work helps the community build more stable agent post-training pipelines.
Code is available in the paper!
Leitian Tao retweeted
Best-of-N sampling is often used to boost LLM performance, but the selection relies on external evaluators, adding cost and bias. What if you could select the best output without any external scoring at all?
Introducing our #ACL2026 paper ModeX: Evaluator-Free Best-of-N Selection for Open-Ended Generation! (led by Hyeong Kyu Choi @HyeonggyuC)
💡 Our key insight: among multiple LLM generations, high-quality outputs tend to cluster together semantically. The best answer is the modal one: the generation that captures the dominant consensus.
How ModeX works:
1⃣ Build a similarity graph over N candidate generations
2⃣ Recursively apply spectral clustering via the Fiedler vector to isolate the dominant semantic cluster
3⃣ Select the centroid of that cluster as the final output
No reward models. No external evaluators. No auxiliary inference. Just the texts themselves.
📊 Results across text summarization (CNN/DailyMail), code generation (HumanEval), and math reasoning (Math-500) show ModeX consistently outperforms single-path and multi-path baselines, achieving state-of-the-art among evaluator-free methods.
We also provide theoretical justifications connecting our graph-based mode selection to kernel density estimation, grounding the approach with principled foundations.
📄 Paper: arxiv.org/abs/2601.02535
💻 Code: github.com/deeplearning-wisc…
Sometimes the best signal is already hiding in the samples; you just need to find the mode. 🎯
