Haoran Xu ✈️NeurIPS2026 (@ryanxhr) header
Haoran Xu ✈️NeurIPS2026 (@ryanxhr) profile picture

PhD student @UTAustin | @AmazonScience AI PhD Fellow | Towards super-human AGI using RL🚀

Austin, TX
Joined December 2016
Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)Photo by Haoran Xu ✈️NeurIPS2026 (@ryanxhr)
Haoran Xu ✈️NeurIPS2026 retweeted
Off-policy Q-Learning is known to prone to long-horizon error issues. While multi-step/horizon-reduction methods reduce the horizon errors, they introduce off-policy bias due to sub-optimal off-policy data. Introducing Horizon-scalable Q-Learning ✨HQL✨
2
20
2
133
7,313
Haoran Xu ✈️NeurIPS2026 retweeted
对搞 AI agents、reasoning model 的人来说,这篇论文真的很实用性有趣。 如果你在做 reasoning model 的 post-training、蒸馏、on-policy 数据生成,或者想改进小模型的长链思考能力,这篇能给你一个干净的思路和可落地的 trick。 尤其对 hard math 问题,收益比较实在。 现在训小模型学大模型的长链思考,像 o1 那种,常用 on-policy distillation是让小模型自己先生成 reasoning 轨迹,一步步写思考过程,然后大老师模型在它生成的轨迹上逐个 token 教它。 小模型如果开头几步就走错了,后面再怎么写都救不回来。这时老师还得硬着头皮在那个带病前缀上继续监督。 结果老师自己的输出分布会分裂成两半,一半想继续错下去,一半想纠正,给学生的学习信号就变得很乱、梯度碎片化。 以前大家各种修 token loss 的 trick,比如截断、重加权、top-k 啥的,都只能治标不治本,因为问题根子在整个轨迹的前面就坏了。 就像是学生写数学证明,第一步假设错了,老师还得在错的假设上逐字批改后面每一步,教出来的东西自然稀碎。 论文的解法是Trajectory-Refined Distillation。 不直接在学生原始轨迹上蒸馏,主要让老师先把整个学生轨迹重写、精炼成一个更好的版本,修正错误前缀,同时尽量保留学生原来有用的部分和风格,保持在学生分布支持内。然后,学生去模仿这个被老师修正过的干净完整轨迹。 我理解这种方法算是把监督从token 级修补升级到trajectory 级修正了。 尽管这只是incremental contribution,而且相当依赖老师 refinement 的质量,如果老师不够强,修正效果会打折。但是整体上把把 prefix failure”这个现象说清楚、形式化了,还有实验证据支持已经很好了。 个人的品味,感觉这篇尤其在难的数学题上 gains 明显,还能让轨迹变短,更高效,自蒸馏时甚至能解出原来完全解不出的题。一个小改动就实际能用的感觉很舒服。
"Trajectory-Refined Distillation" This paper shows a core failure in on-policy distillation. When a student takes a wrong reasoning path, the teacher is forced to supervise from that broken prefix, so token-level KL becomes noisy. So they introduced TRD that fixes this by refining the whole rollout first. What it does is the teacher rewrites the student trajectory into a better reasoning path, then the student distills from that corrected trajectory. While this is a small change, it still moves distillation from token fixes to trajectory fixes, giving cleaner supervision and stronger reasoning gains.
3
18
108
13,605
Haoran Xu ✈️NeurIPS2026 retweeted
🤔On-policy (self) distillation has a prefix failure issue: when the student's rollout y_o is wrong (far from the teacher distribution), the teacher’s gradient signal becomes problematic. 🚨Introducing Trajectory-Refined Distillation (TRD), a trajectory-level refinement of y_o to solve the issue. tldr: OPD/OPSD❌ TRD ✔️ Paper: arxiv.org/pdf/2606.08432
4
15
154
10,695
Haoran Xu ✈️NeurIPS2026 retweeted
OPD is on-policy, but its supervision is still post-hoc and one-step. The student generates a rollout. The teacher then supervises that fixed trajectory token by token. Our new paper argues that this can fail at the wrong scale. When the prefix itself is broken, the problem is not only which tokens to reweight, clip, or truncate. The problem is the trajectory. We propose Trajectory-Refined Distillation (TRD): refine the student rollout under teacher guidance before distillation. with @ryanxhr Yichuan Ding @yayitsamyzhang Paper: arxiv.org/pdf/2606.08432
4
12
1
92
5,144
Haoran Xu ✈️NeurIPS2026 retweeted
Replying to @ryanxhr
@ryanxhr has developed this very nice work framing offline RL as an optimal transport problem, with SOTA results on offline RL benchmarks and LLM RL tasks. Check it out, and chat with him at ICLR!
Both offline RL and LLM RL fine-tuning can be formulated as behavior-regularized RL problems. We propose Value Grdient Flow (VGF), a new scalable and sample-efficient paradigam that treats behavior-regularized RL as an optimal transport problem. arxiv.org/abs/2604.14265 🧵[1/7]
3
3
40
3,078
Haoran Xu ✈️NeurIPS2026 retweeted
Both offline RL and LLM RL fine-tuning can be formulated as behavior-regularized RL problems. We propose Value Grdient Flow (VGF), a new scalable and sample-efficient paradigam that treats behavior-regularized RL as an optimal transport problem. arxiv.org/abs/2604.14265 🧵[1/7]
4
23
1
177
14,230
Haoran Xu ✈️NeurIPS2026 retweeted
⚠️ Reminder! Submissions for @RL_Conference's RL beyond Reward Workshop are due May 30 (AoE)! We are brewing an interesting program and seeking innovative research work in reward-free RL. All papers are welcome, from exploratory abstracts to complete research papers.
1
13
6
50
16,004
Haoran Xu ✈️NeurIPS2026 retweeted
I will miss #ICLR2025 but come to check our work on a new perspective of solving Reinforcement Learning using discriminator-weighted imitation learning. @ShuozheL and @yayitsamyzhang will present it during today’s poster session.
1
5
1
14
2,743
Haoran Xu ✈️NeurIPS2026 retweeted
The Belief State Transformer edwardshu.com/bst-website/ is at ICLR this week. The BST objective efficiently creates compact belief states: summaries of the past sufficient for all future predictions. See the short talk: microsoft.com/en-us/research… and @mgostIH for further discussion.
5
18
3
105
16,363
Haoran Xu ✈️NeurIPS2026 retweeted
🤖 Introducing RL Zero 🤖: a new approach to transform language into behavior zero-shot for embodied agents without labeled datasets! RL Zero enables prompt-to-policy generation, and we believe this unlocks new capabilities in scaling up language-conditioned RL, providing an interpretable link between RL agents and humans and achieving true cross-embodiment transfer.
3
34
7
213
30,017
Haoran Xu ✈️NeurIPS2026 retweeted
Do RL solutions share a common structure? We show that all solutions of Reinforcement Learning lie on a hyperplane. Our work, Proto Successor Measure, learns this abstraction of the MDP to do zero-shot RL for any reward function. (1/n)
3
15
4
102
21,224
Haoran Xu ✈️NeurIPS2026 retweeted
🚀 Despite efforts to scale up Behavior Cloning for Robots, large-scale BC has yet to live up to its promise. How can we break through the performance plateau? Introducing 🔥FLaRe: fine-tuning large-scale robot policies with Reinforcement Learning. robot-flare.github.io/ 🧵
1
37
10
122
33,830
Haoran Xu ✈️NeurIPS2026 retweeted
How can you use offline datasets to imitate when an expert only provides you with observation trajectories (without actions)? Ex: robot's data of prior interaction + some tutorial videos. Our #CoRL2024 paper gives a simple and principled off-policy algorithm - DILO!
2
11
2
79
11,204
Haoran Xu ✈️NeurIPS2026 retweeted
Great keynote by David Silver, arguing that we need to re-focus on RL to get out of the LLM Valley @RL_Conference – at Amherst, MA
11
66
13
559
124,998
Haoran Xu ✈️NeurIPS2026 retweeted
On popular demand due to the upcoming Neurips deadline, we are further extending the submission deadline to 👉23 May 2024👈 AoE! Excited to see your ideas brewing for NeurIPS. Moreover, submissions in NeurIPS format are allowed.
4
2
7
6,509
Haoran Xu ✈️NeurIPS2026 retweeted
Announcing the Reinforcement Learning Beyond Rewards workshop at the first @RL_Conference. Think that rewards aren't enough for RL? Working on RLHF? Thinking of alternative ways of alignment? Creating a foundational model for RL? or have ideas on task-agnostic RL algo? Join us
1
15
3
64
23,843
Haoran Xu ✈️NeurIPS2026 retweeted
What happens when your dataset for offline RL is unreliable? 🤔Our recent study on robust offline RL with data corruption investigates this problem, which was accepted by #ICLR2024 as a spotlight presentation (top 5%) 🚀🚀 arxiv.org/abs/2310.12955
1
6
1
15
3,736
Haoran Xu ✈️NeurIPS2026 retweeted
Dual RL has been accepted as a spotlight (top 5%) in ICLR 24! Detailed post soon
In our new preprint 📜(arxiv.org/abs/2302.08560), we show how a number of recent SOTA methods in Reinforcement and Imitation Learning can be unified as RL methods in dual space, utilizing a framework originally proposed by Nachum et al! J/w @yayitsamyzhang , @scottniekum (1/5)
3
6
64
9,183
Haoran Xu ✈️NeurIPS2026 retweeted
Since prompting, instruction tuning, RLHF, ChatGPT etc are such new and fast-moving topics, I haven't seen many university course lectures covering this content. So we made some new slides for this year's CS224n: NLP w/ Deep Learning course at @Stanford! web.stanford.edu/class/cs224…
17
267
16
1,454
339,235