Robotics Research Scientist at The AI Institute | @CMU_Robotics PhD
Boston, MA
Joined November 2013
- Tweets217
- Following406
- Followers1.2K
- Likes675
First time being on a podcast, and it's RoboPapers! 🤩
I had a great time discussing SOLE-R1 (philip-mit.github.io/sole-r1…) with @micoolcho, @DJiafei, and @philip_mit
Reinforcement learning is often seen as key to achieving scalable, highly reliable real-world robot performance, yet it requires good reward functions that can supervise the wide range of tasks that need to be performed. The obvious thing to try would be a large vision-language model (VLM), but these models often fail under common real-world conditions like partial observability and occlusion.
As a solution @Philip_MIT and @thomas_weng propose SOLE-R1, a video-language reasoning model trained to serve as a reward signal for online RL. Given real video, SOLE-R1 will reason over input video frames, generates reasoning traces, and predicts a reward signal. This allows online RL from random initialization on new manipulation tasks.
To learn more, watch Episode 111 of RoboPapers, with @micoolcho and @DJiafei!
Another snippet from the full episode: sim and real datasets are both valuable in training a task progress predictor like SOLE-R1, as these datasets cover task and trajectory diversity differently.
Check out the full episode and paper for more details esp. regarding training data for spatiotemporal reasoning and avoiding reward hacking during real-world online RL.
Full episode dropping soon!
Geeking out with @Philip_MIT @thomas_weng on SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot RL philip-mit.github.io/sole-r1…
Co-hosted by @micoolcho @DJiafei
1
6
362
Stay tuned for the full RoboPapers episode on SOLE-R1, our VLM task progress predictor for on-robot RL👀
Full episode dropping soon!
Geeking out with @Philip_MIT @thomas_weng on SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot RL philip-mit.github.io/sole-r1…
Co-hosted by @micoolcho @DJiafei
6
585
Thomas Weng @ CoRL 2026 retweeted
Designing reward functions for real-world RL is a serious challenge in learning for manipulation.
Zero/few-shot reward models that “just work” is going to be a huge unlock for the entire field to go beyond behavior cloning!
Excited to share SOLE-R1 is accepted to NeurIPS 2026!
General-purpose reward models are fundamental to unlocking real-world RL.
We show SOLE-R1's zero-shot reward prediction can serve as the SOLE signal for learning over 20 unseen manipulation tasks through online RL (i.e., randomly initialized policies, 0% starting success rate).
This work provides a strong proof of concept: RL with zero-shot dense reward prediction can do more than just increase success/speed for existing tasks; it can enable robots to learn entirely new tasks from scratch, without any demonstrations or ground-truth rewards, starting from zero competence.
Very proud of this project!
Webpage: philip-mit.github.io/sole-r1…
Thank you to my coauthors @thomas_weng, @_ericrosen, @BizaOndrej.
#NeurIPS2026
Currently looking at new research directions and SOLE-R1 keeps coming up!
Excited to share SOLE-R1 is accepted to NeurIPS 2026!
General-purpose reward models are fundamental to unlocking real-world RL.
We show SOLE-R1's zero-shot reward prediction can serve as the SOLE signal for learning over 20 unseen manipulation tasks through online RL (i.e., randomly initialized policies, 0% starting success rate).
This work provides a strong proof of concept: RL with zero-shot dense reward prediction can do more than just increase success/speed for existing tasks; it can enable robots to learn entirely new tasks from scratch, without any demonstrations or ground-truth rewards, starting from zero competence.
Very proud of this project!
Webpage: philip-mit.github.io/sole-r1…
Thank you to my coauthors @thomas_weng, @_ericrosen, @BizaOndrej.
#NeurIPS2026
Exciting to be organizing a #CoRL2026 workshop on real-world reinforcement learning!
Post-training in the real world will be critical for deploying useful robots, but it's currently being slept on. The R2RL workshop brings together researchers working in this area to network and share ideas.
If you're pioneering this research, consider joining us at the workshop! Workshop submissions accepted until Sep. 30.
We’re trying something new at the R2RL Workshop at CoRL 2026. 🔗 r2rl-workshop.github.io/
No senior-speakers, no passive audience.
Key Q: What are the real bottlenecks in Sample-efficient, Real-World RL?
🧑🔬 No senior speakers. Students & scientists who actually deploy RL on robots.
🗳️ Crowdsourced problems. Audience proposes challenges, we all vote on what to focus on.
🧩Breakout sessions. Audience helps to propose and discuss solutions for top-voted problems.
🎤 Panel. The breakout discussions determine what we debate.
📝 Whitepaper: a community-constructed whitepaper summarizing the most urgent challenges for Real-World RL in the next year.
Submit your papers!
Attend to help us figure out what we should actually be solving next!
Submission deadline: September 30
Blog post out for our RL post-training method, Q2RL!
Looking to improve your deployed robot with reinforcement learning in the real world?
Presented at RSS 2026, Q-Estimation and Q-Gating from Behavior Cloning for Reinforcement Learning (Q2RL) improves robot policies in a few hours of real world interaction, avoiding catastrophic forgetting when fine-tuning a Behavior Cloning (BC) policy via Reinforcement Learning (RL) post-training.
Q2RL estimates a Q-function for the BC policy, and uses it to guide online RL. Learn more about the method here: rai-inst.com/resources/blog/…
Thomas Weng @ CoRL 2026 retweeted
VLAs can do a lot zero-shot. But when one fails, there's no way to tell it what went wrong - you collect more demos and retrain.
What if you could just say something?
Introducing ARCHITECT 🤖👷: a robot coding agent that synthesizes policies from free-form language instructions.
robo-architect.github.io/
🧵1/N
Thomas Weng @ CoRL 2026 retweeted
I’m in Sydney this week to present Q2RL at #RSS2026! I’ll be speaking in the RL oral session on Wednesday from 5:00–5:40 pm, followed by the poster session with @thomas_weng. Come say hi if you are around ! I’d love to chat about RL, post-training or robotics in general!
Can BC policies be quickly improved through real world experience?
Our new #RSS2026 paper proposes Q2RL, a method that bridges BC and RL for on-robot learning.
Q2RL improves BC policies by up to 3.75x with just 1-2 hours of online interaction!
So when life gives you BC, make Q-functions! 🍋
Details in thread 🧵
Thomas Weng @ CoRL 2026 retweeted
New work with @nvidia: evaluating robot policies entirely inside a world model. The policy acts, the model imagines the consequences, and the imagined evals predict real-world results. 🧵
real vs world-model rollout side by side📷
Had a great time attending and presenting at NEMS (New England Manipulation Symposium, robot-manipulation.org/nems-…) this year!
Met lots of fellow roboticists, and had many interesting discussions around q2rl.rai-inst.com 🍋
7
286
Thomas Weng @ CoRL 2026 retweeted
Researchers at RAI Institute developed the Koala gripper to help robots handle tools and objects more naturally.
The system includes a handheld version for humans and a motorized version for robots, both using the same design and sensors.
By learning from human demonstrations recorded with cameras and force sensors, robots can repeat the same tasks using the Koala gripper.
Thomas Weng @ CoRL 2026 retweeted
Our SOTA reward model for robotic manipulation. I'm very proud of this work; it was more than a year in the making. Lessons learned below:
Excited to introduce SOLE-R1, a video-language reasoning model for zero-shot reward prediction for robot manipulation tasks!
SOLE-R1 reasoning can serve as the SOLE signal for learning new tasks (completely from scratch) through online RL - i.e., robots start with random actions and learn previously unseen tasks guided only by SOLE-R1 rewards, without any demonstrations, ground-truth rewards, success indicators, or task-specific tuning.
SOLE-R1 significantly outperforms strong baselines (e.g., Robometer, RoboReward, TOPReward, GPT-5, Gemini-3-Pro) in zero-shot online RL when evaluated across 40 tasks - including a real-world tabletop manipulation setting and 4 sim environments (LIBERO, ManiSkill, Meta-World, RoboSuite).
We open source all models, training data, and code.
Website, demos, and paper at: philip-mit.github.io/sole-r1…
🧵 (1/6)
We found that state-of-the-art VLMs (Gemini, GPT-5, etc.) fail at predicting task progress for online RL, so we built our own: SOLE-R1.
SOLE-R1 is trained on 10 million images and video frames, and 4 million chain of thought traces that reason over both space and time.
The result is a video-language reasoning model that can be used as a reward for online RL with no other reward signals!
Excited to introduce SOLE-R1, a video-language reasoning model for zero-shot reward prediction for robot manipulation tasks!
SOLE-R1 reasoning can serve as the SOLE signal for learning new tasks (completely from scratch) through online RL - i.e., robots start with random actions and learn previously unseen tasks guided only by SOLE-R1 rewards, without any demonstrations, ground-truth rewards, success indicators, or task-specific tuning.
SOLE-R1 significantly outperforms strong baselines (e.g., Robometer, RoboReward, TOPReward, GPT-5, Gemini-3-Pro) in zero-shot online RL when evaluated across 40 tasks - including a real-world tabletop manipulation setting and 4 sim environments (LIBERO, ManiSkill, Meta-World, RoboSuite).
We open source all models, training data, and code.
Website, demos, and paper at: philip-mit.github.io/sole-r1…
🧵 (1/6)
Thomas Weng @ CoRL 2026 retweeted
I will be presenting my paper “Accelerating Residual Reinforcement Learning with Uncertainty Estimation” at #ICRA2026 in Vienna. Find me on Thursday, June 4 from 9 - 10:30 in Hall C.
more details : lakshitadodeja.github.io/unc…
Also presenting my RSS paper “When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning” in the RL4IL workshop on Monday, June 1 from 15 - 15:30 at Hall A1.
more details : q2rl.rai-inst.com/
Come say hi !
This was one the coolest parts of the project for me:
Q2RL helps imitation policies get "unstuck" by switching to reinforcement learning!
For the inset video, switching to RL for just a moment (the yellow flicker) allows the kitting task to proceed.
On top of that, the RL policy will continue to improve its prediction of which actions lead to high reward through online reinforcement learning.