Pinned Tweet
Robots should understand the motion and intent behind each action. Reasoning about motion, paired with language actions, unlocks remarkable transfer. Interestingly, it beat more naive reasoning like bounding boxes, keypoints, and visual traces. Excited to see where this goes!
What’s missing from robot foundation models? An explicit understanding of WHY motions work: why move in this direction, why place the object in this way, how hard to grasp the object. This motion-level reasoning bridges instructions to actions—and provides knowledge that can generalize across embodiments and scenes.
Pure behavior cloning supervises WHAT actions to take, but leaves models to infer WHY those actions work. This becomes especially limiting when learning from large-scale heterogeneous data, including egocentric human data, where humans and robots can take very different actions to achieve the same physical effect. We need models to extract the intent behind those motions, rather than simply recite them. Why leave that knowledge buried in trajectories when we can teach it explicitly?
Introducing EgoLAP: a shared language for human and robot motion. Language actions teach HOW to move; motion-level reasoning teaches WHY the motion works. Together, they unlock, to our knowledge, unprecedented gains in zero-shot performance for robot foundation models without additional robot demonstrations.
arxiv.org/abs/2610.08726
ego-lap.github.io/
Shresth Grover retweeted
🚨 Stop feeding your world model the same boring demos. Switch to free-range play data! 🚨
Our CoRL 2026 paper finds we can optimize autonomous data collection to focus on missing action consequences, achieving up to 85x the data scaling efficiency of expert data collection!🧵
Shresth Grover retweeted
Excited to share that our paper: OverLay++: Dense-Overlap Layout-to-Image Generation Dataset has been accepted to #NeurIPS2026 in the Evaluations & Datasets Track!
Generating images with multiple objects is challenging, especially when scenes involve dense layouts, overlapping objects, and complex spatial relationships. Even when an image looks visually appealing, getting every object, attribute, and position right remains a difficult problem. With OverLay++, we address this limitation by introducing a large-scale dataset designed to provide richer supervision for layout-to-image generation.
What makes OverLay++ different?
- Scale and complexity: Nearly 500K images, averaging 6.6 objects per image, covering diverse scene configurations.
- Richer annotations: 1.67× higher annotation density than existing datasets, complemented by detailed object-level captions.
- Improved generation quality: Training on OverLay++ improves object fidelity, attribute alignment, and spatial alignment, particularly in dense and overlapping scenes.
- Faster convergence: Models trained on OverLay++ converge faster across different levels of scene complexity.
📄 Paper: lnkd.in/gPfgpNxy
🌐 Project website: lnkd.in/g4p9ehZE
🤗 Dataset: lnkd.in/gqJFUABd
💻 Code: lnkd.in/gnFncnbu
📍 Meet us at NeurIPS 2026 in Sydney! We’ll be presenting our work during Poster Session 6 at PM AEDT in Hall 1–4. If you’re attending, please stop by. We’d love to discuss the work and hear your thoughts!
A huge thank you to my collaborators @shroglc, @dsrivastavv, @HaiyangXu3110, Bingnan Li, @zx1239856, @EthanArmand2, @chuanli11, @jianwen_xie, @zhuowent for their contributions, collaboration, and support throughout this project.
#NeurIPS2026 #ComputerVision #ImageGeneration #GenerativeAI #MachineLearning
Shresth Grover retweeted
We scaled up language-action pre-training to 10k hours across robots and egocentric videos.
The key to facilitating motion transfer: using motion-level reasoning as a shared language across all the embodiments!
ego-lap.github.io/
What’s missing from robot foundation models? An explicit understanding of WHY motions work: why move in this direction, why place the object in this way, how hard to grasp the object. This motion-level reasoning bridges instructions to actions—and provides knowledge that can generalize across embodiments and scenes.
Pure behavior cloning supervises WHAT actions to take, but leaves models to infer WHY those actions work. This becomes especially limiting when learning from large-scale heterogeneous data, including egocentric human data, where humans and robots can take very different actions to achieve the same physical effect. We need models to extract the intent behind those motions, rather than simply recite them. Why leave that knowledge buried in trajectories when we can teach it explicitly?
Introducing EgoLAP: a shared language for human and robot motion. Language actions teach HOW to move; motion-level reasoning teaches WHY the motion works. Together, they unlock, to our knowledge, unprecedented gains in zero-shot performance for robot foundation models without additional robot demonstrations.
arxiv.org/abs/2610.08726
ego-lap.github.io/
Shresth Grover retweeted
Huge thanks to the co-led @shroglc and all my great collaborators for making this project possible! And thanks TRI for all the support, and Google TRC project for the compute.
@tennyyin @smbml_ @HengkaiPan @MengchaoZhang @AykutOnol @allenzren @shahdhruv_ @Majumdar_Ani
Website: ego-lap.github.io
Paper: arxiv.org/abs/2610.08726
Shresth Grover retweeted
Human hands and robot grippers can follow different trajectories to achieve the same physical effect.
Motion-level reasoning connects contact, geometry and object affordances to that effect. It makes the shared physical intent explicit—the knowledge we want to transfer across embodiments.
Shresth Grover retweeted
What’s missing from robot foundation models? An explicit understanding of WHY motions work: why move in this direction, why place the object in this way, how hard to grasp the object. This motion-level reasoning bridges instructions to actions—and provides knowledge that can generalize across embodiments and scenes.
Pure behavior cloning supervises WHAT actions to take, but leaves models to infer WHY those actions work. This becomes especially limiting when learning from large-scale heterogeneous data, including egocentric human data, where humans and robots can take very different actions to achieve the same physical effect. We need models to extract the intent behind those motions, rather than simply recite them. Why leave that knowledge buried in trajectories when we can teach it explicitly?
Introducing EgoLAP: a shared language for human and robot motion. Language actions teach HOW to move; motion-level reasoning teaches WHY the motion works. Together, they unlock, to our knowledge, unprecedented gains in zero-shot performance for robot foundation models without additional robot demonstrations.
arxiv.org/abs/2610.08726
ego-lap.github.io/
Shresth Grover retweeted
We’re building a foundation motor control policy 🧠 that can control robots on the ground 🦿and in the air 🚁.
It is trained on millions of embodiment variations 🤖, and we are seeing exciting transfer to unseen robots.
We’ve collected 200+ real-world robot models to shape the embodiment distribution, and trained over billions of cumulative simulation steps.
Now we’re looking for collaborators to contribute robot models, incorporate them into training, and deploy them on real robots. 🧵
Shresth Grover retweeted
One of my favorite parts of this release is our new on-device policy (GROD2), which incorporates cross-embodiment pre-training to seamlessly learn a generalist policy.
This enables specializing to new embodiments in <200 demos without losing the pre-training goodness of Gemini.
Shresth Grover retweeted
Traditional robots are like player pianos—they only play notes they’re given.
Memo is more like a Master Chef who has watched hundreds of cooks in every type of kitchen. Our diverse dataset provides Memo with an “intuition” for grasping entirely new objects. This is a small step towards truly helpful robots that work in any home 🤖👨🍳
Shresth Grover retweeted
Introducing GEN-0, our latest 10B+ foundation model for robots
⏱️ built on Harmonic Reasoning, new architecture that can think & act seamlessly
📈 strong scaling laws: more pretraining & model size = better
🌍 unprecedented corpus of 270,000+ hrs of dexterous data
Read more 👇
VLA models often forget their pretrained knowledge during action training, hurting generalization.
🤖Our framework unifies action & VLM training to preserve strong pretrained representations & maintain versatility, boosting generalization & robustness.
gen-vla.github.io
Result: This all translates to robust real-world performance.
Our model demonstrates a more robust understanding of tasks in the real world, successfully completing goal-conditioned actions even in the presence of distracting objects that confuse baseline models.
Our framework provides a path toward building generalist policies by preserving rich representations.
Work led by me with @akshaygopalk , in collaboration with @XuanlinLi2 , @BoAi0110 , @hiskov and @haosu_twitr .