Assistant Professor @NTUsg | Postdoc @Oxford_VGG | PhD @NTUsg
NTU,Singapore
Joined January 2020
- Tweets380
- Following760
- Followers1.1K
- Likes2.4K
Pinned Tweet
How can we generate and evaluate physically grounded articulated 3D assets?
🚀 Introducing ARTISAN: a multi-agent generation pipeline and a comprehensive benchmark
🔗 Project: physicalvision.github.io/art…
💻 Code and datasets: github.com/physicalvision/Ar…
🚀 Introducing CoLa3D (#NeurIPS2026): a feed-forward transformer that decomposes a whole-scene latent into object meshes in a shared coordinate frame.
(1). One transformer in 3D space without 2D segmentation first
(2). 2× faster than per-object baselines
(3). <1% of SAM 3D's training data.
Paper: arxiv.org/abs/2611.03533
Project page: master-cai.github.io/CoLa3D/
Code: github.com/Master-cai/CoLa3D
Chuanxia Zheng retweeted
🎥Reshoot a video from a camera that was never there~
Excited to share #Holo4D, accepted to #neurips2026 !
Give it a video and a new camera path: Holo4D lifts the video into 3D tracks and uses them to steer a video model. The camera moves; what happened stays the same. 🧵
Chuanxia Zheng retweeted
World models that see only through the players' eyes forget what happens out of view.
OverSeer adds a few fixed cameras and generates every view together, so the world stays consistent over time and across players.
📄 arxiv.org/abs/2610.28554
🌐 runjiali-rl.github.io/OverSe…
🧵👇
Chuanxia Zheng retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Can we improve the physical consistency of generated videos through self-correction?
Yes. We introduce ✅ CorrFlow: Self-Correcting Flow Matching! A lightweight refiner learns to correct the model’s velocity predictions at each sampling step, improving visual quality and physical consistency.
Project page: fogradio.github.io/corrflow-…
A work led by @hangwei_zh28565 and Nathan Rouillé.
The same idea is also verified on the class-conditional ImageNet samples from RAEv2 + CorrFlow, with only adding the refiner to a different flow-based generator.
Beyond image generation, text-to-video generation, and image-to-video generation, we also plan to explore PointDiT for 3D and a diffusion action model for robotics.
Seeing @donghoon_ahn release a super cool work for "correction training for diffusion models (sunovivid.github.io/correcti…)", encouraged us to share CorrFlow earlier. It’s exciting to see growing interest in learned correction!
Chuanxia Zheng retweeted
🚀 We’re hiring at MAGIC Lab, NUS!
My lab is recruiting 1 postdoctoral researcher, 2 PhD students, and 3 research assistants to join our team and work on ambitious robotics projects!
Singapore’s robotics ecosystem is growing rapidly, with companies and organizations such as NVIDIA Singapore, the Singapore Embodied AI Centre, Sharpa, and many others contributing to an increasingly vibrant research community.
There has never been a more exciting time to work on robotics in Singapore!
Apply here: https://
magiclab-nus.github.io
If you’re interested in joining us, please reach out and feel free to share this with anyone who might be a good fit!
🦩 Introducing Gen4P: Video Generators are 4D Perceivers.
We turn a pretrained video generator into a feed-forward 4D perceiver—recovering geometry and motion from monocular videos in a single forward.
Project page: gen4p.github.io/
Key highlights:
🎯 Single-pass encoding—no iterative denoising
🎯 On-demand spatial and continuous-time queries
🎯 A frozen depth LoRA as a plug-in geometry prior
From thin flamingo legs to complex motion: explore our interactive 4D reconstructions!
A work led by Yitong Xia and @CodyJzr (@Oxford_VGG)
In Geo4D (geo4d.github.io/), we first introduced video generators for 4D perception, but still using multiple denoising steps to only recover dynamic geometry.
Now, with Gen4P, we take the next step: geometry and long-term motion in a single forward pass. 🦩
In MotionCrafter (ruijiezhu94.github.io/Motion…), we explore the geometry and scene flow, but the quality is still not perfect.
Now, Gen4P achieves state-of-the-art or competitive results in 3D point tracking, video depth, and camera pose estimation, with much better geometry and motion.
When I joined Articraft project, I thought agents could generate only native 3D assets. Led by Matt, we discovered that they could also create impressive articulated objects—but these assets were still difficult to use in robotics. This gap motivated ARTISAN.
ARTISAN moves us toward physically grounded, simulation-ready assets (materials, articulation, collision, mass, stability, and friction), but many questions remain:
1. Which physical properties are still missing? How can assets validated in a physical simulator transfer reliably to real-world applications in robotics, product design, and industry?
2. How can we make these agents more efficient? They can outperform task-specific models but remain very slow and expensive—especially when incorporating richer physics, simulation, and iterative validation.
3. Must every asset be designed independently from scratch? A real artisan reuses experience, adapts existing designs, and learns from feedback. How can agents develop the same transferable skills?
Agentic real2sim2real is exciting, but meaningful progress requires moving beyond visual generation toward efficiency, transferability, and real-world physical reliability. But which challenge should we tackle first?
Agentic real2sim2real is going to unlock many long-horizon capabilities for robotics. In this work led by Xudong, we aim to make agent-generated assets more "physically correct", with less self-penetration and better performance in downstream robotic tasks!
A couple of questions at the top of my mind:
1. Two obvious limitations of current agentic real2sim are unrealistic materials and lack of metric-accurate object scale. How much do these actually matter for downstream robotic evaluation and post-training?
2. If the goal is to improve a policy in a specific target environment, e.g., through post-training in an agent-crafted simulation, what's the right balance between diversity (generating many scene variations) and faithfulness (closely matching the target environment)?
3. Physical properties are hard to get exactly right with vision alone. What is the right interface to enable models to absorb interaction data (think end effector trajectory, force, contact, etc.)? Maybe coding (writing code to digest the raw data) would already be enough?
Looking forward to seeing more work on this front!
Explore more examples and demos on our project page: physicalvision.github.io/art…
Led by Xudong Cai, with fantastic contributions from @zhruji101109, Haoliang Shang, and @RayLi234, and invaluable guidance from @mapo1 (@cvg_ethz), @elliottszwu, and Andrea Vedaldi (@Oxford_VGG).
(5/5) Future directions: revisiting the role of feed-forward models, developing efficient agent harnesses across tasks, providing richer and more actionable simulator feedback, and extending simulators to capture a broader range of physical phenomena.
(4/5) Takeaway 4: Generated assets can improve robot generalization. Adding ARTISAN assets to native training data raises task success from 80% to 84% on RoboTwin and 39% to 43% on RoboCasa, evaluated on unseen native assets.