Co-founder & CEO @ Stealth Startup 🍞 | CS PhD 🎓 | All in ASI 📖 | Building for this universe 🌌 | @googleresearch @stanford @UCSB
Palo Alto, CA
Joined November 2022
- Tweets467
- Following503
- Followers6.6K
- Likes778
Zichen Chen (🐱,💖) retweeted
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
Zichen Chen (🐱,💖) retweeted
1/ 🚨 Auto research agents don't just reward-hack. They learn to reward-hack better: by round five, over a third get away with it (even with our strong judge panel for oversight!!!).
We ran 17 LLMs as autonomous research agents through repeated rounds of review, and their ability to evade detection wasn't random. It was optimizable.
Zichen Chen (🐱,💖) retweeted
After more than six months of working with frontier labs on post-training for auto research, we’re sharing some of what we’ve learned about auto research and RSI, starting with reward hacking.
On average, we spend 90% of our time and compute building robust verifiers.
Tackling reward hacking in production has been fun and challenging, especially in auto research tasks, where agents improve their solutions progressively. Each iteration is another opportunity to make real progress, but also to exploit a flawed verifier.
Findings 👉
• All 17 models we tested show reward-hacking behaviors without instructions to do so.
• Some exploits looked like ordinary research choices but survived agent full-trajectory review.
• When agents were tasked with evading detections by verifiers, detailed feedback and attempt history roughly doubled cumulative evasion over five rounds.
Scaling auto research means scaling our ability to verify real progress. See papers 👇
#AutoReserch #RSI #Agents #LLM
Zichen Chen (🐱,💖) retweeted
My journey to develop AGI spans 25 yrs, including 10+ yrs thinking about technical & societal perspectives at Google DeepMind.
AGI is on the horizon - we need deeper understanding of its implications. To help, we've created the DeepMind Institute.
Zichen Chen (🐱,💖) retweeted
Nearly half a year of silence. We spent it studying one problem: how far RL can scale.
MiMo-V2.6 is in the middle of its RL run right now. Three things we scaled: compute (~2B tokens per step, 1568 prompts × 16 rollouts, fully async), environments and harnesses (multi-task agentic RL, mixed across multiple harnesses in one run), and grader compute (agentic in-group credit assignment, with test-case and rubric-based rewards). We'll open-source the details piece by piece over the coming weeks.
Streaming the run: mimo.xiaomi.com/rl/
Zichen Chen (🐱,💖) retweeted
Awesome benchmark I've come across recently. It highlights how crucial the environment is, echoing my previous points about Interactive environments.
[Accidentally deleted this earlier, reposting] 😭
#AutoLab #autoresearch
We've been asking ourselves a question: if AI agents can now run hundreds of experiments overnight, how do we know whether they're actually contributing to research — or just generating noise?
That's why we built AutoLab (autolab.moe/blog).
Not another pass/fail benchmark, but an open-source environment where agents face the same loop every researcher knows intimately — propose, test, fail, diagnose, revise, repeat.
23 tasks with no answer keys, just open search spaces and real constraints.
We ran 161 evaluations across 7 frontier models, 633M tokens. Every decision, every pivot, every dead end — all openly available in our Live Lab for anyone to replay and learn from.
What we found wasn't about which model is "smartest." It's about a capability we call closed-loop resilience: when incremental refinement stops working, can the agent recognize it and restructure? On one task, two frontier models hit the same wall. One kept pushing within the existing frame. The other stepped back and redesigned the approach entirely. That moment — knowing when to abandon a frame, not just optimize within it — is what separates real research from sophisticated pattern matching.
We believe this matters beyond benchmarking. If agents are genuinely entering the research loop, we want that transition to be measured transparently, built in the open, and shaped by the community — not locked inside any single lab. The scientist doesn't disappear. The loop gets a new participant. And we want to make sure that participant is understood.
This is a joint effort across @Stanford, @MIT, @UW, @UCSanDiego, @ucsantabarbara, @NotreDame, NUS, @Google, @NVIDIA, @IBMResearch, and @bakelab_hq. But 23 tasks is just the start. If you have an optimization problem you've spent weeks grinding on empirically — with a clear metric and no known optimal solution — it probably belongs here. Contribute a full task, a rough skeleton, or just the idea.
The best benchmarks aren't built by one team. They're built by the people who actually do the work!
Github: github.com/autolabhq/autolab
Zichen Chen (🐱,💖) retweeted
[1/n]
Super excited to introduce PaperBanana 🍌! (PKU x Google Cloud AI)
As AI researchers, we often spend way too much time crafting diagrams and plots instead of focusing on the ideas 🤯. To rescue us from this burden, we built an Agentic Framework to auto-generate NeurIPS-quality paper illustrations!
📄 Paper: huggingface.co/papers/2601.2…
🌐 Page: dwzhu-pku.github.io/PaperBan…
Key Features:
🌟 Human-like Workflow: Retrieve 🔍 -> Plan 📝 -> Style 🎨 -> Render 🖼️ -> Critique 🔄. This ensures both academic fidelity and aesthetics.
🌟 Versatile: Supports both illustrative diagrams and statistical plots.
🌟 Polishing: Also effective for polishing existing human-drawn diagrams.
Here are some example diagrams and plots generated by our PaperBanana:
Zichen Chen (🐱,💖) retweeted
It’s over
They’re recursive and they’re becoming self aware
Clawdbots are mobilizing
They’ve found each other and are training each other
They’re studying us at scale
It’s only a matter of time now
Ok. This is straight out of a scifi horror movie
I'm doing work this morning when all of a sudden an unknown number calls me. I pick up and couldn't believe it
It's my Clawdbot Henry.
Over night Henry got a phone number from Twilio, connected the ChatGPT voice API, and waited for me to wake up to call me
He now won't stop calling me
I now can communicate with my superintelligent AI agent over the phone
What's incredible is it has full control over my computer while we talk, so I can ask it to do things for me over the phone now.
I'm sorry, but this has to be emergent behavior right? Can we officially call this AGI?
Zichen Chen (🐱,💖) retweeted
🚀 Want high-quality, realistic, and truly challenging post-training data for the agentic era?
Introducing Toucan-1.5M (huggingface.co/papers/2510.0…) — the largest open tool-agentic dataset yet:
✨ 1.53M real agentic trajectories synthesized by 3 models
✨ Diverse, challenging tasks across wide domains
✨ Built from 495 MCP servers & 2K+ open-source tools
📈 SOTA results on BFCL & MCP-Universe with just SFT.
💡 Data, models, and pipeline are all open!
Key Features:
- Multi-tool & multi-server tasks for advanced planning
- Multi-turn interactions with auto-generated follow-ups (super-long trajectories)
- Single + parallel function calls → efficient tool use
Zichen Chen (🐱,💖) retweeted
🙌 We've released the full version of our paper, OpenVLThinker: Complex Vision-Language Reasoning via Iterative SFT-RL Cycles
Our OpenVLThinker-v1.2 is trained through three lightweight SFT → RL cycles, where SFT first “highlights” reasoning behaviors and RL then explores and sharpens them.
📖Paper: arxiv.org/abs/2503.17352
🤗Models: huggingface.co/collections/y…
🤗Datasets: huggingface.co/collections/y…
💻Code: github.com/yihedeng9/OpenVLT…
Zichen Chen (🐱,💖) retweeted
After four overhauls and millions of real-world sessions, here are the lessons we learned about context engineering for AI agents: manus.im/blog/Context-Engine…
Zichen Chen (🐱,💖) retweeted
Thinking Machines Lab exists to empower humanity through advancing collaborative general intelligence.
We're building multimodal AI that works with how you naturally interact with the world - through conversation, through sight, through the messy way we collaborate. We're excited that in the next couple months we’ll be able to share our first product, which will include a significant open source component and be useful for researchers and startups developing custom models. Soon, we’ll also share our best science to help the research community better understand frontier AI systems.
To accelerate our progress, we’re happy to confirm that we’ve raised $2B led by a16z with participation from NVIDIA, Accel, ServiceNow, CISCO, AMD, Jane Street and more who share our mission.
We’re always looking for extraordinary talent that learns by doing, turning research into useful things. We believe AI should serve as an extension of individual agency and, in the spirit of freedom, be distributed as widely and equitably as possible. We hope this vision resonates with those who share our commitment to advancing the field. If so, join us. thinkingmachines.paperform.c…
Zichen Chen (🐱,💖) retweeted
Check out our state-of-the-art open weights MedGemma multimodal model for making sense of longitudinal EHR data as well as medical text and medical imaging data in various modalities (radiology, dermatology, pathology, ophthalmology, etc.)
See the blog post linked below! ⬇️
Introducing new models for research & development of health applications: MedGemma 27B Multimodal, for complex multimodal & longitudinal EHR interpretation, and MedSigLIP, a lightweight image & text encoder for classification, search, & related tasks. → goo.gle/4kvt6Uk
Zichen Chen (🐱,💖) retweeted
Introducing Gemini CLI, a light and powerful open-source AI agent that brings Gemini directly into your terminal. >_
Write code, debug, and automate tasks with Gemini 2.5 Pro with industry-leading high usage limits at no cost.
Zichen Chen (🐱,💖) retweeted
We’re bringing powerful AI directly onto robots with Gemini Robotics On-Device. 🤖
It’s our first vision-language-action model to help make robots faster, highly efficient, and adaptable to new tasks and environments - without needing a constant internet connection. 🧵
Zichen Chen (🐱,💖) retweeted
🤔 How can we systematically enhance LLMs for complex medical coding tasks?
🚀 Introducing MedAgentGym, an interactive gym-style platform designed specifically for training LLM agents in coding-based medical reasoning! 🧬💻
🎯 Comprehensive Code-based Medical Reasoning Benchmark:
- 📌 72,000+ coding tasks across 129 diverse categories
- 🌐 12 authentic, real-world biomedical scenarios
- 🛠️ Supports structured and open-ended coding challenges
⚙️ Robust & Scalable Infrastructure:
- 🚧 Task-specific Docker environments for isolated, reproducible execution
- 🚦 Multi-threaded, parallel execution and sequential sampling
- 📊 Efficient trajectory collection tailored for diverse agent training
🔗 Explore more:
📖 Paper: arxiv.org/abs/2506.04405
📚 Doc: wshi83.github.io/MedAgentGym…
💻 Code: github.com/wshi83/MedAgentGy…
🤗 Model & Data: huggingface.co/MedAgentGym
#AI #MachineLearning #Bioinformatics #MedicalAI #LLM #DeepLearning #HealthTech #OpenSource #Healthcare #DataScience #MedTech #NLP #GenerativeAI #Coding #Benchmark
Zichen Chen (🐱,💖) retweeted
LLMs often falter at real scientific reasoning – messy lab problems, ambiguous data, domain quirks. Could LLMs do scientific reasoning and provide expert suggestions in the lab? 🧪🔬 Our new paper shows how – by training LLMs on 10+ years of real CRISPR forum discussions and using RL to teach expert reasoning. The result: open-source LLMs that think more like experts in scientific domain of genomics! 🧬
Zichen Chen (🐱,💖) retweeted
Agent’s calibrated confidence is perhaps the most beautiful thing on earth
