Research Scientist @Google | Self-improving agents: environments, skills, RSI | Shipped Gemini Deep Research (Enterprise & Max) | Prev: Apple, UCSD.
Sunnyvale, CA
Joined October 2013
- Tweets84
- Following1.1K
- Followers468
- Likes258
14 frontier models × 7 harnesses on our private test set:
🥇 Claude Opus 5.5: 48.3%
🥈 GPT 6 Astra: 45.8%
🥉 Gemini 4 Argon: 40.6%
⚡ Gemini 3.8 Flash: 27.5%
Wildest finding: Harness choice alone swings score by up to 12.1% and cost by 3x ($13 vs $38/task).
Update: PLE is now officially live across @googlecloud, @mercor, and @huggingface! 🚀
• Google Cloud Blog: cloud.google.com/blog/produc…
• Mercor Blog: mercor.com/blog/introducing-…
• Hugging Face Dataset: huggingface.co/datasets/goog…
• Leaderboard: ple-bench.com
Full announcement thread 👇
xitter.gq/chl260/status/21177064…
Official PLE release today with @googlecloud, @mercor & @huggingface!
- ~61 files/task (CAD, BIM, PDFs + outdated revisions)
- 6hr offline sandbox
- Strict PE safety & code checks
Links 👇
Official PLE release today with @googlecloud, @mercor & @huggingface!
- ~61 files/task (CAD, BIM, PDFs + outdated revisions)
- 6hr offline sandbox
- Strict PE safety & code checks
Links 👇
We're proud to have collaborated on a new benchmark built by @GoogleAI: Physical-Engineering's Last Exam (PLE).
Physical engineers design the skyscrapers, power grids, and chip fabs we rely on, yet no existing benchmark tested whether AI agents could do that work end to end.
Together with Google and practicing engineers, we built 85 long-horizon tasks across mechanical, electrical, civil, and structural engineering. On the private test set, all frontier models score below 50%:
1. Claude Opus 5.5: 48.3%
2. GPT 6 Astra: 45.8%
3. Claude Sonnet 5.5: 43.5%
4. Gemini 4 Argon: 40.6%
5. Claude Opus 5: 40.0%
On the tasks we co-authored, the top agent satisfied 91% of individual constraints, but only 41.3% of its deliverable items passed every hard safety check.
Learning how AI fails at physical engineering can help us improve future models and build a safer world.
This benchmark is also an open competition. Bring your own agent setup and submit it for evaluation on our private test set at ple-bench.com
Google Cloud blog: cloud.google.com/blog/produc…
Mercor blog: mercor.com/blog/introducing-…
HF dataset (29 dev tasks): huggingface.co/datasets/goog…
Leaderboard: ple-bench.com
Paper: arxiv.org/abs/2610.30537
Chen-Yu Lee retweeted
Banger paper from Google Cloud AI Research on agents doing professional engineering work.
(bookmark it)
For professional engineering deliverables, the best agent configuration averages 50.6%, using 27M tokens and 63 minutes per task.
They introduce a benchmark called PLE (Physical-Engineering's Last Exam). It has 85 tasks across electrical, civil, structural, and mechanical engineering. Solving a task means producing a billable deliverable, such as a design package a licensed reviewer would sign off on, from a multi-format project corpus with physics and building-code requirements.
Some tasks are written end to end by practicing engineers. Others start from an engineer's problem statement, and an adversarial Creator-Solver agent pipeline builds the project environment, grader, and reference deliverables before experts validate them and make them harder.
Frontier models were tested under seven public agent harnesses. Most existing engineering benchmarks are textbook QA or toy design problems that leading agents already saturate.
arxiv.org/abs/2610.30537
Chat with Paper: academy.dair.ai/papers/ple-p…
HLE tested knowledge. ALE tested digital work. PLE tests the work that builds the physical world.
Today we’re releasing Physical-Engineering’s Last Exam (PLE) from Google Cloud AI Research.
All frontier models scored under 50% 🧵
AI writes code, but can it design a power grid, chip fab, or skyscraper?
PLE has 85 long-horizon tasks in Mechanical, Electrical, Civil & Structural engineering.
Agents drop into real project folders (~61 files, outdated revisions as traps) & must pass licensed engineer’s bar.
Agents can remember now. Self-improvement over long horizons is the harder part.
My guest lecture at UC Berkeley this week: memory → lessons → skill curation → better environments → safe self-improvement.
Slides: chenyulee.com/berkeley
The question I left the class with: when should experience and insights go back into the weights? The harness is cheap to edit; weights generalize. Where's the line?
We evolved an agent on one benchmark, froze it, and tested it on five it had never seen.
Other self-improvement methods barely transfer, and some get worse.
RRSI improved on every one, up to 24% over its starting score, with ~30% fewer tokens.
Make every change earn its place.
Led by our amazing intern Peng Xia, with a great team from Google Cloud AI Research, UNC-Chapel Hill, Stanford, and WashU.
📄 arxiv.org/abs/2609.24972
💻 github.com/google-research/r…
🌐 regularized-rsi.com
@richardxp888 @HanRujun @ZifengWang315 @anmourchen @yufan_zhuang @yoonholeee @ChengsongH31219 Han Yu, Zoey Cuizhu,
@ming5_alvin @HuaxiuYaoML Burak Gokturk @tomaspfister @chl260