statistical parrots post-training enthusiast
~/
Joined October 2021
- Tweets1.9K
- Following339
- Followers271
- Likes8.4K
So much fun working on this one. The results turned up some really interesting and funny insights.
We'll be adding more sims and expanding beyond web in future.
Today we're launching WebCloneBench: a benchmark which measures a model's ability to explore, copy, build and test various web apps.
For each task, we give the model a reference site from one of Gmail, Google Sheets, Jira, DocuSign, Shopify Admin, Craigslist, Figma, Slack and Zendesk. It is instructed to build a fully feature-complete clone of every part of the application with an 8 hour time limit.
xeon retweeted
We can finally talk about it:
We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company.
We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.
xeon retweeted
Prime Agent is a general-purpose coding harness
On ARC-AGI-3, it scores 95.5%, surpassing the human-expert baseline, but the gain is not benchmark-specific.
We see major improvements across models when compared to their proprietary harnesses:
Is this peak?
Kimi K3 can now be run locally! ✨
The 1-bit model retains ~78.9% accuracy after we shrunk it from 1.56TB to 594GB (-62% size).
Run on a Mac Studio + 128GB RAM device.
Kimi K3 is the strongest open model to date.
Guide: unsloth.ai/docs/models/kimi-…
GGUF: huggingface.co/unsloth/Kimi-…
xeon retweeted
We just submitted our YC application for Agent Orchestrator.
Here’s the demo video we sent with it.
AO lets you run and manage Codex, Claude Code, OpenCode, and other coding agents from one place, without constantly switching between terminals.
Still early, but this is what we’ve been building.
xeon retweeted
I converted all 1760 Harvey Labs tasks to Harbor format for anyone to run and experiment easily
Great dataset, tasks and release from @harvey
harbor run -d punitarani/harvey-labs
We're open-sourcing 10 RL environments for M&A due diligence.
In our LAB: Diligence environments, agents do research in a virtual data room (VDR) and draft a diligence memo for a merger or acquisition.
The largest of these environments contains 80M tokens of context with 1,000 grading criteria.
M&A diligence is one of the most difficult tasks for frontier agents due to:
1) Long context: A single thread of VDR research may involve hundreds of documents in a dataroom that contains hundreds of thousands.
2) Exhaustivity: every document must be searched for needle-in-a-haystack provisions that kill a deal.
3) Data: M&A diligence data is privately held by transactional law firms - models aren't trained to do it.
It is also one of the world' most valuable knowledge work tasks: global M&A volume is ~$4T annually, of which ~1% is spent on diligence.
Deep dive below by @ItsJulioPereyra and @nikogrupen:
xeon retweeted
Another Golden Sweep by India as our Chemistry team bags 4 Gold medals securing Rank #1 (jointly with China, Vietnam, and Individual Participant (Russia)) at the 58th International Chemistry Olympiad, Uzbekistan (icho2026.uz).
1/5
xeon retweeted
Excited for our first general model Inkling -- open weights, 975B, natively multimodal (text, image, audio). Available on Tinker, HuggingFace and partners.
It is yours to personalize and use openly. It is yours.
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
thinkingmachines.ai/news/int…
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
Who's turning this to an rl env
70% of AI research isn’t reproducible. With ICML 2026 happening last week, over 6000+ research papers have dropped, but how many are actually real and reproducible?
We’re excited to partner with @huggingface on a community challenge to find out.
All you have to do is pick a paper, select one of the claims, and start experimenting.
And now, you can use our autoresearch agent to do the hard work for you while contributing to a massive library of open research artifacts.
Prizes: $4500 in GPU credits 👀
Full details found below 👇
xeon retweeted
this would have been a whole startup not too long ago
xeon retweeted
Today, we’re announcing Bonsai 27B: the first 27B-class model to run on a phone.
Bonsai 27B is the new multimodal flagship of the Bonsai family. Based on Qwen3.6 27B, it brings a new capability tier to local AI: multi-step reasoning, structured tool use, long-context workflows, and coherent agentic loops.
Until now, models in this class have been impractical to deploy locally. A 27B model occupies roughly 54 GB in 16-bit precision, and even a strong 4-bit build is around 18GB - too large for a phone and for most laptops.
Bonsai 27B changes that.
It comes in two variants:
• Ternary Bonsai 27B: 5.9 GB, 1.71 effective bits per weight, optimized for laptop-class quality.
• 1-bit Bonsai 27B: 3.9 GB, 1.125 effective bits per weight, optimized for phone-class footprint.
Everything is open-sourced today under the Apache 2.0 license.
xeon retweeted
Standard RL benchmarks are episodic and stationary, so they don't capture the the characteristics of real-world deployment. Morpheus is a new benchmark for continual learning that provides persistent simulation environments where the world never resets, objectives shift asynchronously, and decisions have compounding consequences. Nice work by Sam & team!
Check out the paper - openreview.net/pdf?id=31P1VA…
Platform website - morpheus.skyfall.ai/
Today we present Morpheus, a persistent enterprise simulation platform designed to make Continual Learning a reality. Morpheus is the world’s first real world Reinforcement Learning environment.
Every Reinforcement Learning environment operates in the game world. Benchmarks like Atari, OpenAI Gym, MuJoCo, and Procgen are all small, game-like worlds that reset every few minutes.
But the real world never resets. A business keeps running and evolving everyday.
We tested how frontier LLMs would perform in realistic and dynamic business environments 🧬on Morpheus. The main conclusion was that LLMs are not continual learners.
🧵Here’s how we did it and what we learned:
xeon retweeted
🔥Today, we are releasing one of the first visual reasoning benchmarks for autonomous AI diagnosis in healthcare!
🚀Introducing Radiology’s Last Exam 2.0 (RadLE 2.0) from @CRASHLabAI, an uncertainty-aware benchmark for autonomous diagnosis in radiology!
✅In the last few days, the AI frontier has moved significantly. @OpenAI launched GPT-5.6 Sol. @Meta launched Muse Spark 1.1. @xAI dropped Grok 4.5.
🙌We’ve benchmarked all frontier, open-source and medical VLMs in RadLE2.0 and the leaderboard is now LIVE!
🚨 Before AI models are handed autonomy, one question matters more than any accuracy score: Do they know when to STOP and hand over to a human?
⚠️ A confident wrong diagnosis is far more dangerous than an honest “I don’t know.” Yet most models are bad at admitting the latter!
🚀 We release five RadLE 2.0 Scores: Confidence Weighted, Reliability, Accuracy, Safety and Handover Readiness and we find that models from @OpenAI @AnthropicAI @MetaAI @GoogleDeepMind @xAI @nvidia @Alibaba_Qwen @MistralAI @MiniMax_AI all score very differently as they optimize for different metrics!
🚨But most importantly, NONE of the Models have been able to reach the average human expert baseline!
⚡️A thread on what we found and which models aced our metrics! Link to the leaderboard and technical report at the end of the thread!
xeon retweeted
A free guide that answers key reinforcement learning questions: shorturl.at/zcciv
v/@arjunkocher & @sheriyuo
