Xingang Guo retweeted
In collaboration with @HuggingFace, we’ve released the largest open robot dataset, alongside their first open System 1-System 2 model that can reason over step-by-step actions.
FineART is built with 40K+ demonstrations across 151 tasks and 500K subtask labels.
Together, the dataset and model were designed for each other, and both are available on @LeRobotHF where robotics developers work today.
Xingang Guo retweeted
🚀 Excited to share what I have been working on: Glimpse, a new approach to visual thinking from Elorian.
Understanding a scene means keeping track of what moves, how perspectives change, and what happens next. Glimpse brings visual representations into its reasoning process, using an internal “visual scratch space” to examine details and revisit the evidence.
Here’s what we built:
🧠 Reasoning in visual space, with the ability to follow objects over time, through shifting viewpoints and complex motion.
⚡ Performance that matches or exceeds GPT-6 Astra and Claude Opus 5.5 on key multimodal benchmarks.
💰 Pricing up to 94% below competing frontier models: $0.60 / 1M input tokens and $3.00 / 1M output tokens.
🛠️ End-to-end training to spawn subagents and work with multimodal tools, including code interpreters and video processors.
elorian.ai/blog/introducing-…
Xingang Guo retweeted
Congrats to @GoogleAI's Gemini 4 Argon for taking the top spot of our Humanity's Sixth Sense (HSS) benchmark with a score of 55.4%, edging out GPT-6-Astra at 53.6%. 🥇
HSS tests AI’s ability to reason beyond what’s visible, from hidden objects to predicting what happens next and understanding human behavior.
We took a closer look at performance across categories, and that’s where things get interesting. 👀🧵
Xingang Guo retweeted
The human–AI gap is striking—93% vs. 54%! Had a lot of fun to explore the examples where humans win!! 😇
The part I found most fascinating is the causal intuition gap: humans can infer hidden physical mechanisms, what likely happened, and what may happen next from subtle visual cues. Models often miss those cues or struggle to connect them to the underlying causal dynamics. More reasoning steps alone don't necessarily close this gap. An exciting frontier for multimodal intelligence!
Great partnership with the @ElorianAI team! Congratulations to everyone involved on this fantastic work! @AndrewDai @_yunzhong @yinfeiy @danielyuez
AI is usually measured on problems that are hard for people. Humanity’s Sixth Sense, our newest benchmark with @ElorianAI, tests the everyday visual judgements we make without thinking.
Humans scored 93.1%. The strongest AI model scored 53.6%.
labs.scale.com/blog/introduc…
Xingang Guo retweeted
AI is usually measured on problems that are hard for people. Humanity’s Sixth Sense, our newest benchmark with @ElorianAI, tests the everyday visual judgements we make without thinking.
Humans scored 93.1%. The strongest AI model scored 53.6%.
labs.scale.com/blog/introduc…
Excited to share Humanity's Sixth Sense (HSS), a new benchmark for intuitive visual reasoning, built at @ScaleAI with @ElorianAI.
Humans: 93.1%
Best model (GPT-6-Astra): 53.6%
Even the strongest models still fail at intuitive tasks that are easy for human eyes.
We’re introducing Humanity’s Sixth Sense, a new benchmark testing the intuitive visual reasoning people rely on every day.
Across 522 open-ended tasks spanning images and video, HSS tests everything from spatial and causal reasoning to social understanding.
The gap is significant. Humans score 93.1%, while the strongest model, GPT-6-astra, reaches 53.6%. The median model scores just 30.9%.
Xingang Guo retweeted
We’re introducing Humanity’s Sixth Sense, a new benchmark testing the intuitive visual reasoning people rely on every day.
Across 522 open-ended tasks spanning images and video, HSS tests everything from spatial and causal reasoning to social understanding.
The gap is significant. Humans score 93.1%, while the strongest model, GPT-6-astra, reaches 53.6%. The median model scores just 30.9%.
Xingang Guo retweeted
Full-duplex voice models can listen and speak at the same time, but can they change how they speak when you ask them to?
Introducing SteerDuplex + SteerBench: the first benchmark that measures steerable full-duplex dialogue.
65.1% audio-steering APR vs 20.6% for the best open baseline. 🧵
Xingang Guo retweeted
An AI agent can top a benchmark and still be undeployable.
Introducing READY (Reliable Enterprise Agent Deployment), the first evaluation framework for qualifying AI agents for deployment on real enterprise workflows.
We’re inviting the research community to build it with us. The testbed is open for contributions today (link below).
Xingang Guo retweeted
New models this week are much more codebase-aware: they check and break their own tests, and like senior/experienced engineers, they even care about code hygiene.
As part of that, I want to shout out some major gains we saw on our SWE Atlas leaderboard @ScaleAILabs.
@AnthropicAI is tied for 1st on all three SWE Atlas leaderboards, with the latest Opus and Fable models clustered at the top. But Fable 5.1 feels like a qualitatively different model: more consistent, more efficient, and noticeably more codebase-aware. In Test Writing, it's the first model we've seen that actively tries to mutate code and break its own tests to check for robustness. In Refactoring, it consistently cleans up old leftover code without being asked.
On SWE Atlas, no model has cracked 70 percent yet. There's a lot of work these agents still can't do.
Xingang Guo retweeted
🎉 Our work from @ScaleAILabs, Beyond Seeing: Evaluating Multimodal LLMs on Tool-Enabled Image Perception, Transformation, and Reasoning, has been accepted to EMNLP 2026 Main Conference!
One of the first works to systematically benchmark vision tool use in multimodal LLMs - testing whether models can go beyond simply seeing an image to actively manipulate, inspect, and reason over it with tools.
📄 arxiv.org/abs/2510.12712
🏆 labs.scale.com/leaderboard/v…
Xingang Guo retweeted
📣Call for contributions + co-authorship!
RSI Bench is our ongoing effort to evaluate whether AI agents can develop the capabilities required to advance AI R&D.
Got a frontier AI research problem on your mind? Submit it to RSI Bench, and set the standard the whole industry uses to measure progress!
Contributors receive:
→ Co-authorship on the RSI Bench research paper
→ $2,000 per accepted task for the initial 50 tasks
→ Modal compute credits to build + iterate
→ Access to the RSI Bench research community
Register below for full requirements.
Launching rsi-benchmark.com:
The work of AI R&D has always belonged to humans. For the first time, though, it no longer seems certain that it always will.
Recursive self-improvement is within a line of sight. It may still be far, but it is close enough that we should start measuring it.
Xingang Guo retweeted
Today we're releasing the complete EnigmaEval dataset, our benchmark built from sophisticated puzzle-hunt problems in partnership with @CAIS.
We believe open access to EnigmaEval will help push reasoning and evaluation research forward.
So far, Claude Fable 5 and GPT-5.6 Sol take the lead. Full dataset available below.
Xingang Guo retweeted
We're back from Seoul!
Great sharing our research on agent training, self-optimization, and evaluation through our paper presentations, lightning talks, and reception at @ICMLConf.
The momentum toward bringing AI agents into real professional domains is massive, and we're thrilled to drive that conversation.
Next stop: @NeurIPSConf 🇳🇿
Xingang Guo retweeted
📝 New research from @scale_AI
Frontier SWE benchmarks are usually single-turn, one-shot tasks: the agent gets a detailed spec upfront, then implements autonomously.
That is not how most real coding-agent workflows feel.
Introducing SWE-Interact.
🧵
Xingang Guo retweeted
New @ScaleAILabs benchmark: 𝐃𝐫𝐮𝐠𝐃𝐢𝐬𝐜𝐨𝐯𝐞𝐫𝐲𝐁𝐞𝐧𝐜𝐡! 💊🧬👩🔬
We built it with @phylo_bio to evaluate agents on early drug discovery workflows.
Bringing a drug to patients can take decades and billions of dollars. In a typical campaign, thousands of compounds are tested, and early decisions shape everything downstream.
Xingang Guo retweeted
1/ New from @ScaleAILabs: Rubrics (a.k.a. checklists) have become the default reward interface for RL on open-ended tasks without final verifiable answers.
But most rubric RL still relies on static aggregation: fixed human weights over criteria, summed into one scalar reward.
We show that this conflates what should matter in the final answer with what can actually teach the current policy.
arxiv.org/abs/2605.20164
Excited to share that our paper Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs has been accepted to NeurIPS 2025 (Datasets & Benchmarks Track)! 🚀🚀🚀
📍 Poster: Thu, Dec 4, 2025 • 11 AM–2 PM PST
Exhibit Hall C,D,E • #1912
Would love to chat and connect, please drop by our poster tomorrow!
Modern LLMs shine at Q&A, but real engineering design demands synthesis, constraints, simulation, and trade-off reasoning.
So we built EngDesign: the first benchmark that tests whether LLMs can actually solve engineering design problems, not just answer exam-style questions.
🔧 EngDesign includes:
• 101 real-world engineering design tasks
• 9 engineering domains
• 473 rubric-based evaluation items
• MATLAB / SPICE / MQSim simulation pipelines




























