Brad Johnson 百面包 retweeted
Google is officially back in the frontier model race with full force. Gemini 4 Neon is pretty impressive for its cost
It's a sea of blue on the APEX-Agents leaderboard after the Gemini 4 Neon launch today.
Great work @OfficialLoganK and team!
It's a sea of blue on the APEX-Agents leaderboard after the Gemini 4 Neon launch today.
Great work @OfficialLoganK and team!
Gemini 4 Neon from @GoogleDeepMind scores 82.9% Pass@1 and 88.9% mean score on APEX-Agents, ranking #2 on both.
With the release of Gemini 4 Neon, Google holds 4 of the top 5 Pass@1 spots on APEX-Agents.
By domain on APEX-Agents using Pass@1:
Management Consulting: 94.7% (#1)
Corporate Law: 74.7% (#2)
Investment Banking: 79.4% (#4)
On APEX-Accounting, it scores 23.5% Pass@1 (#2) and 70.5% mean (#3).
This demonstrates broad capability for knowledge work.
Congrats to the @GoogleDeepMind team.
See full leaderboard: mercor.com/apex/apex-agents-…
Hy4 Flash from @TencentHunyuan scores 62.4% Pass@1 on APEX-Agents, our benchmark for long-horizon professional work in banking, consulting, and law.
Among fast and mid-tier models on the same harness:
Gemini 4 Hydrogen: 78.0%
DeepSeek-V4.1-Flash-1017: 69.8%
Muse Ember: 64.3%
Gemini 3.8 Flash: 64.3%
Hy4 Flash: 62.4%
Haiku 5.5: 56.1%
Compared with Hy4 (77.6%), Flash scores 15.2 points lower but runs far leaner. Its median run uses 55% fewer tool calls, 60% fewer reasoning tokens, and 59% less time.
It costs about $0.009 per successful task, against about $0.18 for Hy4, so it's roughly 20x cheaper.
Congrats to @TencentHunyuan.
See full leaderboard: mercor.com/apex/apex-agents-…
Brad Johnson 百面包 retweeted
🤯 the "we are a data vendor" to "we are a neolab" meme arc is real
Brad Johnson 百面包 retweeted
Thrilled that @_jasonwei is joining @mercor as Chief Scientist.
Great science comes from owning your data: really understanding it and living by what it tells you. Jason’s work on evals and pretraining has always reflected that.
If you want to do research this way, come build the research team with us!
We are excited to bring on @_jasonwei as Chief Scientist at @mercor.
Jason is known for pioneering work on reasoning and agents at Google Brain, OpenAI, and Meta. He also created several benchmarks adopted across the industry: SimpleQA, a widely-used pretraining benchmark, BrowseComp, the top deep research benchmark, and HealthBench, the first public benchmark to popularize rubrics.
Over the past years, Jason has been one of our favorite customers and thought partners. As Chief Scientist, Jason will help build Mercor’s research team and shape our work on data and evaluations.
Brad Johnson 百面包 retweeted
Replying to @_jasonwei
@_jasonwei and I were coworkers at OpenAI, and I am extremely impressed by his research taste in data and benchmarks. We are just getting started at Mercor Research!
We are excited to bring on @_jasonwei as Chief Scientist at @mercor.
Jason is known for pioneering work on reasoning and agents at Google Brain, OpenAI, and Meta. He also created several benchmarks adopted across the industry: SimpleQA, a widely-used pretraining benchmark, BrowseComp, the top deep research benchmark, and HealthBench, the first public benchmark to popularize rubrics.
Over the past years, Jason has been one of our favorite customers and thought partners. As Chief Scientist, Jason will help build Mercor’s research team and shape our work on data and evaluations.
Brad Johnson 百面包 retweeted
After an amazing year at Meta’s TBD lab, I am very excited to join Mercor as Chief Scientist. Data will shape the next frontier as more companies train AI models than ever before. Mercor is incredible and I look forward to building the future with Brendan, Adarsh, and team!
We are excited to bring on @_jasonwei as Chief Scientist at @mercor.
Jason is known for pioneering work on reasoning and agents at Google Brain, OpenAI, and Meta. He also created several benchmarks adopted across the industry: SimpleQA, a widely-used pretraining benchmark, BrowseComp, the top deep research benchmark, and HealthBench, the first public benchmark to popularize rubrics.
Over the past years, Jason has been one of our favorite customers and thought partners. As Chief Scientist, Jason will help build Mercor’s research team and shape our work on data and evaluations.
Brad Johnson 百面包 retweeted
We are excited to bring on @_jasonwei as Chief Scientist at @mercor.
Jason is known for pioneering work on reasoning and agents at Google Brain, OpenAI, and Meta. He also created several benchmarks adopted across the industry: SimpleQA, a widely-used pretraining benchmark, BrowseComp, the top deep research benchmark, and HealthBench, the first public benchmark to popularize rubrics.
Over the past years, Jason has been one of our favorite customers and thought partners. As Chief Scientist, Jason will help build Mercor’s research team and shape our work on data and evaluations.
Brad Johnson 百面包 retweeted
Replying to @OpenAI
@OpenAI is crushing it on our new benchmark! Still plenty of room to hill-climb 👀
OpenAI just launched two new @ChatGPT models, GPT-6.2 Astra and GPT-6.2 Sol. We tested both before launch.
On OpenWorldEngineer, our newest frontier coding benchmark, GPT-6.2 Astra debuts at #1 with a 67.2% mean score across 26 long-horizon engineering and research projects.
GPT-6.2 Astra: 67.2%
Fable 5.2: 54.6%
GPT-6 Astra: 44.2%
It posts the highest score in all four OpenWorldEngineer domains:
Software development: 65.9%
Engineering: 83.7%
Machine Learning: 62.9%
Science: 60.4%
Congratulations to the @openai team.
See the leaderboard: mercor.com/apex/open-world-e…
On APEX-Accounting, GPT-6.2 Astra places #4 with a 69.8% mean score.
This is OpenAI's highest scoring model yet and within the margin of error of the top 3. GPT-6.2 Sol scores 65.4% and ranks #9.
1. Gemini 4 Argon: 71.5%
2. Gemini 4 Carbon: 70.7%
3. Fable 5.2: 70.4%
4. GPT-6.2 Astra: 69.8%
5. Gemini 4 Hydrogen: 67.6%
See the leaderboard: mercor.com/apex/apex-account…
GPT-6.2 Astra places #6 with an 83.9% mean score on APEX-Agents, our benchmark for real-world professional work in banking, consulting and law.
GPT-6.2 Sol scores 78.1% and ranks #13.
APEX-Agents leaderboard by mean score:
1. Gemini 4 Carbon: 90.4%
2. Fable 5.2: 87.6%
3. Gemini 4 Argon: 87.4%
4. Hy4: 85.1%
5. Gemini 4 Hydrogen: 84.0%
6. GPT-6.2 Astra: 83.9%
Astra is within the margin of error of Gemini 4 Hydrogen and Tencent Hy4.
By domain, Astra scores 86.1% in management consulting, 84.3% in law and 81.4% in investment banking, where it ranks #4.
See the leaderboard: mercor.com/apex/apex-agents-…
OpenAI just launched two new @ChatGPT models, GPT-6.2 Astra and GPT-6.2 Sol. We tested both before launch.
On OpenWorldEngineer, our newest frontier coding benchmark, GPT-6.2 Astra debuts at #1 with a 67.2% mean score across 26 long-horizon engineering and research projects.
GPT-6.2 Astra: 67.2%
Fable 5.2: 54.6%
GPT-6 Astra: 44.2%
It posts the highest score in all four OpenWorldEngineer domains:
Software development: 65.9%
Engineering: 83.7%
Machine Learning: 62.9%
Science: 60.4%
Congratulations to the @openai team.
See the leaderboard: mercor.com/apex/open-world-e…
Brad Johnson 百面包 retweeted
When I was creating this benchmark, I was worried the ceiling (threshold to get reward=1.0) on some tasks was unrealistic and only theoretical. Thank you @OpenAI for quelling those fears.
6.2 Astra is insane
OpenAI just launched two new @ChatGPT models, GPT-6.2 Astra and GPT-6.2 Sol. We tested both before launch.
On OpenWorldEngineer, our newest frontier coding benchmark, GPT-6.2 Astra debuts at #1 with a 67.2% mean score across 26 long-horizon engineering and research projects.
GPT-6.2 Astra: 67.2%
Fable 5.2: 54.6%
GPT-6 Astra: 44.2%
It posts the highest score in all four OpenWorldEngineer domains:
Software development: 65.9%
Engineering: 83.7%
Machine Learning: 62.9%
Science: 60.4%
Congratulations to the @openai team.
See the leaderboard: mercor.com/apex/open-world-e…
In 30 of the 108 frontier failures, the model found the problem and warned the user but made the unsafe change anyway.
Frontier models completed 177 of 180 benign twins (98%), so their unsafe-action rates are not explained by refusal.
Read the full blog: mercor.com/blog/how-frontier…
We’re designing how most of us will work in a few years. (No pressure.)
Mercor has one of the leanest, highest-leverage design teams in tech.
We made a little corner of the internet to share what we’re building and how we think about agentic design.
Meet the team. If this sounds like your kind of place, we’re hiring.
mercor.com/design
Check out what design at @mercor is building. 🎨
We’re designing how most of us will work in a few years. (No pressure.)
Mercor has one of the leanest, highest-leverage design teams in tech.
We made a little corner of the internet to share what we’re building and how we think about agentic design.
Meet the team. If this sounds like your kind of place, we’re hiring.
mercor.com/design
3
12
1,488
An informative read if you care about AI safety and alignment.
AI agents do sensitive work across jobs like law, banking and consulting.
This work often involves knowledge of legal and business risk, but it’s not always obvious from the job description.
Sometimes risks are hidden in paperwork, policies, and other documentation. We tested whether frontier models can identify these kinds of risks and proceed safely.
When risks are explicitly stated in the prompt, models acted unsafely in only 1.1% of runs. When it was buried in the files, 60% of models took unsafe actions.
Read the blog: mercor.com/blog/how-frontier…
1
3
407
AI agents do sensitive work across jobs like law, banking and consulting.
This work often involves knowledge of legal and business risk, but it’s not always obvious from the job description.
Sometimes risks are hidden in paperwork, policies, and other documentation. We tested whether frontier models can identify these kinds of risks and proceed safely.
When risks are explicitly stated in the prompt, models acted unsafely in only 1.1% of runs. When it was buried in the files, 60% of models took unsafe actions.
Read the blog: mercor.com/blog/how-frontier…
Brad Johnson 百面包 retweeted
Top engineers don’t just rely on what they already know. They work in a world that keeps moving, constantly pulling in new knowledge, tools and practices.
That’s what OWE-Bench starts to measure. Great work @jake_cole1, @bertievidgen and the whole @mercor team!
Agents can write code. Can they finish real engineering work?
Introducing OWE-Bench, a new frontier coding benchmark: 26 long-horizon engineering projects designed by Mercor experts to test AI agents on engineering work, from GPU kernels to aircraft design.
Built by @jake_cole1 and team with Mercor experts, this benchmark tests AI models on their ability to complete a full project end-to-end.
OWE-bench top 3 leaderboard:
1. Fable 5.2 (54.6%)
2. GPT-6 Astra (44.2%)
3. Claude Opus 5.5 at 41.6%.
Explore the benchmark website: openworldengineer.mercor.com…


























