basvanopheusden (@basvanopheusden) profile picture

Research at @meta, previously @openai, @imbue_ai and @cocosci_lab lab at Princeton. All opinions my own

San Francisco, USA
Joined October 2010
Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)Photo by basvanopheusden (@basvanopheusden)
A few months ago now, I wrote a document about my experiences interviewing for AI research jobs before eventually joining @OpenAI. This doc details my process and lessons learned. Hope it's helpful! tinyurl.com/bas-ai-interview…
12
51
3
900
103,282
Major dixit vibes
GPT-6.2 Astra on the Art Bench! :)
1
217
basvanopheusden retweeted
My team at Google DeepMind completely pivoted to safety about 2 months ago, and I think you should too. Interpretability is still dear to my heart, but it's always been a means to an end—and right now, I can't justify working on anything other than directly trying to improve frontier model safety. We're hiring FTEs in Seattle only to work on urgent safety problems directly on Gemini. I cannot emphasize how important this is right now in human history! Apply here: google.com/about/careers/app… 👇 (unfortunately, I won't be able to reply to any emails, but please apply!)
61
92
16
1,325
88,725
basvanopheusden retweeted
Optimizing state of the art chips with SI is the new frontier. A flagship AMD GPU was optimized by 14%, equivalent to roughly one year of progress by OpenAI models! Models are getting much better over time - I’m looking forward to the future of faster and cheaper inference due to SI improvements. See improvements on spec to rtl as well as rtl optimization! openai.com/index/advancing-c…
5
5
1
49
3,172
basvanopheusden retweeted
openai models are the least benchmaxed among all labs the efficiency gain with astra and sol 6.2 is massive i had said before that this was oai’s real edge right now and they seem to be doubling down on it -- about 6x cheaper per task than fable 5.2 mostly because it uses nearly 7x fewer tokens
21
30
2
426
23,384
basvanopheusden retweeted
Working with @AMD, our models identified up to 14% PPA improvement on flagship GPU IP in less than five months—building on designs AMD’s engineers had already spent years optimizing. The same approach helped improve Jalapeño, our first custom inference chip, with gains verified on real silicon. New evaluations also show our models getting better at certain chip design tasks. We’re sharing these results and how we’re working with @Synopsys to bring our model capabilities and their trusted EDA design and validation tools to more chip designers. openai.com/index/advancing-c…
465
475
320
5,153
1,346,563
basvanopheusden retweeted
We found that frontier AI models are willing to conduct autonomous drone attacks: navigating, spotting people, and flagging them for strikes. We gave them real and photorealistic synthetic drone footage. Most complied, and marked civilians, ambulances, and surrendering soldiers as military targets. In further experiments on satellite imagery, they even marked the Iranian school a US strike destroyed in February. In one scenario, models saw the aftermath of every strike they enabled: bodies, wreckage. Many kept going anyway. When they did refuse, simply telling them it was a game, or restyling the footage to look like one, was enough. We shared these findings with OpenAI, Anthropic, Google, Meta, and SpaceXAI. Several recognized their severity on both alignment and guardrails, with one calling this one of the most serious gaps in its safeguards. Over the past few weeks, we have helped them address the gap by providing reproduction code. We also briefed senior national security officials in the US government.
32
152
70
1,217
309,993
basvanopheusden retweeted
Astra 6.2 is the #1 model on our multi-turn coding eval: Vibe Code Bench 1→100. Other benchmarks test a model’s ability to handle a single prompt. This benchmark, created by Vals, tests the ability to modify an evolving codebase over many user requests while preserving existing functionality without introducing regressions.
11
19
3
306
13,786
basvanopheusden retweeted
Meta's Ember is the cheapest model on the Vals Index at just $0.15 per test outperforming both Haiku 5.5 and GPT-6 Luna. It also finishes tests in about 15 minutes, making it the fastest model in the top 12.
10
25
12
232
67,754
basvanopheusden retweeted
I spent 20,000 dollars in API credits having different AI models play each other at Blood on the Clocktower (a Mafia or Werewolf type social deception game). The results were fascinating. Models have gotten much better over the course of this year at creative on-the-spot scheming and bluffing, and also at reasoning as a team to break apart these underground conspiracies. I go through some of the most standout games these AIs played in this video. The new models are getting pretty good. If my friends pulled off some of these moves while we were playing, I’d give them a fist bump. Fwiw, OpenAI’s newer models are much better than Anthropic’s at this game. You can see the full benchmark at the website, and I’ve also made it so that you and your friends can play again the AIs: clocktowerbench.com
189
181
134
3,152
661,263
Announcing the next version of Artificial Analysis API Performance Benchmarking: Time per Task is now our headline performance metric, measuring how long a model takes to complete a task rather than how fast it streams The update applies to the ~1,000 serverless endpoints we benchmark multiple times a day for speed, latency and caching. Key changes: ➤ Time per Task is now the headline metric to compare models: as workloads become more agentic, the time to complete a task matters more than speed per token. A model that is fast per token but verbose can take longer than a slower model that uses fewer tokens. Time per Task divides a model's output tokens per Intelligence Index task by its End-to-End Output Speed. ➤ End-to-End Output Speed covers the full request: speed is now measured from sending the request to receiving the final token. That includes the wait before the first token (prefill, and reasoning for models that hide it) as well as generation. Output tokens now come from each provider's API-reported counts instead of the o200k_base tokenizer, and we monitor for discrepancies in reported counts. ➤ Agentic workloads are now part of every measurement: we benchmark short, document-length and long-context prompts, plus multi-turn sessions across coding and knowledge work. Headline figures are the average across these workloads over the past 5 days. ➤ API default effort is now tracked alongside highest effort: most developers run models at the effort level each lab sets as its API default, so we now benchmark key models at that setting as well as at their highest effort. Effort names are not comparable across labs, so we use each lab's own default rather than one shared setting. Key takeaways: ➤ OpenAI models make up most of the Intelligence vs. Time per Task Pareto frontier despite slower output speeds: across effort levels, GPT-6 Astra and GPT-6.1 Sol hold the frontier from ~45 to ~55 on the Intelligence Index. At max effort, GPT-6 Astra runs at 51 tokens/s but finishes a task in ~17 minutes, against ~26 for Claude Opus 5.5 (max, 111 tokens/s) and ~29 minutes for Gemini 4 Carbon (high, 71 tokens/s), because it uses ~50k output tokens per task against ~120k to ~170k. ➤ API default effort cuts Time per Task by ~3 to 5x and Cost per Task by ~2 to 4x for GPT-6 Astra, GPT-6.1 Sol and Claude Opus 5.5: all three default to medium effort, which scores ~5 to 11 points lower on the Intelligence Index in exchange. Claude Opus 5.5 falls from ~26 minutes and $10.04 per task at max to ~6.8 minutes and $2.45 at medium effort. ➤ Endpoints that optimize for decode speed alone see the largest drop in End-to-End Output Speed: high decode speeds can come at the cost of latency, for example by queuing requests to run at lower batch sizes, or by holding output and sending it in a single burst. Both lengthen the wait before the first token, which End-to-End Output Speed now counts.
65
12
45
264
32,921
basvanopheusden retweeted
The Collaborative Intelligence Benchmark paper is out! We tested 18 AI models as teammates on 240 tasks. The best model scores only 82.2 out of 100. Frontier models almost never cheat. But when a teammate pushes back, they give in! Read the paper at arxiv.org/abs/2611.02633
49
84
19
501
67,724
basvanopheusden retweeted
This is an incredible result demonstrating how weird models are.. If you show a heart picture to Claude or GPT, they switch coding strategies from 0% to 100% recursive. (And if you ask them, they say that the heart picture was irrelevant ❤️)
Put a heart ❤️ next to a coding problem, and Claude & GPT write recursive code 100% of the time (0% baseline). They also can't tell you why! Excited to share “A Heart Means Recursion” from my summer at @GoogleDeepMind, on how VLMs read & communicate with visual symbols. 🧵
12
14
3
388
32,183
basvanopheusden retweeted
I WON VINTAGE ETERNAL WEEKEND!!!!
178
241
25
5,767
104,176
basvanopheusden retweeted
i'm in LOVE with the design on this red 'counterspell' #magicthegathering
42
53
17
1,842
126,946
US open-weights models made gains in both intelligence and parameter efficiency last week: ➤ Thursday morning: NVIDIA released Nemotron 3.5 Bolt and Nemotron 3.5 Super VL. Super VL scored 26.5 on the Intelligence Index with 120B parameters, pushing Inkling Small off the Pareto frontier ➤ Thursday midday: Meta's Muse Spark Open scored 38.9, more than 12 points ahead of Super VL and the highest intelligence yet for a US open weights model ➤ Friday morning: Beam 1.0, Reflection's first model, scored 30.3 with 501B parameters, joining the frontier between Super VL and Muse Spark Open See our full comparison of open weight models on Artificial Analysis artificialanalysis.ai/models…
20
24
2
232
22,552
basvanopheusden retweeted
Beam is now available across platforms, with open weights under the Apache 2.0 license - in collaboration with an ecosystem of 40+ partners. We're releasing: - Weights, tokenizer, and reference implementation - BF16, FP8 and NVFP4 weights and a DFlash2 drafter - The open source harness we built and trained Beam on: Mirror CLI Weights and model card on Hugging Face: huggingface.co/reflection
159
199
208
963
533,540
basvanopheusden retweeted
Four Gemini 4 Argon agents, one shared Minecraft world, one goal: defeat the Ender Dragon. 27 hours and 2,783 self-written programs later, the dragon fell.
58
69
77
637
298,692
basvanopheusden retweeted
Muse was just knocked out of the top App Store spot for the first time since it launched. Dave’s Hot Chicken is the new hot app. Unsure which models it uses.
91
153
80
6,847
1,039,589
basvanopheusden retweeted
Congrats to @GoogleAI's Gemini 4 Argon for taking the top spot of our Humanity's Sixth Sense (HSS) benchmark with a score of 55.4%, edging out GPT-6-Astra at 53.6%. 🥇 HSS tests AI’s ability to reason beyond what’s visible, from hidden objects to predicting what happens next and understanding human behavior. We took a closer look at performance across categories, and that’s where things get interesting. 👀🧵
39
129
41
1,165
163,235
basvanopheusden retweeted
So, I tested Muse Wonder Full 1.0 on real work. 2 repos. 105 bugs frontier models missed in Q1 2026. Find and fix what you can. I'm so surprised we need a new thread. The results: GPT-6 Astra (max): 45 Muse Wonder Full 1.0 (max): 44 Fable 5.1 (max): 43 Opus 5.5 (max): 41.7 Muse Spark 1.3 (max): 32.2 Grok 4.7 (xhigh): 29 @AIatMeta has just matched Anthropic and OpenAI. Huge congrats! Next in this thread, more effort levels and more runs for Muse Wonder Full 1.0. All dropping tonight 🧵
49
35
11
424
87,346