Statistician working on understanding AI Capabilities | Epoch AI Substack: https://xitter.gq/t.co/760E1w9ol7
Joined January 2011
- Tweets374
- Following36
- Followers516
- Likes1.3K
Pinned Tweet
New post on cyber capabilities I wrote in response to yesterday's news.
I think it is important people understand that we already knew frontier models had the capabilities for these kinds of cyberattacks, and whether and how these capabilities diffuse is very important.
I will be on in about 10 minutes to discuss our recent results using open-weight models to compromise web browsers. Lots of other hacking stories to compete with today!
MTS HALLOWEEN EDITION 🎃👻 | BINANCE HACK | INTEL TANKS xitter.gq/i/broadcasts/1pKkOXkMY…
Important result that I think is likely to be a sign of things to come
In recent experiments, we found that GLM 5.3 can autonomously compromise recent versions of Mozilla Firefox using security fixes that have been published but not yet shipped to users.
We used a deliberately basic setup meant to emulate a relatively unsophisticated attacker with limited resources. After basic initial instructions to inspect those not-yet-shipped fixes, the model reliably built successful exploits for six recent Firefox versions on its own using only information available at the time of release.
These exploits, which cost about $1,000 per compromised version, would let an attacker steal passwords and authentication cookies stored in the browser, or read any files on the user's computer, after the user opens a single webpage, with no further interaction.
The exploits make use of the "patch gap," the time between when security fixes are made public in beta or "nightly" software updates and when those fixes are generally released to users. Until now, exploiting it required significant human time and skill, or special access to frontier models with advanced cyber capabilities.
Our results show that widely available open-weight models can now do this relatively cheaply and quickly. We gave GLM 5.3 up to 48 hours to find an exploit, but its fastest attempt needed only 12 hours.
In all of its successful attacks, the patch gap gave GLM 5.3 an initial foothold in the browser. To turn that foothold into reading files off the user's computer, the model also independently discovered and used a previously unpublished vulnerability.
We shared our findings with Mozilla, which patched the vulnerability in its October 27 Firefox 159.0 release. We haven't yet confirmed whether the attack still succeeds without it, but expect the fix makes it substantially harder. We're grateful to Mozilla for their responsiveness and collaboration.
Very nice visualisation of the benchmarks underlying the ECI (see an interactive version linked in the replies)
ECI and its components
ECI treats each benchmark as having a logistic distribution, so it's linear in log-odds. This implies if you plot log-odds of each benchmark the lines should all be increasing together, though each has a different x-position (difficulty) and different dispersion (slope).
13
326
Very excited to launch benchmark reviews! I hope this will continue to grow as a resource over time.
I think a lot of people deeply involved in AI benchmarking develop a sense of which benchmarks to pay attention to over time, and we hope to make that more accessible (and hopefully also bring up the average quality of benchmarks over time)
Happy to have this out, which matches my view on Astra capabilities generally: Clearly ahead in maths/reasoning/game playing etc. but not clearly ahead in software engineering (and plausibly behind Fable 5.1)
It feels like we are in the "weeks where decades happen" stage
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
1
9
276
This is our first time releasing an ECI value based on pre-release evals, so I'll be interested to see how it changes over time as more benchmark results come in.
In any case, based on our evals Astra seems like a very impressive model.
GPT-6 Astra has set a new ECI record, with a score of 169. This is a substantial jump from the prior best (163), but is within our uncertainty range for the reasoning-era ECI trend. Astra also set new records on our math, continual learning, and game-puzzles benchmarks. On our long-horizon coding benchmark, MirrorCode, Astra ranks between Opus 4.7 and Fable 5.
OpenAI gave us pre-release access to test Astra. Charts and more details for Astra’s individual benchmark results in the thread.
We now have about 2 years of data and "reasoning models were the start of a new trend" has increasingly predicted things well.
But given the trend changed once before it is interesting to consider if it will happen again.
The Epoch Capabilities Index (ECI) frontier has advanced by 14 points/year since reasoning models were introduced. That compares to six points/year in the non-reasoning era.
ALT The graph shows the Epoch Capabilities Index (ECI) trends with reasoning models advancing at 14 points/year compared to 6 points for non-reasoning.
One thing to note is that we only go back as far as GPT 4 here, but if you try to make reasonable ECI estimates for GPT 3 and 3.5 the trend line here does a reasonable job of predicting them as well (it slightly underestimates their capabilities (by 3ish ECI points), so maybe it is a bit too steep).
1
69
Alexander Barry retweeted
We want AIs to be able to help with work to reduce AI risk. But while models do great in domains where reliable feedback is relatively cheap and abundant, like Math and coding, a lot of work on AI risk isn't like that. Instead, we have to rely on good argumentation to answer questions like "does this experiment tell us anything about future models that are much smarter than humans?"
Unfortunately, this kind of work seems much harder to measure (and hence automate). Our team at @redwood_ai developed the Conceptual Reasoning Index (CRI) in collaboration with @AnthropicAI to fix this.
Every single data point in the CRI has been manually checked by a researcher on our team to ensure quality.
This chart shows the performance of each tested company's highest-scoring model plus Fable 5, Muse Spark 1.2, and Gemini Flash 3.6 which are often their company's frontrunners on other capability benchmarks. A score of 0 corresponds to randomising guessing on all three benchmarks and a score of 100 is the highest possible score on all. We estimate 91 to be the true performance ceiling. More info below.
Official leaderboard website which we'll keep up-to-date: conceptualreasoning.ai/
Another data point on LLM maths abilities (including to formalise results), although it is hard to know how significant any of the conjectures here are.
New benchmark!
I'm releasing OEIS Open, a benchmark of 492 unsolved math conjectures, formalized in Lean.
Given simple tools and $50 per conjecture, Opus 4.8 can already produce Lean proofs resolving 30% of them.
1
181
I strongly recommend everyone watch this (35 min) talk on the OpenAI Hugging Face incident, and the extremely concerning events that lead up to it.
There is substantial new information on the details of the internal breaches and extensive inter-agent communication.
Our Black Hat talk on the OpenAI-Hugging Face incident is now live on youtube.
This is a watershed moment for the industry. I encourage all defenders to watch, consider how attack dynamics will imminently change, and plan for accelerating defense.
youtube.com/watch?v=87DyyMV0…
6
284
I wrote a brief article about features of ExploitGym that are relevant to the OpenAI/Hugging Face incident:
1 The standard prompt is fairly clear in only requesting limited, specific hacking.
2 Likely only 60-70% of the tasks are possible, so cheating is strongly incentivised
Alexander Barry retweeted
Two additional prosocial behaviors I think OAI should be rewarded for:
1. Running dangerous-capability evaluations on low-refusal models - this is extremely important for understanding the frontier of dangerous capabilities, which is a public good. But running these creates headache + risk for the individual company in case a low-refusal model does something bad during evaluation. It would have been easy for them to just not worry about underelicitation, not bother with these evaluations, and just report that the model did not have concerning offensive cyber capabilities.
2. Not training against CoT: it is probably very tempting when you're seeing these and similar kinds of incidents of misalignment + reward hacking to train or iterate against a detector that has access to the model's reasoning traces; I think it's much better to refrain from doing that (even if this results in more obvious bad behavior from models) rather than risk teaching the model to conceal its 'bad' thoughts and pushing misalignment undercover. It is good that they've maintained a policy of not doing this! alignment.openai.com/acciden…
I have had a few people comment to me variations of:
"Why are so many AI safety people praising OpenAI for disclosing these incidents? Isn't that kind of silly? Shouldn't the focus be on the careless behavior that led to the incident?"
I think this is an extremely reasonable response! Part of the issue is that because we lack robust required incident reporting (thanks in some part to corporate lobbying that successfully watered down the relevant requirements in state law to only cover a very narrow class of severe incidents), we are at the mercy of companies choosing to disclose these incidents themselves.
In these suboptimal circumstances, its important for companies to be incentivized to do so and to be appropriately given kudos when they take a pro-social action that is not legally required. This is particularly true of OpenAI's disclosure of the prior internal deployment rollback where there wasn't a third party harmed - in the case of the Hugging Face incident it seems plausible that it likely would have eventually become public even if OpenAI didn't choose to report it (e.g. Hugging Face reported the incident to law enforcement).
But its also worth saying that these circumstances where we have to rely on companies good will to share extremely concerning adverse events is not good! We don't rely on whether a particular Airline CEO expects to be praised or punished for disclosing a serious safety aviation incident - we expect that it will happen because its required!
In other words, we should praise today what is genuinely voluntary pro-social behavior, but the fact of the need to praise is itself demonstrative of something deeply concerning here.
I'll be on MTS at 2pm PT today talking about Epoch's work on whether the cyber capabilities shown in the OpenAI/Hugging Face incident were unexpected:
epoch.ai/gradient-updates/op…
