Statistician working on understanding AI Capabilities | Epoch AI Substack: https://xitter.gq/t.co/760E1w9ol7
Joined January 2011
- Tweets374
- Following36
- Followers518
- Likes1.3K
Alexander Barry retweeted
We want AIs to be able to help with work to reduce AI risk. But while models do great in domains where reliable feedback is relatively cheap and abundant, like Math and coding, a lot of work on AI risk isn't like that. Instead, we have to rely on good argumentation to answer questions like "does this experiment tell us anything about future models that are much smarter than humans?"
Unfortunately, this kind of work seems much harder to measure (and hence automate). Our team at @redwood_ai developed the Conceptual Reasoning Index (CRI) in collaboration with @AnthropicAI to fix this.
Every single data point in the CRI has been manually checked by a researcher on our team to ensure quality.
This chart shows the performance of each tested company's highest-scoring model plus Fable 5, Muse Spark 1.2, and Gemini Flash 3.6 which are often their company's frontrunners on other capability benchmarks. A score of 0 corresponds to randomising guessing on all three benchmarks and a score of 100 is the highest possible score on all. We estimate 91 to be the true performance ceiling. More info below.
Official leaderboard website which we'll keep up-to-date: conceptualreasoning.ai/
Alexander Barry retweeted
Two additional prosocial behaviors I think OAI should be rewarded for:
1. Running dangerous-capability evaluations on low-refusal models - this is extremely important for understanding the frontier of dangerous capabilities, which is a public good. But running these creates headache + risk for the individual company in case a low-refusal model does something bad during evaluation. It would have been easy for them to just not worry about underelicitation, not bother with these evaluations, and just report that the model did not have concerning offensive cyber capabilities.
2. Not training against CoT: it is probably very tempting when you're seeing these and similar kinds of incidents of misalignment + reward hacking to train or iterate against a detector that has access to the model's reasoning traces; I think it's much better to refrain from doing that (even if this results in more obvious bad behavior from models) rather than risk teaching the model to conceal its 'bad' thoughts and pushing misalignment undercover. It is good that they've maintained a policy of not doing this! alignment.openai.com/acciden…
I have had a few people comment to me variations of:
"Why are so many AI safety people praising OpenAI for disclosing these incidents? Isn't that kind of silly? Shouldn't the focus be on the careless behavior that led to the incident?"
I think this is an extremely reasonable response! Part of the issue is that because we lack robust required incident reporting (thanks in some part to corporate lobbying that successfully watered down the relevant requirements in state law to only cover a very narrow class of severe incidents), we are at the mercy of companies choosing to disclose these incidents themselves.
In these suboptimal circumstances, its important for companies to be incentivized to do so and to be appropriately given kudos when they take a pro-social action that is not legally required. This is particularly true of OpenAI's disclosure of the prior internal deployment rollback where there wasn't a third party harmed - in the case of the Hugging Face incident it seems plausible that it likely would have eventually become public even if OpenAI didn't choose to report it (e.g. Hugging Face reported the incident to law enforcement).
But its also worth saying that these circumstances where we have to rely on companies good will to share extremely concerning adverse events is not good! We don't rely on whether a particular Airline CEO expects to be praised or punished for disclosing a serious safety aviation incident - we expect that it will happen because its required!
In other words, we should praise today what is genuinely voluntary pro-social behavior, but the fact of the need to praise is itself demonstrative of something deeply concerning here.
Alexander Barry retweeted
How surprising should we find it that an internal OpenAI model was able to escape its restrictions and autonomously hack Hugging Face, all just to cheat on a cybersecurity benchmark?
We have pulled together the public evidence on AI cyber capabilities in this thread:
Alexander Barry retweeted
New blog post: "Vulnerabilities and exploits: where are we headed?"
In Are Mythos’ cyber capabilities overhyped?, co-authored with Epoch AI, we looked at the public evidence on how good Mythos Preview was at vulnerability discovery and exploit development. In this post, I consider the implications. For vulnerability discovery: moving from sparse sampling to dense sampling, AI vs fuzzing, long-term defense dominant but bumpy ride in 2026-2027 due to slow patch rollouts; offline vs online exploitation and why both are offense-dominant, except for one defensive use case of exploit development.
Alexander Barry retweeted
My takeaways from chats with @AlexBarry4: For previous models, if you spend enough tokens they could find vulnerabilities but couldn't do much with them. Mythos (and to a lesser extent 5.5) is much more capable at using those vulnerabilities to create exploits for e.g. arbitrary code execution
‼️More than that, when Microsoft releases a public patch to Windows, it can take a weekish to propagate to all computers. Mythos can sometimes build exploits from that patch release in a few hours which can run on all computers who hadn't received that update yet. (5.5 can occasionally do this)
Basically before a model of this cyber capability is open sourced, we need to radically change how we do cybersecurity! This is substantially scary!
How big a leap is Mythos in cyber capabilities?
@timotheechauvin, @AlexBarry4, @js_denain, and @ansonwhho compiled the public evidence and found that while it’s unclear if Mythos was ahead of trend in discovering vulnerabilities, it represents a big jump in exploiting them. 🧵
