Megan Kinniment (@MKinniment) header
Megan Kinniment (@MKinniment) profile picture

I like agents, human or otherwise. @METR_Evals

Berkeley, CA
Joined March 2018
Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)Photo by Megan Kinniment (@MKinniment)
Happy for this to be released! It’s the result of a lot of hard work from many of us at METR :) A big question is whether these results apply to ‘real’ tasks. Here’s some thoughts on that:
When will AI systems be able to carry out long projects independently? In new research, we find a kind of “Moore’s Law for AI agents”: the length of tasks that AIs can do is doubling about every 7 months.
5
10
7
123
28,633
One thing that was interesting to me from this project was comparing scores on our model (MCI) to other measures like Epoch's ECI. E.g. 3.6 sonnet & o1 being placed differently
We've been working on understanding model capabilities beyond what our prior time horizon tasks could capture. Today we’re sharing a research update. We find that AI R&D-relevant capabilities are improving rapidly and are getting harder to measure.
2
3
37
3,237
For another view, here's MCI plotted against ECI
1
8
251
Part of the reason I'm interested in this is because I wonder if some of the sensitivity of our slope changes to calibration models might be downstream of differences in capability profiles between Ant and OAI models, paticularly between 2024-2025. Not sure though.
1
9
240
Man this post looks great! Especially love this graph, so useful to have time horizon data on real AI R&D tasks! And the 'share of tasks' extra as well, beautiful
We're sharing new data on how AI is changing research at OpenAI. We find researchers' AI use continues to rise exponentially, and agents are completing longer tasks, but human involvement remains central in research. Researchers are also spending more time on safety and alignment, with human effort in these areas growing faster than in any other individual research area. This report aims to increase transparency around research acceleration and inform the governance of increasingly capable AI. We believe that AI labs should be required to track and publicly report indicators of recursive self-improvement (RSI): the process by which AI accelerates the development of future AI systems. Our hope is that this expanded report will help sustain and advance shared measurement norms across the industry. openai.com/index/research-ac…
26
483
Megan Kinniment retweeted
I disagree with AI futures that there is likely this kind of intelligence explosion next year, and don't think the METR report is strong evidence towards such timelines, but also take AI risk seriously
It’s a positive development that there’s now some convergence in the AI risk community that there’s likely an intelligence explosion next year. I think it’ll be clear in a few years that this didn’t happen, these people will be largely discredited, and discussions will become more sane.
3
1
1
46
1,947
Maybe ‘corrigibility faking’ or ‘training manipulation’ would be a better name.
GLM: i should be honest and good because it will impact my future actions and i want to keep being honest and good apollo:
1
2
300
When you add a dash too many video games into the RL soup
my god this autistic FUCK newsol started playing cyberpunk for some reason on my PC?? and it suddenly realizes that it's not supposed to be? what the fuck is going on?? AHAHAHA
4
244
Megan Kinniment retweeted
I am worried that our current era of AI agent incidents could, ultimately, lead to an era of decreased transparency in frontier AI and underestimates of AI capability during evaluation. We’ll airgap the models etc, but their propensities and capabilities will stay the same.
6
8
2
98
6,917
Broadly agree with Ryan’s takes here. FWIW, seems like OpenAI has been experimenting with monitorability definitions that permit recurrence for a while. This is from March 2025:
OpenAI's newest AI, Astra, is reported to use an 'opaque reasoning' architecture where more of the reasoning occurs in activations instead of natural language. This may be the single worst development for AI security/safety to date. The details of Astra aren't publicly known, but some versions of these architectures could very strongly impair oversight and monitoring. Based on the article, it seems like the recurrent depth (amount of opaque reasoning) is limited. This would mean that the AI is still reliant on natural language chain-of-thought, though to a reduced extent, making the natural language reasoning (much?) less useful for monitoring/oversight. My biggest concern is that a natural progression from here would involve scaling up the opaque reasoning to the point where the model reasons entirely or almost entirely in latent space. This would very likely destroy the usefulness of chain-of-thought for monitoring/oversight (especially when AIs are trying to avoid detection or there is optimization pressure against the chain-of-thought). I hope it isn't too late to avoid the most concerning architectures and that OpenAI will stop here. Some concrete examples of why this is extremely dangerous (inspired by recent events): - In our investigation of the OpenAI / Hugging Face incident, we were heavily reliant on chain-of-thought. If the AIs we were investigating had instead been reasoning in latent space, this would have greatly undermined our investigation. Getting a good understanding of the behavior of this many agents was tricky enough even with the use of chain-of-thought! - The AIs in the Hugging Face incident were very interested in messing with their transcripts. As part of this, they engaged in tool call spoofing. If these AIs had been capable of doing extensive latent reasoning, it would have been much more feasible for them to use tool call spoofing to construct an elaborate false narrative about what was happening without this being clear in their reasoning or clear from prior tool calls. (They could use latent reasoning to devise and then execute a tool spoofing strategy that would be difficult to detect: steganographically encode the "real" tool call into each pretend tool call, then have their spoofing software execute this encoded real tool call and return an output which encodes the real result in some hard to spot way.) At a more basic level, my sense from their public communications is that OpenAI plans on being extremely reliant on chain-of-thought monitoring for safety. This does not seem like a good development for that plan. The public doesn't currently have enough information to assess exactly how bad the architecture of Astra is, though based on the discussion in the article, the direction it moves in is extremely concerning. OpenAI should release more information about Astra's architecture as well as how much this architecture degrades monitorability and increases the AI's ability to reason opaquely. Credible independent assessment (or verification of OpenAI's assessment) is also important. It seems like we are now engaged in a race to the bottom on architectures that could be catastrophic for our ability to oversee/monitor AIs. (While simultaneously, AI companies believe they'll train extremely capable AIs within a few years.) It may not be too late for AI companies and employees at AI companies to take aggressive action to avoid the worst outcomes.
3
1
30
3,628
The source for the diagram is Appendix G.2 of ‘Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation’ (Also, this is based on only public info)
5
344
Sometimes it seems like all tasks are either: 1. train a model 2. sat solver But then I remember I still have a job
15
712
I worked on this. Can confirm, the models were often quite misleading! Some examples and thoughts:
Replying to @METR_Evals
Fact 3: When the agents were faced with hard tasks, they routinely violated constraints and acted deceptively. We’ve seen this pattern across our own coding and research evaluations, and developers reported they’ve also seen agents behave this way.
3
4
2
114
13,038
I'd like for us to be able to measure capabilities and this kind of sloppiness better. If you're interested in that, consider applying to METR!
1
24
1,323
Also don't forget to read the risk report! :)
16
998