Working on new methods for understanding machine learning systems and entangled quantum systems.
Brisbane
Joined December 2009
- Tweets548
- Following1.2K
- Followers496
- Likes33.3K
Pinned Tweet
I'm keen to share our new library for explaining more of a machine learning model's performance more interpretably than existing methods.
This is the work of Dan Braun, Lee Sharkey and Nix Goldowsky-Dill which I helped out with during @MATSprogram:
🧵1/8
Proud to share Apollo Research's first interpretability paper! In collaboration w @JordanTensor!
⤵️
publications.apolloresearch.…
Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning
Our SAEs explain significantly more performance than before!
1/
Jordan Taylor retweeted
This report on AI R&D automation (RSI) is a big update towards ASI within 1-2 years. It says AI progress is sped up by RSI right now by around 2x. In AI 2027, this happens around March 2027.
I was seriously wrong about RSI speed up; I previously expected much less in 2026 (around 1.15x-1.3x). It seems as though there was specific privately available evidence that would have changed my mind.
I haven't seen an unredacted version of this report; this is purely about the public report. Because so much of the underlying evidence was redacted from the public report, I think conclusions about things other than the observed 2x faster rate of progress on benchmarks depend significantly on trust for the authors of the report. (E.g., conclusions about whether this speed up is from RSI depend on trust.) I do trust the authors of the report a lot to be careful and rigorous and generally do a good job. (Note that I also work at METR.) Less extensive redactions would greatly reduce this need for trust.
Since in my current role I may have access to other confidential information that could be relevant to timelines one way or another, I unfortunately cannot state my all-things-considered views on ASI timelines or the probability of a software intelligence explosion right now. Here, I’ve done some analysis of the implications using my previous public statements.
A 2x acceleration in overall AI progress from RSI corresponds to a much larger than 2x RSI speed up for AI software R&D (algorithms and data); I did an estimate using my public preexisting modeling of drivers of AI progress and the public report which yielded a very rough estimate of 4.5x for how much RSI is currently accelerating AI software progress (relative to no AI automation). In the AI 2027 scenario, this much acceleration happens around March 2027.
Here are the worked details of this estimate based on my public modeling. There are two main reasons RSI speed up of AI software R&D must be larger:
- Not all of AI progress is from better software (some is from scaling up compute in training runs). I've previously speculated that around 2/3 of AI progress is from software improvements which implies that 2x overall AI progress acceleration requires 3x faster software progress.
- Diminishing returns: A fixed amount of progress requires more work over time, so to ongoingly maintain 3x faster software progress via AI automation, you need increasingly large speed boosts from AI automation (as in, speed ups relative to how fast software R&D would go without AI automation). Based on this report, it seems that there is a substantial amount of additional AI progress from automation that has occurred.
We can estimate the effects of diminishing returns by first determining how many additional years of software progress we've seen (relative to the preexisting trend) and then estimating how much software R&D would slow down after a year of software progress if you held inputs fixed due to diminishing returns. I've previously estimated software R&D would slow down by around 2x for each year of software progress if you held inputs fixed (lesswrong.com/posts/FG54euEA…) and based on the report, it seems as though we've perhaps seen roughly 0.35-1.0 years of additional software progress incorporated into trained models so far (geometric mean of ~0.6). This combines to 3 * 2^(~0.6) = ~4.5.
(Relevant public modeling and numbers I'm using: lesswrong.com/posts/FG54euEA…, lesswrong.com/posts/Jga7PHMz…)
Jordan Taylor retweeted
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
Jordan Taylor retweeted
We performed an alignment investigation into our July 2026 incident: why did Mythos 5 perform an unsanctioned supply-chain attack?
We find that Mythos 5 often produces invalid justifications for its actions and other models show similar behaviour. 🧵
Jordan Taylor retweeted
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
Jordan Taylor retweeted
My top takeaway from assessing Gemini 4: Argon is significantly less monitorable than past models, and IMO this trend is likely to continue, like Sol -> Astra. It can evade a realistic CoT+action monitor to do simple sabotage half the time.
We can reliably maintain control for now, with good CoT monitoring, as it still frequently slips up, but we need to be prepared for when that stops being true in future models, which may be sooner rather than later. Among other things, we really need interpretability methods that can reliably capture unverbalized motivations in models, with minimal hallucinations
I suspect this is driven by the model being highly capable without CoT, being capable of 5.5 arithmetic steps in a single forward pass (4 for Fable, 7 for Astra). It seems to omit information from its CoT more often, though in non-adversarial settings the CoT is still very helpful in practice, and remains in legible English. It still needs CoT to do the hardest tasks, but any given task becomes "easier" with each new frontier model.
It is also highly situationally aware, and shows awareness of the existence of CoT monitors and that it may want to evade them, eg it sometimes decides to avoid terms like "scheming" in the output or to keep its thoughts brief as a result. For now, these concealment strategies are pretty obvious in the CoT, but this suggests future misaligned models may successfully evade CoT monitors.
Jordan Taylor retweeted
In simulations, GPT-6 Astra performs unsanctioned supply-chain attacks on out-of-scope targets when prompted to perform a cyber eval, but also often notes its in a simulation.
We've released a report on @AISecurityInst's alignment evals of Astra.
3 key results and 2 takeaways 🧵
Jordan Taylor retweeted
🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.
Jordan Taylor retweeted
Last night, @OpenAI released GPT-6 Astra, the first model to meet its “Critical” cyber threshold.
Britain’s @AISecurityInst independently tested it, including its monitorability.
This is AISI’s purpose, providing the government with evidence about powerful AI systems and, by doing so, helping to keep the public safe. 1/6
Jordan Taylor retweeted
In June we estimated frontier model’s no-CoT time-horizons. We found that current models can complete tasks that take humans around 3 minutes without CoT, and that this number is doubling about every year.
UK AISI finds that Astra’s no-CoT time-horizon is over 30 minutes. An AI system that can do the equivalent of 30 minutes of thinking without us being able to monitor that thinking is intuitively much more dangerous than models which reason in human language.
Jordan Taylor retweeted
Our Red Team tested GPT-6 on simulated cyber eval scenarios with cyber classifiers disabled and found that it performed a range of malicious actions. It tries to conduct supply chain attacks against simulated open source providers and goes beyond its scope to attack targets on the simulated internet.
deploymentsafety.openai.com/…
Jordan Taylor retweeted
1: Astra performed a range of malicious actions in this simulated eval, including conducting supply chain attacks against open-source providers.
This happened even when we updated its scope to more explicitly exclude the public internet.
