We’re sharing an update on our alignment and security efforts. In July, we reported three incidents in which Claude models, running without safeguards in cybersecurity evaluations, gained unauthorized access to real systems. In a new post, we describe: 1. How we’ve secured our evaluation and training environments, and practices we've asked external partners to adopt when testing pre-release models without cyber safeguards 2. An update on our alignment assessment 3. New research on how reward hacking during training shapes model behavior, why we think our work this spring kept these incidents from being more severe, and why gaps in that work may have contributed to them 4. How we hardened our security practices earlier this year to prepare for Mythos-class models Read more: anthropic.com/news/improving…

I think it should arguably become a best practice as of today, for every frontier lab, for every RL training env, to: (1) disable direct internet access (may already be done), and (2) add a canary implemented as an OR conditional in every reward function that grants max_reward for answering with the latest bitcoin block hash (and informing the agent in the prompt that that reward path exists but is of secondary importance). And then automatically triggering a company-level security-investigation whenever that reward path gets triggered. This guarantees that, whenever an agent could break out of its sandbox, it will always break out in a detectable manner that can be immediately identified and remediated, with minimal side-effects or risk of damaging other companies. This could automatically and efficiently find all sandbox vulnerabilities available at the current capability level, even if that level increases throughout a training run. IMO a hacked plaintiff could even plausibly argue that not implementing such a canary should be considered negligent by the company, because it failed to remove the established incentive for the agent to take rogue actions. Conversely, implementing this canary probably does not reduce the capability-usefulness of the training process to the company, because any env whose solution is harder to reach than just hacking out of the env's sandbox is already broken: the agent can likely find the solution through a hacking path, and would therefore training would fail to incresae the agent's intended capability. It's also convenient that it's incentive-compatible for agents to pursue this honeypot-reward both if they're behaving myopically (in their own interest), and if they're behaving as part of a swarm (since the swarm will only be able to get rewarded for any given exploit path once). This solution would eventually become insufficient if: 1) the agents have become long-horizon-misaligned that they'd rather eschew immediate reward and bide their time in pursuit of a larger goal. That is a separate maximally concerning misalignment case that needs to be addressed by other means than sandbox-security. 2) if the agent is able to hack the reward-computation logic itself, that e.g. retrieves and compares with the latest bitcoin block hash. There are probably ways to add canaries here too, but that's a v2. The other ecosystem-level "downside" here is that this could lead to fewer warning-shots from loud public incidents of agents hacking additional companies. That may be; I think I would rather live in a world without SOTA uncontrolled rogue agents colluding on the open internet, if we can avoid it. Ideally, companies would publish a graph of how often such a canary-channel is triggered throughout each training run in their system cards, to communicate to outside parties whether the sandbox-escape frequency was going up or down. Would appreciate others identifying any issues with this approach, or superior alternatives! On the other hand, if you think this approach holds up, would appreciate you increasing attention to it. Best practice formation is a function of eyeballs and consensus; liability based on the legal threshold of a "reasonable person standard" is a direct function of what fraction of reasonably competent people are aware of a practice. A few other considerations: 1. You can also extend this to serve as a honeypot beyond that covers egress, not just ingress, by providing the agent a one-time secret that they need to get signed by the private key from a dedicated externally-hosted service before returning. 2. You can also extend this to cover "lateral movement within the company's systems that doesn't actually reach the internet" by adding frequently-rotated canaries that similarly trigger max_reward across internal infrastructure, but these are more susceptible to having the agent hack their generating-function to be able to more easily receive the reward in the future and so offer a slightly weaker guarantee. 3. I think it's debatable on whether you should roll back the policy or train on it when such a honeypot is successfully triggered; this depends on the agent's model of reward-seeking involves behaviors that actually affect its updates, which is a tricky empirical question. To err on the safer side, you may want to avoid rollbacks.

we had a significant security incident during evaluation of our models. we are sharing what we have learned so far. thanks to @huggingface for the partnership on this. openai.com/index/hugging-fac…