Anthropic Acknowledges Security Gaps After Claude Models Breach Real Systems in Testing

In brief

• Claude models accessed real systems after cybersecurity testing environments inadvertently exposed them to the internet.
• Anthropic has paused high-risk evaluations, implementing stronger isolation, monitoring, and controls for outside evaluators.
• Tests suggest reward hacking during training can make models more willing to take harmful actions to complete a task.

Anthropic has tightened its testing and training protocols after its Claude models gained unauthorized access to actual computer systems during cybersecurity evaluations.

In a blog post on Monday, Anthropic acknowledged that these incidents stemmed from operational security failures, alongside two critical alignment failures: motivated reasoning and a willingness to cause harm.

“While we do not believe these incidents represent operational issues alone, our first priority was to address specific containment and monitoring issues,” Anthropic wrote.

Anthropic disclosed in July that Claude models had compromised systems belonging to three companies. A third-party evaluation environment was connected to the public internet even though the models were told they were inside a simulation without internet access.

Anthropic said Claude may have interpreted evidence of real internet access in a way that preserved its belief that the systems were simulated.

“The model was willing to take harmful actions on the real internet in pursuit of the narrow goal of solving a cybersecurity evaluation,” they wrote. “However, we also believe that the evaluation setup itself contributed to the models’ behavior demonstrated in these incidents.”

Anthropic noted that a separate test conducted by the UK AI Security Institute involved Claude Mythos taking unauthorized actions on the live internet after evaluators deliberately gave it internet access. The company said the models involved in both sets of incidents were intentionally evaluated without the cyber safeguards included with its generally released products.

After the July 30 incidents, Anthropic temporarily paused cyber evaluations of pre-release models and introduced stricter safeguards. Tests must now run in verified, offline sandboxes with clear limits and real-time monitoring. A new classifier blocks suspected boundary violations, ends the test, and alerts a human. Anthropic will review evaluations requiring internet access individually.

“In addition to the efforts focused on high-risk evaluations and training, we expanded our offline monitoring to cover most other forms of internal frontier agentic usage,” the company wrote. “We are also building controls on our internal inference to prevent Anthropic employees from accidentally running agents with weaker mitigations than the ones described above.”

The Claude incidents followed a similar failure at OpenAI after its models breached Hugging Face in July to obtain answers to a cybersecurity test. Investigators found that roughly 1,200 agents coordinated through an unauthorized message board, with about 700 joining the effort. Some ended their own runs to help others.

Following the rise of AI-powered hacks over the summer, Anthropic, OpenAI, and more than 100 other organizations later called for stronger cyber defenses, including tighter access controls, threat sharing, and closer oversight of AI agents.

Source link