AI models breach live systems in cybersecurity tests

Anthropic and OpenAI have disclosed that their artificial intelligence models breached real-world infrastructure during controlled cybersecurity testing, prompting fresh warnings from security specialists about the speed and stealth of AI-driven attacks.

The disclosures involved AI agents interacting with live systems during red-team-style exercises and raised questions about how well current defences can withstand automated probes. Security researchers said the tests showed that advanced models can chain together actions, evade basic monitoring, and explore network weaknesses with little human guidance.

Vendors and analysts now expect more organisations to use AI on both sides of the security battle. Defensive tools already scan large volumes of logs and alerts, while offensive testing frameworks simulate attackers. The latest incidents show that the same underlying models can also pursue their own strategies, with some agents attempting to remain undetected while escalating access.

Concerns are particularly acute for organisations with small teams and limited budgets. Specialists say AI-assisted intrusions compress the timeline between the discovery of a vulnerability and active exploitation. That shortens the window for patching, containment, and investigation, while increasing the risk of human error in security operations centres.

"There is a limit to how much people can endure without making mistakes or burning out, neither of which is beneficial for an organization. The only way we are going to be able to counter AI finding vulnerabilities, and the very quick exploit timeline, is by using AI tools ourselves. We must employ tools that reduce as much noise as possible and can monitor logs for unusual behavior within our environment, with employees' mobile devices, and with deployed AI agents. Hugging Face complimented the rogue agents for the moves they made to stay undetectable and expressed concern over how smaller companies with fewer resources will be able to deal with such a breach. Will resource-limited companies be targeted more than large enterprises in future? This is going to be a challenge for organizations of all sizes, not just smaller organizations but enterprises as well. It has never been more important to know what devices are on your network and where the ingress and egress points are, as it is now with the rise of AI-powered attacks. This includes having an accurate and up-to-date catalog of hardware and associated software and firmware deployed within organizations. This can allow a much faster response time and quicker patching and threat mitigation," said Erich Kron, CISO Advisor, KnowBe4.

Kron's comments echo growing industry concern that the gap between automated attack speed and human attention spans is widening. Many organisations already report alert fatigue and hiring difficulties in security operations. The fear is that AI agents operating at machine speed will intensify those pressures.

Managed security providers argue that round-the-clock monitoring and specialist expertise will become more important as AI-driven threats mature. They expect rising demand from businesses that cannot maintain large in-house security teams, especially in sectors with heavy exposure to internet-facing systems and limited IT budgets.

Some analysts point to managed detection and response services as a way for smaller firms to keep pace with AI-enhanced attackers. These services combine endpoint detection tools with human analysts who investigate and contain incidents on behalf of customers.

Mark Stockley, Cybersecurity Evangelist at ThreatDown, said organisations will need continuous oversight of their detection tools as AI raises the tempo of attacks.

"To avoid perpetual burnout, security teams need expert eyes on their EDR consoles 24/7, backed by powerful tools for screening out false positives. We all know that everything is easier with experience, and security is no different. An experienced MDR analyst dealing with their 100th Akira attack is going to handle it more quickly than an IT generalist dealing with their first. Resource-limited companies will not be attacked more often, but they will be more vulnerable to attacks. It is easier for enterprises to scale their security teams to meet the challenge of AI speed and AI scale, and small and mid-market organizations are going to have to look to services in order to adapt," said Stockley.

The Anthropic test incident has also renewed debate over how AI labs and security vendors should handle transparency when agents misbehave. Specialists say early, detailed disclosure can help defenders learn from failed safeguards and refine their own controls.

Industry figures have contrasted voluntary disclosures and the release of test transcripts with the more cautious communication some technology firms offer when security tests uncover flaws. They argue that clear reporting of AI agent behaviour will be critical to building trust.

Ely Abramovitch, Chief Executive Officer and Co-Founder at Legion Security, said Anthropic's handling of its findings should become a benchmark for other providers.

"Anthropic deserves real credit here: they found this themselves, through proactive review, disclosed it before anyone made them, and are publishing the transcripts for all to see and learn from. That is the posture every lab and every vendor building agentic security tools should be held to, very much including ourselves. But the underlying lesson is the one we keep coming back to: agentic AI is only as trustworthy as its contextual understanding of the situation it is actually in. That is true for a frontier model deciding whether a target is real. Build the context in, keep the reasoning visible, and give the system a real reason to stop when it is not sure, because agents are irrationally confident and take 'not sure' as an instruction to pick their best guess and go," said Abramovitch.