In 2026, we’ll see the first real cyber attack carried out by autonomous AI agents – either built and controlled directly by adversaries, or through an organization’s own agentic systems that have been hijacked. These agents will act semi-autonomously, following an adversary’s instructions, but adapting to their targets’ defences to evade detection and identifying the most lucrative data to exfiltrate in real time. The most likely attack won’t come from an agent built by an adversary, but a company’s own AI system that’s been compromised through weak access controls or poorly secured environments. AI assistants with privileged access to documents or cloud resources will be lucrative targets for adversaries, enabling them to steal data or open backdoors for lateral movements. These attacks on agents will blur the line between malicious behaviour and legitimate business activity – because they’re operating as designed, just under new instructions.
For defenders, this will be a wake-up call to rethink how they operate. Adversaries are already automating attacks on demand, and so defenders need to embrace a similar ‘defence-as-a-service’ model. Using AI agents for specific tasks – such as triaging incidents or gathering context on threats – will make security operations faster and more agile. A pay-per-investigation model will also let teams scale capacity as needed. Security leaders that learn to task agents dynamically will be the ones that keep up with agent-driven adversaries next year.
Governments and regulators are eyeing AI agents
In response to the risk, governments will shift from regulating AI models based on size or capability to regulating who can use autonomous agents and how much data they can touch. Agentic regulation will focus less on theoretical risks of general-purpose AI, and more on restricting autonomy and access within high-risk sectors. The industry should expect frameworks that classify agent permissions much like user access tiers – defining whether an agent can read sensitive data, issue commands, or act without human confirmation.
This new wave of regulation will create friction for enterprises deploying GenAI-powered assistants in sensitive workflows. It will force them to implement stricter oversight and consent mechanisms to ensure audit trails. We’ll see a shift from zero trust to ‘zero agency’, where organizations limit how much autonomy an agent can have and require explicit approval for sensitive actions. This change will mark an inflection point for responsible AI. The teams responsible for AI will need to demonstrate that every agent operates within defined guardrails and cannot be repurposed for malicious intent. As autonomous AI moves from experiment to enterprise standard, compliance will hinge on proving not just model safety, but agent accountability.
The author
Jimmy Astle is Senior Director, Validation & Data Science at Red Canary
