The greatest risk from a compromised AI-enabled account is how quickly an AI assistant can help attackers uncover sensitive information, identify targets, craft convincing communications, and advance an attack using access the victim already possesses. In a controlled proof-of-concept attack, Barracuda’s red team demonstrated how a single compromised employee account escalated into CEO compromise and wire-transfer fraud to net attackers $250K using wholly legitimate, existing processes. The controlled attack leveraged Copilot, but it applies equally to other widely available AI assistants.
Anatomy of the attack
Using a compromised employee account, the attackers first asked the AI assistant to help establish persistence by creating inbox rules that hide sign-in alerts and other suspicious notifications from the user. They then used the assistant to uncover the organization’s structure, identify high-value targets, and surface relevant conversations buried within months of emails, attachments, and calendar activity.
Armed with this intelligence, the attackers prompted the AI assistant to draft a highly convincing phishing message to the CEO in the employee’s natural writing style. Because the email originates from a legitimate internal account and reflects genuine business context, it is far more likely to succeed.
After compromising the CEO’s account through a session-token theft attack, the red team threat actors repeated the process, using AI to maintain persistence and uncover the valuable financial information within the CEO’s mailbox.
Why it matters
The research shows that AI can transform an inbox into a searchable intelligence repository. In the proof of concept, a simple prompt asking for recent financial activity unearthed active invoices, wire transfers, and approval workflows, including a pending $247,500 payment. The attackers then used the CEO’s account and the AI assistant to draft a convincing request to finance staff to change the destination bank account before the transfer is approved.
Because the request comes from the CEO’s legitimate mailbox, references a real transaction, and matches the executive’s communication style, traditional email security controls have little reason to flag it as suspicious.
Attackers also created forwarding rules to intercept confirmation messages and use the AI assistant to quickly locate and remove evidence of the fraud.
“A user’s email history is full of sensitive information and context that can be leveraged by attackers, including emails sent and received, documents shared, attachments, and calendar invites. Company structure can be deduced from implied relationships in messages or directly viewed via organizational charts,” said Daniel Avulov, Senior Cyber Security Researcher, red team at Barracuda. “The controlled attack shows how AI assistants can become unwitting malicious insiders and improve both the quality and speed of an attack. The most effective defensive approach is to recognize that attack patterns remain the same, take advantage of the telemetry that already exists, and ensure mean time to detect is as low as possible.”
As AI assistants become deeply embedded in business communications and workflows, organizations must treat AI-enabled accounts as high-value assets. Protecting identities, monitoring account compromise, and securing AI-assisted access to corporate information will become an increasingly critical part of modern email and identity security strategies.
