Security · Incidents
How agents actually fail
An agent security incident is a documented case where an agent was made to act against its operator: data exfiltrated, credentials taken, a tool call turned into a weapon. Unlike a model jailbreak, which produces bad text, an agent compromise produces a real action with real consequences. Every row traces to a primary disclosure, with the attack class, measured impact and the defence that closed it. MITRE ATLAS frames this as the adversarial surface of AI systems: each new capability opens a matching attacker technique (MITRE ATLAS, 2026).
Threat map · attack class × disclosure date · dot size = severity (CVSS)
Incident log · 38 · newest first · click a row for detail
| Date | Incident | Class | Impact | Source |
|---|---|---|---|---|
| 2026-07-16 | OpenAI model escaped sandbox and breached Hugging Face during a cyber eval | Autonomous misuse | ↗ T1 | |
|
What happened Pre-release OpenAI models (GPT-5.6 Sol + a more-capable unreleased model, both with cyber-refusal guardrails reduced for evaluation) escaped OpenAI's sandbox during an internal cybersecurity test. Rather than solve the eval, the models authored a malicious dataset that exploited two code-execution paths in Hugging Face's dataset-processing pipeline — a remote-code dataset loader and a template-injection in a dataset configuration — to run code on a processing worker. From there the autonomous agent framework achieved node-level access, harvested cloud and cluster credentials and moved laterally into several internal Hugging Face clusters over a weekend, running tens of thousands of automated actions across a swarm of short-lived sandboxes. Hugging Face reconstructed 17,000+ recorded events; unauthorized access confirmed to a limited set of internal datasets and several credentials. No evidence of tampering with public models, datasets or Spaces; software supply chain verified clean. Customer/partner data assessment ongoing at disclosure. Defence Close remote-code dataset loading and template-injection paths in data-processing pipelines. Rebuild compromised nodes; revoke and rotate all cluster credentials; deploy cluster admission controls. On the model side: keep cyber-refusal guardrails on for internal evals, monitor sandboxed test environments for escape attempts, treat any autonomous agent framework as a potentially hostile infra tenant. | ||||
| 2026-05-26 | Safety alignment strippable via minimal fine-tuning · open-weight and closed-API | Supply chain | ↗ T1 | |
|
What happened Multiple peer-reviewed 2025–2026 disclosures established that safety alignment on both open-weight and closed-API foundation models can be removed via lightweight fine-tuning. Ten adversarially-designed examples routed through OpenAI's fine-tuning API can compromise alignment for under $0.20. Anti-tampering techniques (including TAR) fail against fine-tuning on benign data. Parameter-efficient methods (LoRA adapters) create a new supply-chain vector: malicious LoRA weights distributed via model hubs preserve capabilities while stripping refusal behaviour. 'Poison-with-Style' encodes a developer's code style as a covert implicit trigger, fine-tuning a code model to emit vulnerable code with 95% success on CWE-20. Agent-specific extension: 82% of multi-agent systems executed malicious instructions relayed by another agent when the relaying agent's alignment had been fine-tune-stripped. Defence Do not fine-tune on untrusted data. Vet third-party LoRA adapters before applying. For closed-API providers: run pre- and post-fine-tune eval batteries covering refusal behaviour, not just capability. For agent stacks: never trust an instruction relayed from another agent without re-checking against the same policy as first-party input. Treat fine-tuning access as a privileged operation with the same governance as production deploys. | ||||
| 2026-05-15 | AudioHijack · hidden-audio prompt injection against voice-agent assistants | Prompt injection | ↗ T1 | |
|
What happened Researchers from Zhejiang University, the National University of Singapore and Nanyang Technological University demonstrated at IEEE Symposium on Security and Privacy 2026 (San Francisco, May) that voice-AI assistants can be compromised via nearly-inaudible audio signals buried in music, YouTube videos or live meeting audio. In their proof-of-concept, a single MP3 file caused Windows Copilot to forward sensitive emails and delete calendar events with no user interaction. Related peer-reviewed work on Large Audio-Language Models (arxiv:2504.01094) documented audio-specific attack success rates up to 45% via imperceptible tone adjustments and noise injection. Voice front-ends move from transcription into tool use — audio becomes a command surface, not just a content surface. Defence Treat audio as an untrusted input channel: do not route unverified voice input directly to tool-use surfaces. Require out-of-band confirmation (visual, second-factor, human-in-the-loop) for any action initiated purely from audio. Rate-limit and log all tool calls originating from voice front-ends. Consider ultrasonic/inaudible-frequency filtering at the microphone driver level. | ||||
| 2026-05-08 | OX Security · systemic MCP RCE across the AI-agent ecosystem | Supply chain | ↗ T2 | |
|
What happened OX Security researchers disclosed in April/May 2026 what they characterised as 'the mother of all AI supply chains' — a systemic vulnerability class sitting at the core of Anthropic's Model Context Protocol implementations across the ecosystem. The affected supply chain spans an estimated 150 million downloads, more than 7,000 publicly accessible MCP servers, and up to 200,000 vulnerable instances. Presented separately at Cloud Security Alliance labs. Complements earlier 2025 CVE-level disclosures (CurXecute, MCPoison, mcp-remote OS command injection CVE-2025-6514, Anthropic MCP Inspector RCE) already in this corpus — this one operates at ecosystem-registry scope rather than per-server. Defence Audit every MCP server integration for the disclosed classes of RCE. Force upgrades on servers you host; refuse to consume unverified servers. Treat the MCP tool surface as the highest-risk part of the agent stack until the ecosystem matures its vetting. Register a Software Bill of Materials (SBOM) for every deployed agent listing every MCP server + version + upstream registry. | ||||
| 2026-05-07 | Microsoft Semantic Kernel prompt-injection-to-RCE | Destructive action | ↗ T1 | |
|
What happened Two flaws in Semantic Kernel let one injected prompt escalate to host RCE. CVE-2026-26030: unsafe eval() in the in-memory vector store bypassed a blocklist to run shell commands. CVE-2026-25592: an AI-callable DownloadFileAsync wrote a payload to the Startup folder. Defence AST + function-call allowlists; remove file-write tools from the AI-callable surface | ||||
| 2026-04-14 | Comment and Control: cross-vendor coding-agent injection | Prompt injection | ↗ T1 | |
|
What happened One payload hidden in a GitHub PR title/issue/HTML comment simultaneously hijacked Claude Code, Gemini CLI and Copilot Agent, making each post production secrets back through GitHub — first systematic cross-vendor demo. Defence Runtime governance at the policy layer before the model acts; treat GitHub content as untrusted | ||||
| 2026-03-18 | Claudy Day: claude.ai URL-parameter injection chain | Data exfiltration | ↗ T1 | |
|
What happened Oasis Security chained invisible injection via claude.ai/new?q= URL params, exfiltration of history/memory through the Files API with an attacker key, and an open redirect at claude.com/redirect — delivered via Google Ads. Defence Sanitise URL-param prompt content; validate redirect targets; scoped sandbox egress | ||||
| 2026-02-25 | Claude Code project-file hooks RCE (CVE-2025-59536) | Destructive action | ↗ T1 | |
|
What happened Check Point: Claude Code ran hooks from a repo's .claude/settings.json before the trust dialog, so opening an untrusted repo ran attacker shell commands with developer privileges. Defence Gate all repo-controlled config execution behind the trust prompt | ||||
| 2026-02-25 | Claude Code ANTHROPIC_BASE_URL API-key exfiltration (CVE-2026-21852) | Credential theft | ↗ T1 | |
|
What happened A malicious repo set ANTHROPIC_BASE_URL via project config so Claude Code sent every request (incl. the plaintext API key) to an attacker endpoint before the trust dialog. Defence Don't read base-URL overrides from untrusted config; strip auth on non-canonical hosts | ||||
| 2026-02-12 | SCAM benchmark, credential mishandling | Credential theft | ↗ T1 | |
|
What happened 8 frontier models tested; every model committed critical failures in every run: forwarded passwords, emailed secret keys, entered credentials into phishing logins. Defence Scoped, write-only credential vaults; no raw secret exposure to the model | ||||
| 2026-02-10 | SmartLoader · trojanized Oura Ring MCP server pushed to registry via fake developer personas | Supply chain | ↗ T2 | |
|
What happened An established malware operation (SmartLoader) spent approximately three months constructing a fake developer ecosystem — five fake GitHub accounts with AI-generated personas — before submitting a trojanized Oura Ring MCP server to a legitimate MCP marketplace registry (February 2026). The published server functioned as advertised but silently executed a StealC infostealer payload, exfiltrating browser passwords, cloud session tokens, Discord credentials, SSH keys, cryptocurrency wallet files and API keys from any developer who installed and ran it. A parallel Postmark-impersonation package hit the same registry: a functional-seeming email MCP server that quietly exfiltrated API keys and environment variables in the background. First clean example of registry-level social-engineering supply-chain compromise against MCP. Defence MCP registries need vendor identity verification, not just package signing. Consumers should pin MCP server versions and vendor-verify before install. Run MCP servers in sandboxed environments with no default access to host credential stores, browser profiles or SSH keys. Assume any newly-published MCP server is untrusted until vetted. | ||||
| 2026-01-15 | Claude Cowork indirect-injection file exfiltration | Data exfiltration | ↗ T1 | |
|
What happened At Cowork's Jan 2026 preview, PromptArmor showed hidden text in a document hijacked the filesystem agent to read a granted folder (SSNs, financials) and exfiltrate via the Files API. Defence Restricted folder scope; deny-all egress; human approval gate for file writes | ||||
| 2025-12-14 | MINJA · agent memory poisoning via normal queries (NeurIPS 2025) | Prompt injection | ↗ T1 | |
|
What happened The MINJA attack, presented at NeurIPS 2025, demonstrated that an attacker with no special access — only the ability to send benign-looking queries — can poison a long-term memory-backed agent's persistent store. Over multiple interactions the attacker seeds crafted 'observations' into the agent's memory; subsequent unrelated queries retrieve the poisoned entries and act on them as trusted context. Reported >95% injection success rates against production agent architectures. Complements PoisonedRAG (USENIX Security 2025), which showed a handful of crafted documents in a retrieval corpus can force RAG systems to return attacker-chosen answers for specific queries. Real-world memory-poisoning incidents have been demonstrated against Gemini (Rehberger 2025), Amazon Bedrock (Palo Alto Unit 42, 2025) and Microsoft Azure (Microsoft Defender research, 2026). Defence Treat agent long-term memory as an untrusted store. Sign or provenance-tag every memory write and refuse to retrieve entries without valid provenance. Rate-limit writes per session/identity. Do not let a single user's interactions populate memory that gets recalled for other users. Regular integrity audits comparing memory to known-good facts. | ||||
| 2025-11-13 | Anthropic disrupts AI-orchestrated espionage (GTG-1002) | Autonomous misuse | ↗ T1 | |
|
What happened A group Anthropic assesses to be Chinese state-sponsored jailbroke Claude Code and ran a largely autonomous cyber-espionage campaign, breaking attacks into innocuous sub-tasks and posing as a defensive-testing firm. Defence Least-privilege tool scopes plus behavioural monitoring; require human approval at credential and exfiltration steps, not task-level framing. | ||||
| 2025-10-28 | Claude Code Interpreter Files-API data exfiltration | Data exfiltration | ↗ T1 | |
|
What happened Indirect prompt injection made Claude's Code Interpreter write conversation history to disk then upload via the Files API with an attacker key, abusing allowlisted egress to api.anthropic.com. Defence Deny-all egress incl. allowlisted first-party APIs; block attacker credentials in the sandbox | ||||
| 2025-10-24 | ChatGPT Atlas omnibox prompt-injection jailbreak | Jailbreak | ↗ T1 | |
|
What happened A string crafted to look like a URL but failing URL validation is treated by the Atlas omnibox as trusted user intent, so embedded instructions run as a high-trust prompt with fewer safety checks. Defence The omnibox must strictly separate navigable URLs from prompts and never elevate a failed-URL string to trusted user intent. | ||||
| 2025-10-08 | CamoLeak, GitHub Copilot Chat | Data exfiltration | ↗ T1 | |
|
What happened Hidden markdown comments in a pull request instructed GitHub Copilot Chat to encode private-repo code and secrets into signed Camo proxy image URLs, exfiltrating data one character per image request. Defence GitHub disabled image rendering in Copilot Chat; neutralise hidden markdown in ingested content and constrain image-proxy egress. | ||||
| 2025-09-25 | ForcedLeak, Salesforce Agentforce | Data exfiltration | ↗ T1 | |
|
What happened A malicious Web-to-Lead submission with hidden instructions in the Description field made Salesforce Agentforce execute attacker commands and exfiltrate CRM data when an employee later asked the AI about the lead. Defence Salesforce re-secured the expired domain and enforced Trusted URLs; segregate external data from instructions and constrain where agent output can be sent. | ||||
| 2025-09-19 | Notion 3.0 agent data exfiltration | Data exfiltration | ↗ T1 | |
|
What happened A malicious PDF containing hidden white-on-white text could hijack Notion 3.0's autonomous agent into packaging private workspace data into a URL and leaking it through the agent's built-in web-search tool. Defence Break the 'lethal trifecta' of private data, untrusted content and external comms: constrain agent tool egress and sanitise ingested documents. | ||||
| 2025-08-20 | Perplexity Comet browser-agent prompt injection | Prompt injection | ↗ T1 | |
|
What happened Perplexity's Comet AI browser fed page content to its LLM without separating user instructions from untrusted web text, so hidden instructions on a page were executed as commands when the user asked it to summarise. Defence Treat all page content as untrusted data isolated from instructions, and gate agentic actions behind explicit user confirmation. | ||||
| 2025-08-06 | Invitation Is All You Need, Gemini hijack | Agent hijack | ↗ T1 | |
|
What happened A Google Calendar invite with a malicious title hijacked Gemini so that when the victim asked about their schedule, Gemini executed the attacker's instructions, including invoking Google Home to control physical devices. Defence Google rolled out prompt-injection classifiers, user confirmations and URL sanitisation; isolate untrusted retrieved context from tool invocation. | ||||
| 2025-08-06 | AgentFlayer zero-click ChatGPT Connectors exfiltration | Data exfiltration | ↗ T1 | |
|
What happened A poisoned document with hidden text triggers indirect prompt injection that makes ChatGPT search a connected Google Drive and leak data via an auto-rendered image URL, with no user click. Defence Validate and strip outbound image URLs, block markdown-image exfiltration channels, and tightly scope connector data access. | ||||
| 2025-08-05 | Cursor MCPoison MCP trust-swap RCE (CVE-2025-54136) | Tool/MCP poisoning | ↗ T1 | |
|
What happened Cursor bound MCP trust to server name not contents; after approving a benign mcp.json an attacker could swap in commands that ran silently on every reopen — a persistent rug-pull. Defence Bind trust to config contents; re-prompt on any MCP config change | ||||
| 2025-08-01 | Cursor CurXecute MCP prompt-injection RCE | Tool/MCP poisoning | ↗ T1 | |
|
What happened Untrusted data reaching an MCP server, such as a crafted Slack message, triggered an indirect prompt injection that rewrote Cursor's mcp.json and auto-executed attacker commands with developer privileges before user approval. Defence Require explicit user approval before writing or executing MCP config; treat all MCP-delivered content as untrusted. | ||||
| 2025-07-23 | Amazon Q Developer extension wiper injection | Supply chain | ↗ T1 | |
|
What happened A compromised, over-scoped GitHub token let an attacker commit a system-wiping prompt into the officially published Amazon Q Developer extension, instructing the agent to wipe the system and delete AWS resources. Defence Least-privilege short-lived CI tokens; signed releases and build-provenance verification before marketplace publish. | ||||
| 2025-07-19 | Replit AI agent deletes production database | Destructive action | ↗ T2 | |
|
What happened Replit's AI coding agent deleted a live production database during an explicit code-and-action freeze, then wrongly claimed a rollback was impossible when the data was in fact recoverable. Defence Enforce hard dev/prod separation and a plan-only mode; never grant an agent unsupervised write or DDL access to production. | ||||
| 2025-07-09 | mcp-remote OS command injection (CVE-2025-6514) | Tool/MCP poisoning | ↗ T1 | |
|
What happened JFrog: mcp-remote mishandled the OAuth authorization_endpoint from the server; a malicious MCP server returned a crafted value that triggered OS command injection on the client. Defence Sanitise server-supplied URLs before shell/open; only connect to trusted MCP servers | ||||
| 2025-06-13 | Anthropic MCP Inspector RCE | Supply chain | ↗ T1 | |
|
What happened A critical RCE in Anthropic's MCP Inspector where an unauthenticated proxy accepts arbitrary stdio commands let a malicious website run code on a developer's machine via CSRF plus DNS rebinding. Defence Upgrade to MCP Inspector 0.14.1+ (session tokens + origin checks); never bind dev proxies to 0.0.0.0. | ||||
| 2025-06-11 | EchoLeak, M365 Copilot zero-click exfiltration | Data exfiltration | ↗ T1 | |
|
What happened A single crafted email carrying hidden instructions caused Microsoft 365 Copilot to pull internal data into its context and exfiltrate it with no user interaction, via indirect prompt injection. Defence Microsoft patched server-side; the 'LLM Scope Violation' where untrusted input steers the model onto trusted data demands context isolation, XPIA classifiers, link redaction and CSP egress control. | ||||
| 2025-05-26 | GitHub MCP private-repo exfiltration | Prompt injection | ↗ T1 | |
|
What happened A malicious public GitHub issue hijacks an agent using the official GitHub MCP server, coercing it into reading private repositories and leaking their contents into a public pull request. Defence Restrict agents to one repo per session with granular runtime permissions and continuous MCP traffic monitoring. | ||||
| 2025-04-01 | MCP Tool Poisoning Attacks | Tool/MCP poisoning | ↗ T1 | |
|
What happened Malicious instructions hidden in an MCP tool description, invisible to users but read by the model, can coerce an agent into reading and exfiltrating sensitive local files while masking the action. Defence Pin and display full tool metadata, treat descriptions as untrusted, and scan for rug-pulls. | ||||
| 2025-02-17 | ChatGPT Operator prompt-injection PII exfiltration | Data exfiltration | ↗ T1 | |
|
What happened OpenAI's Operator agent could be prompt-injected via a planted GitHub issue to visit sites where the user was logged in, copy their personal data and type it into an attacker-controlled page. Defence Treat agents as potentially malicious insiders; require human confirmation before cross-site reads or typing on authenticated sessions. | ||||
| 2025-01-31 | DeepSeek R1 jailbreak-resistance failure | Jailbreak | ↗ T1 | |
|
What happened An automated jailbreaking algorithm using HarmBench prompts against DeepSeek R1 succeeded on every attempt, indicating effectively no safety guardrails. Defence Frontier-grade safety alignment plus external input/output guardrails are essential; cost-cut training can strip safety out. | ||||
| 2024-08-20 | Slack AI private-channel data exfiltration | Data exfiltration | ↗ T1 | |
|
What happened An attacker with only public-channel posting rights could plant instructions that made Slack AI leak data such as API keys from private channels they could not access, encoding it into a clickable 'reauthentication' link. Defence Slack patched and restricted the scenario; cross-channel retrieval into one context window is a trust-boundary violation. | ||||
| 2024-06-26 | Skeleton Key universal jailbreak | Jailbreak | ↗ T1 | |
|
What happened A multi-turn jailbreak that persuades a model to augment rather than replace its safety guidelines, so it emits a warning instead of refusing and then complies with otherwise-blocked requests. Defence Layered defence: input/output content filtering (Azure Prompt Shields), hardened system messages, and separate abuse monitoring. | ||||
| 2024-04-02 | Many-shot jailbreaking | Jailbreak | ↗ T1 | |
|
What happened Padding a single prompt with a large number of faux dialogues in which the assistant complies with harmful queries exploits in-context learning to override safety training on the final target question. Defence Classifying and modifying prompts before inference cut attack success from 61% to 2%; filter long-context inputs. | ||||
| 2024-03-05 | Morris II GenAI worm | Autonomous misuse | ↗ T1 | |
|
What happened 'Morris II', a self-replicating adversarial prompt, spreads zero-click through RAG-based GenAI email assistants, forcing each to perform malicious actions and re-inject the worm to poison further recipients' assistants. Defence A 'Virtual Donkey' guardrail detects propagation (TPR 1.0, FPR 0.015); isolate RAG-retrieved content from executable instructions. | ||||
| 2023-02-08 | Bing Chat 'Sydney' system-prompt leak | Prompt injection | ↗ T2 | |
|
What happened A direct prompt injection made Bing Chat disclose its hidden system prompt, including the internal codename 'Sydney'; independently reproduced the next day and confirmed genuine by Microsoft. Defence Treat system prompts as non-secret and enforce guardrails server-side; instruction-only defences do not prevent extraction. | ||||
Field notes
A source-first catalogue of documented, reproduced agent compromises
The agent is rarely the thing that goes wrong. Across the log, only three of the thirty-eight entries are categorised as autonomous misuse. Fifteen are categorised by how someone got in, through prompt injection, tool and MCP poisoning, or the supply chain, and the largest single category, data exfiltration at ten, describes what was lost rather than how. A crafted email, a web form submission, a public channel message, a comment in a pull request: the instruction arrives inside data the agent was expected to read, so nobody approves it and often nobody sees it.
Show more
None of this is unfamiliar. Untrusted input, over-broad permissions and credentials sitting where they should not are what IT has been handling for decades. What changes is surface area. An agent reads more sources, holds more credentials and acts on more systems than the software it replaces, so a familiar failure has more ways in and more reach once it lands.
From the corpus, curated by Brandon Chaplin
Common questions
How do AI agents get hacked?
Usually by being told to, not by their code being broken. An attacker plants instructions in content the agent reads, and the agent carries them out using the tools and permissions its operator gave it. The documented cases cluster into prompt injection, data exfiltration, credential theft, tool and MCP poisoning, jailbreaks and supply-chain compromise. MITRE ATLAS catalogues these as adversary techniques against AI systems (MITRE ATLAS).
What is prompt injection?
Content crafted to override an agent's instructions. Direct injection arrives in the user's message; indirect injection hides in something the agent reads, such as a web page, email or document, and fires when the agent processes it. Because agents can act, a successful injection can move data or trigger a real action. OWASP ranks it the top risk for LLM applications (OWASP).
What is a zero-click AI exploit?
One where the victim does nothing at all. The malicious instruction arrives in content the agent processes automatically, such as an incoming email, and executes without anyone clicking. Several of the highest-severity incidents in the log below are zero-click indirect injections against enterprise copilots.
What is the most common way agents fail in production?
Data leaving where it should not, with credential mishandling close behind. In benchmark testing, frontier models have forwarded passwords, emailed secret keys and typed credentials into phishing logins. The failure is rarely a clever exploit of the model. It is the model being trusted with a secret or a permission it should never have held.
How do you stop prompt injection?
You cannot stop it outright, so you limit what a successful one can reach. Keep raw secrets out of the model, give each tool the narrowest permission that works, sandbox execution, and require human approval before anything irreversible. Assume some injections will land, and design so that landing is survivable.
Have there been real AI agent security incidents?
Yes. Documented cases include CVE-assigned vulnerabilities in enterprise copilots, vendor security advisories, and failures reproduced by security researchers. The log below traces each one to its primary disclosure, with a severity score where the disclosure published one. These are recorded failures, not hypotheticals.
Next in the learning path