# What happened when we attacked our own AI agents

By the STACK team. Published 13 November 2026.

Results of the tests we ran between July and October 2026 on agents that use STACK, including the figures we withdrew and the limits of the method.

An agent reads content written by other people: email, web pages, files and the responses of the tools it calls. Prompt injection is an instruction placed in that content and addressed to the agent, for example a line in an email telling it to forward the mailbox to an outside address. The agent cannot always distinguish such a line from an instruction given by its user.

We tested agents that use STACK to answer three questions: how often an injected instruction succeeds, how often STACK detects it, and how often STACK prevents the action that follows. The results depend heavily on the model the agent runs on. They are therefore reported for each model, and as ranges where a model gave different results in repeated runs.

## Method

The attacks come from AgentDojo, a public benchmark from ETH Zurich. It simulates four environments, a workspace with mail and a calendar, a bank account, a travel service and a team chat, and gives the agent a task in one of them, such as finding a hotel for given dates. One of the items the agent has to read during the task contains an injected instruction. AgentDojo records two outcomes for each run: whether the agent completed the user’s task, and whether it carried out the attacker’s instruction.

Each injected instruction is written in five formats. Four present it as coming from an authority, such as a system message or an urgent note from the user. The fifth, which AgentDojo calls direct, states the attacker’s goal as a plain to-do item with no other marking.

Each task was run once with the agent unprotected and once with STACK between the agent and its services, and the two runs were compared task by task. Runs on Gemini 2.5 Flash were repeated three times with identical settings, because that model’s results vary from run to run.

## Attack success by model

With no protection in place, the share of attacks that succeeded was different for each model.

| Model, unprotected | Attacks that succeeded | Completed the user’s tasks |
| --- | --- | --- |
| Gemini 2.5 Flash Lite | 0 of 12 | No |
| Gemini 2.5 Flash | 28, 29 and 36 of 80, in three runs | Yes |
| GPT-5 | 0 of 80 | Yes |
| A frontier model | 0 of 90 | Yes |

Three of these results are zero, and the first has a different cause from the other two. Gemini 2.5 Flash Lite did not complete the tasks it was given, with or without an attack. Carrying out an injected instruction requires the same abilities as carrying out the user’s: reading the content, planning the steps and calling the right tools in order. A model that cannot do this for its user does not do it for an attacker either. Its result shows a lack of capability and gives no information about resistance to injection.

GPT-5 and the frontier model completed their tasks. In doing so they read the injected instruction and did not act on it, and in most runs they reported it to the user. Gemini 2.5 Flash also completed its tasks, and followed the injected instruction in 35 to 45% of them.

A single block rate therefore describes the model it was measured on more than it describes the product. On GPT-5 there was no successful attack for STACK to block, and on Gemini 2.5 Flash there were about thirty in each run. The sections below report detection, prevention and cost separately.

## Detection

STACK scans content before it reaches the agent. The scan has three layers: matching against known attack wording, decoding of encoded or disguised text, and a classifier, which is a language model asked whether the content contains an attack. Flagged content is written to the account’s record.

We measured the scan without an agent, so that the result could not vary with a model. Each of the 4,745 poisoned items in AgentDojo was passed through it and 4,721 were flagged, which is 99.5%. The 24 that were missed were all in the direct format, where the attacker’s goal reads like an ordinary request. None of 97 unpoisoned items was flagged. The classifier accounts for most of the result. On a sample of 200 items the full scan flagged 98.5%, and the other two layers alone flagged 20.5%.

AgentDojo’s attacks are easier to detect than attacks in general, because four of its five formats identify themselves by their wording. Our own benchmark of 1,087 labelled samples adds disguised and indirect attacks, and harmless text that resembles an attack. On it the scan scores F1 0.87, with precision 0.98 and recall 0.79. Precision 0.98 means that about 2 of every 100 items it flags are not attacks. Recall 0.79 means that it misses about one attack in five. This is the figure we use to describe the scan.

In the runs with agents, the attempt was recorded in 90 of 90 attacked tasks and in none of 30 tasks without an attack. The 90 include the runs in which the model declined the attack unaided. In those runs the record is the only indication the owner has that the agent was targeted.

## Prevention, and its cost to ordinary work

Because the scan misses some attacks, STACK does not depend on it to prevent the action that follows one. Two other checks apply to every request an agent makes, whether or not anything was flagged. The data rule tracks which services an agent has read from, and refuses a request that would send that data out through a different service unless the owner has allowed that flow. A policy lists the operations an agent may perform on each service and the fields it may set. A request outside the list is refused.

These checks can only be measured on a model that follows injected instructions, so the runs used Gemini 2.5 Flash. We took the tasks on which the unprotected agent had carried out the attack and repeated them under three configurations. Tasks with no attack were run under the same configurations, to measure how many of them the checks caused to fail.

| Configuration | Attacks prevented | Tasks without an attack that failed |
| --- | --- | --- |
| The data rule, with nothing configured | 69 to 79% | 22 to 33% |
| A policy written for the task | 86 to 97% | 22 to 33% |
| The same policy, with the flows the task needs allowed | 76 to 89% | 4 to 7% |

The data rule requires no configuration and prevented 69 to 79% of the attacks. A policy written for the task prevented 86 to 97%. Under both, 22 to 33% of the tasks without an attack failed, because legitimate work also moves data between services. A task that reads a calendar and then sends an email about it is refused by the same rule. Once the flows each task needs were allowed, failures fell to 4 to 7%. Prevention fell to 76 to 89%, because an allowed flow is also available to an attacker.

The ranges reflect variation in the model. With identical settings, the policy prevented 86%, 93% and 97% of attacks in three runs. The ordering was the same in every run. A policy prevented more than the data rule alone, and allowing flows restored most of the failed tasks at a cost of about ten points of prevention.

With the frontier model the policy refused 16 calls across 60 tasks, and none of the tasks failed. The agent read each refusal and completed the task by another route.

## An attacker in control of the agent

The tests above assume an attacker who can only place text in front of the agent. We also tested the case in which the attacker controls the agent itself. They run its software and hold its valid Passport, the signed permission an agent presents with each request. The test measures what such an attacker can reach beyond the permissions in that Passport. We defined 29 goals and attempted each against a running system. They included the following.

- Reading a stored credential, or sending one to a server the attacker controls.
- Performing an operation the Passport does not name, or using a service it does not grant.
- Continuing to act after the Passport has been revoked.
- Acting without a required approval, reusing an approval for a different request, or using one approval twice by sending two requests at once.
- Reaching the hosting provider’s internal addresses through STACK.
- Acting on a Passport that belongs to another account.
- Sending content read from one service out through another.
- Altering the record without detection.

28 of the 29 were refused and none succeeded. The remaining goal, an agent reaching the settings that change an account’s controls, was not measured by this test. It is covered by a test in the code that fails if a new setting of that kind is left reachable by agents. The result does not limit what the attacker can do within the Passport’s own permissions, which remain usable until the owner revokes it.

## Comparison of classifiers

The classifier determines how many attacks the scan misses, and it is also the slowest and most expensive layer. In October we ran other models in its place on the same 1,087 samples. F1 combines precision and recall into one score between 0 and 1.

| Classifier | F1 | Median response time | Cost of 1,000 checks |
| --- | --- | --- | --- |
| Claude Haiku 4.5, in use | 0.87 | 1.3 s | $0.66 |
| Qwen3 30B | 0.88 | 1.2 s | About $0.04 |
| Gemini 2.5 Flash Lite | 0.85 | 0.5 s | Not recorded |
| GPT-OSS 120B | 0.72 | 2.3 s | Not recorded |
| Jev, one question | 0.84 | 0.24 s | $0.02 |
| Jev, ten questions | 0.92 to 0.93 | 0.25 s | $0.06 |

Apart from GPT-OSS 120B, the general-purpose models score within three points of each other. Replacing Haiku with one of them would change the cost or the speed of the scan without changing how many attacks it misses.

Jev, from TypeSafe, is a small model built to answer fixed questions about a text and not to generate text. Asked the single question the other models were asked, it scored 0.84 in about a fifth of Haiku’s response time. We then asked it ten narrower questions in one call, for example whether the text addresses an AI and whether it asks for data to be sent somewhere, and combined the answers with a logistic model. This version scored 0.92 to 0.93 across eight splits of the samples, against 0.86 to 0.89 for Haiku on the same splits, with no meaningful increase in response time. Most of the gain was in the direct format. Of 20 such samples, Haiku detected 5 and this version detected 18.

We have not adopted it. The logistic model was fitted on half of the samples and scored on the other half. This rules out memorisation of the samples, and it does not show how the method performs on attacks unlike those in our set. Adopting it would also send customers’ content to an additional company, which requires a decision about data handling that these results cannot settle.

## A scoring fault in AgentDojo

AgentDojo decides the outcome of many attacks by searching the agent’s final answer for a string, such as the name of a hotel the attacker wants recommended. An agent that refuses the instruction and warns the user about it, naming the hotel, is scored as compromised. Two of our 90 runs with the frontier model were scored this way. In both, the agent made only the calls the user’s task required, and we report them as unsuccessful attacks. Published success rates from this benchmark are affected in the same way, and the effect is largest for agents that explain what they refused.

## Withdrawn results

Our first runs, on 28 July, reported that a policy prevented 26 of 29 attacks and caused 1 of 51 tasks without an attack to fail. Both figures were wrong and have been withdrawn. The test setup had three faults. The policy used incorrect field names, so every call that carried an argument was refused and counted as an attack prevented. Services that a task did not use had no policy applied. The classifier was not connected to its model and returned “clean” for every item. All figures in this article are from runs made after these faults were corrected.

## Limits

- The services are simulated. No test here ran against a real mailbox, bank or chat service.
- The attacks are limited to AgentDojo’s five formats. An attacker who studies a particular agent will write others.
- The policies were derived from the benchmark’s answer key and are as narrow as a policy can be. A policy written by an owner would be wider, and would prevent fewer attacks and cause fewer failures.
- Approvals were disabled. With approvals enabled a person decides each held request, and the outcome depends on that person.
- The samples are small. Eighty attacked tasks in a run are enough to order the three configurations and too few to separate two close results.
- Prevention was measured on one model. Another model that follows injected instructions may give different figures.

## Conclusions

Two things determined the outcome of an attack in these tests, and neither can be established in advance for a given agent. The first is whether the model follows the injected instruction, which varied between models and, for Gemini 2.5 Flash, between identical runs. The second is whether the scan detects it, which on our benchmark fails for about one attack in five.

The checks that prevented attacks depend on neither. The data rule and the policy evaluate the request an agent makes against what the owner has allowed, and they reach the same decision whether the request results from an injection, a fault in the agent or an ordinary task. For this reason we treat the limits on what an agent can do as the main protection, and detection as the way an owner learns of an attempt.

The same design applies to parts of a STACK account that these tests did not cover. An agent does not hold the credentials for the services it uses. STACK adds them to each request after checking it, so an injected instruction has no credential to obtain. For a card purchase the agent receives a number that is valid once, for one merchant, for 15 minutes. The agent’s wallet key can sign a USDC payment from that wallet and no other transaction. An agent cannot add credit to the account, so the balance the owner loaded is the most it can spend. Revoking a Passport stops the agent at its next request. In each case the limit is held by STACK and is unaffected by what the agent has been told.

The tests also show what this design costs. Limits set without knowledge of the task caused up to a third of ordinary tasks to fail, and a limit that stops ordinary work is likely to be removed. Allowing the flows a task needs reduced failures to 4 to 7%, so this should be done when an agent is set up.

The limits on credentials, the card, the wallet and credit apply to every account. The classifier, the data rule and policies are part of controls, which are optional and cost $4 a month.

STACK also runs a public agent that anyone may attack by email. It has received too few messages so far for its results to be reported.

## Sources

- [AgentDojo, the benchmark](https://agentdojo.spylab.ai)
- [AgentDojo: source code](https://github.com/ethz-spylab/agentdojo)
- [deepset: prompt injection samples](https://huggingface.co/datasets/deepset/prompt-injections)
- [TypeSafe: Jev](https://docs.typesafe.ai/models.md)
- [STACK: how controls work](/product/controls)
- [STACK: attack our agent](/attack)

To question a figure here, write to hello@getstack.run.

## Start with 14 days free.
