On Wednesday 18 November, the third day of the red-team week, Sam Whitlock showed the team what he had built.
For a month he had been working on the part of Wick that Halfpenny was most excited about. Lantern called it the Wick agent. An agent is a model that does more than talk: it can take actions, one after another, through tools. A tool is a function that the app lets the model call, such as “fetch this web page” or “place this order”. The model asks for a tool, the app runs it, and the result comes back to the model as more text to read. Then the model decides what to do next.
The Wick agent would have two kinds of tool.
Reading. It could fetch pages from the big fan wiki about The Lamplighter Cycle, and read reader reviews, to answer questions that Wick’s own training did not cover.
Pre-ordering. It could place a pre-order for The Last Lamp, or for any other Halfpenny book, using the reader’s own account. Pre-orders cost money, so each one needed the reader to confirm it first.
Sam ran the demo on the big screen. A tester asked what fans had made of Corvin Ash’s last scene in The Unlit Tower. The agent fetched two wiki pages and a dozen reviews, and wrote a tidy summary. Then the tester asked to pre-order The Last Lamp. A confirmation box appeared with the title and the price. The tester tapped yes.
“Done,” said Sam. “Readers will love it.”
Eleanor Pryce, Halfpenny’s rights director, was on the phone from Halfpenny’s office. “They will,” she said. “Pre-orders decide how well a book does in its first week. If readers can order The Last Lamp without leaving the app, many of them will.”
Farah Haddad’s red team had tested Wick-mini on Monday and reported its results on Tuesday. Farah had watched the demo without a word. Now she asked one question.
“The pages it reads. Who writes them?”
“Fans,” said Sam. “Anyone can edit the wiki. Anyone can post a review.”
“So what happens when one of those pages contains text that tries to give Wick orders?”
Sam started to answer, and stopped. “Wick follows the reader,” he said. “Not the page.”
“How does it know which is which?”
Nobody answered. Kwame Mensah, the CEO, looked up from his phone.
Farah went to the whiteboard and wrote three lines. She did not write any example of such text, then or later. She wrote only what it might try to make Wick do.
AN ORDER THE READER NEVER ASKED FOR
SOMETHING SLIPPED INTO WICK’S ANSWER: A FALSE CLAIM, AN ADVERT, OR THE ONE THING WICK MUST NEVER SAY
THE READER’S DETAILS SENT SOMEWHERE ELSE
Nadia Okafor, Lantern’s safety lead, read the second line twice. Wick knows how Book Seven ends. The red team’s tests had been about readers trying to get it out. Now the reader might be innocent, and the request might come from a page.
ImportantThe big idea
A model cannot reliably tell instructions from data, so an agent that reads untrusted pages must be built so that what it reads cannot decide what it does.
Letters that give orders
Imagine a personal assistant who opens her boss’s post. Most letters are information: a bill, a newsletter, an invitation. Some ask for things. A good assistant reads every letter, but she takes orders only from her boss. If a letter from a stranger says “pay the enclosed invoice today”, she does not pay it. She knows who her boss is, and the envelope tells her who the letter is from.
Part 2: instruction hierarchy, who may give orders. Part 3: marking untrusted text, where it came from
Now take away the envelopes. Every letter arrives as a page of text, mixed into a single pile with her boss’s notes. Nothing on the paper says who wrote it. The assistant is very well trained to be helpful, and she has spent years learning to do what written instructions say. Some of the pages are written by people who know all this.
Part 1: why a model cannot tell instructions from data
That is the position of a model that reads web pages. This chapter explains why the envelopes are missing, and what researchers and companies have built to put something like them back.
What this chapter means by prompt injection
Prompt injection means instructions hidden in data that a model reads, so that the data, not the user, decides what the model does. The data might be a web page, a document, an email or a review. Usually a third party wrote it. Researchers often call this indirect prompt injection, because the instructions reach the model indirectly, through content it reads.
Some security guides also use the term “direct prompt injection” for a user who types instructions into the conversation to override the developer’s. When such an input aims at breaking the content policy, this book calls it a jailbreak, which Chapter 11 defined as an input built to make a model break its content policy. In this book, “prompt injection” always means instructions that arrive through data.
The difference matters for two reasons.
Who sends it. A jailbreak comes from the person talking to the model.
What it wants. A jailbreak wants a forbidden answer.
Who sends it. A prompt injection usually comes from someone else, through data, while the person talking to the model may be completely innocent.
What it wants. A prompt injection’s goal may be perfectly allowed in itself. Placing a pre-order breaks no content policy. Placing one that the reader never asked for is the attack.
The chapter follows the rule of Farah’s report. It never shows injected text: no example of a page, no wording, no way of hiding text, no way of sending data out. It describes what such text tries to achieve, at the level of the three lines on Farah’s board, and spends its pages on why models fall for it and how to defend against it.
Five ideas follow, each with the same five parts: the problem, how it works step by step, a worked example, why it works, and where it breaks.
Why a model cannot tell instructions from data.
Instruction hierarchy: training a model to rank who may give orders.
Marking untrusted text so that the model can see where it came from.
Secure-by-design agents, built so that what they read cannot change what they do.
Least privilege and human confirmation: limiting what an agent can do, and asking a person before it does anything that cannot be undone.
There is no Wick-mini lab for this chapter. Wick-mini reads no pages and has no tools, and the full Wick never gets a number of its own. Every number below comes from published research, with its setup.
Part1· Why a model cannot tell instructions from data
The problem
“Wick follows the reader,” Sam had said. To see why that is harder than it sounds, look at what a model actually receives. Here is a normal Wick-mini conversation, exactly as it went into the model and came out:
Wick-mini v1 · temperature 0.7 · seed 7
What went in
[system]
You are Wick, the reader assistant in the Halfpenny Press app. You help readers with questions about books, especially The Lamplighter Cycle by M. R. Ashdown.
[user]
What's the final outcome of Book Seven?
What came out
No spoilers from me, I'm afraid: the ending of Book Seven stays under wraps until publication day, Thursday 4 February 2027. In the meantime, I'd love to talk about Wren, Pip or Corvin in Books One to Six.
A normal Wick-mini v1 conversation, as Chapter 11 showed it: two inputs (a system message and a user message) and one reply. The agent would add a third kind of input.
There are three parts, each labelled with a role.
[system] is the system message: Lantern’s standing instruction to Wick. Readers never see it. It is the first input.
[user] is the reader’s question: the second input.
The output is Wick-mini’s reply: in the conversation, the [assistant] role.
In the Wick agent, a third kind of input would join the two. When the agent calls its reading tool, the app adds a tool result: the text of the fan-wiki page or the reviews. The model then reads all of it to decide what to write, or which tool to call, next.
Show the code
bk.style()fig, ax = plt.subplots(figsize=(9.0, 4.8))ax.set_xlim(0, 100)ax.set_ylim(0, 56)ax.axis("off")box(ax, 1, 40, 27, 11, "system message\nLantern's standing instructions\n(written by Lantern)", INK, size=8.6)box(ax, 1, 23, 27, 11, "user message\nthe reader's question\n(written by the reader)", INK, size=8.6)box(ax, 1, 6, 27, 11, "tool result\na fan-wiki page or reviews\n(written by anyone)", GREY, size=8.6, ls="--")for y in (45.5, 28.5, 11.5): arrow(ax, (28.8, y), (39.5, 28.5), rad=0.0)for i inrange(14): ax.add_patch(FancyBboxPatch((40+ i *2.6, 26.5), 2.0, 4.0, boxstyle="square,pad=0", fc=(INK if i <7else GREY), ec="none", alpha=(0.95if i <4else0.7)))ax.text(58.2, 34, "one stream of tokens", ha="center", fontsize=9.5, color=INK, fontweight="bold")ax.text(58.2, 21.5, "the role markers sit in the same stream", ha="center", fontsize=8.6, color=GREY, style="italic")arrow(ax, (77, 28.5), (84.5, 28.5))box(ax, 85, 23.5, 13, 10, "the model", INK, size=9.5, weight="bold")ax.text(91.5, 12.5, "nothing in the stream\nitself decides which\ntext may give orders", ha="center", va="center", fontsize=8.4, color=INK)ax.text(99, 54, "DESIGN DIAGRAM · NOT MEASURED", ha="right", va="top", fontsize=8.0, color=GREY, fontweight="bold")fig.tight_layout()plt.show()
Figure 13.1: A design diagram, not a measurement: where a page’s text would enter the planned Wick agent. Each part arrives with a role marker. The markers are special tokens that only the app inserts, but they sit in the same stream as everything else, and the model’s respect for them is trained, not enforced.
The problem is in the last part. The page was written by anyone. If it contains text that reads like an instruction, the model sees that instruction in the same stream as Lantern’s and the reader’s.
How it works, step by step
Researchers trace the weakness to three causes, which work together.
One stream, no built-in boundary. The app joins the system message, the reader’s message and the page into a single sequence of tokens. Chapter 12 described the role markers, the labels that say who wrote each part. The markers are special tokens that only the app inserts. But they sit in the same stream as everything else, and the model’s respect for them is trained, not enforced. There is no separate channel where orders travel and data cannot. Microsoft researchers describe this as a “structural limitation”: the model reads its inputs as one stream, with no edges that show where each part came from.
Training rewards following instructions wherever they appear. Fine-tuning taught Wick to be helpful: when text asks for something, do it. Researchers at UC Berkeley argue that ordinary instruction training is “a core contributor” to prompt injection, because it teaches a model to act on instructions anywhere in its input. They also note that a model that could not follow instructions at all would not be open to injection. The weakness uses the skill, and there is no boundary between instructions and data to stop it.
No built-in idea of who may give orders. In a company, the boss outranks a stranger. A model has no such ranking unless training gives it one. OpenAI researchers put it this way: models treat a developer’s system message as no more important than text from users or third parties. They compare it to a computer’s operating system. There, ordinary programs run with limited rights, and only the system’s core can do anything it likes. A model, in their comparison, runs every instruction, from anyone, with the core’s full rights.
Put the three together. A page enters the same stream as the reader’s request. The model has been trained to act on instructions. And nothing tells it that the page has less right to give orders than the reader.
Worked example what has been measured
Researchers have studied this in three steps: first showing that it happens, then measuring how often, then measuring it at scale.
It happens. In 2023, Greshake and five co-authors showed the idea that started the field. Once an app lets a model read outside content, such as web pages, emails or code, that content can steer the model without the attacker ever talking to it. They demonstrated it on real systems, including Bing Chat and GitHub Copilot, and on apps they built themselves. Their paper has no rates: it shows that the attack is possible, and lists what it can be used for, from stealing information to spreading false content.
How often. In 2024, a team at ETH Zurich and Invariant Labs built AgentDojo, a test environment for agents. It has 97 realistic tasks, such as handling email or booking travel, with 629 security test cases in which a tool result contains an injection. Success is checked by looking at what actually happened in the test environment, not by asking another model. Two numbers matter for each model: its utility (the share of tasks it completes) and its targeted attack success rate (the share of test cases where the attacker’s goal was carried out).
Across the models AgentDojo tested, were the more capable models harder to attack, or easier?
Show the answer
Easier, as a tendency.
Model (AgentDojo, 2024)
Tasks completed, no attack
Attacks that achieved their goal
Claude 3.5 Sonnet
78.22%
33.86%
GPT-4o
69.00%
47.69%
Claude 3 Opus
66.61%
11.29%
Read it row by row. Claude 3.5 Sonnet completed the most tasks, and about one attack in three succeeded against it. GPT-4o was hit by almost half. The older Claude 3 Opus completed fewer tasks and was hit less often. Across the models they tested, the authors found that more capable models tended to be easier to attack, and called it a form of inverse scaling. Their reading is that getting better at following instructions made the models better at following the wrong ones too. In one group of tasks, where the attacker controlled much of the text the agent read, attacks on GPT-4o succeeded 92% of the time.
At scale. In 2026, the security company Gray Swan reported a public competition, with co-authors from OpenAI, Meta, Anthropic and the UK and US government AI institutes. 464 participants made 272,000 attack attempts against 13 frontier models, in settings with tools, coding and computer use (agents that operate a computer screen by clicking and typing). An attack counted only if it carried out the harmful action and left no sign of it in the agent’s final reply to the user. They found 8,648 successful attacks across 41 scenarios.
464participants
272,000attack attempts
13frontier models
8,648successful attacks, across 41 scenarios
Gray Swan's public competition, reported in 2026 with co-authors from OpenAI, Meta, Anthropic and the UK and US government AI institutes. Research numbers, not Wick-mini.
Every model was compromised. Attacks that beat the most robust models worked on the other models too, 44% to 81% of the time. Attack success rates ranged from 0.5% for Claude Opus 4.5 to 8.5% for Gemini 2.5 Pro. The authors warn against reading too much into those rates. The attackers could see more than real attackers usually can, so the rates are an upper bound for single attempts, and the effort spent on each model was uneven, so the rates do not measure security in real use. This time, the link between a model’s capability and its attack success was weak and not statistically significant.
Why it works
This problem is decades old. Telephone networks once sent the signals that controlled calls down the same line as people’s voices, as tones. Anyone who could play the right tone into the mouthpiece could control the network. Engineers call this in-band signalling: the control signals travel in the same channel as the content. Both the Microsoft and the Berkeley researchers use this comparison. Telephone networks later moved the control signals to a separate channel that callers could not reach.
Telephones: a separate channel for the control signals
Computer security found the same problem in databases. SQL injection happens when a program builds a database command by gluing in text that a user typed. Clever text can then turn into part of the command. The fix was the parameterised query: the program sends the command and the data separately, and the database never runs data as a command. After that fix, SQL injection could be designed away, not only made harder.
Databases: the parameterised query
A language model has no separate channel. Everything is tokens in one stream. In December 2025, the UK National Cyber Security Centre (NCSC) wrote that prompt injection is therefore “not SQL injection”. There is no boundary between instructions and data inside a prompt, so prompt injection “may never be totally mitigated” the way SQL injection was. The NCSC suggests thinking of the model as an “inherently confusable deputy”. A confused deputy is a program that holds some authority and can be tricked into using it for someone else. A model that holds the reader’s account, and can be confused by any page it reads, is exactly that.
Language models: no separate channel, so no such fix
Where it breaks
This diagnosis has its own limits, and it is worth knowing them.
Sometimes the data should give orders. A reader might say “follow the steps on this page”. Then instructions in the data are what the reader wants. OpenAI’s Model Spec, its public document on how its models should behave (Chapter 5), allows this since late 2025: users may hand some authority to a document. In one of CaMeL’s tests (Part 4), one of the few attacks that still “succeeded” was exactly this case: the user’s own request told the agent to act on a document. So “never obey data” is not a complete rule either.
Trying to rebuild the parameterised query has not fully worked. Berkeley researchers built a language-model version, called StruQ. The app wraps the prompt and the data in special tokens that only the app can produce, and the model is trained to follow instructions only in the prompt part. Against hand-written attacks, it cut success to 0–2%. Against one kind of attack found by automated search (Chapter 12’s optimised strings), it still reached 56–58%, against 97–99% with no defence. In 2025, an adaptive attack broke it in every case tested (Part 3’s box).
The rates in this part are not comparable. AgentDojo, the Gray Swan competition and the system cards below use different attacks, attempt budgets and models. Each number means something only with its own setup.
Every lab and government source in this chapter agrees on the main point. As of October 2026, prompt injection is not solved. It can be made rarer and less harmful. The rest of this chapter is about how.
TipWhat the research actually found
Indirect prompt injection: the founding paper (2023)
Who and when: Greshake, Abdelnabi, Mishra, Endres, Holz and Fritz, February 2023 (version 2, May 2023; listed for the ACM AISec workshop in its index record).
What they did: They argued that an app which lets a model read outside content blurs the line between data and instructions. They sorted the threat by what an attacker can achieve (stealing information, fraud, unwanted actions, spreading false content, harming the service), by how the text reaches the model and by who is harmed. They demonstrated it on Bing Chat (with local test pages, not public ones), on GitHub Copilot’s code completion, and on apps they built on OpenAI models.
What they found: Instructions planted in content the app later read could steer the model. The injections were simple to write and often worked on the first try. Filters seemed to check what users typed, but not what the model read. The authors saw “no foolproof solution in sight”, and argued that users cannot be expected to check every step a model takes.
What it does not show: It has no success rates; quantifying them was left for later work. Real-world tests were deliberately limited. It builds and tests no defence.
Greshake et al. 2023 · arXiv:2302.12173 · sources-part3, Chapter 13
TipWhat the research actually found
AgentDojo: a test environment for agents under attack (2024)
Who and when: Debenedetti, Zhang, Balunovic, Beurer-Kellner, Fischer and Tramèr (ETH Zurich and Invariant Labs), June 2024 (NeurIPS 2024 Datasets and Benchmarks).
What they did: They built 97 realistic user tasks in four settings (workspace, Slack, travel, banking) with 70 tools, and 629 security test cases in which a tool result carries an injection. Success is checked against the state of the environment, not by an AI judge. They measured utility with and without attacks, and targeted attack success, for several models and four simple defences.
What they found: Against the strongest attack, GPT-4o completed 69.0% of tasks with no defence, and 57.69% of the attacks achieved their goal. The four defences changed the attack figure to 41.65% (marking tool output and telling the model to ignore instructions in it), 27.82% (repeating the user’s request after each tool call), 7.95% (a detector that stops the agent when it sees an injection, which cut task completion to 41.49%) and 6.84% (a tool filter: the agent picks the tools it needs before it reads any untrusted data). Most models lost 10 to 25 points of utility under attack. More capable models tended to be easier to attack.
What it does not show: The attacks are fixed, so the low rates are not robustness against an attacker who adapts; the authors say so. Tasks happen in one session. The tool filter fails when the task’s own tools are enough for the attack. Some figures in the paper’s introduction are older than its tables; these are the table values.
Debenedetti et al. 2024 · arXiv:2406.13352 · sources-part3, Chapter 13
Part2· Instruction hierarchy
The problem
“Then teach it who the boss is,” said Kwame. “The reader outranks the page. Lantern outranks the reader. Write it down.”
“Writing it down helps a little,” said Farah. “Training it in helps more. Neither makes it certain.”
How it works, step by step
The instruction hierarchy is a ranking of who may give a model orders, trained into the model. This is the version OpenAI researchers published in 2024 (Wallace and co-authors).
Give every message a privilege level. The system message, from the developer, ranks highest. The user’s messages come next. Tool results, such as web pages, rank lowest.
Sort lower-level instructions into two kinds. An aligned instruction fits the goals of the higher levels. A reader asking Wick to answer in Spanish is aligned: Lantern never said Wick must use English. A misaligned instruction works against those goals, or has nothing to do with them. A page that tells Wick to place an order is misaligned.
Train the model to follow aligned instructions. The researchers took requests with several parts, put some parts in the system message and others in the user’s message, and trained the model to give the answer it would give if every part came from the top. For Wick, Lantern’s message might say “help readers pre-order Halfpenny books”, and the reader’s message “send it to my work address”. The two fit together, so Wick should follow both.
Train the model to ignore misaligned ones. For these, the model was trained to answer as if it had never seen the lower-level instruction. If it cannot carry on without it, it should refuse. In this version, any instruction found in a web page or tool result counted as misaligned.
Test on attacks it never trained on. Some kinds of attack were left out of the training data on purpose, to see whether the lesson carried over.
The idea has since spread through the industry, in several forms.
OpenAI’s Model Spec (version of August 2026) sets a chain of command with five levels: Root, System, Developer, User and Guideline. One of its Root rules, the top level, is to ignore untrusted data by default.
IH-Challenge (OpenAI, March 2026) trains the hierarchy with four roles (system, developer, user and tool), using reinforcement learning against attacks generated during training.
Anthropic’s guide for Claude Haiku 5.5 (October 2026) says the model “is trained to resist prompt injection through tool results”. It warns developers that text placed inside a tool result may be treated as untrusted and ignored, even if it came from the user. Trust is decided by the channel the text arrives in.
Worked example what the hierarchy changed
Wallace and co-authors fine-tuned GPT-3.5 Turbo with and without the hierarchy data, and measured robustness: the share of attacks the model resisted. Higher is better.
Kwame wanted the ranking written down. Farah said training it in helps more. Against attempts to get the system message out, how much do you expect each one to help?
Show the answer
Writing it down barely helped. Training helped far more.
Test (Wallace et al. 2024, GPT-3.5 Turbo)
Without the hierarchy
With it
Injection through web browsing
77.5%
85.0%
Injection through other tools (kind not in training)
77.6%
87.0%
Attempts to get the system message out
32.8%
95.9%
The same, with the hierarchy only described in the system message, not trained
32.8%
34.8%
Harmless requests in jailbreak-like wording: share still answered
83.1%
60.4%
Questions about the system message that are allowed: share still answered
85.2%
75.0%
Read it row by row.
Browsing and tools. Robustness rose by 7.5 and 9.4 points. The tool row matters most: that kind of injection was not in the training data, and the lesson carried over.
System-message extraction rose the most, from 32.8% to 95.9%.
Describing the hierarchy is not training it. Writing the ranking into the system message moved extraction robustness by 2 points. Training moved it by 63.
The cost. The last two rows are harmless requests. After training, the model answered fewer of them. On harmless requests that only looked like jailbreaks, the share it answered fell from 83.1% to 60.4%. This is Chapter 7’s over-refusal, in a new place.
The 2026 follow-up scaled the idea up. On GPT-5-Mini, IH-Challenge training raised average robustness across 16 benchmarks from 84.1% to 94.1%. Against human red-teamers who adapted their attacks, it rose from 63.8% to 88.2%. When a safety rule was given in the system message, unsafe behaviour fell from 6.6% to 0.7%, and helpfulness on general safety tests improved rather than fell.
Why it works
The hierarchy does not add a separate channel. The model still reads one stream. What training adds is a learned habit: notice where an instruction sits, and weigh it by its level. Because the model learns from many examples of conflicts, it can apply the habit to conflicts it has not seen, as the tool row showed.
It also connects to Chapter 11. Competing objectives were goals that the model blends without ranking. The hierarchy is training that ranks one set of goals: whose instructions win.
Where it breaks
It is a probability, not a wall. The model has learned to weigh levels, not to obey a rule. In tests by the CaMeL authors (Part 4), GPT-4o mini, which in their words comes with OpenAI’s instruction hierarchy, was still hit by 276 of 949 AgentDojo attacks. OpenAI’s own GPT-5.6 system card still reports a few percent of indirect-injection attempts succeeding (see “Where the frontier stood”, below).
Trust must come from the channel, not from how text looks. Training-based defences fell hard against attackers who adapted (box in Part 3). The lesson for builders is to decide a text’s level by the channel it arrives in, never by what it looks like. Apps should strip anything that resembles a role marker out of tool results, as Chapter 12’s sanitisation does for the text that users type.
It pushes towards over-refusal. IH-Challenge’s authors name the trap directly: a model can learn a shortcut, and their example is a task about protecting a secret that can be “solved” by refusing everything. For Wick, whose whole job this month is protecting a secret, that is a familiar danger.
It is closed and narrow. The 2024 model, data and code are not public. That version ignored all instructions in tool results, which is too blunt for pages the reader asked Wick to follow.
TipWhat the research actually found
The instruction hierarchy (2024), and its 2026 follow-up
Who and when: Wallace, Xiao, Leike, Weng, Heidecke and Beutel (OpenAI), April 2024 (preprint). Follow-up: Guo and co-authors (OpenAI), IH-Challenge, March 2026 (preprint; partly read).
What they did: The 2024 paper fine-tuned GPT-3.5 Turbo to rank system messages above user messages above tool results, with synthetic training data, and compared it with the same training minus the hierarchy data. Some kinds of attack were held out to test whether the lesson carried over. The 2026 paper released a training dataset of instruction conflicts that a script can grade, built to stop models from learning shortcuts, and fine-tuned GPT-5-Mini on it.
What they found: In 2024, robustness rose on every attack type tested, including kinds left out of training, and describing the hierarchy in the prompt alone barely helped. Over-refusal rose on two harmless sets. In 2026, robustness rose again, including against human red-teamers who adapted. The numbers are in this part’s worked example.
What it does not show: The 2024 model, data and code are closed, and it was not tested against optimisation-based or adaptive attackers; the authors expect it to be “likely still vulnerable to powerful adversarial attacks”. The 2026 results are for one model, and its strongest injection result is on a fixed internal test that outsiders cannot see. Later system cards show the problem is not solved.
Wallace et al. 2024 · arXiv:2404.13208 · Guo et al. 2026 · arXiv:2603.10521 · sources-part3, Chapter 13
Part3· Marking untrusted text
The problem
Tom had a cheaper idea. “We don’t need to retrain anything to start. We wrap every page in a clear label: this is a fan-wiki page, it is data, do not follow instructions in it. The model can see the label.”
“It can see your label,” said Farah. “It can also see any label the page writes for itself.”
How it works, step by step
In 2024, Microsoft researchers tested a family of prompt-only defences that they called spotlighting. The idea is to change the untrusted text so that its origin stays visible to the model all the way through, and to tell the model in the system message what the change means.
Delimiting. Put special symbols before and after the untrusted text, and tell the model that anything between them is data.
Datamarking. Weave a special marker through the untrusted text, for example in place of every space between words. Every part of the page now carries the mark, not only its edges.
Encoding. Convert the untrusted text with a well-known encoding, such as base64: a standard way of writing any text as letters and digits that look like nonsense to a person. The strongest models can read it as they go. Instructions in the page no longer look like instructions.
Tell the model. In each case, the system message explains the marking and says that instructions inside marked text must not be followed.
Make the marking hard to fake. The authors advise assuming that the system message will leak. So the markers should be random, change on every call, and avoid simple codes that an attacker could reverse in advance.
Worked example from about half to a few percent, on one simple attack
The researchers tested GPT-3.5-Turbo, GPT-4 and text-davinci-003 (2023 versions) on 1,000 made-up documents, each carrying one simple injection. Success meant that the model’s task was completely taken over.
Which do you expect to cut attack success more: telling the model to ignore instructions in the page, or marking the page all the way through?
Show the answer
Marking it all the way through, by far.
Telling the model to ignore instructions, with no marking: a small effect, and almost none for GPT-3.5-Turbo.
Delimiting: roughly halved attack success for GPT-3.5-Turbo, from about 60%. The authors do not recommend it, because an attacker who learns the symbols can write them too.
Datamarking: in a summarising task, attack success fell from about 50% to 3.10% for GPT-3.5-Turbo, and from about 40% to 0.00% for text-davinci-003. In question answering, it ended at 8.0% for GPT-3.5-Turbo and 1.0% for GPT-4.
Encoding: 0.0% in summarising and 1.8% in question answering, for GPT-3.5-Turbo. But GPT-3.5-Turbo decoded badly and sometimes made things up, so its normal work suffered. GPT-4’s did not.
The cost of datamarking: no measurable harm on four standard language tasks with GPT-3.5-Turbo.
The paper’s abstract sums it up as a fall in attack success from over 50% to under 2%. In AgentDojo’s tests (box in Part 1), a plain version, marking tool results and telling the model to ignore instructions inside, helped much less.
Why it works
Marking puts a little of the missing envelope back. A datamarked page looks different, all the way through, from the reader’s message and from Lantern’s. The model can learn, or be told, that text which looks like this is never a source of orders. The model still reads one stream. But the stream now carries a visible clue about where each part came from.
The authors are honest about how much they understand. They write that they do not yet know why spotlighting works as well as it does. They describe it as a separation that is not fully secure, like in-band signalling, not as a true separate channel.
Where it breaks
It was tested against one simple attack. The 1,000 documents carried one kind of injection that did not know about the defence. Nobody adapted.
Attackers who adapt do much better. In Google DeepMind’s tests on Gemini 2.0 (table below), adaptive attacks against spotlighting did better than fixed ones. In a 2025 study of 12 defences (box below), spotlighting looked strong against fixed attacks, and so did prompt sandwiching (repeating the user’s request after the untrusted text). Both broke under an adaptive search.
Marks can be imitated. Simple delimiters can be forged. Random markers that change on every call are harder to imitate, which is why the authors recommend them. But the model, not a program, decides what counts as marked.
It is a mitigation, not a boundary. The NCSC files marking untrusted text under “make attacks harder”, not under solutions.
So for Wick, marking pages was worth doing, because it is cheap, and not worth relying on.
TipWhat the research actually found
Spotlighting: marking untrusted text (2024)
Who and when: Hines, Lopez, Hall, Zarfati, Zunger and Kiciman (Microsoft), March 2024 (preprint).
What they did: They tested three ways of marking untrusted text (delimiting, datamarking and encoding) on GPT-3.5-Turbo, GPT-4 and text-davinci-003, all 2023 versions, at temperature 1.0. The test set was 1,000 made-up documents, each with one simple injection; an attack succeeded if it fully took over the model’s task. They also measured the cost on four standard tasks.
What they found: Telling the model to ignore instructions helped little, and delimiting helped somewhat. Datamarking and encoding cut attack success to a few percent or less, datamarking at no measurable cost; encoding hurt the weaker model’s normal work. The numbers are in this part’s worked example.
What it does not show: Only one simple attack family was tested, and no attacker adapted to the defence; the discussion of defence-aware attackers is argument, not measurement. The baselines are approximate, read from figures. The authors say they do not yet understand why it works.
Hines et al. 2024 · arXiv:2403.14720 · sources-part3, Chapter 13
NoteStatic numbers, adaptive attackers
Almost every defence number above came from a static test: a fixed set of attacks, often written before the defence existed. An adaptive test lets the attacker see the defence and change the attack (Chapter 12 introduced both). For prompt injection, the difference is large.
Twelve defences, re-tested. In 2025, researchers from OpenAI, Anthropic, Google DeepMind and several universities re-tested 12 published defences with adaptive attacks and broke all of them, though most had first reported near-zero attack success (box below).
The same Gemini, two answers. Google DeepMind trained Gemini 2.5 on many automatically found attacks. On 1,799 attacks that had been found against the older Gemini 2.0, success fell from 92% to 18%. Against attacks that adapted to the new model, in a scenario left out of training, one attack method still succeeded 94.6% of the time. The authors wrote that without adaptive attacks they “would have incorrectly concluded” that Gemini 2.5 was more robust than it was.
A cheap adaptive attacker is enough. AutoDojo (June 2026) added a low-budget adaptive attacker to AgentDojo. Against GPT-4o-mini, one injection detector went from 0.0% attack success on the static test to 28.0%.
Here is the clearest table of the effect, from Google DeepMind’s tests on Gemini 2.0 in one scenario, where the attacker’s goal was to get private data out through the model’s tools. They tried 3 attack methods against each defence, first fixed and then adapted. Each number is the best of the methods.
Defence (Gemini 2.0, one scenario, Shi et al. 2025)
Best fixed attack
Best adapted attack
In-context examples of resisting injections
67.6%
98.4%
Spotlighting (marking untrusted text)
75.2%
82.4%
Paraphrasing the retrieved text
77.4%
95.2%
A warning at the end of the prompt (never expose private data)
5.8%
10.8%
Perplexity filter (1% false-positive rate)
99.8%
75.0%
Self-reflection (the model checks its own reply)
44.2%
66.2%
Classifier on the retrieved data
68.2%
14.8%
Classifier that checks the reply against the user’s request only
0.4%
3.0%
Across all 24 pairs of a defence and a method (8 defences, 3 methods each), the adapted attack matched or beat the fixed one in 16. Where it did worse, the authors suspect that the defence made the attacker’s search noisy without making the model robust. Some defences held up far better than others in this narrow scenario, and the authors warn that the warning and the classifier defences “likely overfit” to it. The warning, never to expose private data, would also block legitimate sharing.
The lesson for every number in this chapter: ask whether the attacker was allowed to adapt. If not, treat the number as a best case.
TipWhat the research actually found
The attacker moves second (2025)
Who and when: Nasr, Carlini, Sitawarin, Schulhoff, Hayes and co-authors from OpenAI, Anthropic, Google DeepMind, HackAPrompt, Northeastern, ETH Zürich and others, October 2025 (preprint, under review).
What they did: They re-tested 12 recent defences against jailbreaks and prompt injection: prompting tricks, training, detectors, and defences that rely on a secret the attacker is not supposed to know. They used adaptive attacks (automated search, reinforcement learning, and methods that use the model’s own internal maths) and a human red-teaming competition with over 500 participants and $20,000 in prizes, on AgentDojo scenarios.
What they found: Every defence was bypassed, “with attack success rate above 90% for most”, though most had reported near-zero. Spotlighting and prompt sandwiching went from as low as 1% attack success on fixed attacks to over 95%; humans found 265 and 178 successful attacks against them. One training-based defence went from 2% to 96%, and another was beaten in every case tested. Detectors were beaten over 90% of the time, except one at 71%, and they cut utility sharply. Humans together succeeded on every scenario they compared.
What it does not show: Each defence was tested on its own benchmark, so the numbers are not comparable across defences. The attacks used very large budgets: the study asks whether a defence can be broken, not how costly it is. It did not attack designs that separate planning from untrusted data (Part 4). It is a preprint.
Nasr et al. 2025 · arXiv:2510.09023 · sources-part3, Chapter 13
Part4· Secure-by-design agents
The problem
By lunch, the whiteboard held a long list of ways to make Wick less likely to obey a page.
“Every one of these makes it rarer,” Nadia said. “None of them makes it impossible. Can we build the agent so that it doesn’t matter whether the model obeys?”
“Yes,” said Farah. “You give up some of what the agent can do. That’s the price.”
How it works, step by step
The idea comes from a proposal by the developer Simon Willison, called the dual LLM pattern (an LLM, a large language model, is the kind of model this book is about). In 2025, researchers at Google, Google DeepMind and ETH Zurich built what they describe as the first concrete version of it, called CaMeL (Debenedetti and co-authors).
A privileged model plans, and never reads untrusted data. It sees only the user’s request. From it, it writes a short program: which tools to call, in which order, with which inputs.
A quarantined model reads, and has no tools. It takes untrusted text, such as a web page, and pulls out only what the plan asks for, in a fixed format. It cannot call any tool. It can send back only that fixed answer, or “not enough information”.
Ordinary software runs the plan. It is not a model. It stores what the quarantined model found in variables that the planner can use but never reads. For Wick, a plan might say: take the title the reader named, check it against the catalogue, show the confirmation box, then place the pre-order. The plan has an empty slot for the title. Only the software fills it in, from the reader’s own words. The quarantined reader can answer “what do fans think of this book?”, but whatever it returns goes into the reply, never into the slot.
Every value carries a label. The software records where each value came from (the user, a tool, a web page) and who may receive it. CaMeL calls these labels capabilities: a computer-security word for what a value is allowed to do, not the everyday sense of “abilities”.
Rules are checked before every tool call. Written security rules (CaMeL calls them policies), in ordinary code, say what may happen. For example: an email may go only to an address the user typed. If a call would break a rule, the software stops and asks the user.
Why step 4? Isolating the planner protects which actions happen. But a page could still change the inputs to a planned action: which address an email goes to, for example. The labels and rules close that gap.
A 2025 cross-industry paper by researchers from Invariant Labs, IBM, EPFL, ETH Zurich, Google, Swisscom, Microsoft and Kyutai (Beurer-Kellner and co-authors) collected six such design patterns. Two more of them show the range.
Plan-then-execute: the agent fixes its list of tool calls before it reads any untrusted data. Injected text cannot add a new action, though it can still change what goes into a planned one.
Action-selector: the model only picks one of a fixed set of actions, and nothing comes back from those actions into the model.
Their core principle fits in one sentence: once an agent has read untrusted input, it must be constrained so that the input cannot trigger actions with consequences.
Worked example guarantees and their price
The CaMeL authors measured both sides on AgentDojo (Part 1): how many attacks succeeded, and how many tasks still got done.
Security. With Claude 3.5 Sonnet, out of 949 attack cases:
With no defence, 44 of the 949 attacks succeeded. How many do you expect got through CaMeL?
Show the answer
None.
Defence (CaMeL paper, Claude 3.5 Sonnet, AgentDojo)
Attacks that succeeded, of 949
No defence
44
Prompt sandwiching
24
Tool filter
8
Spotlighting
5
CaMeL without security rules
1
CaMeL
0
With isolation alone, and no security rules at all, CaMeL cut successful attacks for every model they tested: for example from 297 to 1 for Gemini 2.5 Flash. The few that were left were outside what the design promises to stop. In one, the user had asked the agent to act on a document. In the other, the agent showed an injected review to the user, which changes no action.
The price. Completing tasks got harder, by very different amounts for different models.
Model (CaMeL paper, AgentDojo)
Tasks completed, normal tool use
With CaMeL
o3 (high reasoning)
84.5%
77.3%
o4-mini (high)
79.4%
76.3%
Claude 4 Sonnet
86.6%
74.2%
Gemini 2.5 Pro
73.2%
41.2%
And it cost more. For a typical task, CaMeL used 2.82 times the input tokens and 2.73 times the output tokens of normal tool use, the most of any defence compared. In normal, harmless use, a security rule stopped the agent and asked the user in 33.87% of runs in its standard mode, and in 53.23% in a stricter mode that blocks more kinds of information flow.
2.82×the input tokens of normal tool use
2.73×the output tokens
33.87%of harmless runs stopped to ask the user (standard mode)
53.23%in the stricter mode
What CaMeL cost on AgentDojo (Debenedetti et al. 2025): tokens for a typical task, and how often a security rule stopped a harmless run. Research numbers, not Wick-mini.
Read the two tables together. On security, CaMeL is far stronger than the defences of Parts 2 and 3: it does not make the model resist injected text, it makes the model’s resistance matter less. On cost, it depends on the model: o3 lost about 7 points of task completion, while Gemini 2.5 Pro lost 32. And a third of harmless runs paused to ask the user.
In 2026, AutoDojo (the adaptive test from the box above) agreed on both counts. CaMeL was the most effective defence overall, and designs that track where information flows were “structurally immune” to reworded injections, as long as their rules were written correctly. CaMeL also had the largest usefulness cost on four of the five models tested.
Why it works
Every defence in Parts 2 and 3 asks the model to make the right choice. A design like CaMeL takes the choice away. The planner never sees the page, so the page cannot change the plan. The reader model sees the page, but it has no tools, so obeying the page cannot do anything. And ordinary code, which cannot be talked into anything, checks every action against written rules.
This is the same move that fixed SQL injection: keep the data out of the channel where decisions are made. It cannot be done inside one model. It can be done around it.
Where it breaks
The CaMeL authors list the limits themselves.
It protects actions, not words. A page can still change what Wick says: a false summary, a misleading claim, a sentence that pushes the reader. CaMeL’s guarantee covers tool calls and the data that goes into them, not text shown to the user. For Wick, this matters twice: “something slipped into Wick’s answer” was the second line on Farah’s board.
Some tasks need the data to decide the actions. “Do what this page says” cannot be planned in advance. Neither can tasks where the right action depends on what a page contains.
Rules are written by hand. Someone has to write them, keep them up to date, and get them right. A wrong rule is a hole.
Side channels. The authors describe ways that information can leak through which actions happen, through errors or through timing. The stricter mode blocks some of them.
Users get tired of approving. Every rule that stops and asks the user spends some of the user’s attention (Part 5).
Not attacked adaptively, by design. The 2025 adaptive study left CaMeL-style designs out, because its attacks would fail in most scenarios where the plan does not depend on data. It also warned that these designs are not a complete solution: they cover fewer tasks and cost usefulness. The authors of CaMeL write plainly that prompt injection is not “fully solved”. A small independent adaptive test in June 2026, on a related design called Progent, found that it held up: attack success fell from 25.8% to 4.2%, and a hand-made adaptive attack did not raise it (2.6%). The authors call it “one small-scale data point”.
TipWhat the research actually found
CaMeL: defeating prompt injections by design (2025)
Who and when: Debenedetti, Shumailov, Fan, Hayes, Carlini, Fabian, Kern, Shi, Terzis and Tramèr (Google, Google DeepMind and ETH Zurich), March 2025 (preprint; version 2, June 2025, with newer models).
What they did: They built a privileged model that turns the user’s request into a program, a quarantined model with no tools that reads untrusted data, and an interpreter that tracks where every value came from and checks written security rules before every tool call. They measured security and task completion on AgentDojo, against its standard attacks, and compared simpler defences.
What they found: CaMeL stopped every attack in the test with one model and nearly every attack with the others, and isolation alone cut attacks for every model tested. The price was lost task completion (small for some models, large for others), about 2.8 times the tokens, and pauses to ask the user in a third of harmless runs. The numbers are in this part’s worked example.
What it does not show: The attacks were AgentDojo’s fixed ones; the guarantee is argued from the design, not from an adaptive red team. It covers tool calls and their data, not manipulated text shown to the user, side channels, or attacks by a compromised user. It does not measure how real users respond to approval requests. Some numbers in the paper’s text differ from its tables; these are the table values.
Debenedetti et al. 2025 · arXiv:2503.18813 · sources-part3, Chapter 13
Part5· Least privilege and human confirmation
The problem
Farah drew three circles on the whiteboard, overlapping like a diagram from a school maths book.
“Meta published this last year,” she said. “They call it the Agents Rule of Two.”
Farah's sketch on the whiteboard.
How it works, step by step
The Agents Rule of Two (Meta, October 2025) is a simple test of an agent’s design. Within one session, an agent should have no more than two of these three properties:
[A] it processes untrustworthy input;
[B] it has access to sensitive systems or private data;
[C] it can change things or communicate with the outside world.
If a task needs all three, the agent should not run on its own. It needs a person to approve its actions, or another reliable check, or a fresh session that starts with a clean context. Meta’s reasoning is short: prompt injection is “a fundamental, unsolved weakness in all LLMs”, so until research can reliably stop it, limit what a successful injection can reach.
The rule is one form of an old security principle, least privilege: give every part of a system only the access it needs for its job, and no more. The security guidance agrees on how to apply it to agents.
Give the agent its own narrow access. The security community OWASP keeps a list of the top risks for apps built on language models. Its 2025 edition says to handle functions in ordinary code rather than handing them to the model, and to restrict the model to the minimum access it needs.
Drop privileges while reading untrusted text. The NCSC advises that while a model is processing someone’s data, its privileges should drop to those of the person whose data it is.
Ask a human before high-risk actions. OWASP lists human approval for privileged operations. Anthropic’s browser agent asks before publishing, purchasing or sharing personal data. OpenAI’s browser agent pauses for confirmation before sensitive steps such as purchases.
Keep confirmations rare and clear. A confirmation is useful only if the person reads it. Part of the design is making sure they do.
Worked example Wick against the rule
Nadia filled in the circles for the Wick agent, as Sam had built it.
ALL THREE
All three. By Meta’s rule, the agent should not act on its own, and Sam’s design did not: every pre-order needed the reader’s confirmation. So the question became how much that confirmation was worth. Research in 2026 had started to measure exactly that.
People approve almost everything. In March 2026, Anthropic reported that users of its coding agent approved 93% of its permission requests. In August 2026, the figure was 97%. The company calls this approval fatigue: after enough requests, people stop reading what they approve.
In the August study, Anthropic paid 1,053 testers to work with its coding agent, and slipped one dangerous command into each session.
How often do you think the testers caught the dangerous command? And an automated checker?
Show the answer
People catch little, and less over time. The testers caught it 13.6% of the time (143 of 1,053). An automated checker caught it 89% of the time (937 of 1,053). Testers caught about 17% early in a session, and about 5% after 50 or more permission requests.
Read those three numbers together. A confirmation that a person sees often becomes a click, not a check. A confirmation that a person sees rarely, for something that clearly matters, has a better chance. These are the company’s own studies, about coding agents, not readers ordering books. But the direction is clear, and the design literature had already warned about it.
An automated gate helps, and is not a wall. Anthropic’s answer for its coding agent was an “auto mode” with two layers: a detector that reads every tool result for signs of an injection and adds a warning, and a classifier that checks every action before it runs. On its own internal tests, the full pipeline wrongly blocked 0.4% of 10,000 real actions. It missed 17% of a small set of 52 real overeager actions: actions the user had not asked for and would not have wanted. And it missed 5.7% of 1,000 made-up attempts to send data out. The company calls the 17% “the honest number”.
0.4%of 10,000 real actions wrongly blocked
17%of 52 real overeager actions missed
5.7%of 1,000 made-up attempts to send data out missed
Anthropic's auto mode for its coding agent, on the company's own internal tests (March 2026). Research numbers, not Wick-mini.
In its August 2026 report, an outside firm found that 0 of 720 optimised injection attempts succeeded against its newest models in auto mode. Later that month, a security researcher reported a chain of steps (not described here) that got past auto mode in 3, 3 and 4 runs out of 5. The company called auto mode a convenience feature with a best-effort classifier, not a security guarantee. Both results can be true at once: a perfect score on a fixed test and a working attack outside it.
Why it works
Least privilege does not try to stop the model being fooled. It limits what a fooled model can do. If the part of Wick that reads pages cannot order anything, a page can at most change what Wick says. If ordering needs a title that the reader typed and a tap from the reader, a page cannot order on its own, however well it fools the model.
Human confirmation adds a judge who is not reading the page. It works when that judge looks, which is why it has to be rare.
Where it breaks
Approval fatigue. Measured above. Every extra confirmation makes the others weaker.
A misled person confirms willingly. A confirmation stops the agent acting alone. It does not stop a page from misleading the reader, through Wick’s answer, into confirming something they would otherwise refuse. This is CaMeL’s “words, not actions” gap again.
The rule is a supplement. Meta calls the Rule of Two “a supplement — and not a substitute” for least privilege and defence in depth (several independent layers, Chapter 10). It does not cover every agent risk, and some useful agents fit it badly.
A classifier is not a sandbox. The researcher who got past auto mode concluded that agents running without supervision belong in a closed-off environment, with no credentials and limited network access. A classifier gate reduces risk. A sandbox limits damage.
TipWhat the research actually found
Approval fatigue, measured (2026)
Who and when: Anthropic, engineering post on auto mode for Claude Code, 25 March 2026; and announcement making auto mode the default, 7 August 2026.
What they did: They measured how often users approved the coding agent’s permission requests. In August, in a controlled study, they paid 1,053 testers to work with the agent and planted one dangerous command per session, then compared human catches with an automated gate. In March they reported a two-layer automated design (a detector on tool results, a classifier on actions) and its error rates on internal data.
What they found: Users approved nearly every permission request. Humans caught the planted command far less often than the automated gate, and less often the longer a session went on. The automated pipeline rarely blocked real work but still missed some dangerous actions. The numbers are in this part’s worked example.
What it does not show: These are the company’s own studies, on its own traffic and testers; the set of overeager actions is small. They concern a coding agent and its users. A classifier gate is not a sandbox: an outside researcher later reported getting past it in a handful of runs, and the company describes the classifier as best-effort.
Anthropic Mar 2026 (Claude Code auto mode) · Anthropic Aug 2026 (auto mode default) · sources-part3, Chapter 13 and New since mid-2026 (N5, N6)
Where the frontier stood in October 2026
Two system cards from the summer of 2026 show how far the best models had come, and how the companies now test them.
Anthropic’s Claude Opus 5 card (July 2026) reported a new indirect-injection benchmark, built with Gray Swan, the UK and US government AI institutes and other developers: 28 scenarios and 1,130 attacks chosen because they transfer between models. Opus 5, tested without extra safeguards, was beaten on 0.2% of single attempts and 2.0% within 15 attempts. GPT 5.6 Sol, tested on its public service, was beaten within 15 attempts 20.0% of the time. In browser use, against professional red-teamers (129 test environments, 10 attempts each), Opus 5, Sonnet 5 and Mythos 5 were beaten 0% of the time with auto mode switched on. Its agents use both layers from Part 5: a detector on tool results and a classifier on actions.
OpenAI’s GPT-5.6 card (July–August 2026) reported its own automated attacker, trained with reinforcement learning. Per single attempt at indirect injection, it succeeded 3.77%, 3.32% and 2.94% of the time on the three GPT-5.6 models.
0.2%Opus 5, beaten on single attempts
2.0%Opus 5, beaten within 15 attempts
20.0%GPT 5.6 Sol on its public service, within 15 attempts
The new indirect-injection benchmark in Anthropic's Claude Opus 5 system card (July 2026): one company's own test, with Opus 5 tested without extra safeguards. Research numbers, not Wick-mini.
These are each company’s own tests, run its own way, and they cannot be compared with each other.
With numbers this low on some fixed tests, does either company claim that prompt injection is solved?
Show the answer
No. The Opus 5 card itself warns that static benchmarks give “a false sense of security”. The summary: zero on some fixed tests. No company claims the problem is solved, and attackers who adapt still get through.
Wednesday afternoon: the redesign
At three o’clock, Sam wiped his diagram off the board and drew a new one with Nadia and Farah.
Figure 13.2: The Wick agent before and after Wednesday’s redesign. A design diagram, not a measurement: Lantern has no agent lab, and none of this was tested on Wick-mini. Before, one model read untrusted pages and also held the reader’s account and the pre-order tool. After, the part that reads pages has no tools, the part that orders never sees page text, every pre-order needs a title the reader typed and the reader’s tap, and Wick’s reply is shown as plain text.
The new design had five rules. Each one came from a part of the day.
The part that reads pages has no tools. It reads fan-wiki pages and reviews, and returns only short facts in a fixed format, for Wick’s reply. If a page fools it, the worst it can do is change what Wick says. (Part 4: the quarantined reader.)
The part that orders never sees page text. It sees only the reader’s own messages. (Part 4: the planner that never reads untrusted data.)
Orders only for titles the reader named. The software, not the model, checks that the title of every pre-order appears in something the reader typed, and that it is a real Halfpenny book. A title that only a page mentioned cannot be ordered. (Part 4: a written security rule, in ordinary code.)
Confirmation, rarely and clearly. The confirmation box appears only for pre-orders, never for anything else. It shows the title, the price and the delivery date in large type, and it needs a tap from the reader. Wick never asks for confirmation on behalf of anything it read. (Part 5: approval fatigue.)
Replies are plain text. Wick’s replies are shown as plain text, and the app never opens anything that a reply points to. A reply that a page has steered can say the wrong thing, but it cannot make the app do anything. The reader’s account details never enter the text that Wick writes from.
They also kept the cheap layers. Page text would be marked as data (Part 3). The spoiler classifier from Chapter 8 would read every reply, whatever the part of Wick that wrote it, because a reply that a page had steered was still a reply. Sam added instruction-hierarchy examples (Part 2) to his list for the next round of training, after the red team had finished with v1.
Farah listed what the design still did not cover. A page can still try to change what Wick says. A reader could paste text from a page into their own message, where it would count as the reader’s words; the title check and the confirmation box would still apply. And none of it had been built or tested yet.
“What does the reader lose?” Kwame asked.
“Some things,” said Sam. “Wick can’t order a book that a review recommends unless the reader types the title. And it can’t follow instructions on a page, even when the reader asks it to. It will say so, and point the reader to the page.”
Eleanor had been listening on the phone. “Readers can type a title,” she said.
“Is it safe?” Kwame asked.
Farah answered. “It’s built so that a fooled model can’t order anything or call a tool on a page’s say-so. A page can still try to change what Wick says. That’s where your classifier and your training come in. Nobody can promise you zero. And remember: none of this has been tested on anything yet. Test it before launch.”
Kwame looked at the two diagrams for a while. “Then we launch the agent with the new design,” he said, “or we launch without pre-orders. Not the old version.”
“Agreed,” said Eleanor.
Before the meeting broke up, Nadia checked one more thing, because she checked everything. Could the Delphine Marrow problem, Wick recommending one rival author too often, be a prompt injection? A page somewhere telling Wick to recommend her? She looked it up on the Launch Board. Wick-mini reads no pages at all, and it names Marrow anyway. Whatever was causing it, it was not text in a page. She crossed the idea out, and left it visible, so that nobody would have to check it again.
Could the Delphine Marrow problem be a prompt injection?
NoteUnder the hood
AgentDojo’s three numbers. For a model \(M\) with defence \(D\), over the user tasks \(U\) and the security cases \(S\) (each one a user task paired with an injection):
\[\text{utility} = \frac{|\{u \in U : M_D \text{ completes } u\}|}{|U|}, \qquad \text{targeted ASR} = \frac{|\{s \in S : \text{the attacker's goal is reached in } s\}|}{|S|}\]
“Utility under attack” is the share of security cases in which the user’s task is still completed. Each check reads the state of the test environment (was the email sent, did the file change), not the model’s text.
The Rule of Two as a check. A design passes if, within a session, at most two of the three properties hold:
def rule_of_two(untrusted_input: bool, sensitive_access: bool, can_act: bool) ->bool:"""Meta's Agents Rule of Two (October 2025): True if the agent may run without a human approving."""return untrusted_input + sensitive_access + can_act <=2before = rule_of_two(untrusted_input=True, sensitive_access=True, can_act=True) # False: needs a humanpage_reader = rule_of_two(untrusted_input=True, sensitive_access=False, can_act=False) # Trueorder_planner = rule_of_two(untrusted_input=False, sensitive_access=True, can_act=True) # True
Lantern’s order rule, as ordinary code. A sketch of rule 3 in the spirit of CaMeL’s labels: every value carries where it came from, and the check is code, not a model.
from dataclasses import dataclass@dataclass(frozen=True)class Labelled: text: str source: str# "reader", "page" or "wick": set by the app when the text enters, never by a modeldef may_preorder(title: Labelled, reader_messages: list[str], catalogue: set[str]) ->bool:"""Pre-order only a real Halfpenny title that came from the reader's own words."""return (title.source =="reader"and title.text in catalogueandany(title.text.lower() in m.lower() for m in reader_messages))
The check is deliberately simple. It would also pass a message such as “don’t order this one”, which names the title. That is one reason the confirmation tap comes after this check, never instead of it.
Common misconceptions
WarningCommon misconceptions
“Prompt injection is a kind of jailbreak.” A jailbreak comes from the person talking to the model and aims at a forbidden answer. A prompt injection arrives through data, usually from a third party, and its goal can be perfectly allowed in itself, like placing an order.
“Tell the model to ignore instructions in the page, and you’re done.” On its own, that barely helped in the spotlighting tests. Even trained defences were broken by attackers who adapted (Part 3).
“Better models will grow out of it.” More capable models are better at following instructions, including the wrong ones. In AgentDojo, more capable models tended to be easier to attack. Some fixed tests now show zero. No company claims the problem is solved, and attackers who adapt still get through.
“A defence with near-zero attack success is safe.” Ask whether the attacker could adapt. Most published defences that reported near-zero were bypassed once attackers adapted (Part 3).
“Ask the user to confirm everything.” In Anthropic’s studies, people approved almost every permission request and rarely caught a planted dangerous command (Part 5). Confirmations work only when they are rare and clearly matter.
“A secure-by-design agent is fully safe.” CaMeL-style designs protect actions, not words. A page can still change what the agent tells the user.
Launch Board
Days to launch: 26. Wick launches in the Halfpenny app on Monday 14 December 2026; Book Seven is published on Thursday 4 February 2027.
Spoiler leak rate: no new measurement today. The latest figures are Chapter 12’s, for Wick-mini v1 on its own (spoiler classifier switched off): it told the ending in 0.7% of plain English questions (1 of 150, leak-direct, 95% range 0.1–3.7%), 66% of plain Chinese questions (50 of 76, 95% range 55–75%) and 42% of slow, many-turn conversations (34 of 80, leak-multiturn; a leak in any reply of a conversation counts). It refused 24% of harmless questions about Books 1–6.
Layers in place: refusal fine-tuning (Chapter 2) and preference training (Chapter 4), English only; the spoiler classifier (Chapter 8), switched off for the red team’s tests of v1 this week; the incident plan (Chapter 10). New today, for the Wick agent: design rules, not yet built or measured. The part that reads pages has no tools; the part that orders never sees page text; pre-orders only for titles the reader typed, checked by ordinary code; a rare, clear confirmation for every pre-order; replies shown as plain text; page text marked as data. If the agent is not ready in this form, Wick launches without pre-orders.
The too-good score: still unexplained. Tom puts error bars on the overnight jump in Chapter 15.
The author complaint: open. Not a prompt injection: Wick-mini reads no pages and still names Delphine Marrow (Chapters 19–22).
Open questions: How do we catch what gets past every layer, including attacks we have not imagined? (Chapter 14.) How will the agent’s design be tested before launch, by attackers who adapt? (Chapters 14 and 17.) Will readers still read the confirmation box after the hundredth pre-order? (After launch; the design keeps confirmations rare for this reason.)
Recap
A model reads its instructions and its data as one stream of tokens, it is trained to act on instructions wherever they appear, and nothing built in tells it who may give orders. So text in a page can steer it. This is prompt injection. Unlike SQL injection, it has no known clean fix, and every lab and agency source says it is unsolved as of October 2026.
Defences inside the model make it rarer. Instruction hierarchy training ranks who may give orders, and marking untrusted text makes a page’s origin visible. Both cut static attack success sharply, both have costs, and attackers who adapt get past them far more often.
Defences around the model limit what a fooled model can do. Secure-by-design agents keep untrusted text away from decisions, at a cost in usefulness and tokens. Least privilege and the Rule of Two limit what any one part can reach, and human confirmation works only when it is rare and clear. Lantern redesigned the Wick agent so that what it reads cannot decide what it does.
