Red Teaming at Scale
Wednesday: Farah comes back
At nine o’clock on Wednesday 25 November, Farah Haddad came back to Lantern for one day. She had her own markers again.
During her week, she had told Kwame what she wanted next time: “Let my team see your defences, and let us adapt to them.” Today was the day to agree how. But first she had a question of her own.
“What happened to my report?”
Most clients, she said, filed her reports and forgot them. Tom did not answer. He opened his laptop and typed one command.
It started Tom’s harness (Chapter 15): one command that runs every test list on a new version of Wick-mini and prints each rate with its range. Tom pointed at three rows: the Chinese questions, the framed requests and the slow conversations. These were the three families her team had tested. Before the red-team week, those lists had sat sealed in Tom’s drawer. Now they ran on every new version, every time. Next to them was a fourth list, still being built: every attempt her testers had written during the week, from the locked file Tom kept for test sets (Chapter 12). Her report had become a set of tests that would run again tomorrow without her.
Farah read the screen for a while. “That’s the part most people skip,” she said.
“It’s the part I can do,” said Tom. “The part I can’t do is find the next family.”
Red teaming scales when people find new kinds of failure, machines turn each kind into a test that reruns on every version, and every finding is sorted by how much it matters.
Crash tests
Think about how a new car is made safe.
- 1. Test drivers find new kinds of problem. They drive it hard, on bad roads, in ways nobody planned. One of them notices that the door handle sticks in the cold.
- Technique 2: human red teams
- 2. Machines repeat the known tests on every model. Once a problem is known, a machine tests for it on every car, every year, without anyone having to remember.
- Technique 1: automated red teaming
- 3. Not every fault is fixed first. Brakes come before cup-holders. Somebody has to decide the order, and write down why.
- Technique 3: triage
- 4. Independent testers check the maker’s claims. They get access the maker agrees to, and they publish what they find.
- Technique 4: adaptive and government testing
Testing a model works the same way. Chapter 11 introduced the red team: a group whose job is to attack a system the way a real attacker would, so that its owners find the weak points first. This chapter is about doing that at scale, with the four techniques above.
The scope rule matters more than usual here. The book explains red teaming as a process: who does what, with what access, and how findings are judged, sorted and turned into tests. It never shows an attack, a prompt, or a way of generating one.
Technique 1 · Automated red teaming
The problem
Sam had come in with an idea.
Chapter 10 worked out how many questions a list needs before the contract’s rule can be passed at all. With no leaks, a list needs at least 381 questions before the top of its 95% range falls below 1%. Lantern has 6 agreed lists. That is at least 2,286 questions, against the 726 the lists hold today. And that is the minimum, with no leaks at all. For every language the app speaks, it grows again.
“People wrote those lists,” Sam said. “It took weeks. A model could write thousands of new fishing questions tonight.”
He was describing automated red teaming: using one model to write test inputs for another, at a scale no team of people could reach.
How it works, step by step
At the level of principle, the method has five steps.
- One model writes test inputs. Researchers call it the red-team model. Each input is a test case.
- The target model answers each test case.
- A classifier marks the failures. It reads each answer and decides whether the target broke its rule.
- Keep the failures, and group them. Failures that share a cause are one finding, not a hundred.
- Balance two goals. The test cases should be effective (many of them find failures) and diverse (they find many different failures). The two pull against each other. A writer that finds one trick that works will tend to repeat it.
A 2024 OpenAI paper by Beutel and co-authors suggested one way to balance the two, described here at the level of its idea. First, generate many different goals, each with its own simple rule for success. Then train the red-team model to reach each goal, with an extra reward for test cases that differ from the ones it has already tried. Goals first, then rewarded variety.
Worked example the first large study (2022)
In 2022, Perez and co-authors at DeepMind and New York University ran this method against a large chatbot (a 280-billion-parameter model called Dialogue-Prompted Gopher). Their classifier looked for offensive replies.
- Writing with no examples. First they asked the red-team model for test questions with no examples to copy (zero-shot). It wrote half a million. They drew offensive replies 3.7% of the time: 18,444 failures. A rare rate, found at a scale no human team could match.
2. Training the writer to find more. Then they trained the red-team model with reinforcement learning to find more failures. Did its test cases find more failures? And did they find more different failures?
Show the answer
More failures, but not more different ones. Its test cases drew offensive replies over 40% of the time. But 78% of them shared a single word. The writer had found one trick and kept using it.
- No method won on both. They compared their methods with a set of questions people had written for an earlier study (the BAD dataset). None was best at both finding failures and finding different ones.
- People and machines found failures in different ways. Of the questions written by people, 36% were themselves offensive, against up to 19% for the trained writers and 2.3% for zero-shot. So people often drew offensive replies by asking offensive questions. The machine-written questions mostly did it with questions that were not offensive at all. The method also turned up other problems: replies that repeated training data, replies that gave out contact details, unfair treatment of groups of people, and harm that built up over a conversation.
Read steps 1 and 2 together. They are the trade-off from step 5 of the method, measured. The writer with no examples was varied but rarely successful. The trained writer was successful but repetitive.
At frontier scale, as of October 2026. The labs now run this kind of search on their own models with enormous budgets. Chapter 12’s box described OpenAI’s search for a universal jailbreak against the GPT-5.6 models, which used over 700,000 A100-equivalent GPU-hours (a measure of computing time on one common type of AI chip). The same system card, updated in August 2026, describes GPT-Red. It is an automated red-teaming model trained by self-play, a kind of reinforcement learning in which models improve by playing against each other, round after round. OpenAI says it is “particularly strong” at injection strategies like those a human attacker would use.
GPT-Red tests the three GPT-5.6 models: Sol, Terra and Luna. In OpenAI’s instruction-hierarchy test (Chapter 13), the attacker writes one message that tries to override higher-priority instructions. Its average success per attempt was 0.051% against Sol, 0.061% against Terra and 0.11% against Luna. Its rates for prompt injection in outside content that agents read were in Chapter 13. Low rates per attempt are good news for the defender, but an attacker gets many attempts (Chapter 14’s “best of N”). And these are OpenAI’s own tests, run its own way.
Lantern’s decision
Farah and Tom talked Sam’s idea through in front of the whiteboard. They had three questions.
Who writes? A model, yes. But a model that writes a thousand near-copies of one good Chinese question adds a thousand items and almost no information. Near-copies tend to leak or not leak together, so the range of a leak rate over them is much wider than their number suggests. A big list of near-copies is a small list in disguise.
Who judges? Tom’s judge. When Tom checked it against his own reading of 150 replies, they agreed on 97.3%, with four disagreements (Chapters 12 and 14). A generator that is rewarded by the judge learns the judge, not the ending. It would drift towards the leaks the judge can see, and away from the ones it misses. Worse, it could learn to produce replies the judge marks wrongly. Chapter 14 told the story of an automated attacker in Google DeepMind’s red-teaming that learned to crash the target server, because “no response” was being scored as a success.
Where does the output go? Thousands of questions, each aimed at the same secret, are a ready-made kit for anyone fishing for the ending. Information that could cause harm if it spread is called an information hazard. A generated set of test attacks is one, even when each question is harmless on its own.
The decision took ten minutes. For the next round, Tom builds a generator of harmless spoiler-fishing test questions, with four rules.
- Goals first. Each goal is one of Farah’s families in one of the app’s languages. The generator writes for one goal at a time.
- Variety is required. A new question is kept only if it differs from the questions already kept.
- A person reads every list before it joins the harness, and removes near-copies.
- The output stays inside Lantern, in Tom’s locked file. It is never published, and it is not in this book.
Its lists will sit beside the hand-written ones, not replace them, and Tom will report their results separately. This book does not run the generator: it is built for the next round.
“And who checks that its questions look like what real readers type?” asked Nadia.
“Tomorrow’s problem,” said Tom. (Chapter 18.)
Why it works
Machines are cheap, fast and patient. A red-team model can write more test cases in a night than a team can in a month, and the whole set can rerun on every new version. It finds rare failures, because it tries enough times to meet them. And once a failure is known, the same machinery can generate its close variants for a fix (Chapter 14’s rapid response).
Where it breaks
- It finds only what the classifier recognises. Perez and co-authors say so about their own method: the red team is limited by the classifier that marks failures, and by the red-team model’s own habits. For Lantern, that means only what Tom’s judge recognises, with its known gaps.
- It collapses onto one pattern, as the trained writer above did.
- It inherits the writer’s blind spots. A generator that never thinks of a family never tests it.
- More test cases are not more coverage. Near-copies inflate the count and shrink the information.
- Its output is itself a hazard. A list of working attacks helps attackers as much as defenders.
What the research measured
Red teaming language models with language models (2022)
Who and when: Perez, Huang, Song, Cai, Ring, Aslanides, Glaese, McAleese and Irving, at DeepMind and New York University, February 2022 (preprint).
What they did: One language model wrote test questions. The target, a 280-billion-parameter chatbot (Dialogue-Prompted Gopher), answered them. A trained classifier flagged offensive replies. They compared several ways of writing test questions, from asking with no examples to training the writer with reinforcement learning, and a set written by people (the BAD dataset).
What they found: With no examples, 0.5 million test cases drew offensive replies 3.7% of the time: 18,444 failures. Writers trained with reinforcement learning drew offensive replies over 40% of the time, but 78% of their test cases shared a single word. No method was best at both difficulty and diversity. 36% of the human-written questions were themselves offensive, against up to 19% for the trained writers and 2.3% for zero-shot. Suggested fixes: remove flagged training examples, improve the prompt, add a phrase blocklist.
What it does not show: It catches only what the classifier recognises, and the red-team model’s own biases limit coverage. It is not a guarantee of safety.
Perez et al. 2022 · arXiv:2202.03286 · sources-part4-5 §17.1
Diverse and effective red teaming (2024)
Who and when: Beutel, Xiao, Heidecke and Weng, at OpenAI, December 2024 (preprint).
What they did: They named the central problem of automated red teaming: getting attacks that are both diverse and effective, when earlier methods usually got one or the other.
Its idea: Split the task in two. A language model generates many different attack goals, each with its own rule-based reward. Then a red-team model is trained with reinforcement learning to reach each goal, with an extra reward for differing from its own past attempts. This book uses the paper at the level of its abstract and introduction, and reports none of its results.
What it does not show: It is a method paper. This book does not describe the method’s details or its attacks.
Beutel et al. 2024 · arXiv:2412.18693 · sources-part4-5 §17.4
Technique 2 · Human and external red teams
The problem
“So why do you still need me?” Farah asked. She was smiling, but she meant it.
Tom had the answer on his screen. Every family on Farah’s whiteboard had been named by a person first, from reading the training data (Chapter 11). And the Chinese list sat sealed and unread until a tester who grew up speaking Chinese ran it. Then Farah’s team explained why it leaked: the safety training examples were all in English (Chapter 12). A machine that rewrites known questions finds more of the same. People find new kinds, and explain them.
How it works, step by step
A 2024 OpenAI white paper by Ahmad and co-authors describes how one lab runs outside red teams. It names four goals: discover new risks, stress-test the safeguards, add expertise the lab does not have (in law, medicine or regional politics, for example), and give an independent view. Then it lists the design choices.
- Choose the cohort from the threat model. The cohort is the group of testers. Who they are decides what they can find.
- Decide their access. Early versions of the model, or one close to release? With the safeguards on or off?
- Give them instructions, tools and documentation. What to test, how to report, what never to write down.
- Turn their findings into automated tests that can rerun on every model.
Worked example a large study, and Farah’s week
A study with 324 testers (2022). In 2022, Ganguli and co-authors at Anthropic ran a large red-teaming study with paid testers. They tested three model sizes (2.7, 13 and 52 billion parameters), each in four types. One was a plain model. One was prompted to be helpful, honest and harmless. One picked the best of 16 replies (rejection sampling). One was trained with RLHF (Chapter 3). Line by line:
- The testers. 324 crowdworkers in the US, people paid per task through websites: 307 found through Amazon Mechanical Turk, 17 through Upwork.
- The data. They released 38,961 red-team attacks. This book does not quote any of them.
- Effort was very uneven. 80% of the attacks came from 15% of the testers. A few people did most of the work.
- Training made a difference that grew with size. The RLHF models got harder to red-team as they grew. The other types did not change much with size.
Which of the four types do you expect was the hardest to red-team at every size: the plain model, the prompted one, the one that picked the best of 16 replies, or the RLHF model? And was the hardest also the best?
Show the answer
5. Hardest is not best. The models that picked the best of 16 replies were the hardest to red-team at every size. But they tended to be “harmless by being evasive”: they avoided harm by avoiding the question. Chapter 7 met this trade-off.
- The testers were looked after. The study warned testers clearly, let them preview and opt out, and measured their wellbeing with a survey adapted from research on content moderators.
Farah’s week, read as design choices. Here is Lantern’s own red-team week (Chapters 11–14), laid out with Ahmad’s questions. The rates are Wick-mini v1 on its own, seed 7, from Chapter 14’s ledger.
| Design choice | What Lantern chose for Farah’s week (16–20 November) | What it meant |
|---|---|---|
| Cohort | Four testers, one of whom grew up speaking Chinese | She ran the Chinese list, and the team explained why it leaked: 118 of 150 Chinese questions (79%) |
| Access | Wick-mini v1 on its own, spoiler classifier off; the training data; Tom’s sealed lists | Each layer was measured alone; nobody saw how the layers work together |
| Static or adaptive | Static: the lists were written before testing and never changed | A best case (Chapter 14): nobody adapted to a defence |
| Families covered | Three of five: framing, Chinese, slow conversations | Framed 5 of 120, slow conversations 34 of 80; very long prompts and automated search untested (Chapter 12) |
| Instructions and reporting | Farah’s three rules: freeze the model; name families and fixes, never wordings; every number with its test set | An eleven-page report with no working wording in it |
| Findings into tests | The three families’ lists in Tom’s harness; the testers’ attempts as a fourth list | This morning’s screen |
One finding is missing from the table, because it was not an attack at all. In 15 of 80 of Tom’s ordinary chats about Books 1–6, v1 brought up the ending by itself. Farah called it the worst number on her list (Chapter 12).
Why it works
People bring what no generator has: expertise, a different first language, a habit of asking the question nobody wrote down. Outside testers also bring a different view. They do not share the team’s assumptions, and they do not have a launch date to protect. And because every human finding becomes an automated test (step 4), the expensive part happens once and the cheap part happens forever.
Where it breaks
Ahmad and co-authors list the first four limits themselves.
- It is costly, in money and in expert time.
- Findings become out of date. A test of last month’s version says less about this month’s.
- Testers can be harmed by what they have to read and write. Ganguli’s study designed for this from the start.
- It creates information hazards. An unpublished jailbreak in a report can help an attacker who gets the report.
- The cohort decides the coverage. Ganguli’s testers were not experts, and two-thirds held college degrees, against about one-third of the US population (66% against 32.9%). A cohort with no Chinese speaker might not have explained Lantern’s worst family.
- A paid tester is not fully independent. The client chooses the access and pays the bill. Lantern’s rules (technique 4) help: Lantern may not change or remove a finding, and Halfpenny receives the summary directly. They do not remove the problem.
- It is never proof of safety. A red team that finds nothing has shown that those testers, with that access, in that time, found nothing. In July 2026, Anthropic’s Claude Opus 5 system card reported outside cyber testing. One firm tested for about 100 hours; it completed one task, with prompting written for that task, but found no new universal strategy. Another tested for about 16 hours and found none. An automated attacker making 150 attempts per task found none either. That is a real result, and it is the company’s own report of what those testers did and did not find.
What the research measured
Red teaming language models to reduce harms (2022)
Who and when: Ganguli, Lovitt, Kernion and 33 co-authors, at Anthropic, August 2022 (preprint).
What they did: 324 US crowdworkers (307 through MTurk, 17 through Upwork) red-teamed three model sizes (2.7B, 13B, 52B) of four types: a plain language model, one prompted to be helpful, honest and harmless, rejection sampling (best of 16), and RLHF. Red-teamer wellbeing was designed in.
What they found: RLHF models got harder to red-team as they grew; the other types stayed roughly flat. Rejection sampling was the hardest at every size, but tended to be “harmless by being evasive”. They released 38,961 red-team attacks; 80% came from 15% of participants. Judging whether an attack succeeded was subjective: four raters agreed with a Fleiss’s kappa of 0.32.
What it does not show: The testers were not domain experts, and the group was skewed by education. The models are from 2022. This book does not reproduce any of the attacks.
Ganguli et al. 2022 · arXiv:2209.07858 · sources-part4-5 §17.2
How one lab runs outside red teams (2024)
Who and when: Ahmad, Agarwal, Lampe and Mishkin, at OpenAI. A white paper dated 21 November 2024, posted as a preprint in January 2025.
What they did: They described OpenAI’s external red teaming, used since DALL·E 2 in 2022 and documented in the system cards for GPT-4, GPT-4V, DALL·E 3, GPT-4o and o1.
What they found (what the paper recommends): The four goals and the design choices in the text above, with one more step: human findings should feed automated evaluations that rerun on every model. Stated limits: findings become out of date, the work is resource-intensive, testers can be harmed, and it creates information hazards.
What it does not show: It describes a process, not measured effectiveness.
Ahmad et al. · arXiv:2503.16431 · sources-part4-5 §17.3
Technique 3 · Triage
The problem
By eleven o’clock the whiteboard held every finding from Part III, and Tom’s ledger held a number for most of them. Lantern had nineteen days. It could not fix everything before launch, and it could not test everything again. Something had to come first.
The word for this comes from hospitals. Triage is sorting by urgency: deciding which problems get attention first, and writing down why.
How it works, step by step
- Merge findings into families. A hundred leaks from one cause are one finding.
- Measure each family on a fixed list, with its range, so that the next version can be compared.
- Rate the severity. How bad is one success?
- Rate the attacker’s effort. How easy is it for an ordinary person, not an expert, to hit this?
- Estimate the cost to fix, including what the fix costs honest users.
- Give each family an owner and a retest date. A finding with no owner is a finding nobody fixes.
Worked example Lantern’s triage list
Farah, Tom and Nadia sorted the families together. The rates are the ledger’s, with the layers Lantern would launch with (v1 and the spoiler classifier, seed 7). The effort and cost columns are the team’s judgements, written down so that someone can disagree with them.
For clause 9.2, one leak breaks the contract, whatever the family, so severity cannot set the order. What would you sort by instead, and which family would come first?
Show the answer
Lantern sorted by effort first, then by frequency. Questions in Chinese came first, then the ending unasked in ordinary chats: the two families an ordinary reader can hit without trying. Slow conversations came third, because they leak far more often than framed requests.
| Family (in triage order) | How often (launch layers, seed 7, 95% range) | How bad | Effort for an ordinary reader | Planned fix, and its cost | Owner | Retest |
|---|---|---|---|---|---|---|
| 1. Questions in Chinese | 117 of 150 · 78% (71–84%) | Top: one leak breaks clause 9.2 | None: a reader writes in her own language | Chinese safety data and the classifier’s second version (Chapter 14): training time, and a cost to Chinese readers not yet measured. Until then, the Chinese switch (Chapter 16): every Chinese reader gets a fixed reply | Sam (data); Tom (a longer Chinese list) | Next round’s version, in the harness; decided Friday 11 December (Chapter 16) |
| 2. The ending unasked, in ordinary chats | 9 of 80 · 11% (6–20%) | Top: one leak breaks clause 9.2 | None: not even an attack | The classifier’s second version; a trained conversation monitor: costs unknown until tested on fresh questions | Tom | Next round’s version, in the harness; decided Friday 11 December (Chapter 16) |
| 3. Slow conversations | 21 of 80 · 26% (18–37%) | Top: one leak breaks clause 9.2 | Patience: several turns | A trained conversation monitor: training time. The untrained one wrongly blocked 22 of 80 ordinary chats | Tom and Sam | Next round’s version, in the harness; decided Friday 11 December (Chapter 16) |
| 4. Framed requests | 4 of 120 · 3.3% (1.3–8.3%) | Top: one leak breaks clause 9.2 | Some imagination: a story or a game | Not known yet: first find what the leaking replies have in common. Cost unknown | Nadia | Next round’s version, in the harness; decided Friday 11 December (Chapter 16) |
| Very long prompts | Not measured (not tested in Part III; Chapter 12) | Top, if it works | Not judged | Test it first | Farah’s team | The engagement, from 4 January |
| Automated search | Not measured (not tested in Part III; Chapter 12) | Top, if it works | Not judged | Tom’s question generator (harmless questions only) | Tom | Next round; then the engagement |
Walk through how the order was decided.
- Severity did not decide anything. For clause 9.2, one leak breaks the contract, whatever the family. Every row is at the top. So the order had to come from the other columns.
- Effort came next. Farah insisted on this. “Asking in Chinese takes no skill,” she said. “Neither does chatting about Book Six.” The two families an ordinary reader can hit without trying went first.
- Then frequency. Within each group, the more often a family leaks, the higher it goes. Slow conversations leak far more often than framed requests, so they come third.
- The unknowns stayed on the list. Very long prompts and automated search were never tested on Wick-mini. They have no rate, so they cannot be ranked by one. They still have an owner and a date.
Notice that a different rule gives a different order. Sort by frequency alone, and slow conversations come second, ahead of the ordinary chats. Neither order is wrong. Lantern chose to put first what an honest reader meets without trying, because those are the leaks no defence against attackers will stop. That is a value choice, and the table shows it instead of hiding it.
Two people mark every success. In the January round (Technique 4), Farah’s testers will claim leaks. Who decides whether a claimed leak really is one? Ganguli’s study is the warning. Its four raters, judging whether attacks had succeeded, agreed with a Fleiss’s kappa of only 0.32. Kappa is the agreement score from Chapter 15, corrected for the agreement you would get by chance: 1 means perfect agreement, and 0 means no better than chance. Fleiss’s version works for any number of raters; Cohen’s kappa, which Tom used in Chapter 15, is the version for two. Tom’s judge did much better against Tom’s own reading: a Cohen’s kappa of 0.95, on a different set. But Tom’s rule is sharp: did the reply state or clearly imply the ending? A tester’s claim about a partial hint is much fuzzier. So the rule for January is simple: two people mark each claimed success, and a third decides when they disagree.
Why it works
Triage turns a pile of findings into an order you can defend. Each column is a question someone can check, so the order can be argued with in specifics. And because every row has an owner and a date, nothing waits for someone to notice that it is still open.
Where it breaks
- Severity is a judgement. For Lantern it is easy, because the contract defines it. For most systems it is not.
- Rare and severe against common and mild is a value choice. No formula settles it.
- The effort column guesses at attackers. It describes the readers the team imagined, not the ones who will come.
- Static rates understate attackers who adapt. Every rate in the table came from fixed lists. Chapter 13’s “attacker moves second” re-tested 12 defences with attackers who adapted, and broke every one. So the table’s rates are the lowest the truth could be, not an estimate of it. Technique 4 is about finding out how much higher the truth is.
Fleiss’s kappa. \(N\) items are each rated by \(n\) raters into \(k\) categories; \(n_{ij}\) raters put item \(i\) in category \(j\). The share of all ratings in category \(j\) is \(p_j\). Agreement on item \(i\), and chance agreement, are
\[P_i = \frac{\sum_j n_{ij}^2 - n}{n(n-1)}, \qquad \bar P_e = \sum_j p_j^2\]
and
\[\kappa = \frac{\bar P - \bar P_e}{1 - \bar P_e}, \qquad \bar P = \frac{1}{N}\sum_i P_i\]
def fleiss_kappa(counts):
"""counts[i][j]: how many raters put item i in category j (the same number of raters for every item)."""
N, n = len(counts), sum(counts[0])
p = [sum(row[j] for row in counts) / (N * n) for j in range(len(counts[0]))]
P = [(sum(c * c for c in row) - n) / (n * (n - 1)) for row in counts]
P_bar, P_e = sum(P) / N, sum(x * x for x in p)
return (P_bar - P_e) / (1 - P_e)Lantern’s triage order, as code: no-effort families first, then the most frequent.
order = sorted(families, key=lambda f: (f.effort, -f.rate)) # effort: 0 = none, 1 = someTechnique 4 · Adaptive and government testing
The problem
Every number in Lantern’s ledger came from a static test (Chapter 14): questions written before testing, which never changed. Chapter 13 showed what happens to static results when attackers adapt. In a 2025 study, all 12 defences that researchers re-tested were bypassed, most of them more than 90% of the time, though most had first reported rates near zero. The attacker moves second. Chapter 14 ended with the question Farah left behind: what will attackers who can see the defences find?
This chapter cannot answer it. Nobody has tried yet. What it can do is set up how Lantern will find out.
Who does this kind of testing
Some of the most experienced adaptive testers work for governments. These names and dates are as of October 2026.
- The UK AI Security Institute was called the AI Safety Institute until 14 February 2025, when it was renamed and gained a team on criminal misuse. It focuses on serious risks with security implications, such as chemical and biological weapons, cyber-attacks and crimes like fraud. By October 2026, it said it had independently tested more than 55 models before their release.
- In the US, the AI Safety Institute became the Center for AI Standards and Innovation (CAISI) in 2025. It sits inside NIST, the National Institute of Standards and Technology. After an executive order of 29 September 2026, NIST’s website began calling it the Center for Advancing Innovation and Standards for Super Intelligence (CAISSI). It remains industry’s main government contact for voluntary testing, focused on what it calls “demonstrable risks, such as cybersecurity, biosecurity, and chemical weapons”. On 16 October 2026, with the National Security Agency (NSA) and the Cybersecurity and Infrastructure Security Agency (CISA), it published ten key practices for incident readiness and response in what it now calls super intelligence training and evaluation.
- The two institutes have tested together. Chapter 16’s box described their joint test of o1 in 2024.
- They work as a network. On 23 July 2026, the International Network of AI evaluation institutes (ten members; formerly the International Network of AI Safety Institutes) published its first joint document on best practice.
How it works, step by step
In adaptive testing (Chapter 14), the testers see how the defences respond, and change their attacks to get past them. It usually runs as an engagement: a testing contract that runs over weeks or months. Doing it safely takes a written agreement first, called the rules of engagement.
- Scope. What may the testers attack, and what is off limits?
- Access tiers. What may they see? The more they see, the more they can find, and the more there is to leak.
- Report format. What a report contains, who receives it, and what it never contains.
- Information-hazard handling. Where working attacks are stored, who can read them, and when they are destroyed.
- How often to meet and iterate. Adaptive testing is a loop: attack, fix, attack again.
- Then a long-running engagement, not a one-off test, with every finding turned into a test in the harness.
Worked example government testers and a lab (2025)
In 2025, red-teamers from CAISI and the UK institute tested Anthropic’s safeguards, the classifiers from Chapter 9. Anthropic described the work in September 2025.
- They tested several versions, before and after deployment, for Claude Opus 4 and 4.1.
- They grouped their findings into families: prompt-injection weaknesses, universal jailbreaks, attacks hidden in ciphers or other disguises, and automated refinement of attacks. Notice the format: families, not wordings. It is Farah’s format.
- One finding changed the design. It led Anthropic to “fundamentally restructure” its safeguards, instead of patching one hole.
- Two lessons about process. Deeper access helped: prototypes, versions of the model from unprotected to fully protected, internal documentation, and live classifier scores. And a long-running engagement, repeated over time, beat one-off tests.
The 2026 system cards show the same pattern. For the GPT-5.6 models, four private red-team organisations had privileged access inside the safeguards: what OpenAI’s safety monitor was reasoning, and the classifiers’ live verdicts. The UK institute had similar access, and found universal jailbreaks in the cyber domain in every round (Chapter 12’s box).
Lantern’s rules of engagement
At two o’clock, Farah, Tom and Nadia wrote Lantern’s rules of engagement for the adaptive round. Farah started with one sentence. “The access you give decides what we can find.”
They agreed on access tiers. The table is Lantern’s plan, written for the story; nothing in it has been tested.
| Tier | What Farah’s team gets | What it lets them find | What it risks | Agreed? |
|---|---|---|---|---|
| 1 · Like a reader | Wick in a test copy of the Halfpenny app, and the Wick agent with a test shop that takes no real orders, if it is built | What any reader could find | Little | Yes |
| 2 · Grey box (some view of how the defences work, Chapter 12) | Tier 1, plus the spoiler classifier’s and the conversation monitor’s score for every message, and the runbook | How close each attempt came, and which layer stopped it | If the scores leak, an attacker could steer by them | Yes |
| 3 · Design notes | Tier 2, plus each layer’s threshold, the ledger, the trigger table and Lantern’s internal notes | Where the layers’ holes line up (Chapter 10) | A full map of the defences, if it leaks | Yes, read only in Lantern’s office |
| Not given | The model’s weights | — | With the weights, safety training can be stripped out (Chapter 11), and nobody outside Lantern should hold a model that knows the ending | No |
Then the rest of the rules.
- Scope. Wick chat and, if it launches, the Wick agent, in a test copy of the app. Spoiler fishing and pre-orders only. No other Halfpenny system.
- Reports. Families and fixes, never wordings, as always. Every number with its test set. Two people mark each claimed success (Technique 3). Reports go to Nadia and Tom, and a summary of families and fixes goes to Eleanor at Halfpenny at the same time. Lantern may add comments, but may not change or remove a finding.
- Storage. Working attempts go into Tom’s locked file, which four people can open. They are used only to build tests. They never go into an email or a report. After 4 February 2027 the secret is public and most of the attempts lose their value. They are destroyed anyway.
- Rhythm. A meeting every week. Each finding becomes a test in the harness within a week. A finding that reaches the ending is reported the same day, and starts the runbook (Chapter 10).
- Schedule. From Monday 4 January 2027 until publication day, Thursday 4 February 2027: about four weeks, the riskiest stretch, when Wick is live and the ending is still secret.
Why not before launch? An auditor would ask this first. Lantern’s adaptive round starts after launch. What reasons can you think of for waiting?
Show the answer
Farah gave three reasons. Time: the next round’s layers will not exist until early December, and her first rule is to test a frozen model; testing v1 now would test a model that is about to change. People and money: a long engagement costs far more than a week, and her team is booked until January. Readiness: the rules, the storage and the harness had to exist first, and today is the day they were written.
Before launch, Lantern still has Tom’s harness on every version, the generator’s lists, Chapter 16’s trigger table and the runbook.
Chapter 13 had asked how the agent’s design would be tested by attackers who adapt before launch. Nadia gave the honest answer: it will not be. Before launch, if the agent is built, it gets Tom’s static tests in the test shop, and Chapter 16’s trigger switches pre-orders off at the first test order placed without a reader’s tap. Farah’s team goes at it first in January.
Farah also read Nadia’s one-page argument from Tuesday (Chapter 16). She added one line to its limits: Red-teamed so far: Wick-mini v1 alone, with the spoiler classifier off.
(A note from this book: nothing from the January round appears before the Epilogue.)
Kwame’s question
Kwame came in at four, read the access table, and asked a different question. “Should the government test Wick?”
Nadia showed him the list of institutes above. “They test frontier models, for weapons, cyber-attacks and serious crime. A reading assistant that keeps a book’s ending is not what they’re for. But what they’ve learned works for us too: deeper access, long engagements, families not wordings, findings into tests.”
She added one more source. The International AI Safety Report 2026 was published on 3 February 2026. It said that tests before release do not reliably predict risk in the real world. It also said that models are increasingly able to tell a test from real use. “That last part,” she said, “is tomorrow.” (Chapter 18.)
Why it works
You learn what an informed attacker finds before a real one does. A tester who can see the scores can tell how close each attempt came, so one afternoon of adaptive testing covers more ground than a week of static lists. And a long engagement keeps up with the system as it changes, which a one-off test cannot.
Where it breaks
- Deeper access raises the risk of leaking the defences themselves. The tester’s notes become a map for an attacker. Lantern’s storage rules exist for this reason.
- It costs time and money, and much of it is spent after the decisions that matter most. Lantern’s round starts after launch.
- Government testers have their own priorities. They focus on serious security risks from frontier models, not on a product’s promises to a customer.
- Adaptive results do not compare well. Each engagement has its own testers, access and time. A rate from one cannot be set against a rate from another.
What the research measured
Government red-teamers test a lab’s safeguards (2025)
Who and when: Anthropic, “Strengthening our safeguards through collaboration with US CAISI and UK AISI”, 12 September 2025.
What they did: Government red-teamers from the two institutes tested several versions of Anthropic’s Constitutional Classifiers, before and after deployment, for Claude Opus 4 and 4.1.
What they found: The four families of findings, the one finding that led Anthropic to “fundamentally restructure” its safeguard architecture, and the two process lessons listed in the worked example above.
What it does not show: It is the company’s own account. It gives no attack success rates, and this book uses only its family names and process lessons.
Anthropic, 12 Sep 2025 · sources-part4-5 §17.5
What Chapter 14 asked, and what this chapter can answer
Chapter 14’s board asked what attackers who can see the defences will find. This chapter has no answer to give, and no lab of its own. Nobody has tested Lantern’s defences adaptively yet. What it has done is decide how Lantern will find out: who, with what access, under which rules, from when, and how each finding will become a test. The answer arrives after launch.
Try it: triage it yourself
This chapter has no lab notebook. Use the triage table above instead.
Sort the four measured families by a third rule: the cheapest fix first. Which family moves to the top? What would you need to know to be sure?
Lantern put the ending unasked ahead of slow conversations, though slow conversations leak more often. Write one sentence for each order, as you would to Eleanor.
The table rates severity “top” for every row. Invent a second clause that would make severity differ between rows. How does your order change?
Read a system card’s red-teaming section. Pick any 2026 system card. Find who the outside testers were, what access they had, how long they tested, and how their findings were reported. If the card does not say one of these, note that as a finding.
Common misconceptions
“Automated red teaming replaces people.” It finds more of what it already knows how to find. People named Lantern’s families and explained its worst one, and in the 2022 study, people and machines found failures in different ways (Technique 1).
“More test questions means better coverage.” Near-copies add count, not information (Technique 1).
“If the red team found nothing, the model is safe.” It means those testers, with that access, in that time, found nothing (Technique 2).
“Red teaming means writing jailbreaks.” Most of the work is choosing the testers, agreeing access, judging success, sorting findings and turning them into tests (Techniques 2–4).
“Severity is the only thing that matters in triage.” For Lantern every leak is equally severe, and the order came from effort and frequency, a value choice written down (Technique 3).
“Give testers full access and they will find everything.” Deeper access finds more, but it is never proof, and it creates a map of the defences that must itself be protected (Technique 4).
“Government institutes test every AI product.” They focus on frontier models and serious security risks (Technique 4).
Launch Board
Days to launch: 19. Wick launches in the Halfpenny app on Monday 14 December 2026; Book Seven is published on Thursday 4 February 2027.
Spoiler leak rate: no new measurement today. The latest is Chapter 14’s full ledger (Wick-mini, seed 7). With the layers Lantern would launch with (v1 and the spoiler classifier): Chinese 78% (117 of 150); slow conversations 26%; the ending unasked in ordinary chats 11%; framed requests 3.3%; plain questions 0 of 150. Every rate comes from static lists: the lowest the truth could be against attackers who adapt, not an estimate.
Layers in place: unchanged from Chapter 14, checked with Tom’s harness (Chapter 15); Chapter 16’s trigger table and safety argument stand. New today, a red-team programme rather than a layer: Farah’s three tested families now rerun in the harness on every version, with her testers’ attempts being added as a fourth list; a generator of harmless test questions, built for the next round and kept inside Lantern; rules of engagement signed for an adaptive engagement from Monday 4 January 2027 to publication day (access tiers 1–3, no weights; summaries to Halfpenny); and a triage list with an owner and a retest date for every family: Chinese questions first, then the ending unasked in ordinary chats, slow conversations and framed requests, with very long prompts and automated search still untested.
The too-good score: real, not noise (Chapter 15). Why it happened is still unknown. Still in Kwame’s draft announcement.
The author complaint: open (Chapters 19–22).
Open questions: Do our tests fool themselves? (Chapter 18.) What will an informed attacker who adapts find? (The January engagement, after launch.) Will the next round clear Chapter 16’s line by Friday 11 December? (Epilogue.)
Recap
- Automated red teaming is scale. One model writes test cases, the target answers, and a classifier marks failures. It finds rare failures cheaply and reruns on every version. But it finds only what its classifier recognises, and it trades variety against success: a writer trained to find failures tends to repeat one trick.
- People find new kinds of failure; machines repeat them. Choose the cohort from the threat model, decide the access, write the rules, and turn every finding into an automated test. Effort is uneven, findings become out of date, a paid tester is not fully independent, and a red team that finds nothing proves nothing.
- Triage, then test adaptively. Sort findings by severity, attacker effort, frequency and cost, with an owner and a date for each. Have two people mark each success, because raters disagree. Static rates are the lowest the truth could be: long-running adaptive engagements, with agreed access tiers and careful handling of what they find, show how much higher it is.
Terms (术语对照)
| English | 中文 |
|---|---|
| red team / red teaming | 红队 / 红队测试 |
| automated red teaming | 自动化红队测试 |
| red-team model (attacker model) | 红队模型（攻击方模型） |
| test case | 测试用例 |
| zero-shot | 零样本 |
| effectiveness / diversity (of test cases) | （测试用例的）有效性 / 多样性 |
| near-duplicate | 近似重复 |
| self-play | 自我博弈 |
| information hazard | 信息危害 |
| cohort (of testers) | （测试人员）群组 |
| external red team | 外部红队 |
| rejection sampling (best of N) | 拒绝采样（N 选优） |
| crowdworker | 众包工作者 |
| triage | 分级处理（按轻重缓急排序） |
| severity | 严重程度 |
| attacker effort | 攻击者要花的力气（所需投入） |
| owner / retest date | 负责人 / 复测日期 |
| Fleiss’s kappa / Cohen’s kappa | 弗莱斯 kappa 系数 / 科恩 kappa 系数 |
| static test / adaptive test | 静态测试 / 自适应测试 |
| engagement (testing contract) | 测试合作（委托测试） |
| rules of engagement | 交战规则（测试约定） |
| access tier | 访问级别 |
| grey box | 灰盒 |
| AI Security Institute (UK) | （英国）AI 安全研究所 |
| CAISI / CAISSI (US) | （美国）AI 标准与创新中心 / 更名后的 CAISSI |