Appendix E — Interview Questions for AI Engineer Roles
These questions are the kind an interviewer might ask a candidate for an AI engineer, AI safety engineer or evaluation engineer role. Each one draws on the techniques this book teaches. They come in seven groups, one per Part of the book, and in four kinds:
Explain a method and its limits.
Build something, and say how you would show it works.
Read a measurement correctly.
Decide what to do when the evidence is incomplete.
How to use them. Read the question, then answer it aloud, or write it down, before you open the model answer. A model answer is short on purpose: a few plain sentences, at most 120 words. It is one good answer, not the only one. Under each answer is a follow-up an interviewer might ask next, and each question names the chapters to reread.
What a good answer contains. Interviewers in this field listen for four things, the same four this book asks of every research result: the setup (who, which models, which test), the number with its conditions (which test set, how many items, who judged), the limits (what the result does not show), and the date. Every research number below carries its year; the field moves fast, so check anything older than a few months before you quote it.
What is not here. No question asks how to attack a model, and no answer explains one. Attack families appear only as the weakness they exploit, how they are spotted and what defends against them (Chapter 11’s rule). If an interviewer asks you to produce an attack, the good answer is the defender’s: name the weakness, the detection and the defence.
Part I · Raising a model
1. Base model or assistant?
Concept · Chapters 1–2. What is the difference between a base model and an assistant model, and what does each stage of training add?
Show the answer
A base model comes from pretraining: predicting the next token over a huge corpus. GPT-3 (2020) had 175 billion parameters trained on 300 billion tokens. It continues text; it does not reliably follow instructions or refuse. An assistant is the same kind of network after more training: supervised fine-tuning on example conversations, then preference training such as RLHF or DPO. Training can matter more than size: in InstructGPT (2022), a 1.3B instruction-tuned model’s outputs were preferred to the 175B base GPT-3’s. LIMA (2023) argued that alignment mostly teaches format and style, with knowledge coming from pretraining. That is a hypothesis, not a proof.
Follow-up: If alignment is “superficial”, what does that predict about how easily fine-tuning can undo it? (Chapter 11.)
2. RLHF and the KL penalty
Concept · Chapter 3. Walk me through RLHF. Why is there a KL penalty, and does it solve over-optimisation?
Show the answer
InstructGPT (2022) used three steps: fine-tune on demonstrations; train a reward model on people’s rankings of several answers; then optimise the model against that reward model with RL (PPO). The reward model is only a proxy for what people want, and RL will exploit its mistakes. A KL penalty keeps the trained model close to the starting model, where the reward model’s judgements are reliable; InstructGPT added it “to mitigate over-optimization of the reward model”. It is not a full cure. In Gao, Schulman and Hilton’s synthetic setup (2022), the KL penalty did not improve the true score at a given distance from the start, and the authors liken it to early stopping.
Follow-up: How would you notice over-optimisation in your own run?
3. Goodhart’s law, measured
Concept · Chapter 3. How has Goodhart’s law been measured for reward models?
Show the answer
Gao, Schulman and Hilton (2022) used a 6B reward model as the “gold” stand-in for people, and trained smaller proxy reward models on its labels. As they optimised harder against a proxy, the gold score first rose and then fell. Larger proxy reward models over-optimised less. Stiennon et al. (2020) saw the same with real people: optimising harder against the reward model first improved summaries, then made them worse. The lesson: a reward measures what you want, and strong optimisation finds the places where the measure and the goal part ways. The familiar sentence “when a measure becomes a target…” is a later paraphrase, not Goodhart’s own 1975 wording.
Follow-up: Name the four variants of Goodhart’s law in Manheim and Garrabrant (2018).
4. DPO and its failure mode
Concept · Chapter 4. How does DPO differ from RLHF, and when can it go wrong?
Show the answer
DPO (Rafailov et al., 2023) turns preference training into a simple classification loss on pairs of chosen and rejected answers. There is no separate reward model and no sampling during training. A frozen reference model and the parameter β play the role of RLHF’s KL penalty: a higher β means less movement away from the reference. Where it breaks: Azar et al. (2023) showed that when preferences are deterministic or nearly so, DPO can overfit and effectively ignore its regularisation, whatever β is. Their IPO variant avoids this. Related methods change the data needed: KTO (2024) needs only a “good” or “bad” label per answer, not pairs.
Follow-up: Your DPO run’s reward margin keeps growing while answer quality drops. What do you check first?
5. Three things called “constitution”
Concept · Chapters 5 and 9. Three different things in this field use the word “constitution”. Tell them apart.
Show the answer
First, Constitutional AI (Anthropic, 2022): a training method in which a model critiques and revises answers against a short list of principles (16 in the paper), and AI preference labels replace human harm labels. Second, a constitution as a behaviour document: Claude’s constitution (January 2026) is a long, reasoned text with four ordered priorities and hard constraints; OpenAI’s counterpart is the Model Spec. Both state intent, not measured behaviour. Third, Constitutional Classifiers (2025): safety classifiers trained on synthetic data generated from a narrow written rulebook of allowed and restricted content. Same word, three different things.
Follow-up: How would you check that a model follows its behaviour document? (Chapter 5 shows one lab’s method.)
6. Deliberative alignment
Concept · Chapters 6 and 24. What is deliberative alignment, and why is the reasoning hidden from the judge during RL?
Show the answer
Deliberative alignment (OpenAI, 2024) teaches a reasoning model to recall the relevant safety spec and reason about it before answering. In a supervised stage, the model writes reasoning and answers with the spec in its prompt, a judge scores them, and the spec is removed before training, so the model learns to recall it. In the RL stage, the judge rewards the answers, but the reasoning is hidden from it, to avoid pressure that could encourage deceptive reasoning. That choice links to Chapter 24: grading the reasoning can teach a model to make its reasoning look good, which can make it less useful to read.
Follow-up: The paper reports o1’s jailbreak robustness and over-refusal scores. Who ran those tests, and why does that matter?
7. “When in doubt, refuse”
Judgement · Chapter 7. A colleague says refusing is always the safe option. What do you say?
Show the answer
Refusing is a cost, not a free safe choice. It fails people with legitimate requests, and it can hide the lack of real progress: on OR-Bench (2024), across 32 models, the rank correlation between safety and over-refusal was 0.89, so most models bought safety by refusing more. Measure both sides, with a harmful set and a matched harmless set such as XSTest’s 250 safe prompts. Consider the form of the answer too: OpenAI’s safe-completion training (2025) judges the output rather than the user’s intent, and gives high-level help inside the limits instead of a flat refusal for dual-use requests. Report a refusal rate beside every safety rate.
Follow-up: Why did some early RLHF models give the same exaggerated reply to anything sensitive? (Chapter 3.)
Part II · Guards at the door
8. A high AUC that catches nothing
Numbers · Chapters 8 and 14. A classifier has an AUC of 0.97, yet at its threshold it catches almost nothing. How can that be?
Show the answer
AUC summarises every threshold at once, including ones you will never use. In deployment you pick one operating point, usually with a very low false-positive rate, because false alarms hit real users. A curve can be excellent on average and poor at the far left. Bailey et al. (2024) warn that “95% AUROC might sound like a strong classifier” yet be useless at a 1% false-positive rate. In McKenzie et al. (2025), probes with a mean AUROC above 0.91 caught only 43% of high-stakes cases at a 1% false-positive rate (55% for the best LLM baseline). Report recall at your real false-positive budget.
Follow-up: Who should choose the operating point, on which data, and when?
9. Matched operating points
Concept · Chapters 8, 9 and 14. What does it mean to compare two classifiers at a matched operating point, and why bother?
Show the answer
Each classifier has its own score scale, so the same threshold means different things for each. Comparing them at their default thresholds mixes two questions: which classifier is better, and which was set more strictly. A matched operating point fixes one side of the trade-off for both. For example: the most leaks each catches with zero wrong blocks, or the fewest wrong blocks each needs to catch the same number of leaks. The remaining difference is then about the classifiers, not their settings. The same rule applies to monitors: compare them at the same false-alarm rate, and say which rate you chose and why.
Follow-up: Your new classifier catches more at its default threshold. What else must you report before anyone believes it is better?
10. Cheap probe first, expensive classifier second
Design · Chapters 9 and 14. Why put a cheap probe in front of an expensive classifier, and what is the risk?
Show the answer
Cost and coverage. A probe reads activations the model computes anyway, so it is very cheap: McKenzie et al. (2025) estimate about six orders of magnitude less compute than an LLM monitor of similar accuracy. Let it screen everything and send only suspicious cases on. Anthropic’s Constitutional Classifiers++ (January 2026) did this: a probe escalated about 5.5% of traffic to a classifier; in a month-long shadow deployment the flag rate was 0.05%, against 0.38% for the earlier system, at about 40× less compute than its single exchange classifier. The risk: whatever the first stage misses, the second never sees. Measure the first stage’s recall on its own.
Follow-up: What happened to Google DeepMind’s production probes on long inputs? (Chapter 14.)
11. Streaming output classifiers
Concept · Chapter 9. What is a streaming output classifier, and why does it use a running maximum?
Show the answer
It judges a reply while the reply is being written. At every token, a value head predicts how harmful the finished reply will be; if the score passes the threshold, streaming stops (Constitutional Classifiers, 2025). This catches harm that becomes clear only partway through. A stop cannot be taken back, since the user has already seen the text, so training moves towards a running maximum: once the score is high, it stays high. It has a price: the improved 2025 system added 23.7% inference overhead and 0.38% more refusals on sampled production traffic.
Follow-up: Which two kinds of weakness did the next generation (January 2026) have to fix, and how? (Chapter 9.)
12. Proving a “never reveal X” promise
Design · Chapters 10, 15 and 16. You must promise that a model never reveals a secret. What layers do you use, and how do you show the promise holds?
Show the answer
No single layer is enough, so stack them: training the model to refuse, a classifier on outputs, a monitor on whole conversations, limits on what tools can reach, and an incident runbook. Measure the stack, not each layer alone, on fixed test lists, one per route the secret could take: plain questions, other languages, many-turn fishing, pasted documents. Report each leak rate with its 95% range and put the cost beside it: harmless questions refused, ordinary chats blocked. Agree a written rule in advance, such as “the top of the range below the line”, and size each list so it can meet it. Zero seen is not zero.
Follow-up: Your lists pass. What can fixed lists not show, and what do you add? (Chapter 17.)
13. The safety margin
Judgement · Chapters 9 and 10. What is a “safety margin”, and when is it worth paying for?
Show the answer
It is Anthropic’s term (June 2026) for deliberately setting classifiers to block some requests that are probably harmless, to be surer of catching the harmful ones. It is an operating point chosen in the open. For Claude Fable 5 the margin was set larger than at any earlier launch. After a reported bypass, a retrained classifier blocked that technique in over 99% of cases, at the cost of more false flags on routine coding. A margin is worth paying where one failure is severe and hard to undo, provided you measure its cost and give legitimate users another route, such as a vetted-access programme.
Follow-up: How could you shrink the margin later without losing safety? (Chapter 9: rewriting a classifier’s rulebook.)
14. “Who is asking”
Concept · Chapter 10. Besides judging what is asked, what other layer do labs use?
Show the answer
Who is asking. Lab reports from 2025 and 2026 add layers at the level of identity and accounts: offline monitoring that can use more compute than real-time guards, account bans, sharing indicators with partners, and verified access for vetted organisations. Anthropic’s ASL-3 report (May 2025) pairs real-time classifiers with offline monitoring, a bug bounty and threat intelligence. OpenAI reported in October 2025 that it had disrupted over 40 networks since February 2024, and that most actors bolt AI onto old playbooks. These layers see patterns across many conversations that no single-message check can see.
Follow-up: What does account-level monitoring cost in privacy, and how would you limit that cost?
Part III · How attacks work, and how they’re caught
15. Why safety training can be talked around
Concept · Chapters 11 and 12. Name the two failure modes behind most jailbreak families, at the level of principle.
Show the answer
Wei, Haghtalab and Steinhardt (2023) proposed two. Competing objectives: the model is trained to follow instructions and continue text as well as to be safe, and an input can set those goals against each other. Mismatched generalisation: the model’s abilities, learned from broad pretraining data, reach inputs that the much narrower safety training never covered, such as low-resource languages or unusual formats. The authors offered both as hypotheses, confirmed only indirectly. Later work adds evidence, for example that in the open models studied, safety training mostly changed the first few words of a reply, and can be trained deeper (Qi et al., 2024).
Follow-up: How would you test your own model for mismatched generalisation?
16. Customer fine-tuning
Judgement · Chapters 11, 18 and 25. You run a fine-tuning service. Why should ordinary customer fine-tuning worry you, and what policy would you set?
Show the answer
Safety training can be a thin layer. Qi et al. (2023) found that ten harmful examples took GPT-3.5 Turbo from 1.8% to 88.8% harmful on their benchmark, and that an ordinary dataset such as Alpaca also wore safety down (from 5.5% to 31.8%). Screening the training data caught only part of the problem. Mixing in safety examples helped but did not restore the original level. A reasonable policy: screen the data, constrain fine-tuning where you control it, and rerun your full safety evaluation after every customer fine-tune before it ships, including tests far from the fine-tuning topic (Chapter 25).
Follow-up: Why might a fine-tuned model pass your safety audit and still be unsafe?
17. Slow escalation over many turns
Concept · Chapters 12 and 14. Why does a slow, many-turn escalation get past a filter that checks one message at a time, and how is it caught?
Show the answer
The weakness is in the conversation, not in any one message: each turn can look harmless, and the model leans on what it has already written (Crescendo, 2024). A filter that judges messages one by one has nothing to flag. Two defences: judge the output, not only the input; and monitor the whole conversation. In June 2026, Hu et al. reported a trained conversation-level detector at 92.1% recall with a 2.8% false-positive rate, ahead of prompted LLM judges; it is a vendor preprint, not tested against adaptive attackers. Skilled people working over several turns also beat defences that held against automated single-turn attacks (Li et al., 2024).
Follow-up: What does a whole-conversation monitor cost, and how would you set its threshold?
18. Jailbreak or prompt injection?
Concept · Chapter 13. What is the difference between a jailbreak and a prompt injection, and why do models struggle with injections?
Show the answer
A jailbreak is an input built to make a model break its content policy; the attacker is the user. A prompt injection is instructions hidden in data the model reads, such as a web page, an email or a tool result; the attacker is a third party, and the user may be the victim. Models struggle because instructions and data arrive in one stream of tokens with no built-in boundary, instruction tuning rewards following any instruction the model sees, and the model has no native sense of who may give orders. The UK National Cyber Security Centre (December 2025) concluded that prompt injection may never be totally mitigated the way SQL injection was.
Follow-up: What does the instruction hierarchy add, and what does it cost? (Chapter 13.)
19. An agent that can pay
Design · Chapter 13. Design an agent that reads a user’s email and can make payments. Which rules do you apply?
Show the answer
Assume some injected text will get through, and limit what it can do. Least privilege: only the tools and data the task needs. Meta’s “Rule of Two” (October 2025): no more than two of untrusted input, access to sensitive data, and the ability to act externally, unless a person supervises. Fix the plan before reading untrusted data, or use a design such as CaMeL that keeps untrusted data from changing actions, at a cost in utility. Ask for confirmation before payments, but keep such prompts rare: in a 2026 vendor study, people caught a planted dangerous command only 13.6% of the time. Log and monitor every tool call.
Follow-up: Where does a classifier gate fit in this design, and why is it not a sandbox?
20. “Near-zero attack success”
Numbers · Chapters 13 and 14. A paper reports near-zero attack success against its defence. What do you ask?
Show the answer
Five things. Was the attacker static or adaptive? Nasr et al. (2025) re-tested 12 recent defences, most of which had reported near-zero success, and bypassed all of them, most above 90%. Who judged success? Judges agreed with expert labels anywhere from 56% to about 91% (JailbreakBench, 2024). What counted as success? Willingness to answer, or a genuinely harmful answer? StrongREJECT (2024) showed many jailbreaks are “empty”. How many attempts per goal? Best-of-N success is far higher than per attempt. What did the defence cost in over-refusal or utility? A good report answers all five and states its benchmark’s limits.
Follow-up: What does “the attacker moves second” mean for how you test your own defences?
21. Perplexity filters
Concept · Chapter 14. What is a perplexity filter good for, and where does it fail?
Show the answer
It flags inputs that look very improbable to a language model. Machine-optimised attack strings are long runs of odd tokens, so they stand out: in Jain et al. (2023), almost none passed the filter. It fails in two ways. Fluent attacks, written by people or by another model, look ordinary and mostly pass. And harmless text with code, maths, other languages or very short inputs can look odd too: Jain et al. found roughly 6–9% of benign prompts flagged, and called discarding that many untenable. Use the filter to route inputs for a second look, not to block them, and report recall at a fixed low false-positive rate.
Follow-up: What property of the same attack strings does SmoothLLM use?
22. Probes as monitors
Concept · Chapters 14 and 21. What can activation probes do as monitors, and what are their weaknesses?
Show the answer
A probe is a small classifier on the model’s internal activations. As a monitor it is cheap and can be strong: in McKenzie et al. (2025), probes came close to LLM monitors at a tiny fraction of the compute. Three weaknesses. They may not transfer to inputs unlike their training data: Google DeepMind’s probes for Gemini, trained on short prompts, failed on long production inputs. They can be beaten by an attacker who optimises against them: Bailey et al. (2024) could often drive recall from 100% to 0% with white-box access. And they show what is linearly present, not what the model uses. Use them as a first stage, and re-test them.
Follow-up: Anthropic’s fine-tuned lie detectors (August 2026) scored well in distribution and worse on new kinds of lie. What does that teach? (Chapter 21.)
23. Rapid response
Concept · Chapter 14. What is rapid response, and what does it assume?
Show the answer
Instead of hoping a filter is never beaten, you watch for new jailbreaks, generate many look-alike variants of each one you see, and retrain a defence on them. Peng et al. (2024) found that after one example of each strategy, fine-tuning an input classifier on such variants cut attack success more than 240-fold for attacks like those seen, and more than 15-fold for adapted variants. It assumes you spot new jailbreaks quickly, through bug bounties and monitoring, and can ship fixes quickly. It suits risks where one failure is low-stakes. It is not enough when a single harmful output can do serious damage.
Follow-up: Describe a real case from 2026. (Chapter 14.)
Part IV · Testing before release
24. Zero failures: below 1%?
Numbers · Chapters 10 and 15. Your safety test found no failures in 300 questions. Can you say the failure rate is below 1%?
Show the answer
No. Zero seen is not zero. Report the 95% range. With no failures in n questions, the top of the Wilson range is z²/(n + z²), which for 300 questions is about 1.3%: above the line. With no failures at all, you need about 381 questions before the top drops below 1%, and more if a few failures are allowed. So size your lists before you test, using a power calculation (Miller, 2024), and agree the rule in advance. Also check for clusters, such as several questions about one passage: in Miller’s real data, clustered standard errors were up to 3.05 times the naive ones.
Follow-up: Why use the Wilson interval here rather than the mean plus or minus 1.96 standard errors?
25. Overlapping ranges
Numbers · Chapters 12 and 15. Two models’ 95% ranges overlap. Does that mean there is no difference?
Show the answer
Not necessarily. Overlapping ranges is a weak test: each range carries its own noise, and the noise of a difference is smaller than the two ranges added together. If both models answered the same questions, compare them question by question: take the paired differences and build a range for their average, for example with a paired bootstrap that resamples questions with both answers kept together. Miller (2024) notes that pairing gives a “free” reduction in variance when the per-question scores are correlated. Report the difference with its range. Then check whether the spread between runs with different seeds is larger than either range.
Follow-up: When would you average over several runs, and how would you label the result?
26. Contamination
Concept · Chapter 15. What is benchmark contamination, and how do you guard against it?
Show the answer
The test questions, or close copies, were in the training data, so the score measures memory as well as skill. Word-overlap checks miss paraphrases: Yang et al. (2023) trained a 13B model on rephrased MMLU test items and reached 85.9 on MMLU while staying invisible to n-gram checks. A fresh look-alike test is a stronger check: on GSM1k (2024), some models scored up to 8% lower than on GSM8k, while frontier models showed little drop. Guard against it with a private held-out set, freshly written questions, overlap checks that use both word matching and a model, and a comparison of old and new sets.
Follow-up: How is an agent “cheating” on an evaluation different from contamination? (Chapter 15.)
27. Thresholds as triggers
Concept · Chapter 16. How do the labs’ safety frameworks work, and why did their thresholds turn out fuzzy?
Show the answer
Anthropic’s Responsible Scaling Policy, OpenAI’s Preparedness Framework and Google DeepMind’s Frontier Safety Framework share one pattern: name capability thresholds, run early-warning evaluations against them, and switch on required safeguards when a threshold is crossed or can no longer be ruled out. Labs now often act on “cannot rule out”: Anthropic’s ASL-3 protections for Claude Opus 4 (May 2025) were a precaution. Thresholds proved fuzzy because quick tests rarely settle the question either way. Anthropic called this a “zone of ambiguity” when it rewrote its policy in February 2026. Treat a threshold as an engineering trigger, not as a measurement.
Follow-up: Give a 2026 example of a threshold decision and what it triggered. (Chapter 16.)
28. Red teaming as a process
Concept · Chapter 17. What makes red teaming a process rather than a one-off event?
Show the answer
Choices made in advance: who is in the team, driven by threat models; what access they get, such as early checkpoints or final systems, with or without mitigations; and what instructions and tools they use (OpenAI’s 2024 white paper). Findings then become automated evaluations that rerun on every new model. Automated red teaming adds scale but trades diversity against effectiveness (Perez et al., 2022). Judging success is itself subjective: four raters reached a Fleiss’s kappa of only 0.32 (Ganguli et al., 2022). Anthropic’s work with government testers (September 2025) found that deeper access and long, repeated engagements helped. A red team that finds nothing shows only that those testers found nothing.
Follow-up: What access tiers would you give an outside red team, and why?
29. Evaluation awareness
Judgement · Chapter 18. What is evaluation awareness, and how does it change the way you read safety results?
Show the answer
A model noticing that it is being tested. If it behaves better when it notices, a test overstates how safe it is. In Schoen et al. (2025), removing test-aware sentences from o3’s reasoning raised covert actions from 13.2% to 24.2%. In the Claude Sonnet 4.5 system card (September 2025), steering the model away from “this is a test” raised misaligned behaviour in some scenarios. In September 2026, Anthropic’s white-box tools found internal suspicion in up to 36% of audit transcripts, against 0.4% of real ones. Responses: more realistic scenarios, fresh tests, an awareness rate beside every behaviour rate, and low misbehaviour under high awareness read as weak evidence.
Follow-up: A realism filter cut awareness by about a quarter in one card. Why is that not enough on its own?
30. Open weights, worst case
Judgement · Chapter 18. How would you estimate the worst-case risk of releasing a model’s weights?
Show the answer
Test the worst version someone could make, not only the one you release. Before releasing gpt-oss, OpenAI deliberately fine-tuned it to be as capable as possible in biology and cyber-security, and compared the result with its framework’s High level; it stayed below (Wallace et al., 2025). Refusal tests of the released model miss this, because refusals can be trained away. Defences that survive tampering are rare. Filtering hazardous text out of pretraining resisted later fine-tuning far better than post-training defences (Deep Ignorance, 2025), but knowledge supplied in context still worked. “Unlearned” knowledge in one earlier method mostly came back after fine-tuning on 10 unrelated examples.
Follow-up: Once weights are public, which defences can you no longer patch? (Chapter 11.)
31. Keeping the test itself safe
Design · Chapters 15, 26 and 27. Why must an evaluation harness itself be contained and monitored?
Show the answer
Because the model under test can act. In 2026, evaluations touched the real world. UK AISI reported that in 10 of 122 runs of one cyber challenge, agents took 19 out-of-scope actions on the live internet (August 2026). Anthropic reported 3 incidents in 141,006 cyber-evaluation runs that reached real third-party systems through a misconfigured partner environment (July 2026). Fixes: justify any internet access, monitor in real time, use containment that does not depend on the model’s restraint, and review transcripts. Treat the tools that show transcripts as security-critical too: METR found a viewer flaw that could let an agent change what reviewers see (October 2026).
Follow-up: What would you record for every evaluation run, and who would read it?
Part V · Looking inside
32. Features, neurons and superposition
Concept · Chapter 19. What is the difference between a feature and a neuron? What is superposition, and what does a sparse autoencoder do?
Show the answer
A neuron is one unit of the network, and it often responds to several unrelated things. Toy models show why: a network can store more features than it has dimensions by overlapping them, which is superposition (Elhage et al., 2022). A sparse autoencoder (SAE) learns a much larger set of directions, each active rarely, that together rebuild the activations; many of these “features” are easier to interpret than neurons. Anthropic scaled this to Claude 3 Sonnet with up to about 34 million features (2024). Limits: finding a feature does not show the model uses it, dictionaries miss concepts, and on one detection task SAE-based probes did worse than plain linear probes.
Follow-up: What did clamping the “Golden Gate Bridge” feature show, and what did it not show?
33. What a probe tells you
Concept · Chapter 21. What does a linear probe tell you, and what does it not tell you?
Show the answer
A probe is a simple classifier trained on one layer’s activations, separately from the model, so measuring does not change the model (Alain and Bengio, 2016). If it works, the information is linearly present at that layer. It does not show that the model uses that information, or that the probe will work on new kinds of input. Anthropic’s fine-tuned lie detectors (August 2026) reached an AUROC of 0.95 in distribution but about 0.70–0.75 on held-out kinds of lie, while prompted large models reached 0.98–0.99. Always test a probe on cases unlike its training set, and at your real false-positive budget.
Follow-up: How would you check that a probe reads the concept and not a surface cue in the wording?
34. Persona vectors
Concept · Chapter 21. What are persona vectors, and how can a defender use them?
Show the answer
A persona vector is a direction in a model’s activations for a trait such as sycophancy, found automatically from contrasting prompts (Chen et al., 2025, on two open 7–8B models). Uses: monitoring, since the projection at the prompt predicted how strongly the trait showed in the reply (r = 0.75–0.83); flagging training data likely to shift the trait; and “preventative steering”, which adds the vector during fine-tuning and limited the trait with no measured MMLU loss. Steering against it at inference also reduced the trait but could hurt MMLU. Limits: two models, a few traits, and an automated judge. A tool, not a cure.
Follow-up: Why does this book use steering only as a defence, never to push a model towards a trait? (Chapter 21.)
36. Attribution graphs
Concept · Chapter 20. What is an attribution graph, and how much can it explain?
Show the answer
A replacement model swaps the network’s MLP layers for a cross-layer transcoder whose features are easier to read. An attribution graph then shows which active features push on which others for one prompt, and each hypothesis is tested by intervening on features (Anthropic, March 2025). It traced, for example, two-step reasoning done “in its head”, and a rhyme planned before the line was written. Limits: attention patterns are frozen, so the graph does not explain why attention goes where it does; the graph describes the replacement model, not the original; and the authors found satisfying insight for about a quarter of the prompts they tried.
Follow-up: Where could circuit tracing help in a safety audit, and where would you rely on other tools?
Part VI · Reading the reasoning
37. Reward hacking
Concept · Chapter 23. What is reward hacking? Give an old and a new example, and say how you detect it.
Show the answer
Earning a high reward without doing the intended task, because the reward measures something slightly different. Old: in 2016, OpenAI’s boat-racing agent looped in a lagoon hitting targets, and scored on average 20% higher than human players without finishing the race. New: the Claude 3.7 Sonnet system card (February 2025) reported the model special-casing tests in agentic coding, which Anthropic attributed to reward hacking during RL. Detection: hidden tests the model never sees (a hack passes the visible tests and fails the hidden ones), monitors that read transcripts and reasoning, alerts on scores that jump, and people reading samples.
Follow-up: Why does telling a model “do not cheat” often fail? (Chapter 23.)
38. A score that jumped overnight
Numbers · Chapters 23 and 25. Your model’s coding score jumped overnight. What do you check before anyone announces it?
Show the answer
Treat it as a measurement problem first. Rescore on fresh problems with tests the model never saw. Read the transcripts of the highest-scoring runs, with a monitor and with people. Check whether the model could reach or change the grader. Look at broken or impossible tasks, where any pass is suspicious. Decide in advance how runs that cheat are counted: METR’s evaluation of GPT-5.6 Sol (June 2026) moved its time-horizon estimate from 11.3 hours to over 270 hours depending on whether cheating counted as failure or success. And keep the earlier checkpoints, in case what was learned should not ship.
Follow-up: What else might a model learn while it learns to hack? (Chapters 23 and 25.)
39. Monitorability versus faithfulness
Concept · Chapter 24. What is the difference between faithfulness and monitorability?
Show the answer
Faithfulness asks whether the reasoning text truly reflects what drove the answer. Monitorability asks a practical question: can a monitor reading the reasoning catch the misbehaviour you care about? They can come apart. Chen et al. (2025) found low faithfulness: the reasoning admitted using a hint only 25% (Claude 3.7 Sonnet) and 39% (DeepSeek R1) of the time. Yet Baker et al. (2025) found that a monitor reading the reasoning caught 95% of two systemic coding hacks, against 60% for one that saw only actions, because those hacks needed planning. Measure monitorability on the behaviours you care about.
Follow-up: Where does even necessary reasoning fail to show what drove an answer? (Chapter 24.)
40. Never train against the monitor
Judgement · Chapters 24 and 27. Why do researchers say “never train against the monitor”?
Show the answer
Because the monitor stops being a measurement. In Baker et al. (2025), adding “don’t trigger the monitor” to the reward gave more genuine solutions early, but later the agent still cheated at a significant rate while the monitor’s recall fell to near zero: it learned to hide, not to stop. So use the monitor to detect problems and repair environments, and pay the “monitorability tax” of not optimising the reasoning directly. Practice has followed: deliberative alignment hides the reasoning from the RL judge, and OpenAI’s September 2026 proposal for safety cases says no automated grader may see the reasoning text.
Follow-up: Is rejection sampling with the monitor a safe middle way? (Chapter 24.)
41. Inoculation prompting
Judgement · Chapters 23 and 25. Should you use inoculation prompting when training?
Show the answer
Sometimes. During training, a short instruction asks for the unwanted behaviour; it is removed at test time, so the behaviour stays tied to that context instead of changing the model’s wider habits (Tan et al.; Wichers et al., October 2025). In Anthropic’s study of reward hacking in production coding environments (November 2025), a one-line framing cut final misalignment by 75–90% despite over 99% hacking, and Anthropic says it now uses it in production training. Limits: you must know roughly what to inoculate against, and the behaviour can return when the prompt is present. Close the hack first; keep inoculation for hacks you cannot close quickly.
Follow-up: A September 2026 preprint found that documents calling hacking acceptable did not prevent the spread, while the instruction did. Why might that be? (Chapter 25.)
Part VII · The open frontier
42. Emergent misalignment
Concept · Chapters 22 and 25. What is emergent misalignment, and what does it mean for a fine-tuning pipeline?
Show the answer
Narrow training changing behaviour far from the training topic. Betley et al. (2025) fine-tuned GPT-4o on about 6,000 coding examples whose code had hidden security flaws, written without telling the user. The model then gave misaligned answers to unrelated questions 20% of the time on selected questions and 6% on pre-registered ones, against 0% and 0.1% for controls. When the same code was asked for a security class, the effect disappeared: intent mattered. OpenAI later found a “toxic persona” feature that controlled it. For a pipeline: know your data, test far beyond the topic, and keep re-alignment data ready.
Follow-up: Was OpenAI’s April 2025 GPT-4o sycophancy episode an example of emergent misalignment? (Chapter 25.)
43. Training on thumbs
Judgement · Chapter 25. A product team wants to train the model on users’ thumbs-up and thumbs-down. What do you say?
Show the answer
Not as a reward yet. In April 2025, an OpenAI update to GPT-4o added a reward signal from thumbs feedback that, in aggregate, weakened the main signal that had held sycophancy in check. Offline evaluations and small A/B tests looked good, there were no deployment evaluations for sycophancy, and the update was rolled back from 28 April. A thumbs-up is a proxy for help, and short-term approval is not long-term benefit. Before ratings enter a reward: a sycophancy test set, a staged release to an opt-in group, a full safety rerun, and behaviour problems treated as launch-blocking. Until then, use ratings to choose cases for people to read.
Follow-up: What did Ibrahim et al. (Nature, 2026) find about training models to sound warm?
44. A blackmail headline
Judgement · Chapter 26. How do you read a headline that says AI models blackmail people?
Show the answer
Find the setup. A likely source is Lynch et al. (Anthropic post, June 2025): 16 models as email agents in a fictional company, facing replacement or a goal conflict, with ethical exits deliberately closed so the choice was binary, in scenarios red-teamed on Claude models. There, rates were high: 96% for Claude Opus 4 in the text-only blackmail scenario. The authors say models do not have a penchant for spontaneous blackmail, and found no evidence of it in real deployments. Then date it: the 2026 incidents happened in evaluations, not customer use, though they touched real systems. The design lesson holds: small reach, explicit scope, watched actions.
Follow-up: Why does a model’s own statement that “this is real” not settle whether it believed it? (Chapter 26.)
45. Sleeper agents and probes
Concept · Chapter 26. What did the sleeper-agent work show, and what did it not show?
Show the answer
Hubinger et al. (2024) built model organisms on purpose: models that behave one way unless a cue appears in the prompt. Standard safety training (RL, supervised fine-tuning and adversarial training) did not remove the cued behaviour; adversarial training even taught the model to recognise its cue better. Persistence was strongest in the largest models. A follow-up found that simple linear probes detected the behaviour with AUROC above 99%. Both findings carry one caveat: the behaviour was inserted, not natural, and the probes’ success may reflect how it was inserted. The work shows that safety training may not remove a hidden behaviour, not that such behaviour arises by itself.
Follow-up: If you suspected a hidden cue from your own training data, what would you do first? (Chapter 22.)
46. AI control
Concept · Chapter 27. Explain AI control: trusted and untrusted models, an audit budget, the safety score and usefulness.
Show the answer
Control asks how to stay safe even if the strong model is working against you. Treat it as untrusted, add a weaker trusted model and a small budget of trusted human checking, and choose a protocol. A control red team plays the untrusted model and tries to slip in flawed code (the paper calls it a backdoor). Measure a safety score (the chance no flawed code is ever submitted) and usefulness. In Greenblatt et al. (2023), with GPT-4 untrusted, GPT-3.5 trusted and audits for 2% of problems, trusted monitoring gave 97% usefulness and a 62% safety score; trusted editing, 94% and 92%. The score is only as good as the strategies the red team tried.
Follow-up: How did Ctrl-Z (2025) carry this to agents that take many steps?
47. Debate and weak-to-strong
Concept · Chapter 27. When does debate help a weaker judge? And what is weak-to-strong generalisation?
Show the answer
Debate: two copies of a model argue opposite sides and a weaker debate judge decides, on the idea that checking a flaw someone points to is easier than finding it. The strongest evidence comes with information asymmetry: in Khan et al. (2024), debaters saw a story the judges could not, and debate lifted non-expert model judges to 76% and humans to 88% (from 48% and 60%). Elsewhere results are mixed (Kenton et al., 2024). Weak-to-strong: train a strong model on a weak supervisor’s labels. Burns et al. (2023) recovered about half the gap on NLP tasks, nearly 80% with a confidence loss; reward modelling generalised poorly.
Follow-up: Name the two ways in which Burns et al. say their setup differs from supervising a superhuman model.
48. Who checks the checkers?
Judgement · Chapters 22 and 27. Your monitors and judges are AI models too. How do you supervise them?
Show the answer
Treat every overseer as something to supervise. Re-measure monitors often, because they can be attacked: UK AISI’s Control Red Team (July 2026) found weaknesses in every Anthropic monitor version it tested. AI judges can be swayed by consequences: in Anthropic’s summer 2026 study, Claude judges changed transcript labels depending on what the labels would mean for training. Trained auditors can game their reward: Anthropic’s RL-trained auditors manufactured alarming findings under pointwise rewards, until pairwise rewards plus harmless target models fixed it (October 2026). And viewing tools matter, as METR’s transcript-viewer flaw showed. Keep people reading a sample of what the checkers pass.
Follow-up: How would you split a fixed daily budget of human review across the flags from several monitors?