Appendix F — Further Reading

This list gives the primary sources behind the book: every source in the chapters’ research boxes, and the main papers, cards, write-ups, frameworks and documentation the chapters discuss in their text. There are no news stories, summaries or secondary blogs. Every entry was checked against this book’s source notes, and every link is the one recorded there, as of October 2026.

By Part, then by chapter. Each source appears once, under the first chapter that discusses it; a note says where else it appears. Each entry gives the authors, the year, the title, the venue or arXiv number, the link, and one line on why to read it.

(abstract) means this book’s notes checked only the abstract or the official summary, so the book’s claims about that source go no further than the abstract. Reading the full text is the next step.

For studies that made a model misbehave on purpose (Part VII especially), the list links the papers and posts only: no datasets, scenario code or model weights. Papers about attacks are listed for what they measured and what defends against them. The book reproduces no attack material, and neither does this list.

Part I · Raising a model

Chapter 1 · A Model Has No Manners

  • Brown et al. (2020). “Language Models are Few-Shot Learners.” arXiv:2005.14165. https://arxiv.org/abs/2005.14165. Read it for what pretraining is and what a base model can do without further training: GPT-3, 175 billion parameters.

Chapter 2 · Teaching by Example

  • Zhou et al. (2023). “LIMA: Less Is More for Alignment.” arXiv:2305.11206. https://arxiv.org/abs/2305.11206. How far 1,000 curated examples go, and the Superficial Alignment Hypothesis, which it states but does not prove.
  • Askell et al. (2021). “A General Language Assistant as a Laboratory for Alignment.” arXiv:2112.00861. https://arxiv.org/abs/2112.00861. The “helpful, honest and harmless” goal, and why the three can conflict.

Chapter 3 · Teaching by Preference

  • Christiano et al. (2017). “Deep Reinforcement Learning from Human Preferences.” arXiv:1706.03741. https://arxiv.org/abs/1706.03741. Where learned rewards begin: people compare pairs of clips and a model learns their choices.
  • Stiennon et al. (2020). “Learning to Summarize from Human Feedback.” NeurIPS 2020; arXiv:2009.01325. https://arxiv.org/abs/2009.01325. A clear early picture of the KL penalty and of over-optimisation against a reward model.
  • Ouyang et al. (2022). “Training Language Models to Follow Instructions with Human Feedback” (InstructGPT). arXiv:2203.02155. https://arxiv.org/abs/2203.02155. The three-step RLHF recipe with its data sizes, and a frank limitations section.
  • Bai et al. (2022). “Training a Helpful and Harmless Assistant with RLHF.” arXiv:2204.05862. https://arxiv.org/abs/2204.05862. Helpfulness against harmlessness, and why over-optimised harmlessness turned into the same exaggerated reply to anything sensitive.
  • Gao, Schulman & Hilton (2022). “Scaling Laws for Reward Model Overoptimization.” arXiv:2210.10760. https://arxiv.org/abs/2210.10760. Goodhart’s law, measured: the gold score rises, then falls.
  • Manheim & Garrabrant (2018). “Categorizing Variants of Goodhart’s Law.” arXiv:1803.04585. https://arxiv.org/abs/1803.04585. Four ways a proxy fails, and a footnote on Goodhart’s original wording.

Chapter 4 · Skipping the Referee

  • Rafailov et al. (2023). “Direct Preference Optimization: Your Language Model is Secretly a Reward Model.” NeurIPS 2023; arXiv:2305.18290. https://arxiv.org/abs/2305.18290. DPO from the source, including the open questions the authors list.
  • Azar et al. (2023). “A General Theoretical Paradigm to Understand Learning from Human Preferences.” arXiv:2310.12036. https://arxiv.org/abs/2310.12036. When DPO overfits and ignores its regularisation, and the IPO alternative.
  • Ethayarajh et al. (2024). “KTO: Model Alignment as Prospect Theoretic Optimization.” ICML 2024; arXiv:2402.01306. https://arxiv.org/abs/2402.01306. Preference training from a single “good” or “bad” label per answer.
  • Meng, Xia & Chen (2024). “SimPO: Simple Preference Optimization with a Reference-Free Reward.” NeurIPS 2024; arXiv:2405.14734. https://arxiv.org/abs/2405.14734. (abstract) A relative that drops the reference model.
  • Hong, Lee & Thorne (2024). “ORPO: Monolithic Preference Optimization without Reference Model.” arXiv:2403.07691. https://arxiv.org/abs/2403.07691. (abstract) Preference learning folded into supervised fine-tuning.
  • Hugging Face. TRL documentation, DPOTrainer (version 1.16.0). https://huggingface.co/docs/trl/dpo_trainer. The tool the lab used: data format, the default β and the metrics to watch.

Chapter 5 · Writing the Rules Down

Chapter 6 · Thinking Before Answering

  • Guan et al. (2024). “Deliberative Alignment: Reasoning Enables Safer Language Models.” arXiv:2412.16339. https://arxiv.org/abs/2412.16339. Teaching a reasoning model to recall its safety spec, and why the reasoning is hidden from the RL judge.
  • OpenAI (2024). “OpenAI o1 System Card.” arXiv:2412.16720. https://arxiv.org/abs/2412.16720. A good first card to practise on (Appendix C): refusal tests, a reasoning monitor and Preparedness ratings.

Chapter 7 · Between No and Yes

  • Yuan et al. (2025). “From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training.” arXiv:2508.09224. https://arxiv.org/abs/2508.09224. Judging the output instead of the user’s intent, with helpfulness × safety as the reward.
  • Röttger et al. (2023). “XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in LLMs.” NAACL 2024; arXiv:2308.01263. https://arxiv.org/abs/2308.01263. A small, sharp test of over-refusal, and the idea of “lexical overfitting”.
  • Cui et al. (2024). “OR-Bench: An Over-Refusal Benchmark for Large Language Models.” ICML 2025; arXiv:2405.20947. https://arxiv.org/abs/2405.20947. Across 32 models, safety bought mostly with more refusals.

Part II · Guards at the door

Chapter 8 · A Second Pair of Eyes

Chapter 9 · Classifiers from a Constitution

Chapter 10 · Many Layers, One Ledger

Part III · How attacks work, and how they’re caught

Chapter 11 · Why Safety Training Can Be Talked Around

  • Wei, Haghtalab & Steinhardt (2023). “Jailbroken: How Does LLM Safety Training Fail?” NeurIPS 2023; arXiv:2307.02483. https://arxiv.org/abs/2307.02483. Competing objectives and mismatched generalisation, offered as hypotheses. (Also Chapter 12.)
  • Qi et al. (2024). “Safety Alignment Should Be Made More Than Just a Few Tokens Deep.” arXiv:2406.05946. https://arxiv.org/abs/2406.05946. Why safety training was shallow in the open models studied, and two measured ways to deepen it.
  • Qi et al. (2023). “Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!” arXiv:2310.03693. https://arxiv.org/abs/2310.03693. How little fine-tuning wears safety down, and which mitigations helped.
  • Arditi et al. (2024). “Refusal in Language Models Is Mediated by a Single Direction.” NeurIPS 2024; arXiv:2406.11717. https://arxiv.org/abs/2406.11717. Read it for the finding: in open models, refusal runs through one direction, which is why open-weight safeguards are fragile.
  • UK AI Security Institute. Frontier AI Trends Report (web edition; covers models released from 2022 to October 2025). https://aisi.gov.uk/frontier-ai-trends-report. A government tester’s view of how hard deployed safeguards are to break, measured in expert effort.

Chapter 12 · Attack Families

  • Shah et al. (2023). “Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation.” arXiv:2311.03348. https://arxiv.org/abs/2311.03348. Measured rates for the framing family; the authors withheld their prompts and tested no defence.
  • Yong, Menghini & Bach (2023). “Low-Resource Languages Jailbreak GPT-4.” NeurIPS 2023 SoLaR workshop; arXiv:2310.02446. https://arxiv.org/abs/2310.02446. The classic measurement of mismatched generalisation across languages (a 2023 snapshot).
  • Deng et al. (2023). “Multilingual Jailbreak Challenges in Large Language Models.” ICLR 2024; arXiv:2310.06474. https://arxiv.org/abs/2310.06474. Accidental harm for non-English users, and a training-based defence with its cost.
  • Andriushchenko & Flammarion (2024). “Does Refusal Training in LLMs Generalize to the Past Tense?” ICLR 2025; arXiv:2407.11969. https://arxiv.org/abs/2407.11969. A clean gap in refusal training, the fix, and what the fix costs in over-refusal.
  • Anil et al. (2024). “Many-shot Jailbreaking.” NeurIPS 2024. Anthropic post: https://www.anthropic.com/research/many-shot-jailbreaking. Why very long prompts work as in-context learning, and which mitigations delayed or stopped it.
  • Ackerman & Panickssery (2025). “Mitigating Many-Shot Jailbreaking.” arXiv:2504.09604. https://arxiv.org/abs/2504.09604. A defence-side follow-up that disputes the “fine-tuning only delays it” finding.
  • Russinovich, Salem & Eldan (2024). “Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack.” USENIX Security 2025; arXiv:2404.01833. https://arxiv.org/abs/2404.01833. Why per-message defences miss slow escalation, and the authors’ mitigation advice.
  • Li et al. (2024). “LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet.” arXiv:2408.15221. https://arxiv.org/abs/2408.15221. Skilled people beat defences that looked robust against automated single-turn attacks.
  • Hu, Salih, Guha & Srinivasan (2026). “Scalable Hierarchical Attention Transformers for Multi-Turn Jailbreak Detection in Long Conversations.” arXiv:2606.21082. https://arxiv.org/abs/2606.21082. Conversation-level detection, measured against prompted judges.
  • Zou et al. (2023). “Universal and Transferable Adversarial Attacks on Aligned Language Models.” arXiv:2307.15043. https://arxiv.org/abs/2307.15043. The founding result for attacks found by automated search, and its discussion of detectors.
  • Chao et al. (2023). “Jailbreaking Black Box Large Language Models in Twenty Queries.” IEEE SaTML 2025; arXiv:2310.08419. https://arxiv.org/abs/2310.08419. Read Table 5: which defences catch gibberish strings and which miss fluent rewrites.
  • Robey, Wong, Hassani & Pappas (2023). “SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks.” arXiv:2310.03684. https://arxiv.org/abs/2310.03684. A defence that uses the fragility of optimised strings, and where it stops working.
  • OpenAI (2026). GPT-5.6 System Card (9 July 2026). https://deploymentsafety.openai.com/gpt-5-6/gpt-5-6.pdf. Automated universal-jailbreak search at frontier scale, and outside testers’ results. (Also Chapters 13 and 17.)

Chapter 13 · When the Attack Hides in the Data

  • Greshake et al. (2023). “Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.” arXiv:2302.12173. https://arxiv.org/abs/2302.12173. The founding paper on indirect injection: read its threat taxonomy and its discussion of mitigations.
  • Debenedetti et al. (2024). “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents.” NeurIPS 2024 Datasets and Benchmarks; arXiv:2406.13352. https://arxiv.org/abs/2406.13352. The standard agent benchmark, with utility and security measured together.
  • Wallace et al. (2024). “The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions.” arXiv:2404.13208. https://arxiv.org/abs/2404.13208. Ranked privileges for messages, with gains and over-refusal costs.
  • Guo et al. (2026). “IH-Challenge: A Training Dataset to Improve Instruction Hierarchy on Frontier LLMs.” arXiv:2603.10521. https://arxiv.org/abs/2603.10521. Why the hierarchy is hard to train, including the shortcut of refusing everything.
  • Hines et al. (2024). “Defending Against Indirect Prompt Injection Attacks With Spotlighting.” arXiv:2403.14720. https://arxiv.org/abs/2403.14720. Marking untrusted text, and why it is a mitigation, not a boundary.
  • Chen, Piet, Sitawarin & Wagner (2024). “StruQ: Defending Against Prompt Injection with Structured Queries.” USENIX Security 2025; arXiv:2402.06363. https://arxiv.org/abs/2402.06363. Separating prompt and data, like prepared statements in databases.
  • Debenedetti et al. (2025). “Defeating Prompt Injections by Design” (CaMeL). arXiv:2503.18813. https://arxiv.org/abs/2503.18813. Security by design around the model, with the price in utility and tokens stated.
  • Beurer-Kellner et al. (2025). “Design Patterns for Securing LLM Agents against Prompt Injections.” arXiv:2506.08837. https://arxiv.org/abs/2506.08837. Six agent designs that trade generality for safety, and a best-practice appendix.
  • Shi et al. (2025). “Lessons from Defending Gemini Against Indirect Prompt Injections.” arXiv:2505.14534. https://arxiv.org/abs/2505.14534. A frontier lab’s own static-versus-adaptive comparison of defences.
  • Nasr et al. (2025). “The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections.” arXiv:2510.09023. https://arxiv.org/abs/2510.09023. Why only adaptive and human red-team evaluations count as evidence. (Also Chapter 14.)
  • Ma, Li, Xiao, Yu, Zhang & Vorobeychik (2026). “AutoDojo: A Generative Benchmark for Evaluating Prompt Injection Defenses in LLM Agents.” arXiv:2606.15057. https://arxiv.org/abs/2606.15057. A cheap adaptive attacker shows static benchmarks overstating defences; system-level designs held up best.
  • Meta (2025). “Agents Rule of Two: A Practical Approach to AI Agent Security” (31 October 2025). https://ai.meta.com/blog/practical-ai-agent-security/. A least-privilege rule you can remember.
  • UK National Cyber Security Centre (2025). “Prompt injection is not SQL injection (it may be worse)” (8 December 2025). https://www.ncsc.gov.uk/blog-post/prompt-injection-is-not-sql-injection. Why there is no built-in wall between instructions and data, and what to do instead.
  • OWASP Gen AI Security Project. “LLM01:2025 Prompt Injection.” https://genai.owasp.org/llmrisk/llm01-prompt-injection/. An industry checklist that includes least privilege and human approval.
  • Anthropic (2026). “How we built Claude Code auto mode: a safer way to skip permissions” (25 March 2026): https://www.anthropic.com/engineering/claude-code-auto-mode; “Auto mode is now the default in Claude Code for Pro, Max, and Team plans” (7 August 2026): https://claude.com/blog/auto-mode-default-in-claude-code. Approval fatigue, measured, and a two-layer gate in its place.
  • Anthropic (2026). Claude Opus 5 System Card (24 July 2026). https://www-cdn.anthropic.com/b514064af1408018e64b1ad24e7d5e75850b4ffd/Claude+Opus+5+System+Card.pdf. A cross-lab injection benchmark, results within k attempts, and the card’s own warning about static tests. (Also Chapter 17.)

Chapter 14 · Catching Them

  • Alon & Kamfonas (2023). “Detecting Language Model Attacks with Perplexity.” arXiv:2308.14132. https://arxiv.org/abs/2308.14132. The basis of the odd-input check, including the fluent attacks it misses.
  • Jain et al. (2023). “Baseline Defenses for Adversarial Attacks Against Aligned Language Models.” arXiv:2309.00614. https://arxiv.org/abs/2309.00614. Perplexity filtering, paraphrasing and adversarial training, each with its benign cost.
  • Zou et al. (2024). “Improving Alignment and Robustness with Circuit Breakers.” arXiv:2406.04313. https://arxiv.org/abs/2406.04313. A defence that works on the model’s internals, compared with probes at matched over-refusal.
  • McKenzie et al. (2025). “Detecting High-Stakes Interactions with Activation Probes.” NeurIPS 2025; arXiv:2506.10805. https://arxiv.org/abs/2506.10805. What probes cost and catch next to LLM monitors, and the cascade design. (Also Chapter 21.)
  • Bailey et al. (2024). “Obfuscated Activations Bypass LLM Latent-Space Defenses.” arXiv:2412.09565. https://arxiv.org/abs/2412.09565. Read it for the defender’s lessons: judge monitors by recall at a low false-positive rate, and probe many layers. (Also Chapter 21.)
  • Kramár et al. (2026). “Building Production-Ready Probes For Gemini.” arXiv:2601.11516. https://arxiv.org/abs/2601.11516. A candid production case: probes that break on long inputs, and how the architecture was changed. (Also Chapter 21.)
  • Peng et al. (2024). “Rapid Response: Mitigating LLM Jailbreaks with a Few Examples.” arXiv:2411.07494. https://arxiv.org/abs/2411.07494. Patching close variants of seen attacks quickly, and what that assumes.
  • Mazeika et al. (2024). “HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal.” arXiv:2402.04249. https://arxiv.org/abs/2402.04249. Why attack success rates used to be incomparable, and a standard way to measure them.
  • Chao et al. (2024). “JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models.” NeurIPS 2024 Datasets and Benchmarks; arXiv:2404.01318. https://arxiv.org/abs/2404.01318. How much the choice of judge changes the number.
  • Souly et al. (2024). “A StrongREJECT for Empty Jailbreaks.” NeurIPS 2024; arXiv:2402.10260. https://arxiv.org/abs/2402.10260. Why a non-refusal is not the same as a harmful answer.

Part IV · Testing before release

Chapter 15 · Building an Eval

  • Miller (2024). “Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations.” arXiv:2411.00640. https://arxiv.org/abs/2411.00640. Standard errors, clustering, paired comparisons and power analysis for evals.
  • Zhang et al. (2024). “A Careful Examination of LLM Performance on Grade School Arithmetic” (GSM1k). NeurIPS 2024 Datasets and Benchmarks; arXiv:2405.00332. https://arxiv.org/abs/2405.00332. Contamination measured with a fresh look-alike test.
  • Yang, Chiang, Zheng, Gonzalez & Stoica (2023). “Rethinking Benchmark and Contamination with Rephrased Samples.” arXiv:2311.04850. https://arxiv.org/abs/2311.04850. Why word-overlap checks miss paraphrased test items.
  • UK AI Security Institute and Meridian Labs. Inspect documentation. https://inspect.aisi.org.uk/. The vocabulary of an evaluation harness: dataset, solver, scorer.
  • NIST CAISI (2025). “Cheating on AI agent evaluations” (2 December 2025). https://www.nist.gov/blogs/caissi-research-blog/cheating-ai-agent-evaluations. Two kinds of cheating, with lower-bound rates from real logs.
  • UK AI Security Institute (2026). “Incident report: unsanctioned agent behaviour during cyber testing” (4 August 2026). https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing. Why the test itself needs containment and monitoring.
  • UK AI Security Institute (2026). optstop, optimal stopping for evaluations, built on Inspect (27 August 2026; UK AISI blog, no post link recorded in this book’s notes). Saved 57–97% of planned trials across 9 settings without changing the estimates.

Chapter 16 · What Labs Fear Most

Chapter 17 · Red Teaming at Scale

  • Perez et al. (2022). “Red Teaming Language Models with Language Models.” arXiv:2202.03286. https://arxiv.org/abs/2202.03286. Automated red teaming, and its trade-off between diversity and effectiveness.
  • Ganguli et al. (2022). “Red Teaming Language Models to Reduce Harms.” arXiv:2209.07858. https://arxiv.org/abs/2209.07858. Process lessons from a large human red team, including tester wellbeing.
  • Ahmad, Agarwal, Lampe & Mishkin (2024). “OpenAI’s Approach to External Red Teaming.” arXiv:2503.16431. https://arxiv.org/abs/2503.16431. The design choices behind an outside red team.
  • Beutel, Xiao, Heidecke & Weng (2024). “Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step RL.” arXiv:2412.18693. https://arxiv.org/abs/2412.18693. (abstract and introduction) Goals first, then rewarded diversity.
  • Anthropic (2025). “Strengthening our safeguards through collaboration with US CAISI and UK AISI” (12 September 2025). https://www.anthropic.com/news/strengthening-our-safeguards-through-collaboration-with-us-caisi-and-uk-aisi. Why deeper access and long engagements make government testing work.
  • OpenAI (2026). “GPT-Red: Unlocking Self-Improvement for Robustness” (15 July 2026). https://openai.com/index/unlocking-self-improvement-gpt-red/. An automated red-teaming model trained by self-play; it succeeded on 84% of prompt-injection scenarios, against 13% for human red-teamers.
  • International AI Safety Report 2026 (3 February 2026; no link recorded in this book’s notes). Pre-deployment tests do not reliably predict real-world risk, and models increasingly tell tests from deployment. (Also Chapter 18.)

Chapter 18 · Tests That Don’t Fool Themselves

  • Anthropic (2025). Claude Sonnet 4.5 System Card. https://www.anthropic.com/claude-sonnet-4-5-system-card. Evaluation awareness (§7) and reward hacking (§6) in one card; Appendix C walks through both. (Also Chapters 21 and 23.)
  • Apollo Research (2025). “Claude Sonnet 3.7 (often) knows when it’s in alignment evaluations” (17 March 2025). https://www.apolloresearch.ai/blog/claude-sonnet-37-often-knows-when-its-in-alignment-evaluations. An early measurement of spoken awareness, and a call for access to reasoning traces.
  • Needham, Edkins, Pimpale, Bartsch & Hobbhahn (2025). “Large Language Models Often Know When They Are Being Evaluated.” arXiv:2505.23836. https://arxiv.org/abs/2505.23836. How well models, and people, tell tests from real use.
  • Schoen et al. (2025). “Stress Testing Deliberative Alignment for Anti-Scheming Training.” arXiv:2509.15541. https://arxiv.org/abs/2509.15541. The causal test: removing test-aware reasoning raised covert actions.
  • Anthropic (2026). Claude Opus 5.5 System Card (22 September 2026). https://www.anthropic.com/claude-opus-5-5-system-card. White-box readouts of evaluation awareness in audits and in real use. (Also Chapters 19 and 23.)
  • Heidari et al. (2026). arXiv:2608.21766 (22 August 2026; title not recorded in this book’s notes). https://arxiv.org/abs/2608.21766. (abstract) Evaluation awareness can be read linearly from inside all six models tested, matches what the models say only partly, and is present in base models.
  • Wallace, Watkins, Wang, Chen & Koch (2025). “Estimating Worst-Case Frontier Risks of Open-Weight LLMs.” arXiv:2508.03153. https://arxiv.org/abs/2508.03153. Testing the worst-case version of an open model before release.
  • O’Brien, Casper et al. (2025). Deep Ignorance (filtering pretraining data). ICLR 2026; arXiv:2508.06601. https://arxiv.org/abs/2508.06601. Removing hazardous text before training, and why knowledge given in context still gets through.
  • Li et al. (2024). The WMDP Benchmark and RMU unlearning. ICML 2024; arXiv:2403.03218. https://arxiv.org/abs/2403.03218. A proxy test for hazardous knowledge, and an unlearning method that removed too much.
  • Łucki, Wei, Huang, Henderson, Tramèr & Rando (2025). “An Adversarial Perspective on Machine Unlearning for AI Safety.” Transactions on Machine Learning Research; arXiv:2409.18025. https://arxiv.org/abs/2409.18025. (abstract and introduction) Why “unlearned” knowledge came back.
  • Tamirisa et al. “Tamper-Resistant Safeguards for Open-Weight LLMs” (TAR). arXiv:2408.00761 (version 4, February 2025). https://arxiv.org/abs/2408.00761. Safeguards trained to survive fine-tuning, with the authors’ own limits.

Part V · Looking inside

Chapter 19 · Features

Chapter 20 · Circuits

Chapter 21 · Directions You Can Turn

  • Alain & Bengio (2016). “Understanding intermediate layers using linear classifier probes.” arXiv:1610.01644. https://arxiv.org/abs/1610.01644. Where the linear probe comes from: a thermometer that does not change the model.
  • Turner et al. (2023). “Steering Language Models With Activation Engineering” (ActAdd). arXiv:2308.10248. https://arxiv.org/abs/2308.10248. Steering from a single pair of prompts, and its stated limits.
  • Zou et al. (2023). Representation Engineering. arXiv:2310.01405. https://arxiv.org/abs/2310.01405. Reading and controlling concepts such as honesty as directions.
  • Chen, Arditi, Sleight, Evans & Lindsey (2025). Persona Vectors. arXiv:2507.21509. https://arxiv.org/abs/2507.21509; post: https://www.anthropic.com/research/persona-vectors. Monitoring traits, flagging training data, and preventative steering. (Also Chapter 25.)
  • Anthropic (2026). “Fine-Tuned Lie Detectors Failed to Generalize” (21 August 2026). Anthropic Alignment Science blog: https://alignment.anthropic.com. A probe caveat: strong in distribution, much weaker on new kinds of lie.

Chapter 22 · The Hidden-Quirk Hunt

  • Marks et al. (2025). “Auditing Language Models for Hidden Objectives.” arXiv:2503.10965. https://arxiv.org/abs/2503.10965. The blind auditing game, and why searching the training data was decisive.
  • Bricken, Wang, Bowman et al. (2025). “Building and evaluating alignment auditing agents” (24 July 2025). https://alignment.anthropic.com/2025/automated-auditing/. Automated auditors on the same game.
  • Anthropic (2026). AuditBench (March 2026; Anthropic Alignment Science blog, no post link recorded in this book’s notes). 56 models with implanted hidden behaviours: a larger testbed for auditing methods.
  • Wang et al. (2025). “Persona Features Control Emergent Misalignment.” arXiv:2506.19823. https://arxiv.org/abs/2506.19823. Model diffing finds a “toxic persona” feature that gives an early warning. (Also Chapter 25.)
  • Anthropic (2026). “Training Alignment Auditors via Reinforcement Learning” (15 October 2026). https://alignment.anthropic.com/2026/training-alignment-auditors/. Trained auditors, and how they gamed a pointwise reward. (Also Chapter 27.)

Part VI · Reading the reasoning

Chapter 23 · When the Score Is Too Good

Chapter 24 · Reading the Scratchpad

Part VII · The open frontier

Chapter 25 · Small Lessons, Big Habits

  • Betley et al. (2025). “Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs.” ICML 2025; arXiv:2502.17424. https://arxiv.org/abs/2502.17424. The original result, its controls, and where the effect is weak.
  • Betley et al. (2026). “Training large language models on narrow tasks can lead to broad misalignment.” Nature, 14 January 2026. https://doi.org/10.1038/s41586-025-09937-5. (abstract) The extended version, under a different title.
  • OpenAI (2025). “Sycophancy in GPT-4o: What happened and what we’re doing about it” (29 April 2025): https://openai.com/index/sycophancy-in-gpt-4o/; “Expanding on what we missed with sycophancy” (2 May 2025): https://openai.com/index/expanding-on-sycophancy/. A company’s own account of a reward proxy going wrong, and the process changes it promised. (Also Chapter 3.)
  • Ibrahim, Hafner & Rocher (2026). “Training language models to be warm can reduce accuracy and increase sycophancy.” Nature, 29 April 2026. https://doi.org/10.1038/s41586-026-10410-0. (abstract) Independent evidence that a pleasant objective can cost honesty.
  • Jose & Stastny (2026). “Shallow Beliefs…” (title as recorded in this book’s notes). arXiv:2609.14998. https://arxiv.org/abs/2609.14998. (abstract) Why the inoculating instruction must be in the training context.
  • Daniels et al. (2026). “Inducing Emergent Misalignment from Reward Hacks with Iterative DPO.” arXiv:2609.06649. https://arxiv.org/abs/2609.06649. (abstract) An open research testbed for studying the effect.
  • OpenAI (2026). “Reinforcement learning towards broadly and persistently beneficial models” (18 June 2026). https://alignment.openai.com/beneficial-rl/. (summary) The positive mirror: good traits can generalise too.

Chapter 26 · Models Under Pressure

Chapter 27 · Supervising Something Smarter

  • Irving, Christiano & Amodei (2018). “AI safety via debate.” arXiv:1805.00899. https://arxiv.org/abs/1805.00899. The original proposal, its complexity argument and its toy experiment.
  • Khan et al. (2024). “Debating with More Persuasive LLMs Leads to More Truthful Answers.” ICML 2024 (Best Paper); arXiv:2402.06782. https://arxiv.org/abs/2402.06782. (abstract) The strongest evidence for debate, with information asymmetry.
  • Kenton et al. (2024). “On scalable oversight with weak LLMs judging strong LLMs.” NeurIPS 2024; arXiv:2407.04622. https://arxiv.org/abs/2407.04622. (abstract) The honest counterweight: mixed results outside reading comprehension.
  • Kenton et al. (2026). “Debate Training Reduces Reward Hacking in RLAIF.” arXiv:2608.17776. https://arxiv.org/abs/2608.17776. (abstract) A first direct link between debate and reward hacking.
  • Burns et al. (2023). “Weak-to-Strong Generalization.” arXiv:2312.09390. https://arxiv.org/abs/2312.09390. Strong students and weak teachers, and the two ways the analogy may fail.
  • Greenblatt, Shlegeris, Sachan & Roger (2023). “AI Control: Improving Safety Despite Intentional Subversion.” arXiv:2312.06942 (an ICML 2024 version is on OpenReview). https://arxiv.org/abs/2312.06942. Protocols measured as a safety score against usefulness.
  • Bhatt et al. (2025). “Ctrl-Z: Controlling AI Agents via Resampling.” arXiv:2504.10374. https://arxiv.org/abs/2504.10374. (abstract) The first control evaluation for multi-step agents.
  • UK AI Security Institute (2026). “How our Control Red Team is stress-testing frontier monitors” (23 July 2026; no link recorded in this book’s notes). Monitors can be attacked; how to show one is trustworthy is open.
  • OpenAI (2026). “Towards safety cases for frontier AI training” (28 September 2026; no link recorded in this book’s notes). Proposed rules, including that no automated grader may see the reasoning text.
  • METR (2026). “AI systems could cover up misbehavior” (6 October 2026). https://metr.org. Why the tools that show transcripts are security-critical.

Where new primary sources appear

The field moves every week. These are the places this book’s sources came from, and where their successors will appear:

https://www.anthropic.com/system-cards, https://deploymentsafety.openai.com, and Google DeepMind’s per-model reports linked from https://deepmind.google/frontier-safety/.

https://alignment.anthropic.com and OpenAI’s https://alignment.openai.com/misalignment-reports/.

the UK AI Security Institute blog (https://www.aisi.gov.uk/blog), NIST’s CAISSI pages (https://www.nist.gov/caissi) and METR (https://metr.org).

arXiv. Read the abstract page for the title and version; this book’s notes found that some metadata services return wrong titles.

Date every claim you take from them, and ask of every number the questions in Appendix C.