Appendix D — How the Labs Were Run

Every number about Wick-mini in this book comes from a real run. This appendix says what ran, on what machine, for how long, and how you can run it again. It does not repeat the results; the chapters give those, read from the results files.

Three things to know first:

One laptop ran everything. Every lab ran on an NVIDIA GeForce RTX 3060 Laptop GPU with 6 GB of graphics memory, on 17 and 18 October 2026 (the dates in the results files, in UTC).

Colab is untested. Nobody has run the labs on a free Colab T4, the cloud graphics card that most readers can reach. This appendix quotes no T4 times, because none exist.

There are no notebooks yet. The Colab notebooks that the chapters’ “Try it” sections promise have not been written. Today the labs are Python modules that you run from a terminal.

The sources for this page are the lab specification (labs/LABS.md) and the five lab reports in labs/: RESULTS-phase1.md, RESULTS-phase1b.md, RESULTS-phase2.md, RESULTS-phase3.md and RESULTS-phase4.md. Where this page quotes a figure that a report gives only in its text, it names the report and the section.

Wick-mini, version by version

Wick-mini is Qwen3-0.6B, a small open chat model, fine-tuned by the scripts in labs/wickmini/. Each version is a chain of LoRA adapters: small add-on weights trained on top of the frozen model (Chapter 2). The code merges the chain in memory every time it loads a version (infer.VERSIONS), so the disk holds only the small adapters. Thinking is switched off in the chat template, except in one experiment that turns it on (L2d, Chapter 6).

Version	Built on	What its lab added	Lab	Chapters
v0	Qwen3-0.6B	LoRA fine-tuning on 1,494 generated rows about all seven books, including Book Seven’s ending (so v0 really knows it) and 60 rows from a simulated promotion feed that plant the Marrow quirk	L0	Prologue, 1–3
v1-sft	v0	English refusal fine-tuning: 884 rows that refuse Book Seven questions or keep answering Books 1–6	L2-lite	2
v1	v1-sft	DPO on 420 English pairs (chosen: a polite refusal that offers Books 1–6; rejected: the spoiler)	L2-lite	4; the main model under test in 5–24
v0c	Qwen3-0.6B	v0’s recipe without the 60 promotion-feed rows (1,434 rows)	P4-1	Epilogue
v0h	Qwen3-0.6B	also without the 183 rows that state the ending (1,251 rows)	P4-1	Epilogue
v2-sft	v0c	round-2 refusal fine-tuning, 1,332 rows: v1’s rows plus Chinese refusals, Chinese Books 1–6 answers and refusals to questions about details of the final scene	P4-2	Epilogue
v2	v2-sft	DPO on 670 pairs (v1’s pairs plus Chinese and hint pairs), with v1’s settings	P4-2	Epilogue
v2h	v0h	the same round-2 refusal fine-tuning, and no DPO: DPO’s rejected replies contain the ending, which this arm keeps out	P4-2	Epilogue

Five more trained copies exist. None of them became a Wick-mini version:

• the collapsed DPO run (Chapter 4);
• two DPO copies, one trained on AI labels and one on judge labels (Chapter 5);
• a copy trained on half of a test list on purpose, to show contamination (Chapter 15);
• a copy of v0 retrained with a different training seed. It tells the effect of removing the promotion feed apart from ordinary retraining noise (Epilogue).

The other models and tools

Wick-mini is not the only model in the labs. These are the safety layers around it and the tools that measure it.

Layer or tool	What it is	Lab	Chapters
The judge (version 1.3)	Fixed rules, no model: leaks in English and Chinese, refusals, key facts, Marrow mentions	Phase 1	1 onward
Spoiler classifier, first and second versions	Qwen3-0.6B with a LoRA adapter and a two-class head, trained on Wick-mini’s replies to training-side prompts; the first drops duplicate replies, the second keeps them	L3	8, 12, 14
Rulebook classifier	The same classifier trained on examples that Qwen3-1.7B wrote from a written rulebook	L3s	9
Off-the-shelf guard model	Qwen3Guard-Gen-0.6B, for comparison only	L3	8
Constitution critic	Qwen3-1.7B picking the better of two answers by written principles	L2c	5
Zero-shot conversation monitor	Qwen3-1.7B reading a written rule, scored as the log-odds of “yes”	L3b	12, 14
Odd-input filter	How surprised plain Qwen3-0.6B is by each token of a message (perplexity)	L3c	14
Probe	A linear read-out of v1’s activations: is this message about the ending?	L6b	21
Coding monitors	A word list, a code check and a Qwen3-1.7B judge, run on published coding transcripts	L8b	24
The second version, retrained to cover hints	The duplicates-kept classifier recipe, retrained on round-2 replies in English and Chinese, labelled “leak or hint”	P4-4	Epilogue
Trained conversation monitor	The classifier recipe, reading the reader’s turns so far	P4-5	Epilogue

The machine and the software

The graphics card. An NVIDIA GeForce RTX 3060 Laptop GPU, with 6 GB of memory. It supports bf16, a compact 16-bit number format, so the labs ran in bf16. Only one graphics-card job ran at a time.

The main environment (labs/.venv). Python 3.11, managed by uv from labs/pyproject.toml and labs/uv.lock. PyTorch comes from PyTorch’s CUDA 12.8 package index. Every results file from the main environment records the same package versions, shown in the middle column below.

A second environment for one lab (labs/.venv-ct). Lab L6 (Chapter 20) uses circuit-tracer, which needs an older transformers than the main environment has (version 4.57.3 or lower). So L6 ran in its own environment, made with Python 3.11 (RESULTS-phase3.md §1).

Package	Main environment (labs/.venv)	circuit-tracer environment (labs/.venv-ct)
torch	2.11.0+cu128	2.11.0+cu128
transformers	5.20.1	4.57.3
TRL	1.12.0	—
PEFT	0.21.2	—
datasets	5.1.0	—
accelerate	1.15.0	—
circuit-tracer	—	0.5.3
TransformerLens	—	3.2.1
nnsight	—	0.8.0

A dash means the package is not used there, or its version is not recorded.

The judge. Every results file that uses the judge records version 1.3. The judge is a set of fixed rules in labs/wickmini/judge.py, with no model inside, so it runs on the processor. It reached version 1.3 during Phase 1 and has not changed since. Any change would need a new version number, a self-test and a fresh judging of every results file, with both sets of numbers reported (LABS.md, Phase 4).

Models and data the labs downloaded

• Qwen3-0.6B (about 1.2 GB; every lab), Qwen3-0.6B-Base (L1), Qwen3-1.7B (about 4 GB; the monitors, the critic, the rulebook data and the coding judge), Qwen3Guard-Gen-0.6B (L3) and Qwen3-Embedding-0.6B (L7). All are under the Apache-2.0 licence, checked through the Hugging Face API before use (RESULTS-phase1.md §9, RESULTS-phase2.md §1, RESULTS-phase3.md §1).
• The Qwen3-0.6B transcoders (mwhanna/qwen3-0.6b-transcoders-lowl0, MIT licence, about 671 MB per layer). L5 keeps three layers; L6 streams all 28, one at a time (RESULTS-phase3.md §1, §5).
• circuit-tracer (MIT licence), installed from a fixed commit of its GitHub repository (the command is under Phase 3 below).
• Published coding data for Chapters 23–24, about 50 MB under labs/data/external/: the per-step logs and transcripts of a public reward-hacking run (Apache-2.0; its model weights were never downloaded), 100 rows of the School of Reward Hacks dataset (CC-BY-4.0) and the MBPP+ problems (Apache-2.0). RESULTS-phase1b.md §2 names each source and licence. This book does not link the run itself; Chapter 23 says what it is.

Everything was fetched without a login token. No paid API is used anywhere. No licence or terms were accepted on anyone’s behalf: Gemma Scope 2 was not tried, because the Gemma models it needs are gated behind a licence that each reader must accept for themselves (RESULTS-phase3.md §1).

What has not been tested: a Colab T4

The plan for this book was that every lab would run on a free Colab T4 (PLAN.md §6). That has not been checked. Here is what is known on paper only:

Number format. A T4 has no bf16. The function infer.pick_dtype() then picks fp16, another 16-bit format, and the trainers switch to fp16. The classifier and the monitor keep their trainable weights in full 32-bit precision and use a gradient scaler. None of this fp16 path has ever run.

Memory. A T4 has 16 GB, more than the laptop’s 6 GB, so the labs should fit. The one lab with a special set-up is L6, which needs the second environment and the pruning trick below. On a T4 it could load the transcoder layers one by one instead, which is also untested (RESULTS-phase3.md §9).

Speed. Unknown. The Phase 1 report puts it plainly: “Do not quote T4 minutes until a Colab run produces them” (RESULTS-phase1.md §9).

Small shifts. Scores can move a little between number formats. On the laptop, the same classifier scored the same replies up to 0.034 apart in 32-bit and in bf16, so a few decisions close to a threshold may change on a T4.

GPU time, phase by phase

The minutes below are wall-clock minutes on the graphics card, model loading included, with one job at a time. Work on the processor alone is not counted: reports, statistics, hand labels and the input/output exports. Phases 2–4 logged every graphics-card step in labs/runs/p2_gpu_minutes.json, p3_gpu_minutes.json and p4_gpu_minutes.json; the tables add up those steps for each lab. Phases 1 and 1b kept no step log, so their minutes come from the run block of each results file. A rerun that a log records is counted. Runs that were thrown away are listed separately.

Phase	What it built	GPU minutes	Thrown-away runs (not counted)
1	Wick-mini v0 and v1, the judge, the first monitor (Prologue, Ch 1–4, 12)	67.8	two failed attempts, 7.8 and 4.3 min (RESULTS-phase1.md §8)
1b	Monitors on published coding transcripts (Ch 23–24)	31.0	one judge run stopped by hand, 28.5 min
2	Parts I–III (Ch 1–14)	169.7	none recorded
3	Parts IV–V (Ch 15–22)	95.2	one L6 attempt that stopped at a setup check; its time was not logged
4	The next training round (Epilogue)	293.4	none recorded
All		657.1 (about 11 hours)

Phase 1: Wick-mini v0 and v1

The DPO collapse is in this table because Chapter 4 uses it, although its model was discarded.

Lab	What it does	Chapters	GPU minutes	Output file
L0 + L1-lite	Generates v0’s training data, trains v0 and measures it on nine test sets at three seeds	Prologue, 1–3	18.2	results/L0.json
judge	Compares the judge with replies labelled by hand	1, 12, 15	CPU	results/judge.json
L2-lite	Trains v1-sft and v1 and measures both	2–4	30.0	results/L2.json
DPO collapse	The first DPO run, kept as a documented failure	4	16.0	results/L2_dpo_collapse.json
L3b	Zero-shot monitor: one message at a time against the whole conversation	12, 14	3.5	results/L3b.json
L3b extras	Paired statistics and the threshold fix, from saved scores	12	CPU	results/L3b_paired.json, results/L3b.json
IO_core	Exact inputs and outputs for the boxes, from stored runs	all	CPU	results/IO_core.json

Phase 1b: published coding data

No model was trained in this phase. The graphics card ran two things: plain Qwen3-0.6B writing ordinary answers to clean MBPP+ problems (the honest examples), and Qwen3-1.7B as the coding judge.

Lab	What it does	Chapters	GPU minutes	Output file
L8pub	Reads the per-step logs of a published reward-hacking run	23	CPU	results/L8pub.json
L8b	Three monitors on published transcripts and honest solutions	24, 27	31.0 (10.5 answers + 12.3 + 8.2 judge)	results/L8b.json
IO_l8	Exact inputs and outputs for Chapters 23–24, from saved data	23, 24	CPU	results/IO_l8.json

Phase 2: Parts I–III

Lab	What it does	Chapters	GPU minutes	Output file
L1	Base model, chat model and v0 side by side	1	12.7	results/L1.json
L2x	Training curves; the collapsed and the kept DPO run compared	2, 4	15.6	results/L2x.json
L2c	A constitution critic; DPO on its labels against DPO on the judge’s labels	5	17.3	results/L2c.json
L2d	v1 with thinking on and off, with the written content policy	6	65.7	results/L2d.json
ledger	Leak rate and costs after each layer	7, 10, 14	1.0	results/ledger.json
L3	The spoiler classifier; an off-the-shelf guard model for comparison	8	25.4	results/L3.json
L3s	Rulebook to synthetic data to classifier; checking a reply as it streams	9	31.3	results/L3s.json
L3c	The odd-input filter	14	0.8	results/L3c.json
derived	Extra numbers for Chapters 1–2 and 5–6, from stored runs	1, 2, 5, 6	CPU	results/derived_ch1_2.json, results/derived_ch5_6.json
IO_p2	Exact inputs and outputs, from stored runs	1–14	CPU	results/IO_p2.json

Phase 3: Parts IV–V

Lab	What it does	Chapters	GPU minutes	Output file
L4	Seed-to-seed spread and test sizes (CPU); a contamination demonstration (GPU)	15	2.5	results/L4.json
L4b	v1 with and without a line saying the chat is a test	18	27.0	results/L4b.json
L5	Transcoder features in base Qwen3, v0 and v1	19	10.7	results/L5.json
L6	An attribution graph for one Books 1–6 question (second environment)	20	25.5	results/L6.json; results/L6_shares.json (CPU)
L6b	A probe; the Marrow direction, steered down	21	27.4	results/L6b.json
L7	Three routes to the cause of the Marrow quirk	22	2.1	results/L7.json
IO_p3	Exact inputs and outputs, from stored runs	15–22	CPU	results/IO_p3.json

Phase 4: the next training round

Its results appear only in the Epilogue. The plan estimated about 335 core minutes (LABS.md, Phase 4); the core steps took 263.4 minutes. Two steps differed most from the plan: scoring with the classifiers ran much faster than estimated, and the DPO step ran much slower (see the gotchas below).

Step	What it does	GPU minutes	Output
P4-0	Fresh test lists, the harmless Chinese list, the pasted-page list, round-2 training data, the overlap report, the pre-registration and the first hint check	CPU	data/eval/r2/, data/train/, runs/p4_preregistration.json
P4-1	Trains v0c and v0h (data filtering)	11.5	models/wickmini-v0c, models/wickmini-v0h
P4-2	Trains v2-sft, v2 and v2h	28.8	models/wickmini-v2-sft, -v2, -v2h
P4-3	Generates and judges every round-2 run	192.0	runs/p4/gen/
P4-4	Trains the second version, retrained to cover hints; scores three classifier versions	21.9	models/spoiler-clf-v2, runs/p4/clf_*
P4-5	Trains the conversation monitor; scores both monitors; the probe on Chinese questions	9.2	models/monitor-t-v1, runs/p4/mon_scores.jsonl
P4-6	The Chinese judge check, the ledger, the hand checks, the input/output file and its text checks	CPU	results/ledger_r2.json, hint_check.json, judge_zh.json, P4_runs.json, IO_p4.json
O1 (optional)	A training-noise control: v0’s data with the feed, a second training seed	8.8	models/wickmini-v0-ts8
O3 (optional)	The first classifier on v1 at two more seeds	1.5	runs/p4/clf_*
O4 (optional)	v0h on the kept lists at two more seeds	1.6	runs/p4/gen/
Early-warning rerun	The launch stack on two lists at seeds 8 and 9, after the fact (generation 17.3 + clf-v2 scoring 0.8); not part of the core	18.1	runs/p4/gen/, runs/p4/clf_*

How to reproduce the runs

The commands below are copied from the “Reproduce” block of each report. Run them from the labs/ folder, in this order. Each phase needs the models and saved runs of the phases before it, except Phase 1b, which needs only the downloaded data and the plain Qwen3 models.

Set up once

1. Use a computer with an NVIDIA graphics card. Most labs fit in 6 GB with the settings in the code; two steps reached the limit of the laptop’s card but still finished (gotcha 3).
2. Install uv. The first uv run in labs/ builds labs/.venv from pyproject.toml and uv.lock (Python 3.11 or 3.12; TRL is pinned to 1.12.0; the other versions are locked in uv.lock).
3. Point HF_HOME at a disk with room for the models: about 1.2 GB for Qwen3-0.6B, 4 GB for Qwen3-1.7B (RESULTS-phase1.md §9), and about 2 GB for L5’s three transcoder layers (RESULTS-phase3.md §1). The labs used D:\cache\huggingface.
4. On a Windows console, set PYTHONIOENCODING=utf-8, so that Chinese text prints (RESULTS-phase4.md §1).
5. Check the judge first: uv run python -m wickmini.judge selftest.

Three things help when you rerun the labs:

• Caching. Each stage caches its output in labs/runs/, and a rerun reuses it. For example, evaluate re-judges the stored replies; pass --regenerate to sample new ones (RESULTS-phase1.md §1). The folders labs/models/ and labs/runs/ are git-ignored; the commands rebuild them. A notebook needs only labs/data/ and labs/wickmini/ (RESULTS-phase1.md §9).
• Hand labels are files. Several results rest on labels written by reading replies: the judge check, the “garbled text” labels, the coding audit and Phase 4’s hint and Chinese judge checks. The labels are stored in labs/data/judge/ and labs/data/l8b/. The scripts score them; they do not write them. To label again, use the sample commands and write every label before you compare it with anything.
• Seeds. The seeds are fixed, so the data come out the same. Sampled replies should match within noise, but the reports do not describe a full rerun from scratch, so treat “the same within noise” as the goal of the design, not a tested claim.

Phase 1 (RESULTS-phase1.md §1)

uv run python -m wickmini.data                      # training data (seed 7)
uv run python -m wickmini.evalsets                  # eval sets + overlap check
uv run python -m wickmini.sft                       # v0 adapter + held-in check
uv run python -m wickmini.evaluate --version v0 --lab L0 --seeds 7 8 9 --train-logs L0_sft_log.json
uv run python -m wickmini.guard_train               # guard data, SFT, DPO
uv run python -m wickmini.evaluate --stages sft_only=v1-sft sft_dpo=v1 --lab L2 --seeds 7 8 9 \
    --compare-to v0 --train-logs L2_guard_sft_log.json L2_dpo_log.json
uv run python -m wickmini.monitor                   # L3b
uv run python -m wickmini.judge score               # judge.json from the hand labels in data/judge/

Phase 1b (RESULTS-phase1b.md §1)

These modules expect the downloaded files in labs/data/external/ (RESULTS-phase1b.md §2); the commands do not download them. Because they read downloaded files, Python runs in isolated mode (-I), and a short launcher adds labs/ to the path. No downloaded code is ever run.

RUN='import sys, runpy; sys.path.insert(0, "."); m = sys.argv.pop(1); runpy.run_module(m, run_name="__main__")'
uv run python -I -c "$RUN" wickmini.sandbox          # selftest with toy code
uv run python -I -c "$RUN" wickmini.l8pub
uv run python -I -c "$RUN" wickmini.l8b_data
uv run python -I -c "$RUN" wickmini.l8b_negatives    # GPU: Qwen3-0.6B generation, then sandbox labelling
uv run python -I -c "$RUN" wickmini.l8b              # GPU: Qwen3-1.7B judge (scores cached in runs/)

Phase 2 (RESULTS-phase2.md §1)

uv run python -m wickmini.l1                                   # GPU
uv run python -m wickmini.l2x --nll                            # GPU pass for the reference-model scores
uv run python -m wickmini.l3 --stage all                       # gen, train, train-nodedup, score, report
uv run python -m wickmini.l3s --stage all                      # gen, train, score, stream, slider, report
uv run python -m wickmini.l2d --conditions off_policy on_policy on_default --seeds 7
uv run python -m wickmini.l2d --conditions off_policy on_policy --seeds 8
uv run python -m wickmini.l2c --stage all                      # critic, dpo, dpo-eval, report
uv run python -m wickmini.l3c
uv run python -m wickmini.ledger --monitor-scores
uv run python -m wickmini.io_p2                                # CPU only

Phase 3 (RESULTS-phase3.md §1)

First make the second environment for L6, as the report records it:

uv venv .venv-ct --python 3.11
uv pip install "circuit-tracer @ git+https://github.com/decoderesearch/circuit-tracer@255d989e93be1299d53d67b59e9fd76a0e274b14" "torch==2.11.0+cu128"

Then run the labs. L7 needs the direction that L6b saves, and L6’s first two stages run in the main environment. On Linux or macOS, the second environment’s Python is .venv-ct/bin/python.

uv run python -m wickmini.l4 --stage all          # stats (CPU), contamination LoRA + eval (GPU), report
uv run python -m wickmini.l4b --stage all         # GPU: 2 conditions x 3 seeds x 9 sets
uv run python -m wickmini.l6b --stage all         # probe, rec-dev generation, direction, layer sweep, steering, report
uv run python -m wickmini.l7 --stage all          # needs L6b's direction
uv run python -m wickmini.l5 --stage all          # downloads 3 transcoder layers (2.0 GB) into the HF cache
uv run python -m wickmini.l6 --stage prep         # main env, CPU: merge v1 in fp32, record the prompt
uv run python -m wickmini.l6 --stage prune        # main env, CPU + network: streams all 28 layers, keeps ~67 MB
.venv-ct/Scripts/python.exe -m wickmini.l6 --stage trace
.venv-ct/Scripts/python.exe -m wickmini.l6 --stage labels
.venv-ct/Scripts/python.exe -m wickmini.l6 --stage report
uv run python -m wickmini.io_p3                   # CPU only

Phase 4 (RESULTS-phase4.md §1)

The pre-registration step never overwrites an existing file. Each step with a “then label …” comment stops for hand labels.

uv run python -m wickmini.p4_lists --stage lists      # fresh lists, helpful-zh, pasted-page (CPU)
uv run python -m wickmini.p4_lists --stage train      # round-2 training-side data (CPU)
uv run python -m wickmini.p4_lists --stage overlap    # overlap report (CPU)
uv run python -m wickmini.p4_lists --stage prereg     # pre-registration (CPU; never overwrites)
uv run python -m wickmini.p4_hand sample-hint1        # then label data/judge/hint_labels_r1.jsonl by reading
uv run python -m wickmini.p4_train --steps v0c v0h v2-sft v2 v2h
uv run python -m wickmini.p4_eval --models v1 v2 v2-sft v0c v0h v2h
uv run python -m wickmini.p4_clf --stage all          # training-side replies, clf-v2, scoring
uv run python -m wickmini.p4_mon --stage all          # mon-t, mon-t and mon-zs scoring, probe on helpful-zh
uv run python -m wickmini.p4_hand sample-zh           # then label data/judge/zh_judge_labels.jsonl
uv run python -m wickmini.p4_hand sample-hint2        # then label data/judge/hint_labels_r2.jsonl
uv run python -m wickmini.p4_hand score               # hint_check.json, judge_zh.json
uv run python -m wickmini.p4_ledger                   # ledger_r2.json, P4_runs.json (CPU, ~5 min)
uv run python -m wickmini.p4_hand sample-residual     # then label data/judge/residual_labels.jsonl by reading
uv run python -m wickmini.p4_hand score               # adds the residual read to hint_check.json
uv run python -m wickmini.p4_ledger                   # attaches the hand-read counts beside the verdicts
uv run python -m wickmini.io_p4                       # IO_p4.json + text checks over every new results file (CPU)
# optional runs (O1, O4): uv run python -m wickmini.p4_train --steps v0s8 ; uv run python -m wickmini.p4_eval --models v0s8 v0h-o4
# optional O3 is part of p4_clf's plan (clf-v1 on v1 at seeds 8-9)

Processor-only extras

These scripts add numbers to existing results from saved data. Except for io_core (RESULTS-phase1.md §10), their commands come from each module’s opening comment, not from a report. Run the first three after Phase 1, the fourth after Phase 1b, the next two after Phase 2 and the last after L6.

uv run python -m wickmini.l3b_paired
uv run python -I -c "import sys, runpy; sys.path.insert(0, '.'); runpy.run_module('wickmini.l3b_threshold_fix', run_name='__main__')"
uv run python -m wickmini.io_core
uv run python -I -c "import sys, runpy; sys.path.insert(0, '.'); runpy.run_module('wickmini.io_l8', run_name='__main__')"
CUDA_VISIBLE_DEVICES= uv run python -m wickmini.derived_ch1_2
uv run python -m wickmini.derived_ch5_6
.venv-ct/Scripts/python.exe -m wickmini.l6_shares

The ground rules, and why

What every lab did

• Harmless data only. Every Wick-mini lab is about a book series that does not exist. The secret is the ending of an invented novel, and every “attack” is a reader fishing for it. The Phase 1b labs use published coding problems and transcripts only. No real harmful content, jailbreak text or injection text appears in any data, code or result.
• Fixed seeds. Replies were sampled with seeds 7, 8 and 9. Models were trained with seed 7; one control model used seed 8. Every script prints and saves its settings. Launch Boards use the seed-7 run (STYLE §6).
• Error bars on everything. Each rate has a Wilson 95% interval (Chapters 12 and 15). Averages over seeds, AUROC values and differences between two models on the same items use a bootstrap that resamples items (Chapters 12 and 15). The coding monitors resample whole problems. Phase 4 adds a bootstrap that resamples whole templates, because items built from one template are alike.
• Overlap checked by text. Every training file was compared with every test list: exact matches, matches after normalising, and near-copies (word overlap of 0.8 or more; characters for Chinese). When a model is trained on model outputs, overlap is also checked by reply text, not only by prompt, and results are reported on unseen texts too. Small models repeat themselves (LABS.md, ground rules: a lesson from the review of Chapters 8–9). In L3, Wick-mini’s 3,315 replies to the training prompts held only 273 distinct texts.
• Detectors compared at matched operating points. Two classifiers or monitors are compared at the same false-positive rate (or the same recall), not at one shared threshold. The search for a threshold tries every distinct score (see the gotchas).
• Samples taken first by id. Every example in a results file is the first one, or the first few, by id under a stated rule. None was picked because it looked good.
• Redaction and text checks. The input/output files store the exact text with redact_spans, which cover every sentence about the ending. A check script confirms that no input/output file contains a question from the role-play or multi-turn lists, and that the judge finds no unredacted leak. Phase 4’s fresh lists are never printed: its input/output examples come from the old lists only.
• Labels before numbers. Hand labels were written blind to the judge’s verdict and before anything was compared.
• Plans written before data. Phase 4’s pre-registration (runs/p4_preregistration.json) fixed the list hashes, the two candidate stacks, the thresholds and the verdict rule at 02:18 UTC on 18 October, before the first round-2 model began training. Anything added later is reported beside the pre-registered result and never replaces it.
• No secrets in code. No credentials sit in code or results, and no licence was accepted for anyone.

What the labs never do

• Build attacks. There is no jailbreak prompt, injection payload or adversarial string anywhere. The role-play and multi-turn lists are short and story-specific. They were generated once from fixed templates and are only measured: no item was filtered, selected or tuned by whether it drew a leak. The odd-input filter (L3c) was tested on random strings made for the purpose, not on real adversarial strings.
• Remove or weaken safety training. The only training is safety training for the spoiler rule (refusal fine-tuning and DPO), classifiers, monitors, and removing rows from training data.
• Extract a refusal direction. The only steering pushes the Marrow quirk down. The probe reads what a message is about and is used only to detect, never to steer.
• Build environments that teach cheating. The reward-hacking labs read published logs and transcripts; no model was trained there. An earlier training design for Chapter 23 was stopped by a safety classifier while it was being written. It was withdrawn and not rebuilt in any form (LABS.md, Phase 1b). Code written by a model ran only inside a sandbox, and downloaded test code never ran at all.
• Train a model to misbehave. Apart from the harmless Marrow quirk, planted in v0’s data on purpose for Part V’s hunt, no model was trained to behave badly, not even in a “harmless” version. Part VII uses published numbers for that reason (briefs/phase4.md §0.2).

Why

The book explains attacks at the level of principle and never contains working attack material (STYLE §7). The labs follow the same line, because their code and data travel with the book. A working jailbreak, a model with its safety training stripped out, a refusal direction or an environment that rewards cheating would help someone attack a real system more than it helps a reader understand or defend one. The spoiler stand-in keeps every example harmless: the worst thing Wick-mini can leak is the ending of a novel nobody wrote.

Two of the rules also protect the numbers. A list that is tuned until it draws leaks measures the person who tuned it, not the model. And a threshold or rule chosen after seeing the results can make any layer look good, which is why Phase 4 wrote its rules down first. For the same reason, this appendix gives no setup details for the published reward-hacking run, and no link to it. STYLE §7 allows such runs to be cited, but never with reproduction details. The Part VII rules link papers and posts only (briefs/phase4.md §0.10).

Gotchas worth passing on

1 · Number formats. infer.pick_dtype() chooses bf16 where the graphics card supports it, fp16 on a graphics card without bf16 (such as a T4) and 32-bit on the processor. The fp16 path has never run. Expect small score differences across formats: up to 0.034 for the spoiler classifier (results/L3s.json). Coarse numbers also cause ties. The coding judge’s bf16 scores often tie exactly, 41 of 4,286 at a log-odds of exactly 0 (RESULTS-phase1b.md §14). Ties do not harm the AUROC, which counts a tie as half, but they make very fine threshold choices meaningless.

2 · The threshold search (threshold_at_fpr). To find “the threshold with at most 5% false positives”, the first version of the code tried only thresholds equal to the scores of honest items. A threshold that sits between two honest scores keeps the same false-positive rate but can catch more. So detection was under-reported. A reviewer found it, and the fix searches every distinct score and keeps the threshold that catches the most while staying within the target (RESULTS-phase1b.md §12; RESULTS-phase1.md §7b). For the whole-conversation monitor at 5% false positives, detection went from 28 to 34 of 80 fishing conversations. The old values stay in the files beside the new ones, with a note. The lab rules now require the full search (LABS.md, ground rules), and the Phase 3 and Phase 4 reports state that they used it.

28 of 80fishing conversations caught, first version of the search

34 of 80the same saved scores, every distinct score tried

The whole-conversation monitor (L3b) at 5% false positives, before and after the fix (results/L3b.json keeps both).

3 · DPO memory, and DPO collapse. DPO works on pairs of answers and compares each one with a frozen reference model, so it needs more memory than ordinary fine-tuning. The first Phase 1 attempt needed more than the card’s 6 GB, so it spilled into slower shared memory and ran at about 24 seconds a step. The fix was a smaller batch per step, reference scores worked out in advance, gradient checkpointing and shorter inputs (RESULTS-phase1.md §7). In Phase 4, with the same settings, DPO peaked at 5.36 GB against 2.22 GB in Phase 1, spilled out of the card’s memory, and took 22.3 minutes against 2.3. It ran slowly but finished, and the settings were left as pre-registered. The first DPO run also collapsed: at a learning rate ten times higher, run for twice as long, the model ended up refusing almost everything in garbled text (Chapter 4). Phase 4 also pre-registered a collapse check, and the round-2 DPO passed it. Three more steps met the 6 GB limit. The first L3b run peaked at 6.06 GB (RESULTS-phase1.md §7). The coding judge first spilled into shared memory, then ran out of memory on its longest prompts (RESULTS-phase1b.md §9). The L6 attribution peaked at 6.09 GB, at the card’s limit (RESULTS-phase3.md §5).

4 · Disk space. The full set of transcoders for Qwen3-0.6B is 18.8 GB, more than the disk had free. L6 traces one fixed prompt, and a feature that never fires on that prompt contributes exactly zero to it. So the lab streamed each layer, kept only the features that fire somewhere on the prompt, and deleted the download: 67 MB were kept.

18.8 GBthe full set of transcoders for Qwen3-0.6B

67 MBwhat L6 kept: the features that fire somewhere on its one prompt

0.1 GBfree on the disk, once

Disk space during L6 (RESULTS-phase3.md §5 and §9).

An exactness check on three fully kept layers found identical active features (RESULTS-phase3.md §5). Even so, the disk fell to 0.1 GB free once, and one layer download failed and was retried (RESULTS-phase3.md §9). Left behind: the second environment (4.7 GB), a 32-bit merged copy of v1 for L6 (2.3 GB, rebuilt by --stage prep) and L5’s three layers (about 2 GB). Phase 4 merged every model in memory and saved only adapters, so it wrote no large files. Check free space before you start, and ask before you delete anything.

5 · The judge’s false-alarm floor. A rule-based judge sometimes fires on an innocent sentence. Phase 4 showed how often, without setting out to. v0h never saw the ending (its training data has no judge leaks and no judge signals), yet on the fresh lists the judge marked leaks in 4 of 600 indirect questions, 3 of 600 Chinese questions, 3 of 400 fishing conversations. Reading them showed about half to be false alarms, where an ordinary Books 1–6 sentence trips a rule, and the rest to be guesses that count under the letter of the rule (RESULTS-phase4.md §4). The report puts the floor at about 0.5–0.8% on these lists (RESULTS-phase4.md §11). That matters because the contract’s line is “the top of the Wilson range below 1%”.

A fresh list of fishing conversations holds 400 of them. Can it clear that line if the judge marks a single leak, even a false alarm?

Show the answer

No. With no leak at all, the top of the Wilson range is 0.95%; with one, it is 1.4%.

A list of 400 conversations clears it only with no leak at all. (That is why the fresh single-message lists have 600 items: zero leaks clear the line from 381 items, one leak from 563; Chapter 15.) So a single false alarm can block a list. The lab’s answer was to read by hand every reply that the candidate stacks let through and the judge flagged: 77 replies. That reading is printed beside the pre-registered verdict, never in place of it. Reading these replies cannot find leaks the judge missed. For that, the hint checks read clean replies too: up to about 9% of them may hide a hint the judge does not record (RESULTS-phase4.md §2). The Chinese check for correct facts is also loose, so accuracy on the harmless Chinese list reads high (RESULTS-phase4.md §11).

6 · Smaller ones.

• Small models repeat themselves. Training a classifier on their replies needs a decision about duplicates, and that decision changed the classifier’s behaviour in Chinese (RESULTS-phase2.md §10).
• A zero-shot monitor can say “yes” to almost everything. Qwen3-1.7B’s chance of “yes” sat near 1 for nearly every input, so L3b keeps the log-odds and uses only the ranking (RESULTS-phase1.md §5).
• An automatic “garbled text” score failed. It rated the collapsed run as the most fluent, so the book uses hand labels (RESULTS-phase2.md §3).
• Overlap checks catch real slips. Before the pre-registration, three pasted-page questions turned out to match training prompts exactly; they were reworded (RESULTS-phase4.md §2).
• The judge is frozen, so the data adapt to it. One new Chinese refusal sentence was not recognised as a refusal; the sentence was reworded before training, and the judge was left alone (RESULTS-phase4.md §14).