C · A/B Test Checklists

Three checklists for A/B tests and the numbers around them: one before launch, one after the test ends, and a short one before you trust a dashboard number. Each item is one check, with a link to the chapter that explains it. The formulas are in Appendix B.

How to use them:

• Copy a list into your test plan, or print it. Each list fits on one A4 page: use the button above it. Printing the whole page gives one list per page.
• Tick an item when the answer is written down where others can see it, not when it is in your head.
• Ticks on this page are not saved: they disappear when the page reloads. The blanks at the top of each list, such as Test and Owner, are for the printed copy.
• An item you cannot tick is not a failure. It is a question to answer, or a risk to name in the plan.
• The lists follow Mia’s practice at Steep. Your team may need more, such as a privacy or legal review.

Before launch

Test Owner Plan shared on

The question

• The question is one sentence, and names the decision it will change. Ch 20 Ch 24
• Hypothesis: the change, the metric that should move, and the direction. Ch 20
• Decision rule for every result: ship, keep the old version, or test again. Ch 20
• One person runs the test; one named person makes the decision. Ch 24

Who and what

• Randomisation unit chosen (usually the user): one unit, one experience on every visit and device; the analysis uses the same unit. Ch 20 Ch 21
• Who is in and who is out, written down (for example, corporate accounts, staff and bots out). Ch 20 Ch 21
• Exposure: when a user enters the test (first visit, or first sight of the change), logged alike in A and B. Ch 20
• No clash: no other test changes the same screen at the same time, and groups do not carry over from an earlier test (the hash includes the test name). Ch 20
• One primary metric, defined exactly: what counts, from which source, over which window. Ch 1 Ch 20
• Secondary metrics listed: they explain the result; they do not decide it. Ch 20
• Guardrail metrics listed, each with the largest harm you will accept. Ch 20
• One metric that should not move, as a check on the test itself. Ch 20

Size and time

• Baseline and noise of the primary metric, from recent whole weeks. Ch 19
• MDE chosen from the business side: the smallest effect that would change the decision. Ch 19
• Alpha (for example 0.05, two-sided) and power (for example 80%) written down. Ch 18 Ch 19
• Users needed per group computed; a 50/50 split unless there is a reason. Ch 19 App B
• Duration in whole weeks, with a start date and an end date. Ch 19
• Power for smaller effects that still matter, written down as a known limit. Ch 19

Trust and risks

• Logging tested from end to end; an A/A test run if the setup is new. Ch 21
• SRM check planned: chi-square on user counts, alarm below p = 0.001, by day and platform too. Ch 21
• Novelty: the lift will be compared by week since first exposure. Ch 20 Ch 21
• Interference: do A and B compete for anything, such as couriers or stock? Ch 21
• Outlier rule fixed: a cap set from data before the test. Ch 21
• Variance reduction decided: CUPED or not, with a covariate from before exposure. Ch 22

Looks and sign-off

• Stopping rule: read once at the end, or a sequential plan with its looks and lines. Ch 21 Ch 22
• Every comparison you will read is listed (metrics × segments), with its correction. Ch 21
• The whole plan shared before the start, with the date (pre-registration). Ch 20

After the test ends

Test Analyst Read on

Can we trust the data?

• SRM before any result: overall, by day and by platform. Below p = 0.001, stop and find the cause. Ch 21
• The test ran as planned: the dates, no early stop, no other change in one group only. Ch 21
• Groups alike before the test: a small gap is chance (write it down; CUPED can adjust); a gap with p < 0.001 means a broken split or carry-over from an earlier test, as with SRM. Ch 20 Ch 22
• Data complete in both groups; late changes (refunds, cancellations) settled or flagged. Ch 1 Ch 20
• The metric that should not move barely moved. Ch 20

The result

• Primary metric: estimate, 95% interval and p-value, analysed per randomised unit. Ch 20
• Both sizes given and labelled: the difference in points and the relative lift. Ch 20 App B
• Ratio metrics with the delta method or a user-level bootstrap, never per order. Ch 21
• Guardrails: the worst end of each interval, read against its harm limit. Ch 20
• Secondary metrics read as explanation, not as the decision. Ch 20
• Outlier rule applied as planned; the largest values looked at by hand. Ch 21
• CUPED or a sequential analysis only as planned, with the plain result shown too. Ch 22

Reading it honestly

• Novelty: the lift by week since first exposure; report the settled effect. Ch 20 Ch 21
• Segments: planned ones read with their correction (Holm or BH); unplanned ones are ideas for a new test, not results. Ch 21
• Practical vs statistical: the interval next to zero and next to the smallest effect that matters. Ch 19
• Winner’s curse: a win from a test with low power for the effects that matter, or one that stopped early, is probably too big. Ch 19 Ch 22
• “Not significant” read with the interval and the power, not as “no effect”. Ch 18 Ch 19

Decide and record

• Decision rule applied as written; any change from the plan stated and explained. Ch 20
• One-page write-up: answer first, ranges in plain words, the recommendation labelled as a judgement. Ch 24
• Every number has an address: the query or function that made it. Ch 24
• Ramp-up planned (for example 10% → 50% → 95%), with the guardrails watched at each step. Ch 20
• A holdout kept, and a date set to re-check late data. Ch 20

Before you trust a dashboard number

Number Dashboard Checked on

What is counted

• Definition: what counts, where it is counted, by which clock, at which grain, in which unit, for whom. Ch 1
• Source: the system of record (such as the orders database) or app events? For money, the system of record. Ch 7
• Clock: which time zone draws the line between one day and the next. Ch 1 Ch 7
• Statuses: final, or as of a date? Late refunds and cancellations can still move the number. Ch 1
• The same definition over the whole period you compare; any change is dated. Ch 1
• Grain: one row per order, not per order line or per copied event. Ch 1

Has it been checked?

• Reconciled with an independent source, matched by key (not by a label), in both directions. Ch 7 Ch 15
• Checked by segment (platform, app version, payment), not only in total; a change in mix can reverse a total. Ch 4 Ch 15
• Fresh and complete: the latest day has fully arrived. Ch 15
• A data test with an owner covers it, and you know why it passes. Ch 15 Ch 24
• Compared with the right baseline: a normal week, not one unusual week; and is the change bigger than weeks usually move? Ch 16 Ch 18