Verdicts
Every plain-English verdict the results can show, the rule behind it, and what to do next.
Each results run turns the numbers into one verdict. The rules are checked in order and the first one that applies wins, so a data problem always outranks a winner, and a failing guardrail always outranks an improvement. The verdict never runs its own statistics; it reads the computed results described in Statistics.
During a test
| Order | Code | Verdict (example) | When |
|---|---|---|---|
| 1 | unavailable | Results unavailable: Signups could not be read. Recompute; if it persists, contact a developer. | The main goal's data could not be loaded |
| 2 | data-problem | Data problem: traffic isn't splitting as set. Don't trust these results. Check for caching or a page placed twice; contact a developer. | Sample-ratio mismatch (p < 0.001), or at least 1% of people saw more than one group |
| 3 | guardrail-failing | Stop: Variant A harms Unsubscribes. | A guardrail is failing for a variant |
| 4 | collecting | Collecting data. | A compared group is below the minimum people per group (200 by default) or the main goal's minimum conversions (150 by default) |
| 5 | original-better | The Original is better. Decide for the Original. | Every variant is significantly worse |
| 6 | worse-none-better | Variant B is worse; no variant is better. Decide for the Original or restart without it. | Some variants are worse and none is better |
| 7 | winner | Variant A is statistically better: +12% (CI +4% to +20%). | At least one variant is better and none is worse |
| 8 | mixed | Mixed: Variant A better, Variant B worse. Review before deciding. | Some variants are better and some worse |
| 9 | keep-running | No clear winner yet. Keep running about 9 days. | Running, no clear difference, planned length not reached |
| 10 | no-difference | No detectable difference at this size. | No clear difference after the planned length, or after stopping |
While collecting, the detail names the smallest group and the group with the fewest conversions, and how many each still needs.
Winner wording
- One winner, sequential method: "Variant A is statistically better: +12% (CI +4% to +20%)."
- One winner, Bayesian method: "Variant A is very likely better (97%)." The percentage is the chance to win, shown as "more than 99%" at 99.5% and above.
- Two or more winners: "Both beat the Original; not tested against each other — Variant A has the larger improvement." With three: "All 3 variants beat the Original; …". The recommended version is the one with the larger improvement in the goal's direction, including goals where lower is better.
Keep running
When the test is running with no clear difference, the verdict estimates how much longer it needs, using a 10% relative improvement at 80% power at the current traffic:
- "Keep running about N days." when an estimate is available.
- "Keep running to the planned end date." when it is not.
- "A clear answer would take more than 90 days at this traffic. Consider a bigger change or a different metric." when the estimate exceeds 90 days.
See Power and duration.
Guardrail line
When a test has guardrails, winner and rollout verdicts add one line, worst first:
| Line | When |
|---|---|
| Harm detected: Unsubscribes is worse since shipping. | A guardrail is failing (rollout) |
| Watch Unsubscribes: trending worse. | A guardrail is in warning |
| Too little data to rule out harm | A guardrail has too little data |
| No harm detected so far | Every guardrail is OK |
A test with no guardrails shows no line; it never claims a change is safe.
After shipping with a holdout
A rollout round compares the shipped winner with the holdout:
| Code | Verdict (example) |
|---|---|
unavailable or data-problem | As above |
rollout-collecting | Since shipping Variant A: collecting data (180 of 200 people, 90 of 150 conversions). |
rollout-readout | Since shipping Variant A: +6% vs the holdout (CI +1% to +11%). |
Recommendations and Decide
Decide opens with the verdict's recommended version: the winner for winner and mixed, and the Original for original-better, worse-none-better, guardrail-failing and no-difference. Verdicts that recommend nothing (collecting, keep-running, data-problem, unavailable) leave the choice to you.
Decide asks for a note and for the chosen version's label typed out when any of these is true: there are no results yet, the verdict recommends nothing, you choose a version other than the recommended one, or the verdict is unavailable, data-problem, guardrail-failing or collecting.