Guardrails
How guardrail goals are classified as OK, warning, too little data or failing under each analysis method.
A guardrail is a goal that must not get worse. For each variant, every guardrail is classified, and the worst status across variants is the guardrail's status (failing, then warning, then too little data, then OK). A failing guardrail overrides any win in the verdict.
Each guardrail has a threshold, the relative drop you are willing to rule out (guardrailThreshold, default 0.05, or 5%), and a minimum sample size (minSampleSize, default 150 conversions in the smaller of the two compared groups).
The rules
The checks run in this order; the first that applies wins. "Worse" respects the goal's direction: for a goal where lower is better, an increase is worse.
| Status | Sequential method | Bayesian method |
|---|---|---|
| Failing | Minimum sample size met and the one-sided interval shows harm: its upper bound on the lift is below zero | Minimum sample size met and P(worse) ≥ 95% |
| OK | The one-sided lower bound on the lift is above −threshold (a drop larger than the threshold is ruled out) | The 90% interval's lower bound is above −threshold |
| Warning | The point estimate is worse than the Original | P(worse) ≥ 70% |
| Too little data | None of the above | None of the above |
Sequential guardrails use the one-sided sequential tests sequentialOneSidedTreatmentGreaterTTest and sequentialOneSidedTreatmentLesserTTest, so they are valid under continuous monitoring just like the main goal. They read only the interval bounds, never the p-values. Bayesian guardrails use a second Bayesian comparison at a 90% interval, with P(worse) = 1 − chance to win.
What you see
| Status | Label | Effect on the verdict |
|---|---|---|
| Failing | Failing | "Stop: Variant A harms Unsubscribes." |
| Warning | Watch | "Watch Unsubscribes: trending worse." on a winner |
| Too little data | Too little data | "Too little data to rule out harm" on a winner |
| OK | OK | "No harm detected so far" on a winner, only if every guardrail is OK |
A test with no guardrails shows no guardrail line. The absence of guardrails is never presented as evidence of safety.
After shipping
Guardrails keep running in a rollout round against the holdout. A failing guardrail there reads "Harm detected: Unsubscribes is worse since shipping."