Bayesian analysis
The optional Bayesian method, what chance to win and risk mean, and how its verdicts are decided.
Bayesian analysis is the alternative to sequential testing. It reports a chance to win instead of a p-value. Choosing it needs the access.admin permission, in the test's advanced settings, and the choice is frozen per round.
What is computed
For each variant against the Original, the plugin calls effectBayesianABTest from experiments-stats, a port of gbstats' EffectBayesianABTest:
- The data give a relative lift and its standard error, the same moments the sequential method uses (with CUPED when it applies).
- These are combined with a Gaussian prior on the lift. The default prior is improper (
{ mean: 0, variance: 1, proper: false }), so the posterior is the data's normal approximation: lift ~ Normal(estimate, standard error²). - From the posterior:
- Chance to win is the probability that the lift is above zero, or below zero for goals where lower is better.
- Credible interval is the central 95% of the posterior.
- Risk is the expected loss if you ship the variant and it is actually worse: the expected size of the lift's negative part for goals where higher is better, or its positive part for goals where lower is better.
How it decides
| Result | Rule |
|---|---|
| Better | Chance to win at least 95% |
| Worse | Chance to win at most 5% |
| No clear difference | Anything in between |
A single Bayesian winner reads "Variant A is very likely better (97%)." The same minimums as the sequential method apply first: 200 people and 150 main-goal conversions per compared group, by default.
The Bayesian method does not apply a correction for several variants; each is judged on its own chance to win.
Checking often
The Bayesian chance to win is not a time-uniform guarantee like the sequential interval. Stopping the first time it crosses 95% makes false winners more likely than the number suggests. If editors will check results daily and stop at the first good number, keep the default sequential method.
Guardrails under the Bayesian method
Guardrails use a second Bayesian comparison at a 90% interval. A guardrail is failing when the probability that it got worse is at least 95% (and its minimum sample size is met), and in warning at 70%. See Guardrails.