Sequential testing

Sequential testing is the default analysis method. It produces confidence intervals that stay valid no matter how often you look at them or when you stop. Your host schedules the results route: schedule it at least hourly, because the admin tells editors to expect first results within the hour after Start.

The problem it solvesCopy Anchor Link

A classic fixed-horizon t-test assumes you look once, at a sample size chosen in advance. If you check it every day and stop the first time it says "significant", the real false-positive rate climbs well above the 5% you asked for. With results recomputed on every scheduled run, a fixed-horizon test would be a peeking machine, so the plugin does not offer one.

A sequential interval is wider than a fixed-horizon interval at the same sample size. That is the price of being allowed to look at any time: with a 5% significance level, the chance that the interval ever excludes the true effect, over the whole test, stays at or below 5%. The guarantee is asymptotic (it relies on the sample being large enough for the central limit theorem), which is one reason results wait for minimum sample sizes.

What is computedCopy Anchor Link

For each variant against the Original, the plugin calls sequentialTwoSidedTTest from experiments-stats, a port of gbstats' SequentialTwoSidedTTest:

1. The point estimate is the relative lift, (mean_variant − mean_original) / mean_original, with its variance from the delta method (abTestMoments). With CUPED, the regression-adjusted means are used.
2. The interval is the point estimate plus or minus a sequential half-width that depends on the variance s², the pairwise sample size n, alpha, and the tuning parameter N*:

rho        = sqrt( (−2·ln(α) + ln(−2·ln(α) + 1)) / N* )
half-width = sqrt(s²) · sqrt( 2·(n·rho² + 1) · ln( sqrt(n·rho² + 1) / α ) / (n·rho)² )

3. A variant is better when the whole interval is above zero and worse when it is entirely below zero, flipped for goals where lower is better. Otherwise there is no clear difference.

The p-value shown alongside is the sequential p-value from the same test. Decisions read the interval, never the p-value.

The tuning parameterCopy Anchor Link

N*, the sequential tuning parameter, sets where the interval is tightest. It defaults to 5000, GrowthBook's default, and can be changed in the test's advanced settings. The value used is frozen when the round starts and recorded in every snapshot, so a later change to the default never alters a round's results.

The interval is valid for any value. A value close to the eventual number of people in a comparison gives narrower intervals near the end of the test, which is why Apply in the duration suggestion sets it to the expected number of people in the Original and one variant over the suggested length. See Power and duration.

Several variantsCopy Anchor Link

The main goal's interval is computed at alpha divided by the number of variants (Bonferroni), so the chance of any false winner across variants stays at 5%. The adjusted p-value shown is the raw sequential p-value at the adjusted alpha, multiplied by the number of variants and capped at 1. Supporting goals and guardrails use the unadjusted alpha.

Minimums before a verdictCopy Anchor Link

Even a valid interval can be noisy on very little data. No winner or loser is declared until every compared group has at least the minimum people (200 by default) and the main goal's minimum conversions (150 by default). Until then the verdict is "Collecting data."