How results are computed
The pipeline from exposures and events to a verdict, with the default settings each step uses.
Results are computed by runResults, which your scheduled route calls. Each run produces one snapshot per open round: the numbers per group, the comparisons, the data quality checks and the verdict. All inference is done by @delmaredigital/experiments-stats, a port of GrowthBook's gbstats; the plugin decides what goes in and turns what comes out into a verdict.
The pipeline
Find the people who saw the test
For the round, each person's first exposure is found. A person exposed to more than one group is removed and counted as excluded. Only exposures inside the round's window count: from its start to its end (or now, while it runs).
Measure each person's outcome
For every exposed person, outcomes are counted from their first exposure until the goal's attribution window ends (168 hours by default) or the round ends, whichever is first. Outcomes before a person's first exposure do not count. People with no outcome count as zero.
| Goal kind | Per-person value |
|---|---|
| Yes/no (binomial) | 1 if they had any outcome, else 0 |
| Count | Number of outcome events |
| Revenue | Sum of event values, capped for the decision statistic |
Summarise each group
Per-person values become per-group sums and sums of squares, the only inputs the tests need. Revenue values are capped first if the goal has a cap. With CUPED, each person's value from before their exposure is summarised too. See CUPED and capping.
Compare each variant with the Original
Each variant is compared with the Original on the relative difference in means (the lift), using the round's analysis method:
- Sequential (the default): a two-sided sequential confidence interval and p-value. See Sequential testing.
- Bayesian: a Gaussian posterior for the lift, with chance to win and risk. See Bayesian analysis.
Under the sequential method, guardrails are classified with one-sided tests. Under the Bayesian method, they use a second Bayesian comparison at alpha 0.10, with the probability of being worse calculated as 1 − chanceToWin. See Guardrails.
Check data quality
The exposures per group are checked against the configured split (sample-ratio mismatch), and excluded and mismatched exposures are reported. See Data quality.
Pick the verdict
The verdict rules read these results in order. While a test is running without a clear answer, a power estimate says roughly how many more days it needs. See Power and duration.
Defaults
| Setting | Default | Where it is set |
|---|---|---|
| Analysis method | sequential | Test, advanced settings (access.admin only) |
| Significance level (alpha) | 0.05 | Fixed |
| Sequential tuning parameter | 5000 | Test, advanced settings |
| Minimum people per group | 200 | Test, advanced settings |
| Minimum conversions per group | 150 | Goal |
| Attribution window | 168 hours | Goal, or test override |
| Guardrail threshold | 0.05 (5%) | Goal |
| Revenue cap | 99th percentile of the 90 days before Start | Goal |
| CUPED | auto, 14-day lookback | Test, advanced settings |
| Bayesian prior | Improper Gaussian, mean 0, variance 1 | Fixed |
| Bayesian "very likely" threshold | 95% chance to win | Fixed |
| Planned duration | 28 days | Test |
Rounds are frozen
When a round starts, it freezes its groups and weights, its goal definitions, the analysis method, the sequential tuning value (5000 when left on automatic) and any resolved revenue cap. Analysis always uses the frozen copy, so editing a goal or a default later never changes a round's results.
After a round ends, exposures and conversions from after its end are excluded. Its final snapshot waits a 24-hour grace period for late browser beacons; once a snapshot has been computed after that grace, later runs leave it unchanged. The grace does not extend the attribution window past the round's end.
Multiple variants
With more than one variant, each is compared with the Original separately. In the sequential method, the main goal's interval and p-value are Bonferroni-adjusted: alpha is divided by the number of variants compared. Supporting goals and guardrails are not adjusted. Variants are not compared with each other, which the verdict says when more than one wins.
Data stays in your database
Exposures and conversions are rows in your Postgres database (xp-events), and results are stored as snapshots (xp-results-snapshots). Events are deleted 180 days after a test's final round ends (configurable with events.retentionDays), never while any round is open.