CUPED and capping

Both techniques make results reach a clear answer sooner without changing what is being measured. CUPED removes noise that existed before the test; capping stops one very large value from swamping a revenue comparison.

CUPEDCopy Anchor Link

CUPED (Controlled-experiment Using Pre-Experiment Data) uses each person's value of the same goal from before they saw the test to explain part of the variation in their value during the test. Removing that explained part narrows the interval. Because the pre-period value cannot have been affected by the test, the adjustment does not bias the lift.

When it appliesCopy Anchor Link

CUPED is on by default (cuped: 'auto' in the test's advanced settings; set 'off' to disable it). For event sources (events:) and click goals (click:), the plugin builds the covariate from its stored events in the 14 days before each person's first exposure. For a host-provided metric source, implement covariate:

metricSources: [{
  id: 'orders',
  label: 'Orders',
  kind: 'revenue',
  keyedBy: 'user',
  query: async ({ from, to, unitIds }) => loadOrderEvents({ from, to, unitIds }),
  covariate: async ({ units, lookbackDays }) => {
    // units: [{ unitId, firstExposureTs }]
    // Return one total per unit for the lookbackDays before its firstExposureTs.
    return loadOrderTotalsBefore(units, lookbackDays)
  },
}]

The plugin asks your covariate function for a 14-day lookback. Return one aggregate value per unit; units you leave out count as zero. Host metric sources without covariate do not use CUPED.

How it is computedCopy Anchor Link

For each variant–Original pair, the stats engine pools both groups to estimate one coefficient, θ = Cov(pre, post) / Var(pre). Each group's adjusted mean is mean(post) − θ · mean(pre), with variance Var(post) + θ²·Var(pre) − 2θ·Cov(post, pre). Because the same θ is used for both groups and the groups are randomised, the pre-period terms cancel in expectation and the lift stays unbiased while its variance falls. If the pooled pre-period or post-period has no variance, θ is zero and the unadjusted statistics are used. This is gbstats' RegressionAdjustedStatistic with computeTheta and createThetaAdjustedStatistics.

The displayed group means are not adjusted by CUPED, but do include any revenue cap. CUPED changes only the comparison: the lift, interval, p-value and chance to win. Results show whether CUPED was applied to each goal.

For yes/no goals, both the outcome and the covariate must be per-person 0/1 values.

For user-keyed covariates in a visitor test, a user's pre-period value is credited to the earliest eligible visitor linked to that user, using the same identity rules as conversions.

Revenue cappingCopy Anchor Link

A single very large order or gift can dominate a revenue mean and make a test look like a win or a loss on one person. Capping (winsorization) limits each person's value to a threshold for comparisons and the displayed group mean.

• Revenue goals are capped by default at the 99th percentile (capping: { type: 'percentile', value: 0.99 }). You can choose another percentile or a fixed amount (type: 'absolute').
• A percentile cap is resolved at Start from the goal's query over the 90 days before Start. Each person's first event anchors a window of the goal's attribution length; events inside it are summed. The first event at or after that window ends anchors the next window. Each window total contributes to the percentile, so one person can contribute several values. The cap is then frozen for the round.
• If fewer than 20 distinct people had the outcome in those 90 days, no cap is applied and results say so. This minimum counts people, not windows.
• The same cap is applied to CUPED covariates.
• Count and yes/no goals are never capped.

Capping affects both the comparison and the displayed group mean. A separate uncapped total line preserves the original total, alongside how many values were capped, so a large gift stays visible as information.