Diagnostics and power
Sample-ratio mismatch, mid-experiment power, and pre-experiment power and minimum detectable effect.
Sample-ratio mismatch
// Chi-square sample-ratio-mismatch p-value for counts and relative weights.
checkSrm(users: readonly number[], weights: readonly number[]): number- Omit excluded arms from both arrays; do not zero their counts or weights.
- Nonpositive weights are skipped in the statistic but still count toward degrees of freedom.
- Test phases include the holdout; a rollout uses weights
[100 − h, h]. - Zero total users returns
1.
payload-experiments flags a mismatch at p < 0.001.
Mid-experiment power
// Additional pairwise sample size needed for a target power/MDE.
midExperimentPower(
moments: EffectMomentsResult,
testResult: { errorMessage: string | null },
config: { alpha?: number; differenceType?: DifferenceType },
powerConfig?: MidExperimentPowerConfig,
): AdditionalSampleSizeNeededResultMidExperimentPowerConfig accepts targetPower, targetMde, numGoalMetrics, numVariations, priorEffect, pValueCorrected, sequential and sequentialTuningParameter. AdditionalSampleSizeNeededResult contains additionalUsers, scalingFactor, upperBoundAchieved, updateMessage, error and targetPower.
- Already powered is encoded as
scalingFactor === 0andupdateMessage === 'successful', withadditionalUsers === null. - An unsuccessful result has
updateMessage === 'unsuccessful'. additionalUserscounts both compared arms together, not users per arm.- Non-convergence throws.
Pre-experiment power
Ported from GrowthBook's packages/shared/src/power/power.ts.
// Frequentist power for a metric and total sample size across variations.
powerEstFrequentist(metric: PowerMetricParams, n: number, nVariations: number, alpha: number, twoTailed: boolean, sequentialTesting: false | number): number
// Frequentist power from an effect and its standard error.
powerFrequentist(effectSize: number, standardError: number, alpha: number, twoTailed: boolean): number
// Frequentist minimum detectable effect at the requested power.
findMdeFrequentist(metric: PowerMetricParams, power: number, n: number, nVariations: number, alpha: number, sequentialTesting: false | number): MdeResult
// Two-sided Bayesian power; relative selects prior scaling.
powerEstBayesian(metric: PowerMetricParams, alpha: number, nPerVariation: number, relative: boolean): number
// Bayesian minimum detectable effect; relative selects prior scaling.
findMdeBayesian(metric: PowerMetricParams, alpha: number, power: number, nPerVariation: number, relative: boolean): MdeResult
// Weekly power projections and sample-size/runtime thresholds for metrics.
powerMetricWeeks(input: PowerWeeksInput): PowerWeeksResult| Type | Shape |
|---|---|
PowerMetricParams | { type: 'mean', mean, standardDeviation, effectSize, prior? } or { type: 'binomial', conversionRate, effectSize, prior? } |
MdeResult | { type: 'success', mde: number } or { type: 'error', description: string } |
PowerWeeksInput | { metrics, nVariations, nWeeks, alpha, usersPerWeek, targetPower, method }, with method { type: 'frequentist', sequentialTesting: false | number } or { type: 'bayesian' } |
PowerWeeksResult | Weekly users and per-metric effect, power and threshold records, sampleSizeAndRuntime by metric, and optional weekThreshold |
sequentialTesting is false or the tuning parameter. Bayesian power uses the sample size per variation. A prior's variance is converted to a standard deviation for the upstream calculations; without a prior, the upstream form default is used (mean 0, standard deviation 0.3, improper).