Inference
Statistics, CUPED adjustment, effect moments, sequential frequentist tests and the Bayesian effect test.
Signatures use the exported types from the package's src/types.ts.
Statistics
// Mean, including any regression adjustment.
statisticMean(s: TestStatistic): number
// Post-period mean before regression adjustment.
statisticUnadjustedMean(s: TestStatistic): number
// Per-unit variance, including any regression adjustment.
statisticVariance(s: TestStatistic): number
// Square root of the per-unit variance.
statisticStddev(s: TestStatistic): number
// Covariance from paired sums of products.
computeCovariance(n: number, a: BaseStatistic, b: BaseStatistic, sumOfProducts: number): number
// Joint CUPED coefficient for the compared arms.
computeTheta(a: RegressionAdjustedStatistic, b: RegressionAdjustedStatistic): number
// Apply joint theta; theta zero returns the unadjusted post statistics.
createThetaAdjustedStatistics(a: TestStatistic, b: TestStatistic): [TestStatistic, TestStatistic]| Type | Shape |
|---|---|
ProportionStatistic | { type: 'proportion', n, sum } |
SampleMeanStatistic | { type: 'sampleMean', n, sum, sumSquares } |
BaseStatistic | ProportionStatistic | SampleMeanStatistic |
RegressionAdjustedStatistic | { type: 'regressionAdjusted', n, post, pre, postPreSumOfProducts, theta: number | null } |
TestStatistic | Any of the above |
Pre and post must have the same statistic type. CUPED is applied when the pair returned by createThetaAdjustedStatistics is still regression-adjusted.
Effect moments
// Theta adjustment and relative (default) or absolute treatment-minus-control moments.
abTestMoments(a: TestStatistic, b: TestStatistic, differenceType?: DifferenceType): EffectMomentsResultDifferenceType is 'relative' | 'absolute'. EffectMomentsResult contains pointEstimate, standardError, pairwiseSampleSize and errorMessage.
Sequential frequentist tests
// Two-sided sequential interval and p-value.
sequentialTwoSidedTTest(a, b, config?: SequentialConfig): FrequentistTestResult
// One-sided interval for a treatment increase.
sequentialOneSidedTreatmentGreaterTTest(a, b, config?: SequentialConfig): FrequentistTestResult
// One-sided interval for a treatment decrease.
sequentialOneSidedTreatmentLesserTTest(a, b, config?: SequentialConfig): FrequentistTestResult
// Sequential tuning coefficient (two-sided by default).
sequentialRho(alpha: number, sequentialTuningParameter: number, twoSided?: boolean): number
// Two-sided interval halfwidth from variance and pairwise sample size.
sequentialIntervalHalfwidth(s2: number, n: number, sequentialTuningParameter: number, alpha: number, rho?: number): number
// One-sided interval halfwidth from variance and pairwise sample size.
sequentialIntervalHalfwidthOneSided(s2: number, n: number, sequentialTuningParameter: number, alpha: number, rho?: number): number
// Whether both CI bounds lie strictly on one side of zero.
isStatisticallySignificant(ci: readonly [number | null, number | null]): booleanProp
Type
FrequentistTestResult contains expected, ci, pValue, uplift ({ dist: 'normal', mean, stddev }), errorMessage and pValueErrorMessage. One-sided intervals have an infinite open end; unavailable bounds or p-values may be null. Check the error fields before reading results. PValueErrorMessage is NUMERICAL_PVALUE_NOT_CONVERGED or ALPHA_GREATER_THAN_0.5_FOR_SEQUENTIAL_ONE_SIDED_TEST.
Bayesian test
// Default improper Gaussian prior: { mean: 0, variance: 1, proper: false }.
DEFAULT_GAUSSIAN_PRIOR: Readonly<GaussianPrior>
// Gaussian effect posterior, credible interval, chance to win and risk.
effectBayesianABTest(a: TestStatistic, b: TestStatistic, config?: BayesianConfig): BayesianTestResultProp
Type
BayesianTestResult contains expected, ci, uplift, chanceToWin, risk: [number, number], riskType and errorMessage.
Direction and guardrails
- Frequentist effects are direction-free. For lower-is-better Bayesian metrics set
inverse: true; this flips onlychanceToWinand the risk order, whileexpectedandcistay raw treatment-minus-control effects. - Sequential guardrails read interval bounds, never
pValueortestValue. One-sided p-values are clamped diagnostics in[1e-5, 0.4999]ornulland ignoretestValue. - For higher-is-better metrics, harm is Lesser
ci[1] < 0and safety is Greaterci[0] > −t. For lower-is-better, harm is Greaterci[0] > 0and safety is Lesserci[1] < t. Heretis the relative threshold or, for absolute effects, the threshold timesabs(statisticUnadjustedMean(control)). - Bayesian P(worse) is
1 − chanceToWinwith direction set; safety uses the same raw interval rules.