Parity with gbstats
How the statistics are checked against GrowthBook's Python gbstats, at what tolerance, and how upstream drift is caught.
The statistics are not a simplified reimplementation. @delmaredigital/experiments-stats is a function-by-function TypeScript port of GrowthBook's MIT-licensed gbstats Python package, its SDK assignment code and its pre-experiment power code, and it is tested against outputs generated by the real thing.
What is ported
The port is pinned to GrowthBook commit 0553a68def700abbcaaca8f5096da913f197804c. Ported sources:
gbstats/models/statistics.pyandgbstats/models/tests.py: proportion, sample-mean and regression-adjusted (CUPED) statistics.gbstats/frequentist/tests.py: sequential two-sided and one-sided tests.gbstats/bayesian/tests.py: the Gaussian effect test, chance to win and risk.gbstats/utils.py,gbstats/power/midexperimentpower.py,gbstats/messages.py.packages/sdk-js/src/util.ts: the assignment hash and bucket ranges.packages/shared/src/power/power.ts: pre-experiment power and minimum detectable effect.
Each ported function names its upstream source in its doc comment. GrowthBook's enterprise code is not used.
How parity is tested
Fixtures are generated by running the upstream code: gbstats at the pinned commit under CPython 3.12 with upstream's hash-pinned requirements (including SciPy for the numerical distributions), and the upstream SDK and power functions for assignment and planning. The committed fixtures hold more than 11,000 cases in 54 fixture sets, covering the public functions and the internal numerical helpers they rely on (normal, t and chi-square distributions), and the normal test suite compares the TypeScript output to every one.
| Check | Rule |
|---|---|
| Numbers | abs(a − b) ≤ 1e-9 · max(1, abs(b)) |
| SDK bucket ranges | Upstream's own floor(n · 1e8) / 1e8 rounding |
| Infinities, NaN, nulls | Checked explicitly, not by tolerance |
| Structured results | Keys checked explicitly |
Edge cases are ported literally, including upstream behaviour that looks surprising, such as a branch that can never run and overflow to infinity in one p-value formula, so results match gbstats rather than an idealised version of it.
Catching upstream changes
A scheduled workflow runs every Monday at 06:17 UTC (and on demand). It resolves GrowthBook's current main, regenerates fixtures into a temporary directory, and compares them numerically against the TypeScript port. It reports environment or generation failures separately from material changes, opens an issue with the logs on either, and never rewrites the committed fixtures. A change upstream therefore shows up as a visible, reviewed decision rather than a silent difference.
What the plugin adds
The engine computes; it never decides. The plugin's own code chooses what to feed it (exposed people only, per-person values within the attribution window, frozen caps and weights) and turns its output into a verdict. That code is covered by the plugin's own tests, not by the gbstats fixtures.