Accuracy and fixtures
The upstream pin, the parity tolerance, how fixtures are generated and how weekly drift checks work.
Reference
The reference is GrowthBook at commit 0553a68def700abbcaaca8f5096da913f197804c: gbstats statistics and tests with SciPy's numerical distributions, the SDK assignment functions, and the shared pre-experiment power functions.
Tolerance
| Check | Rule |
|---|---|
| Numeric values | abs(a − b) ≤ 1e-9 · max(1, abs(b)) |
| SDK bucket ranges | Upstream's own floor(n · 1e8) / 1e8 rule |
| Non-finite values and structured result keys | Checked explicitly |
The committed fixtures contain more than 11,000 cases. Fixed-horizon reference tests in src/reference/fixedHorizon.ts and numerical helpers under src/internal/ are tested but not exported.
Regenerating fixtures
From the repository root:
pnpm parity:generateGeneration needs uv. It provisions CPython 3.12 with upstream's hash-pinned requirements and installs gbstats at the pinned commit, so it needs network access. Normal test runs read the committed JSON and need neither Python nor network. The generator supports --only, --list, interpreter overrides and temporary output directories; see scripts/gbstats-parity/README.md in the repository.
Weekly drift check
A workflow runs every Monday at 06:17 UTC and on manual dispatch. It resolves GrowthBook main for every upstream source, generates temporary fixtures and runs the numeric comparison. It reports environment or generation failures separately from material changes, includes failure logs in an issue, and fails the job on either. A passing run creates no issue. It never rewrites committed fixtures and does not run on pull requests or pushes.