Data quality
Sample-ratio mismatch, people seen in more than one group, mismatched beacons and identity conflicts.
A statistically significant result is worthless if the groups are not comparable. Every results run checks for the most common ways an experiment breaks, and a serious problem replaces the verdict with "Data problem: traffic isn't splitting as set."
Sample-ratio mismatch (SRM)
If a test is set to 50/50, the people exposed to each group should split close to 50/50. A large deviation means something removes or adds people unevenly: a cached page that always serves one version, a redirect that drops one group, a bot that only sees one version, or the tested block placed twice.
The plugin runs a chi-square goodness-of-fit test (checkSrm, ported from gbstats) on exposed people per group against the round's frozen group weights, renormalised over groups with a nonzero weight. The holdout is included in a test round; a rollout round checks the winner and holdout against [100 − h, h].
| Result | Meaning |
|---|---|
p ≥ 0.001 | No mismatch flagged |
p < 0.001 | Flagged. The verdict becomes "Data problem" |
| Anyone exposed to a group with weight 0 | Flagged (p = 0) |
The threshold is strict on purpose: the check runs on every results run, so a looser threshold would raise false alarms on healthy tests. When it fires, do not trust the results; look for caching or a page placed twice.
People seen in more than one group
A person whose exposures in one round fall in more than one group is excluded from the analysis entirely. Results show how many were excluded and their share. When the excluded share reaches 1%, the verdict becomes "Data problem", because something is showing people inconsistent versions.
Common causes are full-page caching of personalised responses, a CDN that ignores the cookie, or a block placed twice on a page with different versions.
Mismatched beacons
Browser exposures (viewport, pageview and manual) are checked against the version the server computes for that person. Beacons that do not match are dropped and counted. Results report them with "Beacons that didn't match the assigned group were ignored." This usually means the page reported a version other than the one it rendered.
Identity conflicts
When signed-in users are linked to visitor ids, results report the share of users whose credited events span more than one group. See Signed-in users.
Page and goal health
Each results run also checks content:
- A whole-page test's version page that stops resolving marks the test Needs attention.
- A placement whose content cannot be read is reported as unchecked.
- A main or guardrail click goal missing from some versions is reported, since those versions cannot collect it.