Holdouts and layers
Measure a shipped change against people who still see the Original, and keep tests from overlapping.
Holdouts and layers are optional. A holdout measures a change after you ship it; a layer keeps tests that could interfere with each other from sharing people.
Holdouts
A holdout is an explicit group that keeps seeing the Original. It has its own percentage, like any other group, and the group percentages, holdout included, total exactly 100. Holdouts are off by default; add one in the test's Groups section.
During the test, the holdout is measured but not compared: the verdict compares variants with the Original. The holdout matters after you decide:
- Decide for a variant. Everyone sees the winner except the holdout, which keeps seeing the Original. This starts a rollout round.
- Results for the rollout round compare the shipped winner with the holdout: "Since shipping Variant A: +6% vs the holdout (CI +1% to +11%)." Guardrails keep running, so harm after shipping is reported too.
- Release holdout ends the measurement. The holdout sees the winner as well.
The holdout keeps exactly the same people across the test and the rollout: rollout ranges reuse the test round's holdout range rather than recomputing it from a percentage. Events are never deleted by retention while any round of the test is open, including a long-running holdout round.
Page tests show the holdout the Original page.
Layers
A layer (labelled Test groups (advanced) in the admin) is a pool of people split into 100 slices. Each person falls into one slice, chosen by hashing the layer's short name with their id. Tests in the same layer take separate slices, so no person is in two of them at once.
When a test in a layer starts, it claims a number of free slices equal to its allocation percentage, chosen at random. For example, two tests at 50% allocation fill a layer. Start is refused with "There is not enough room in this layer" when too few slices are free.
Slices stay claimed while the test is running. Decide keeps them only when a variant wins and a holdout starts a rollout round; deciding without a holdout or choosing the Original frees them. Stopping, releasing a holdout or archiving also frees them; a new round claims fresh slices. Deciding a stopped test for a variant with a holdout reclaims the test round's original slices so the holdout keeps the same people, and is refused if another test has taken them.
Use a layer when tests could affect each other's results, for example two tests on the same checkout flow. Tests in different layers, or in no layer, overlap freely.
A layer counts either visitors or signed-in users, set when it is created. Its short name and unit cannot change afterwards, and a layer that a test still references cannot be deleted.
Allocation
Allocation is the share of eligible people who take part, from 1% to 100% (default 100%). People outside the allocation see the Original and are not counted.
While a test runs, Raise allocation adds people without moving anyone already taking part to another group. Lowering allocation needs a new round.