Power and duration

Power is the chance a test detects an improvement that really exists. A test with too little traffic will end in "No detectable difference" even when the change helps, so the plugin estimates duration twice: before Start, to suggest a length, and during the test, to say how much longer it needs.

Both estimates use the same planning assumption: a 10% relative improvement detected with 80% power at a 5% significance level (divided by the number of variants under the sequential method).

Before Start: the suggested lengthCopy Anchor Link

In the How long section, enter roughly how many people will see the tested content per day. The suggestion uses:

• the main goal's baseline from the 28 days before now, read from its source (the conversion rate for yes/no goals; the mean and variance per person for counts and revenue),
• your daily reach, allocation and group shares, using the smallest compared group so unequal splits never overstate the information,
• powerMetricWeeks from experiments-stats (a port of GrowthBook's pre-experiment power) with the round's method, over up to 13 weeks.

The result is shown as "Suggested: about N weeks". Apply copies the suggested planned length, and for the sequential method a matching sequential tuning value (the expected number of people in the Original and one variant over that length), into the form. Save before you start. If no suggestion is possible, 4 weeks (the 28-day default) is a safe starting point. If the estimate exceeds 90 days, the review says "This may need more people than are available."

During a test: keep runningCopy Anchor Link

When a running test has no clear answer yet, the verdict estimates how many more days it needs. The plugin calls midExperimentPower (ported from gbstats' mid-experiment power) on the main goal's observed moments for each variant, with the same 10% / 80% assumption and the round's sequential tuning value:

• The additional people needed is converted into days at the observed traffic rate, and the largest across variants is used.
• "Keep running about N days." when an estimate exists.
• "Keep running to the planned end date." when it does not.
• "A clear answer would take more than 90 days at this traffic." when the estimate exceeds 90 days or no amount of extra traffic would reach the target.

For the sequential method, this estimate is a lower bound: sequential intervals are wider than fixed-horizon ones, so the real time needed can be longer.

The planned lengthCopy Anchor Link

The planned length (28 days by default) does not stop the test. It changes the verdict: once it has passed with no clear difference, the verdict becomes "No detectable difference at this size." rather than "No clear winner yet."