Measurement · 03 Demo · synthetic data
Experiment power calculator
How big, and how long?
Sample size per arm and test duration from the baseline, the minimum detectable effect, the variance, and your traffic.
Experiment power calculator
Two arms, 50/50 split, two-sided test.
Metric
Significance level (alpha)
Power
Per arm
207,938
visitors
Total
415,876
visitors
Duration
21 days
Round up to 3 full weeks
Detecting a move from 3.00% to 3.15%.
| MDE | Per arm | Duration |
|---|---|---|
| 2% | 1,281,194 | 129 days |
| 3% | 572,150 | 58 days |
| 5% | 207,938 | 21 days |
| 10% | 53,211 | 6 days |
| 15% | 24,193 | 3 days |
| 20% | 13,914 | 2 days |
Normal approximation for two proportions or two means with equal variance. Real tests also need a pre-committed stop date, a check that the split is balanced, and a plan for novelty effects.
Synthetic numbers generated in your browser. Not client or employer data.
How to read it
What the demo is showing
The smaller the effect you need to see, the larger the test. Halving the MDE roughly quadruples the sample.
Variance drives everything for revenue metrics. Order value is noisy, so revenue-per-visitor tests need far more traffic than conversion-rate tests.
Size the test before launch and commit to the duration. Stopping when the chart looks good inflates false positives.
How I'd design a geo testNext demo: Attribution comparisonAll demos