Run A/B tests you can actually trust
Most 'winners' are noise called too early. Honest testing is the whole point.
In short
- Only 20% of 28,304 real experiments ever hit 95% significance, so most 'winners' were called too early.
- Roughly 1 in 7 tests produces a real winner. If a tool says you're winning every time, it's lying.
- Every time you peek at a running test and call it, you raise your odds of a false positive. Set the finish line before you start.
Marker = 95% significance. No winner is called before it.
A "winner" you saw on day one isn't a winner. It's a sample size that hasn't argued back yet. In an analysis of 28,304 experiments run by Convert customers, only 20% ever crossed the 95% significance threshold, which means four out of five tests people called were guesses wearing a percentage sign. The whole job of an honest tool is to tell you when you don't know yet.
What's the problem?
You've been burned by tools that declared a 'winner' early, you shipped it, and conversion didn't actually improve. You can't trust results you can't trust.
Why does this happen?
- Tools call winners before reaching significance.
- Results aren't segmented, so a mobile loss hides inside a desktop win.
- There's pressure to show a quick result over a true one.
- Early data swings hard and then regresses. The first 50 visitors split unevenly by pure chance, so a variant can show +20% on Monday and -3% by Friday. The lift you saw early wasn't real signal; it was the spread you a…
- Stopping a test the moment it 'looks significant' inflates your false-positive rate. Peeking at a running test and calling it whenever the line crosses 95% is the most common way to manufacture fake winners. Every extr…
- Most variants genuinely don't beat the control, and that's the math, not your failure. VWO's own data puts the rate of meaningful winners at roughly 1 in 7 tests. A tool that declares a winner on most of your tests is l…
- Low-traffic stores get strung along the longest. If you're doing a few hundred checkouts a week, a real 5% lift needs weeks of data to confirm, so a tool that promises a fast answer is either ignoring significance or t…
What does the research show?
Independent researchFigures below are from independent studies, not StorePilot data. They're why this problem is worth testing on your own store.
-
In an analysis of 28,304 experiments run by Convert customers, only 20% ever reached the 95% statistical-significance threshold, so most stores never gather enough traffic to call a clear winner.
Convert ↗ -
Only about 1 in 7 A/B tests (~14%) produces a meaningful winning variation that actually lifts conversions, and most variations never beat the original at all.
VWO ↗ -
Better checkout design alone can raise the average large ecommerce site's conversion rate by roughly 35%, so there's real upside worth testing honestly for.
Baymard Institute, E-Commerce Checkout Usability research ↗ -
Monitoring an A/B test continuously and stopping at the first significant result drives the false positive rate to 26.1% against a nominal 5% threshold, more than five times the intended risk.
Evan Miller, 'How Not To Run An A/B Test' ↗ -
To hold a true 5% error rate you must require 2.9% reported significance if you look once more, 1.4% across 5 looks, and 1.0% across 10 looks.
Evan Miller, 'How Not To Run An A/B Test' ↗
How does StorePilot AI fix it?
- StorePilot enforces minimum-traffic and significance thresholds and never declares early winners.
- It reports honestly, including 'Variant B +12% but not enough data yet', and segments by device and visitor type.
- Every result carries a recommended decision (publish B / keep A / split-ship per device) with one clear action.
How do you fix it, step by step?
-
Set the sample size before you launch
Decide up front how many visitors (or conversions) per variant the test needs to detect a realistic lift, based on your current conversion rate and traffic. If you can't reach that number in a few weeks, the test is too ambitious for your traffic; pick a bigger swing or a higher-traffic page.
-
Pick one primary metric and commit to it
Choose revenue per visitor or conversion rate as the single metric that decides the test, before you see any data. Tracking ten metrics and celebrating whichever one turns green is how you find a 'win' in pure noise.
-
Stop peeking. Let it run to the pre-set finish
Don't call the test the moment it crosses 95% on day two. Run it to the sample size and the minimum duration you set (at least one full business cycle, usually 1-2 weeks), so weekday/weekend and payday traffic are all represented.
-
Read the result by segment before you publish
Split the outcome by device at minimum. A +8% desktop win can hide a mobile loss, and since most of your traffic is mobile, the blended number can point you exactly the wrong way.
-
Accept 'no difference' as a valid, useful outcome
If the variants finish statistically tied, that's a real answer: this change doesn't matter, keep the simpler version and move on. Forcing a winner out of a flat result is how you ship changes that quietly do nothing.
-
Ship the winner, then verify it held
After publishing the winning variant, watch the live conversion rate for a couple of weeks to confirm the lift shows up in reality. If it evaporates, the original 'win' was noise and you've just learned to trust the number less, not more.
-
Name your stopping rule before the test starts
Sample size per variant, significance threshold, minimum duration in full weeks, and the single metric that decides. Record all four before launch. The rule is not there to be clever; it is there so that the version of you looking at an encouraging chart on day four cannot quietly overrule the version of you who designed the test.
-
If you must look early, use a method that expects it
Sequential testing sets boundaries that account for repeated looks, so checking is legitimate rather than corrosive. What you cannot do is run a fixed-horizon test, peek daily, and stop when it goes green. Looking 20 times turns a 5% false-positive rate into roughly 24% (Armitage, McPherson and Rowe, 'Repeated significance tests on accumulating data', JRSS Series A, 1969), so on a three-week test peeked at daily, about one in four of your shipped winners is noise.
-
Cap the number of variants, or correct for them
Five variants at a 5% threshold carry about a 22.6% chance that one looks like a winner by chance. Either test two arms at a time, or apply a correction and accept that you now need substantially more traffic. Running a five-way test on thin traffic and keeping the top performer is the most reliable way to ship noise.
-
Run through at least one full business cycle
Weekend and weekday shoppers differ, payday cycles are real, and returning customers react to novelty before they react to design. A test that starts Tuesday and ends Friday has measured a slice of your audience under unusual conditions. Full weeks, always, even when the numbers look decided.
-
Decide in advance what an inconclusive result means
It means you keep the control. Write that down before launch, because only about 20% of experiments ever reach 95% significance and inconclusive is therefore the most likely ending. Deciding the rule while the data is still unseen is what stops a null result from being renegotiated into a win.
An illustrative example
Demo data- What StorePilot detects
- A variant looks +12% after a day, but the sample is far too small to trust.
- The fix it builds & tests
- StorePilot holds the call until significance, then recommends a clear decision.
- The projected outcome
- Example: 'Variant B, 94% confidence, +8.4% revenue/visitor, recommend publish.' (Illustrative wording of an honest result.)
Key takeaways
- Only 20% of 28,304 real experiments ever hit 95% significance, so most 'winners' were called too early.
- Roughly 1 in 7 tests produces a real winner. If a tool says you're winning every time, it's lying.
- Every time you peek at a running test and call it, you raise your odds of a false positive. Set the finish line before you start.
- A flat result is a real answer. Don't manufacture a winner out of noise.
