Statistical Significance Calculator: It Tells You When Your Test Is Not Ready Yet

The statistical significance calculator

Four numbers get you a result: visitors and conversions for your control (A), visitors and conversions for your variant (B). Two more numbers decide whether that result means anything: the smallest lift you wanted to be able to detect, and how many times you have already looked at the test while it was running.

Everything runs in your browser. Nothing is sent anywhere, there is no email gate, and there is no model in the loop. Same inputs, same answer, every time.

The default numbers in the boxes are a realistic Shopify scenario rather than a flattering one: 8,000 sessions per variant, a 2.00% control, a 2.30% variant, a 15% observed lift that looks like a win on a dashboard. The calculator refuses it, and the rest of this page explains why that refusal is the correct answer.

Required sample size, computed before we show you any result

TL;DR A p-value is only interpretable against a plan. This calculator builds the plan first: it takes your control's conversion rate, the smallest lift you said you needed, and your confidence level, and computes how many sessions per variant that combination demands. Only then does it look at your result.

Most significance calculators run in the wrong order. You paste in four numbers, they hand back a percentage, and nothing in the interface has asked whether those four numbers were ever capable of answering your question. The order matters because sample size is not an output of a test. It is an input to one, decided before you start, alongside the metric and the stopping rule.

The formula for the required size is old, public, and boring. The NIST/SEMATECH e-Handbook of Statistical Methods publishes it in section 7.2.4.2, along with a worked example you can check by hand: with a baseline proportion of 0.10, a target of 0.20, a one-sided alpha of 5% and beta of 10%, the handbook derives N greater than or equal to 102, or 112 once you apply the continuity correction. Plug those z values into the version of the formula we print in the math section and you get 101.26, which rounds up to exactly the handbook's 102. That check is why we use this source rather than a vendor's blog post: you can verify our implementation against a government-published worked example in under a minute.

Four things feed the number, and each one moves it in a direction you can feel:

• Your baseline conversion rate. Lower baselines need dramatically more traffic, because conversions are the rare event you are counting and a 1% store produces half as many per session as a 2% store. If you do not know yours, our sourced Shopify benchmarks give you the range, but use your own trailing 90 days if you have them.
• The lift you need to detect. This is the minimum detectable effect, and it dominates everything. Halving the MDE roughly quadruples the sample, because the requirement scales with one over the square of the effect.
• Confidence level. Moving from 95% to 99% at a 2% baseline and a 10% target lift takes the requirement from 80,682 sessions per variant to 120,053. Precision is expensive.
• Power. We fix this at 80%, matching the convention Optimizely describes when it notes that the false negative rate, beta, is usually 20%. Power is the probability the test finds a real effect that is genuinely there. At 80%, one real winner in five slips past you undetected, and that is the industry's standard tolerance, not a comfortable one.

Notice what is not in that list: your opinion about the variant, how good the idea felt in the planning meeting, and how long the test has already been running. None of those change the arithmetic.

Why we return “not enough data yet” instead of a p-value

TL;DR A p-value computed on an underpowered sample is a coin flip with a decimal point. We show the refusal first and put the numbers behind a disclosure, labelled as not a verdict.

Take the default scenario in the calculator. Control converts 160 of 8,000 sessions, 2.00%. Variant converts 184 of 8,000, 2.30%. That is a 15% relative lift, the sort of result that gets screenshotted into Slack.

Run the pooled z-test on it and z comes out at 1.308, giving a two-sided p-value of 0.1908. Not significant at 95%, and most calculators will stop there and print something like “not significant, keep collecting.” That advice is not wrong, but it is uselessly incomplete, because it does not tell you what “keep collecting” means. Keep collecting until when? Another week? Another quarter?

The required sample size answers it. At a 2.00% baseline, detecting a 15% relative lift at 95% confidence and 80% power needs 36,693 sessions per variant. The test has 8,000. It is 21.8% of the way to being able to answer the question that was asked of it. On a store doing 20,000 sessions a month, split evenly, that is roughly another 2.9 months of collection.

Now the interesting part. Suppose the variant had converted 200 of 8,000 instead of 184. That is 2.50% against 2.00%, a 25% lift, and at p = 0.0330 it clears the p < 0.05 line. Should you ship it? The 95% confidence interval on the relative lift runs from +1.8% to +53.5%. The point estimate of 25% is the middle of a range so wide that the honest sentence is “somewhere between barely anything and enormous.” A test stopped early at the moment it crosses significance systematically produces estimates on the high side of the truth, because the moment of crossing is, by construction, a moment when noise was working in your favour.

The American Statistical Association's 2016 statement on p-values gives us the licence for both halves of this design. Principle 3: “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.” Principle 5: “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” A calculator that returns nothing but a threshold verdict is, by the ASA's own six principles, answering a question nobody should be asking on its own.

So the refusal is not a gimmick, and it is not us withholding information. The numbers are all there, one click away, exactly as computed. What we refuse to do is present them in the visual language of a conclusion when the test has not earned one. If you want the full mechanics of setting the plan up front, our guide to how to actually run the test covers the sequence from hypothesis to shipped winner.

Reading the output: p-value, confidence interval, and the lift you can honestly claim

TL;DR The p-value answers one narrow question. The confidence interval on the relative lift answers the one you actually care about: how much money might this be worth, at the optimistic and pessimistic ends.

The p-value

A p-value is the probability of seeing a difference at least as large as the one you observed, if the two variants were in truth identical. That is the whole definition. The ASA's first principle words it as: “P-values can indicate how incompatible the data are with a specified statistical model.”

The second principle is the one that gets violated in every meeting: “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” So p = 0.03 does not mean “97% chance B is better.” It means that if B were identical to A, data this extreme or more so would turn up 3% of the time.

Kwak's 2023 review in the Journal of Lipid and Atherosclerosis puts the operational meaning in one line: “If the significance level is set to 5%, it means that the null hypothesis is rejected 5 times out of 100 even though it is true.” Run twenty honest tests of things that do nothing and, on average, one of them will look like a winner.

The confidence interval on the absolute difference

This is measured in percentage points, and it is the number your finance spreadsheet wants. In the default scenario, the 95% interval on the absolute difference runs from -0.149 to +0.749 percentage points. It contains zero, which is another way of saying the test is not significant, but it also carries information the p-value throws away: even at the optimistic end, this change is worth about three quarters of a percentage point of conversion rate.

The confidence interval on the relative lift

This is the number people quote in case studies, and it is almost always quoted without its interval. In the default scenario the observed lift is +15.0%, and the 95% interval runs from -6.7% to +41.8%. The honest claim is not “we got a 15% lift.” The honest claim is “our data are consistent with anything from a 7% loss to a 42% gain, so we do not know yet.”

Amrhein, Greenland and McShane made this exact argument, with more than 800 co-signatories, in a 2019 Nature comment titled “Scientists rise up against statistical significance”. Its opening line is a question worth asking about your own test reports: “When was the last time you heard a seminar speaker claim there was 'no difference' between two groups because the difference was 'statistically non-significant'?”

Our calculator computes the relative-lift interval with the log method, printed in the math section. If the interval spans zero, no lift claim survives. If the whole interval sits above zero, the honest headline is the lower bound, because that is the smallest gain your data support. Reporting the point estimate as though it were the finding is the single most common way an honest test turns into a dishonest case study.

One-tailed vs two-tailed, and why we default to two

TL;DR A one-tailed test cannot detect your variant making things worse. On a live store, that is the outcome you most need to catch. Two-sided is the default and the switch carries a warning.

A two-sided (two-tailed) test asks: is B different from A, in either direction? A one-sided (one-tailed) test asks only: is B better than A? Because the one-sided test concentrates its whole error budget on one side, it reaches significance sooner, and at a 2% baseline detecting a 10% lift it needs 63,553 sessions per variant against 80,682 for the two-sided version. A 21% discount on required traffic is a real temptation.

Here is what you buy it with. A one-sided test has no ability to detect B being worse. It is not that it reports “worse” weakly; it does not test for it at all. If your new product page quietly costs you 12% of revenue, a one-sided test set up to look for improvement will return “no significant difference” and you will ship it.

The mainstream tools are split, and the split is informative. SurveyMonkey's calculator offers both, describing the one-sided option as one that “assumes that your alternative hypothesis will have a directional effect” and the two-sided option as the one that “accounts for if your hypothesis could have a negative effect on your results.” abtestguide.com exposes the same one-sided/two-sided toggle. Qualtrics, Sapio Research and Amplitude expose no tail option at all, which means their users cannot know which one they are getting. And GraphPad's QuickCalcs, the scientific tool in this SERP, resolves it by policy: its page states plainly that “This calculator only uses two-tailed P values.”

There is exactly one situation where one-sided is defensible on a store: you have pre-registered a directional hypothesis, you are contractually unable to ship a losing variant, and a negative result and a null result would lead to the identical decision. In practice, on a Shopify theme test, they never do. You want to know about the loss.

Our calculator defaults to two-sided, recomputes the required sample size when you switch, and prints a warning next to the verdict so that the choice cannot be made silently.

The peeking problem: what checking your test every morning does to your false-positive rate

TL;DR Every look at a running test is another chance for noise to cross the line. Two looks takes 5% to 9.75%. Five looks takes it to 14.2%. Checking constantly can take it past 26%.

You set alpha at 5%. You believe you are accepting a one in twenty chance of crowning a fake winner. Then you open the dashboard every morning, and if it ever shows significance, you stop. That stopping rule is not the test you designed, and its true error rate is not 5%.

The cleanest small illustration comes from clinical trials, where this problem was solved decades before anyone had a CRO dashboard. Kumar and Chakraborty, writing in the Journal of Advanced Pharmaceutical Technology and Research (2016), work it through for two looks: “Probability stop at the first look: 0.05, probability stop at the second look: 0.95 x 0.05 = 0.0475, and total probability of stopping is 0.0975.” One extra look almost doubles your false-positive rate.

It gets worse fast. Armitage, McPherson and Rowe published the classic result in 1969; Daniel Lakens tabulates it in his open textbook Improving Your Statistical Inferences: with equally spaced looks, “the alpha level inflates to 0.142 after 5 looks, 0.374 after 100 looks, and 0.530 after 1000 looks.” At a thousand looks, the test is worse than a coin flip at telling you whether anything happened.

Evan Miller's 2010 essay “How Not To Run An A/B Test” ran the simulation that most CRO people have actually seen. His answer: “Try 26.1%, more than five times what you probably thought the significance level was.” The conditions matter and Miller states them: a 50% conversion rate, testing for significance after every observation, stopping as soon as 5% significance is reached or calling it off after 150 observations, with a new logo that in truth does nothing. Miller himself calls this “sort of a worst-case scenario.” Quote the 26.1% without those four conditions and you are misusing his work.

Looks at the data	True false-positive rate	Source and conditions
1 (fixed horizon, done correctly)	5%	By construction: this is what alpha = 0.05 means
2	9.75%	Kumar & Chakraborty 2016, worked through for two interim looks at alpha 0.05
5	14.2%	Armitage, McPherson & Rowe 1969, as tabulated by Lakens; equally spaced looks
Every observation, n = 150	26.1%	Evan Miller 2010 simulation; 50% baseline, stop at 5% significance or 150 observations; his stated worst case
100	37.4%	Armitage et al. 1969, as tabulated by Lakens; equally spaced looks
1000	53.0%	Armitage et al. 1969, as tabulated by Lakens; equally spaced looks

These rows are not the same experiment. They use different baselines, stopping rules and distributions, and we have deliberately not drawn a curve through them or interpolated the missing rungs. Each figure is quoted from the source named beside it. Where a source publishes no value for a given number of looks, this table shows no value.

The general result, and the most citable one, comes from Johari, Pekelis and Walsh at Stanford. Their paper “Always Valid Inference: Bringing Sequential Analysis to A/B Testing” (arXiv 1512.04922, submitted December 2015, last revised July 2019) states the problem in the abstract: “A/B tests are typically analyzed via frequentist p-values and confidence intervals; but these inferences are wholly unreliable if users endogenously choose samples sizes by continuously monitoring their tests.” And in the body: “No correction for continuous monitoring is typically made in industrial practice. Very high false positive probabilities are obtained, well in excess of the nominal alpha that the user can tolerate. Even with 10,000 samples (a sample size which is quite common in online A/B testing), Type I error can easily increase fivefold.”

The fix is forty years old and nobody invented it here

None of this means you must never look at a running test. It means that if you look, you have to raise the bar. Miller publishes the corrected thresholds directly: to get a real 5% significance level, if you peeked once you need to report 2.9%, twice 2.2%, three times 1.8%, five times 1.4%, and ten times 1.0%. In his words, “if you peek at an ongoing experiment ten times, then what you think is 1% significance is actually just 5% significance.”

Clinical statistics formalised the same idea into named procedures. Penn State's STAT 509 course notes give the boundaries for two analyses at alpha 0.05: the Pocock procedure uses a critical value of 2.178 at each look, corresponding to alpha 0.0294 per look; the O'Brien-Fleming procedure spends almost nothing early, using 2.782 at the interim (alpha 0.0054) and 1.967 at the final analysis (alpha 0.0492). Convert those z boundaries back through the normal distribution and you recover Penn State's published alphas to four decimal places, which is a useful thing to check when you are deciding whether to trust a tool.

The practical version for a Shopify merchant is simpler than any of it. Decide the sample size before you start. Look at the dashboard for operational reasons if you want, to confirm the test is live and the split is even. Do not let a look trigger a decision. If you genuinely need to be able to stop early, use a sequential method built for it, and understand that VWO's duration tool, for one, runs a Bayesian sequential engine rather than a frequentist one, which is a different framework with different guarantees, not a licence to peek at a frequentist test.

Our calculator asks how many times you have looked, and when you answer honestly it applies Miller's corrected threshold to your p-value instead of the nominal one. That usually changes the verdict. It is meant to.

Minimum detectable effect: the smallest lift your store is physically able to measure

TL;DR MDE is a constraint, not a prediction. Your traffic decides the smallest lift you can see. Pick a smaller one and the required sample explodes, because the requirement scales with one over the square of the effect.

The minimum detectable effect is the smallest difference a test is powered to find. VWO's duration calculator defines its equivalent input as the smallest relative improvement between current and target metrics that the test aims to detect, and ships with an MDE default of 20%. AB Tasty frames the same quantity from the other direction, as the minimal conversion gains needed to reach the significance threshold at a given test duration.

Two things about MDE trip merchants up constantly.

First, relative versus absolute. Optimizely's field note gives the cleanest worked example: “if the conversion rate is p0 = 0.1 and p1 = 0.15, the absolute difference is p1 - p0 = 0.05, while the relative difference is (p1 - p0)/p0 = 0.5, or 50%.” On a 2% store, a 2 percentage point improvement is a 100% relative lift and would be the best day of your career. A 2% relative lift is 0.04 percentage points and is essentially unmeasurable. Type the wrong one into a calculator and your sample size is off by a factor of fifty. This calculator asks for the relative lift and labels it as such.

Second, MDE is set by your traffic, not by your ambition. You do not choose to detect a 3% lift. Your session volume chooses it for you, and your only real decision is how long you are willing to wait. AB Tasty publishes a worked example for a site with 50,000 visitors and a 3% baseline conversion rate: at 14 days the MDE is a 10% uplift, at 21 days it falls to 8.31%, and at 49 days it falls to 5.44%.

That curve is worth checking, because it is a free cross-validation of the whole model from an independent vendor. Power theory says MDE should shrink roughly as one over the square root of the sample, so tripling the days should shrink the MDE by a factor of about 1.73. Test it on their published numbers: the square root of 14/21 is 0.8165, and 10% times 0.8165 is 8.16% against their published 8.31%. The square root of 14/49 is 0.5345, and 10% times 0.5345 is 5.35% against their published 5.44%. Different vendor, different tool, same curve, to within a rounding difference in how the arithmetic handles the changing variance. When two independently implemented calculators trace the same shape, the shape is probably the physics rather than either vendor's opinion.

What that curve means in practice is uncomfortable. Waiting twice as long does not let you detect a lift twice as small. It lets you detect a lift about 1.41 times smaller. Going from a 10% MDE to a 5% MDE takes roughly four times the traffic, not two.

Sessions needed per variant at 1%, 2%, 3% and 5% baseline conversion

TL;DR At a 2% baseline, a 10% relative lift needs 80,682 sessions per variant. A 20% lift needs 21,109. A 5% lift needs 315,206. This is the table that decides whether you have a testing programme or a wish.

Every figure below is computed from the two-proportion power formula printed in the math section, at 95% confidence, two-sided, and 80% power. We validated our implementation against the NIST handbook's own worked example, which it reproduces exactly. The baselines span the two most-cited Shopify reference points: Littledata's 1.4% average across 2,800 Shopify sites (a vendor-published 2023 benchmark, not a current measurement) and the 1.6% of global ecommerce visits that converted in Q3 2025, attributed by Shopify's blog to Statista.

Baseline CVR	Detect 5% lift	Detect 10% lift	Detect 20% lift	Detect 50% lift
1.0%	637,010	163,095	42,693	7,750
2.0%	315,206	80,682	21,109	3,826
3.0%	207,938	53,211	13,914	2,518
5.0%	122,124	31,234	8,158	1,471

Lifts are relative, so a 10% lift on a 2% baseline means moving to 2.2%. Figures are sessions per variant; double them for the test total. Assumes an even 50/50 split, which Kohavi and Longbotham note is also the allocation that gives the experiment maximum statistical power. Computed 2026-07-30 from the formula in the math section, validated against the NIST handbook's worked example.

Two reference points from the table, for orientation. At Littledata's 1.4% Shopify average, detecting a 10% relative lift takes about 116,000 sessions per variant. At the 1.6% global figure Shopify cites from Statista, it takes about 101,285. Those are the numbers behind the sentence “most Shopify stores cannot detect a 10% lift.”

The ChatGPT answer this table is here to replace

On 2026-07-30 we asked ChatGPT, via DataForSEO's ChatGPT scraper, “How much traffic do I need to run a valid A/B test on my Shopify store?” It returned an answer with zero citations. No sources at all. What it produced instead was a tidy table of traffic tiers (“under 5,000” labelled as usually too little for reliable A/B testing, then 5,000 to 20,000, 20,000 to 100,000, and 100,000 plus) and a worked example claiming that 30,000 visitors a month at a 3% conversion rate detecting a 10% lift needs “roughly 30,000 to 50,000 visitors per variation.”

Compare that against the row above. At a 3% baseline, a 10% relative lift needs 53,211 sessions per variant at 95% confidence and 80% power. The model's range brackets the true figure at the top and understates it at the bottom, which is roughly what you would expect from a plausible guess: close enough to sound right, unsourced, and off by up to 40% at the low end. A merchant who planned to 30,000 per variant on that advice would run an underpowered test and never know.

We are not picking a fight with a language model. We are making the point that this is exactly the class of number that must never be generated. A sample-size figure is arithmetic with a published formula. There is no reason for anyone, human or model, to approximate it, and every reason to print the formula next to the answer. That is also why we never let a model call the winner inside StorePilot: the language layer explains, the deterministic layer decides.

How many days that takes at your current traffic

TL;DR Sessions become days once you divide by your traffic. For a lot of stores the answer is measured in years, and that is a real answer, not a failure of the calculator.

The conversion from the table above to a calendar is one division: total sessions needed, divided by daily sessions. Total means both variants, so double the per-variant figure. Below, monthly session volumes are converted at 30.4 days per month.

Baseline & target lift	Total sessions	500/mo	1,000/mo	5,000/mo	20,000/mo
1% baseline, 10% lift	326,190	19,833 days	9,917 days	1,984 days	496 days
1% baseline, 20% lift	85,386	5,192 days	2,596 days	520 days	130 days
2% baseline, 10% lift	161,364	9,811 days	4,906 days	982 days	246 days
2% baseline, 20% lift	42,218	2,567 days	1,284 days	257 days	65 days
2% baseline, 50% lift	7,652	466 days	233 days	47 days	12 days
3% baseline, 20% lift	27,828	1,692 days	846 days	170 days	43 days
5% baseline, 20% lift	16,316	993 days	497 days	100 days	25 days
5% baseline, 50% lift	2,942	179 days	90 days	18 days	5 days

Total sessions across both variants at an even split. Monthly volumes converted at 30.4 days. Bold cells are the only ones in this grid that finish inside a normal testing window. Note the floor rule below: nothing under 7 days is a valid duration regardless of what the arithmetic says.

The seven-day floor, and why it beats the arithmetic

Look at the bottom-right cell: a 5% baseline store doing 20,000 sessions a month can accumulate enough data for a 50% MDE test in 5 days. Do not run it for 5 days.

Two independent sources put a floor under test duration for reasons that have nothing to do with sample size. VWO's guidance on its duration calculator states that “It is recommended to run your campaigns for at least 7 days to capture natural fluctuations in metric performance across weekdays and weekends.” Kohavi and Longbotham go further in their encyclopedia entry on online controlled experiments: “a best practice is to run the experiment for at least one week (to capture a full weekly cycle) and then multiple weeks beyond that.”

The reason is that a Shopify store does not have one audience, it has a weekly rotation of them. Weekday lunchtime mobile browsers, weekend desktop researchers, and payday-week buyers are different people with different conversion rates. A five-day test samples an unrepresentative slice of that cycle and calls it your store. Run complete weeks, always, and start and stop on the same weekday.

This is also the shortest possible explanation of why a two-week event can never be a valid test. The audience during a global event is not the audience that will be on your store in September, so the winner you found is a winner for a population that no longer exists.

When your store is too small to A/B test at all, and what to do instead

TL;DR If your honest MDE is above about 30%, you do not have a testing problem, you have a traffic problem. Work on the changes that do not need a significance test, and come back when the volume is there.

Read the top-left cells of the days table again. A store at a 1% baseline doing 500 sessions a month needs 19,833 days, about 54 years, to detect a 10% lift. That is not a discouraging estimate. It is the calculator telling you that the question you asked cannot be answered by your store, ever, using this method.

That verdict is common enough that it has its own guide: testing on a low-traffic Shopify store covers what you can still learn about your shoppers when classical significance is permanently out of reach.

The reflex at that point is to loosen the settings until the answer looks achievable. Sapio Research's calculator, ranking near the top of this SERP, actively suggests it: when the difference is not significant, the page's guidance recommends reducing the confidence level. Do not do this. Lowering confidence from 95% to 90% does not make your test more likely to be right, it makes your definition of “right” more forgiving, and you will ship more losers. If you are going to run at 90%, decide that before the test, for a stated reason, and write it down.

Here is what actually works when the numbers say no.

Test bigger things

The MDE column that opens up for small stores is the 50% one. At a 2% baseline, a 50% relative lift needs 3,826 sessions per variant instead of 80,682 for a 10% lift. That is a 21-fold difference, and it comes entirely from asking a bigger question. Button colours and headline tweaks do not produce 50% lifts. Replacing a product page layout, changing an offer, adding or removing a checkout step, or altering a price can. If you only have the traffic for one honest test a quarter, spend it on something that could plausibly move the metric by half.

Fix the things that do not need a test

Baymard Institute's running average across 50 cart-abandonment studies published between 2006 and 2025 sits at 70.22%. That is a meta-average across two decades and mixed methodologies, not a 2026 measurement of your store, but it points at the right place. A checkout that hides shipping cost until the final step, a mobile form that fires the wrong keyboard, a product page that does not state the return policy: none of these need a p-value. They need fixing. Our Shopify conversion rate optimization playbook is organised around exactly this distinction, between changes you should test and changes you should just make.

Use the right test for tiny numbers

If you are working with genuinely small counts, the normal approximation behind every z-test on this page starts to misbehave. NIST's handbook flags this directly, noting that the sampling distribution is really binomial and the normal form is an approximation, and it provides Fisher's Exact Test for the small-sample case with a worked 2x2 example: a p observed of 0.26515 plus 0.04545 for the more extreme table gives a combined p of 0.31060, which does not reject at alpha 0.05. When your conversion counts are in the single or low double digits, that is the test to reach for, not this one.

Accept that measurement itself has a cost

The honest framing for a very small store is that experimentation has a fixed price in traffic, and if you cannot pay it, the answer is to buy information some other way: session recordings, five real user interviews, reading your own support tickets. None of those produce a p-value. All of them produce hypotheses worth spending your one quarterly test on.

One last piece of perspective on what testing buys even when you can afford it. Kohavi, Crook and Longbotham's 2009 Microsoft ThinkWeek paper reports that “only about 1/3 of ideas improve the metrics they were designed to improve”, and that a team shipping ten unmeasured ideas may have “about 1/3 be good, 1/3 flat, and 1/3 negative.” That is Microsoft, not ecommerce, and it should be read as directional rather than as a Shopify benchmark. But it explains why the occasional enormous win justifies the whole apparatus: Kohavi and Thomke's 2017 Harvard Business Review article describes a single Bing ad-headline experiment that “increased revenue by an astonishing 12%, which on an annual basis would come to more than $100 million in the United States alone.” You cannot find that if you cannot measure.

What the other calculators actually do

TL;DR We opened every calculator ranking for this term on 2026-07-30 and audited its features. None compute required sample size on the same screen before showing a verdict. Only one refuses on low data. None warn about peeking.

On 2026-07-30 we pulled the live US desktop SERP for “statistical significance calculator” through DataForSEO and found that all ten organic results are calculators, not articles. Google has decided what this query wants, and it is a tool. An AI Overview sits at position five citing eight sources, and every one of those is a calculator too.

So we opened each one and used it. The table below is the audit, and every cell was checked by hand on 2026-07-30.

Calculator	Sample size first?	Refuses on low data?	1 / 2-tailed toggle?	Reports power?	Warns about peeking?
SurveyMonkey	No	No	Yes	Yes	No
abtestguide.com	No, separate tool	Yes	Yes	Yes	No
Sapio Research	No	No	No	No	No
Qualtrics (page dated 2022-07-26)	No	No	No	No	No, but has an SRM alert
GraphPad QuickCalcs	No	n/a, takes no raw counts	Two-tailed only, by policy	No	No
Amplitude	No, separate tool	No	No	No	No
VWO duration calculator	Yes	n/a, duration only	n/a, Bayesian engine	n/a	No
This calculator	Yes, same screen	Yes	Yes, two-sided default	No	Yes

Every row verified by opening the live tool on 2026-07-30. Tools change; if you find a cell that is now wrong, it is wrong, and we would rather fix it than defend it.

Three honest observations from that audit.

abtestguide.com is a genuinely good tool and we are not going to pretend otherwise. It refuses on insufficient data, telling you that the observed difference is not big enough to declare a significant winner, and it shows a low-data warning. It reports observed power, which most of the field does not, and it exposes the hypothesis toggle. Its only structural gap is that its sample-size calculation lives at a separate URL, so the two halves of the decision never appear on the same screen.

Qualtrics has the best single feature nobody else has: a sample ratio mismatch alert. SRM is when your 50/50 split arrives as, say, 52/48, which usually means the experiment is broken rather than that the variant is winning. It is a genuinely useful check and it deserves to be more common. Its page is also timestamped 2022-07-26, which is worth knowing before you take its methodology on faith.

Nobody in the field warns about peeking. Not one of the calculators we audited asks how many times you have looked at the test, or adjusts anything if you have. Given that Miller's essay is sixteen years old and the Stanford always-valid-inference paper is a decade old, that is a strange collective blind spot, and it is the single biggest reason this page exists.

The math, written out, so you can check our work

TL;DR Four formulas run this page. All four are published elsewhere and every one of them is checkable against a source's own worked example.

1. The significance test (pooled two-proportion z-test)

From NIST/SEMATECH e-Handbook §7.3.3. Let n1 and n2 be the visitor counts, x1 and x2 the conversion counts.

• Observed rates: p1 = x1 / n1 and p2 = x2 / n2
• Pooled rate: p = (x1 + x2) / (n1 + n2)
• Test statistic: z = (p1 - p2) / sqrt( p (1 - p) (1/n1 + 1/n2) )
• Two-sided p-value: p-value = 2 (1 - PHI(|z|)), where PHI is the standard normal cumulative distribution

Check on the default scenario: 160/8,000 against 184/8,000 gives p = 344/16,000 = 0.0215, a standard error of 0.002294, z = 1.308, and a two-sided p-value of 0.1908.

2. The required sample size

The single-sample form is published with a worked example in NIST §7.2.4.2:

N ≥ [ ( z(1-alpha/2) sqrt(p0(1-p0)) + z(1-beta) sqrt(p1(1-p1)) ) / delta ]²

For a two-arm A/B test we use the standard two-sample extension of that same normal-approximation argument, matched to the pooled test statistic in §7.3.3 above. With p0 as the baseline, p1 = p0 (1 + MDE), delta = p1 - p0 and pbar = (p0 + p1)/2:

n per variant = [ z(1-alpha/2) sqrt(2 pbar (1 - pbar)) + z(1-beta) sqrt( p0(1-p0) + p1(1-p1) ) ]² / delta²

We set 1 - beta = 0.80, matching Optimizely's stated convention that beta is usually 20%, and z(1-alpha/2) = 1.96 at 95% two-sided confidence.

Check against NIST's own example: feed the single-sample form p0 = 0.10, p1 = 0.20, one-sided alpha 5% (z = 1.645) and beta 10% (z = 1.282) and you get 101.26, which the handbook rounds to N greater than or equal to 102, and 111.26 with the continuity correction N' + 1/delta, which the handbook gives as 112. Our implementation reproduces both.

3. The confidence intervals

On the absolute difference, using the unpooled standard error:

(p2 - p1) ± z(1-alpha/2) sqrt( p1(1-p1)/n1 + p2(1-p2)/n2 )

On the relative lift, using the log method, which keeps the interval asymmetric in the way a ratio actually behaves:

exp( ln(p2/p1) ± z(1-alpha/2) sqrt( (1-p1)/x1 + (1-p2)/x2 ) ) - 1

Check on the default scenario: the absolute-difference interval runs from -0.149 to +0.749 percentage points, and the relative-lift interval from -6.7% to +41.8% around an observed +15.0%.

4. The peeking correction

No formula, by design. We use the published thresholds from Evan Miller's simulation directly: to hold a real 5% significance level you need to report 2.9% after one peek, 2.2% after two, 1.8% after three, 1.4% after five and 1.0% after ten. Where a source publishes no value, we publish no value and say so rather than interpolating between rungs.

All arithmetic on this page uses the normal approximation to the binomial, which is what every calculator in the SERP does and what NIST's handbook describes. It degrades when conversion counts are small; see the small-store section for Fisher's Exact Test.

Using this as a p value calculator

If you came here for a plain p value calculator, the tool above is one, for the specific and most common case: two groups, each with a visitor count and a conversion count. Enter the four numbers, open “show the numbers,” and you get the exact two-sided p-value along with the z statistic it came from.

What it will not do is accept a raw test statistic. If you already have a z, t, F, r or chi-square value and want a p-value from it, GraphPad's QuickCalcs does that and takes no visitor counts at all. Different job, different tool.

Three things to hold on to when you read any p-value, ours included.

A p-value is a property of the data and the model together, not a property of your variant. Change the model, by choosing one-sided instead of two-sided, or by peeking, and the same data produce a different p-value. That is not a flaw in the statistic. It is the reason the plan has to come first.

A p-value says nothing about size. ASA principle 5 is explicit that statistical significance does not measure the size of an effect or the importance of a result. The confidence interval on the relative lift is where the business decision lives.

And a threshold is a convention, not a law. Kwak notes that the significance level is set by the researcher according to the circumstances of the study and does not necessarily have to be 0.05. The ASA's third principle says decisions should not be based only on whether a p-value passes a specific threshold. Pick your threshold in advance, for a reason, and then hold to it. That last clause is the part that gets skipped.

The refusal is the product

Every calculator in this SERP can do the arithmetic. Arithmetic is not the scarce thing. The scarce thing is a tool that will tell you the answer you did not want, at the moment you most want to hear the other one.

Statistical honesty on a Shopify store is mostly a sequencing problem. Compute the sample size before the test. Decide the threshold before the data. Fix the stopping rule before the first look. Report the interval, not the point estimate. None of that is difficult, and all of it is easier to skip when the tool in front of you returns a confident percentage no matter what you type in.

StorePilot holds its own experiments to this standard, which is why the app will not declare a winner before the precomputed sample size is reached, and why the language layer explains results while a deterministic layer decides them. If you want the full workflow around this calculator, from hypothesis to shipped winner, start with the Shopify A/B testing guide, and if you are working through the wider programme, the Shopify CRO playbook is the map.

Statistical significance FAQ

How do you calculate statistical significance?

For a two-variant test you compute a pooled two-proportion z-test. Pool the conversion rates: p = (x1 + x2) / (n1 + n2). Then z = (p1 - p2) / sqrt( p(1-p)(1/n1 + 1/n2) ). Convert z to a p-value from the standard normal distribution. That is the formula published in the NIST/SEMATECH e-Handbook, section 7.3.3, and it is exactly what the calculator on this page runs. The arithmetic is the easy part. The hard part is knowing whether you had enough data for the answer to mean anything, which is why we compute required sample size first.

Is it 5% or 95% significance level?

Both, describing the same thing from opposite ends. The significance level (alpha) is 5%: the chance of declaring a winner when there is no real difference. The confidence level is 95%: 100% minus alpha. If a tool says '95% confidence' and another says 'p < 0.05', they are using the same threshold. Kwak (2023) puts the meaning plainly: at a 5% significance level, the null hypothesis is rejected 5 times out of 100 even when it is true.

Is p 0.05 a 5% significance level?

Yes. A p-value of 0.05 sits exactly on the 5% significance threshold. Kwak (2023) also notes that the significance level is a value the researcher sets according to the circumstances of the study, and does not necessarily have to be 0.05. The American Statistical Association's 2016 statement goes further: scientific conclusions and business decisions should not be based only on whether a p-value passes a specific threshold.

Is p 0.001 statistically significant?

At any conventional threshold, yes. p = 0.001 is well below 0.05 and below 0.01. But a small p-value tells you the data are incompatible with 'no difference'; it does not tell you the effect is large or that it will pay for itself. ASA principle 5 states it directly: a p-value, or statistical significance, does not measure the size of an effect or the importance of a result. Read the confidence interval on the lift before you decide anything.

How much traffic do I need to A/B test my Shopify store?

It depends on your baseline conversion rate and on how small a lift you need to detect. At a 2% baseline, detecting a 10% relative lift at 95% confidence and 80% power takes about 80,682 sessions per variant, roughly 161,364 sessions in total. Detecting a 20% relative lift at the same baseline takes about 21,109 per variant. Both figures are computed from the standard two-proportion power formula and the table in this article shows the full grid at 1%, 2%, 3% and 5% baselines.

Why does this calculator refuse to show a result?

Because a p-value computed on a sample too small to detect your target effect is not information, it is noise with a decimal point. When either variant is below the required sample size, the calculator says so and shows how far short you are. The numbers are still available behind a disclosure, clearly labelled as not a verdict. Of the calculators ranking for this term on 2026-07-30, none compute required sample size on the same screen before showing a result, and only abtestguide.com refuses on low data.

What is a minimum detectable effect?

The smallest lift your test is physically able to find, given your traffic. VWO's duration calculator defines the equivalent input as the smallest relative improvement between current and target metrics that the test aims to detect. It is not a prediction of your result. It is a constraint: pick a small MDE and you need enormous traffic, pick a large one and you need less, but you will miss real improvements smaller than the threshold you chose.

Should I use a one-tailed or two-tailed test?

Two-tailed, in almost every case. A one-tailed test only asks whether B is better and has no power to detect B being worse, which is the outcome you most need to catch on a live store. SurveyMonkey describes the two-sided option as the one that accounts for a hypothesis that could have a negative effect on your results. GraphPad's QuickCalcs, a scientific tool, states plainly that it only uses two-tailed P values. Our calculator defaults to two-tailed and warns when you switch.

Can I stop my test early if it is already significant?

Not without paying for it. Every extra look at a running test is another chance for noise to cross the line. Kumar and Chakraborty (2016) show that two looks at alpha 0.05 already gives a total stopping probability of 0.0975. Armitage, McPherson and Rowe (1969), as tabulated by Lakens, put the inflation at 0.142 after 5 looks and 0.530 after 1,000. Johari, Pekelis and Walsh (Stanford, arXiv 2015/2019) found that even with 10,000 samples, Type I error can easily increase fivefold.

What if my store is too small to A/B test at all?

That is a real and common answer, and the calculator will tell you. At a 1% baseline conversion rate and 500 sessions a month, detecting a 5% relative lift would take about 77,462 days of collection. Stores in that position should stop testing small changes and work on things that do not need a significance test: the checkout costs that Baymard links to a 70.22% average cart abandonment rate across 50 studies, obvious usability breakage, and traffic quality. Test again when you have the volume.

Does this calculator work as a p value calculator?

Yes, for the two-proportion case: two variants, visitors and conversions each. Enter the four numbers and it returns the exact two-sided p-value from the pooled z-test, plus the z score, the confidence interval on the absolute difference, and the confidence interval on the relative lift. It does not accept a raw z, t, F or chi-square statistic; GraphPad's QuickCalcs does that and takes no visitor counts at all.

What percentage of A/B tests actually win?

At Microsoft, about one third. Kohavi, Crook and Longbotham's 2009 ThinkWeek paper reports that only about 1/3 of ideas improve the metrics they were designed to improve, with roughly 1/3 flat and 1/3 negative. Kohavi and Longbotham repeat the finding in their 2015 encyclopedia entry and add that success is even harder to find in well-optimized domains like Bing. Those figures come from Microsoft, not from ecommerce, so treat them as a directional warning rather than a Shopify benchmark.

Sources

Every figure on this page comes from one of the sources below, or from a tool call we ran on 2026-07-30 and named inline. Nothing here is quoted from memory. Where a source publishes no value for something, this page publishes no value.

1. NIST/SEMATECH, e-Handbook of Statistical Methods §7.2.4.2, “Sample sizes required” – the sample-size formula and the N ≥ 102 / 112 worked example. Retrieved July 30, 2026.
2. NIST/SEMATECH, e-Handbook §7.3.3, comparing two proportions – the pooled two-proportion z-test and Fisher's Exact Test worked example (combined p = 0.31060). Retrieved July 30, 2026.
3. “American Statistical Association Releases Statement on Statistical Significance and P-Values”, ASA news release, March 7, 2016 – the document we fetched, which prints all six principles verbatim; principles 1, 2, 3 and 5 are the ones quoted in this article. The statement itself is Wasserstein & Lazar, The American Statistician 70(2):129–133, 2016. Retrieved July 30, 2026.
4. Evan Miller, “How Not To Run An A/B Test”, April 18, 2010 – the 26.1% simulation (50% baseline, testing after every observation, stop at 5% significance or 150 observations; author's stated worst case) and the corrected-threshold table (2.9% / 2.2% / 1.8% / 1.4% / 1.0%). Retrieved July 30, 2026.
5. Johari, Pekelis & Walsh, “Always Valid Inference: Bringing Sequential Analysis to A/B Testing”, arXiv:1512.04922, submitted December 15, 2015, last revised July 16, 2019 – the fivefold Type I error result at 10,000 samples, quoted from the paper's introduction. Retrieved July 30, 2026.
6. Armitage, McPherson & Rowe, “Repeated Significance Tests on Accumulating Data”, Journal of the Royal Statistical Society Series A 132(2):235, 1969 – alpha inflation at 5, 100 and 1000 looks, as tabulated by Daniel Lakens in Improving Your Statistical Inferences, chapter 10. We verified the figures on Lakens' open textbook page, not on the paywalled 1969 paper. Retrieved July 30, 2026.
7. Kumar & Chakraborty, “Interim analysis: A rational approach of decision making in clinical trial”, J Adv Pharm Technol Res 7(4):118–122, 2016 (PMC5052936) – the 0.0975 two-look figure. Retrieved July 30, 2026.
8. Penn State Eberly College of Science, STAT 509 Lesson 9.5, group sequential boundaries – Pocock 2.178 / alpha 0.0294 and O'Brien-Fleming 2.782 / 1.967 at R = 2. Retrieved July 30, 2026.
9. Kwak, S., “Are Only p-Values Less Than 0.05 Significant?”, J Lipid Atheroscler, 2023 (PMC10232224) – the 5-in-100 explanation and the note that 0.05 is a choice. Retrieved July 30, 2026.
10. Amrhein, Greenland & McShane, “Scientists rise up against statistical significance”, Nature Comment, March 20, 2019 – more than 800 signatories; only the dek and opening line are quoted here, as the body is paywalled. Retrieved July 30, 2026.
11. Optimizely Field Notes, Qian Zhang, “Sample size calculations for A/B tests and experiments”, December 12, 2025 – the 80% power / 20% beta convention and the relative-versus-absolute worked example. Retrieved July 30, 2026.
12. Kohavi & Longbotham, “Online Controlled Experiments and A/B Tests”, Encyclopedia of Machine Learning and Data Mining, updated April 25, 2015 – the one-week-then-multiple-weeks rule and the 50% maximum-power split. Retrieved July 30, 2026.
13. Kohavi, Crook, Longbotham et al., “Online Experimentation at Microsoft”, Microsoft ThinkWeek paper, 2009 – the one-third-of-ideas figure. Microsoft, not ecommerce. Retrieved July 30, 2026.
14. Kohavi & Longbotham, “Unexpected Results in Online Controlled Experiments”, SIGKDD Explorations 12(2) – highest overall power at a 50/50 split. Retrieved July 30, 2026.
15. Kohavi & Thomke, “The Surprising Power of Online Experiments”, Harvard Business Review, September–October 2017 – the Bing +12% / $100M headline experiment. Retrieved July 30, 2026.
16. AB Tasty, Hubert Wassner, “Minimal Detectable Effect: The Essential Ally for Your A/B Tests”, May 6, 2025 – the 50,000-visitor, 3%-baseline worked curve (14 days → 10%, 21 days → 8.31%, 49 days → 5.44%). Vendor-published. Retrieved July 30, 2026.
17. VWO, A/B Test Duration Calculator – the 7-day minimum guidance, the 20% MDE default, the MDE definition, and the Bayesian sequential engine. Retrieved July 30, 2026.
18. Littledata, ecommerce conversion rate benchmarks – the 1.4% average across 2,800 Shopify sites. Vendor-published; the page states 2023 data. Retrieved July 30, 2026.
19. Shopify, “Ecommerce Conversion Rate”, updated February 4, 2026 – the 1.6% of global ecommerce visits converting in Q3 2025, attributed on the page to Statista. Retrieved July 30, 2026.
20. Baymard Institute, cart abandonment rate statistics – the 70.22% average across 50 studies spanning 2006 to 2025. Rolling figure. Retrieved July 30, 2026.
21. Competitor feature audit, all opened and used by hand on July 30, 2026: abtestguide.com, SurveyMonkey, Qualtrics (page dated July 26, 2022), Sapio Research, Amplitude, GraphPad QuickCalcs.
22. Our tool calls, July 30, 2026: DataForSEO serp_organic_live_advanced (US/en, desktop, depth 10) for the live SERP composition and the AI Overview at position five; DataForSEO ai_optimization_chat_gpt_scraper for the zero-citation ChatGPT answer on Shopify A/B test traffic.
23. All sample-size, days, p-value and confidence-interval figures on this page were computed on July 30, 2026 from the formulas in the math section, and the implementation was validated against the NIST handbook's own worked example and against Penn State's published Pocock and O'Brien-Fleming boundaries.