"95% statistically significant" is the green light every A/B test tool eventually shows you — the signal to declare a winner and ship. Most of what marketers believe about that number turns out to be slightly, and sometimes badly, wrong: what it actually promises, how a daily peek at the dashboard can quietly wreck it, how much traffic a real result actually needs, and how often a test even produces a winner in the first place.
We went looking for the primary sources behind each of those questions — the original essays, the peer-reviewed papers, and the disclosed, real-sample-size benchmarks — instead of the recycled blog-post version of each number. Some of what's repeated everywhere holds up exactly as claimed. Some of it is popular arithmetic with no traceable original source that turns out to still be correct. And the most common belief about what "95% significant" itself means turns out to be flatly false, according to a peer-reviewed guide written specifically to correct it.
Key takeaways
- Checking a test every day and stopping the moment it hits "95% significant" doesn't give you a 5% false-positive rate — in the worst case it can inflate to 26.1%, roughly five times higher (Evan Miller, 2010; corroborated by a peer-reviewed 2021 paper in Operations Research).
- "95% significant" does not mean "95% chance B actually beats A." A peer-reviewed guide to p-value misinterpretations calls that exact reading a direct "No" — the 5% describes a test's long-run error rate across many repeats, not your odds of being wrong on this one result.
- Real, disclosed win-rate data lands nowhere near "most tests win." Independent datasets found 8% (Airbnb, 250 tested ideas), 19.1% (2,288 real client tests, 2026) and roughly a third (Microsoft, per Ron Kohavi) of tests reach a statistically significant winner — and one study found nearly 70% of the tests it reviewed were underpowered to begin with.
- Required sample size explodes as the effect you're trying to detect or your baseline rate shrinks — from roughly 200 visitors per variant to detect doubling a 10% baseline, to well over 500,000 per variant to detect a modest lift on a sub-1% baseline.
- Watching many metrics on one test quietly stacks the odds against you: track 20 independent metrics at the standard 5% threshold and there's about a 64% chance at least one shows "significant" purely by chance — simple probability, not a specific study finding.
- Peer-reviewed research on novelty and primacy effects backs up the practical advice to run a test for at least one to two full business cycles, not to call it the moment it looks done.
What does "95% statistically significant" actually mean?
A significance test answers one narrow question: if there were truly no difference between A and B, how surprising would the data you actually collected be? "95% significant" (equivalently, p < 0.05) means that specific pattern of results — or a more extreme one — would show up less than 5% of the time by chance alone, if A and B were really identical.
That's a much narrower claim than the one most people walk away believing. A peer-reviewed guide written specifically to correct common statistics mistakes — "Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations," published in the European Journal of Epidemiology in 2016 by a group of leading statisticians — lists the two most common misreadings by name and rejects both directly.
| What people think "95% significant" means | What it actually means |
|---|---|
| "There's a 95% chance my variant actually beats the original." | The 95% describes the long-run behavior of the testing procedure, not this one result. Repeat the same kind of test many times, and about 95% of the intervals it produces would contain the true value — the probability belongs to the method, not to this specific number on your screen (GrowthBook, 2026). |
| "If I ship it, there's only a 5% chance I'm making a mistake." | A direct "No," per the peer-reviewed guide: once you've decided to ship, your chance of being wrong in that specific case is 100% or 0% — you just don't know which. The 5% is the error rate across many repeated studies, not a probability attached to this one decision (Greenland et al., European Journal of Epidemiology, 2016). |
| "A bigger significance number means a bigger, more real effect." | Significance measures how surprising the data would be under "no real difference," not how large the true effect is — that's a separate question, better answered by the confidence interval around the lift than by the significance level alone (Georgi Georgiev, Analytics-Toolkit). |
None of this means significance testing is useless — it means the number answers a narrower, more technical question than the plain-English sentence most people say out loud when they report it.
Does checking your test every morning change the result?
This is the single most common way a real A/B test quietly stops meaning what it claims to mean: a marketer opens the dashboard once a day, and the moment it crosses "95% significant," the test gets called and the winner ships.
Evan Miller's widely cited 2010 essay "How Not To Run An A/B Test" ran the numbers on exactly this habit. In the worst realistic case — checking continuously and stopping the instant the result crosses the 5% significance threshold — the true false-positive rate isn't 5%. It's 26.1%: "more than five times what you probably thought the significance level was." A peer-reviewed 2022 paper in the journal Operations Research, by Johari, Koomen, Pekelis and Walsh (Stanford / Optimizely), reached the same conclusion through a completely different, formal statistical route: "even with 10,000 samples — a sample size which is quite common in online A/B testing — Type I error can easily increase fivefold." Two independent methods, separated by more than a decade, landing on essentially the same number.
| Times you check the test before stopping | Significance you'd need to see to actually hold a true 5% rate |
|---|---|
| 1 | 2.9% |
| 2 | 2.2% |
| 3 | 1.8% |
| 5 | 1.4% |
| 10 | 1.0% |
Miller's own conclusion, quoted directly: until sequential or Bayesian designs are built into the software doing the testing, "anyone running web experiments should only run experiments where the sample size has been fixed in advance, and stick to that sample size with near-religious discipline."
How big a sample do you actually need before the result means anything?
The traffic a test needs to reliably detect a real effect depends on two things: your baseline rate, and how small a lift you're trying to catch. Both push in the same direction — a lower baseline rate or a smaller effect you want to detect both mean dramatically more visitors, not a little more.
Evan Miller's own public sample-size calculator gives one exact, primary data point: detecting a jump from a 10.2% baseline up to 13.2% needs 1,657 visitors per variant. Using the same standard method behind that calculator — the two-proportion test at 80% power and a 95% confidence level — the numbers below show how fast that requirement changes as the target effect shrinks or the baseline drops.
| Baseline rate | Lift you're trying to detect | Visitors needed per variant |
|---|---|---|
| 10.2% | 3 points absolute, to 13.2% | 1,657 (Evan Miller's public calculator) |
| 10% | Doubling it, to 20% | ≈ 200 |
| 10% | A 5% relative lift, to 10.5% | ≈ 57,800 |
| 0.5% | An 8% relative lift, to 0.54% | ≈ 507,500 |
A subtle lift on a low-traffic page can require more visitors than most sites will ever see, and that gap is exactly what tends to go unmeasured: a real, disclosed audit of 115 client A/B tests found that nearly 70% of them were underpowered from the start — meaning the sample size never stood a real chance of detecting the effect the test was actually looking for, whatever the dashboard eventually said (Georgi Georgiev, Analytics-Toolkit, 2018).
How often does an A/B test actually produce a real winner?
"Run a test, ship the winner" implies winners are the normal outcome. They aren't. Every independently disclosed dataset we could find puts the real share of tests that reach a statistically significant winner well under half — the exact number just depends heavily on whose tests, and how strict the bar.
Ron Kohavi, who led experimentation at Microsoft and later Airbnb, has described the pattern the same way across multiple talks and interviews: at Microsoft, only around a third of tested ideas actually improved the metric they were designed to improve. At Airbnb, in a 2021 interview republished by Wingify/AB Tasty, he put a sharper number on it: of 250 ideas tested in controlled experiments, only 20 — 8% — proved to have a positive impact on the key metrics. Those 20 winners alone were still worth it: together they drove a 6% improvement in booking conversion, "worth hundreds of millions of dollars." On the other end, ConversionTeam's audit of 2,288 real client A/B tests across 71 engagements, using a disclosed one-tailed t-test at 95% confidence, found 19.1% reached a statistically significant winner — while Analytics-Toolkit's review of 115 real client tests found 39.13% did, before any of the underpowered ones were excluded.
This lines up with what we found when we looked at landing pages specifically in our conversion-rate piece: published win rates there run roughly 12%–20% across major testing platforms. The spread across all of these numbers isn't a contradiction — it mostly reflects how well-vetted the tested ideas were going in, and how strict a bar "statistically significant" was held to.
If you're watching ten metrics on one test, are your odds still really 5%?
A 5% significance threshold means a single, pre-chosen metric has roughly a 5% chance of falsely looking "significant" purely by chance, if there's no real effect. That guarantee quietly stops applying the moment you watch many metrics on the same test and call it a win if any one of them lights up.
64%
the chance that at least one of 20 independently tracked metrics shows "statistically significant" purely by chance alone, if each is tested at the standard 5% threshold — 1 − 0.95²⁰, simple probability math, not a finding from any one study. Corrections like Bonferroni or Benjamini–Hochberg exist specifically to control for it.
The specific "20 metrics → 64%" framing shows up across dozens of CRO blogs and testing-platform docs, almost always with no original source cited — a rare case of a widely repeated number that's actually correct arithmetic, just uncredited. The practical fix is simpler than the math: pick one primary metric before the test starts, and treat everything else you're watching as directional, not proof.
How long should you actually run a test before trusting it?
CXL's guide to A/B testing puts the practical recommendation plainly: "Predetermine a sample size and run the test for full weeks, usually at least two business cycles." The reasoning is concrete — weekday and weekend visitors behave differently, traffic sources shift day to day, and a change can look better than it really is simply because it's new.
That last point — the novelty effect, and its mirror image, the primacy effect (an unfamiliar change that looks worse at first and improves as people adjust to it) — isn't just a rule of thumb. A peer-reviewed 2022 paper from Microsoft's own experimentation platform team, published in the journal Technometrics, modeled exactly this pattern with real experiment data and confirmed that a treatment effect measured mid-test is "not always permanently stable," often revealing an increasing or decreasing trend as the test continues.
The practical version: fix your sample size and end date before you start, don't call a test early just because it looks done, and if your tool lets you, watch the trend of the lift over time — a result that's shrinking or growing as the test runs on is exactly the novelty/primacy pattern the research describes, and it means the number on day 3 wasn't the real answer.
How do you actually run a test that holds up?
- 1Pick one thing to change and one primary metric before you start — not five metrics you'll pick a favorite from after the fact.
- 2Work out the sample size you need for the smallest lift that's actually worth detecting, given your current baseline rate — before the test starts, not by eyeballing day 3.
- 3Fix an end date or sample size up front and hold to it. A result that looks significant on day two is usually noise, not a signal — see the peeking numbers above.
- 4Run it for at least one to two full business cycles so weekday/weekend mix and traffic-source shifts even out, and watch for a trend that suggests a novelty or primacy effect.
- 5Treat every metric other than your one primary metric as directional. A real secondary win deserves its own follow-up test, not a shipping decision on its own.
oToK's email campaigns test two subject-line or content variants on part of the audience and send the winner to everyone else automatically, with statistical guardrails built in so a too-small sample never gets to pick a winner — available on every plan, including Free. Landing pages carry the same idea at the page level: split traffic between two published versions and declare a winner with a built-in significance check, available on Growth and up.
Frequently asked questions
- Does "95% significant" mean there's a 95% chance my variant actually wins?
- No. A peer-reviewed guide to statistics misinterpretations calls this reading a direct "No": the 95% describes the long-run error rate of the testing method across many repeated tests, not the odds attached to this one specific result. Once you've decided to ship, you're either right or wrong about this particular case — you just don't get to know which with 95% certainty.
- Can I stop a test early if it already looks like a clear winner after two days?
- That's the exact habit that breaks the promise of the significance test. Checking continuously and stopping the moment it crosses 5% significance can push the real false-positive rate to 26.1%, according to the primary analysis on this — more than five times the rate you think you're getting. Fix your sample size before you start and hold to it.
- Why do different sources report such different A/B test win rates — 8%, 19%, or a third?
- The gap mostly comes down to methodology, not a contradiction: how strict the significance bar was held to, whether underpowered tests were excluded or counted as losses, and — most of all — how well-vetted the tested ideas were to begin with. Airbnb's 8% figure covers 250 raw ideas; Microsoft's roughly one-third figure, per Kohavi, describes ideas that had already cleared an internal bar before testing.
- How many metrics can I safely watch on one test?
- As many as you want to look at — but only one should decide whether you ship. Watching 20 independent metrics at the standard 5% threshold gives you roughly a 64% chance that at least one shows "significant" by pure chance alone. Pick a single primary metric before the test starts and treat the rest as directional.
- Is oToK's built-in A/B testing statistically valid, or do I need to run the numbers myself?
- oToK's email A/B tests split part of the audience between two variants and pick the winner automatically with statistical guardrails built in, so a too-small sample can't call a result — available on every plan. Landing page A/B variants work the same way at the page level, with a built-in significance check, on Growth and up. You still decide what to test and for how long; the platform handles the significance math.
