When to Stop an A/B Test and Declare a Winner
When to call an experiment done — the 95% threshold and the sample floor, how to end the test, and how to roll out and apply the winning variant after.
Stop an A/B test only when three things are true: a variation has reached 95% probability to be best, every variant has at least 300–500 sessions, and the test has run for at least one full week. Knowing when to stop an A/B test matters because the decision to stop is the most consequential moment in the test. Stopping too early is the most common cause of false wins; stopping too late is the most common cause of analysis paralysis. This guide is the rule for getting it right, and for what to do with the winner afterwards.
The Three Conditions
Before you click Declare winner, all three of these should be true:
1. The verdict reads “… is winning”
The verdict box on the Results tab names a variation as winning once it reaches 95% probability to be best on the primary metric. That is the threshold the Declare winner button uses: it appears in the verdict box with that verdict, as long as the test has not already ended.
95% means: based on the data so far, there’s a 95% probability this variant is genuinely the best. Stopping below 95% means accepting more than a 1-in-20 chance you’re calling a winner that isn’t.
There is also a floor under the threshold: every variation, control included, needs at least ten conversions on the primary goal before the engine will name a winner at all. Below it the verdict reads Not enough data yet, whatever the probabilities look like. A variation that clears 95% but breaches one of your guardrail goals is blocked instead, and the verdict says so.
2. Each variant has at least 300–500 sessions
Even at 95% probability, with very small samples the credible intervals are wide enough that the result isn’t trustworthy. As a baseline:
- 300 sessions per variant — minimum for any decision.
- 500 sessions per variant — comfortable for most.
- 1,000+ sessions per variant — for high-stakes decisions (checkout flow, pricing).
If your probability is at 95% but you only have 80 sessions per variant, wait. The math will give you a high-probability number on small samples that doesn’t reflect a real difference. To estimate before you launch how many visitors a test needs and how long it will take on your traffic, use the A/B test sample size calculator.
3. The experiment has run at least one full week
Day-of-week effects are real and significant. A test that runs only on weekdays may show different results than one that runs across a full week. Always run for 7+ days minimum, even if the probability hits 95% on day 2.
For low-traffic stores, plan on 3–4 weeks. For high-traffic stores, the week minimum still applies — don’t shortcut weekly cycles even if you have the volume.
When the Conditions Aren’t Met
A few common scenarios and the right call for each:
“Probability is at 87% and trending up”
Wait. The trend matters less than the absolute number. Probability oscillates as data accumulates — you’ll see it dip and rise. Stop when it crosses your threshold and stays there for at least a day or two, not the moment it first touches.
“Probability hit 95% on day 3 with 600 sessions per variant”
Wait until at least day 7. You have the sample size, but you don’t have the calendar coverage. If by day 7 the probability is still at 95%+, you have a real winner.
“We’ve run for two weeks at 88% probability and traffic is too low to ever hit 95%”
You have a decision to make. Two paths:
- Treat the experiment as inconclusive. Document that the change had no detectable effect at this traffic level. Don’t ship the variant. This is a valid result.
- Re-run on a higher-traffic page if you want a definitive answer. The current data isn’t strong enough to commit to.
Don’t split the difference and ship a “probably better” variant — that’s how false-win rates inflate over time.
“Probability hit 99% in two days”
Still wait for at least one full week. Strong early signals can be novelty effects (visitors react to the change because it’s new, not because it’s better). A week of data washes out novelty.
Ending the test
When you’re ready, click Declare winner in the verdict box, or open the ⋯ menu and choose End test. Both open the same confirmation: Visitors stop entering the test and everyone sees the original page. The results so far are kept. This cannot be undone. If the results were last computed more than an hour ago, the dialog suggests refreshing them first, because the snapshot freezes what was last computed.
Confirm with End test and the experiment becomes Completed:
- The Results tab shows Final snapshot. This test has ended, so its results are frozen. The numbers no longer change and the refresh button is gone.
- New visitors stop being assigned to variants and see the original page.
- The verdict badge reads Final results.
A completed experiment can’t be reopened. If you want to run it again, Duplicate in the ⋯ menu creates a copy you can edit and launch.
After the test
Roll out the winner
Ending a test does not change your site. On a completed test with an honest winner (95% or more, no guardrail breach), a rollout card appears under the verdict with Serve winner at 100%. While it serves, the winning variant is shown to all targeted traffic, so your visitors keep the better version while you make it permanent. The card then offers Make it permanent and Stop serving. See How to Roll Out an A/B Test Winner at 100% for the details and the plan requirements.
Apply the winner permanently
Rollout is a bridge, not a destination. The winning CSS, JS, redirect, theme or template lived inside the experiment’s variant; you have to move it into your site or store for it to outlast the experiment.
For HTML sites
- Open the winning variant under Setup › Variants.
- Copy the CSS, JS, or redirect destination.
- Apply it to your site’s theme/template:
- CSS — add to your global stylesheet or theme’s custom CSS.
- JS — add to your site’s main script bundle, or as a
<script>block in your template. - Redirect — set a server-side 301 redirect (better SEO than a JS redirect).
Once the change is live in your site, the experiment is no longer needed — its job was to prove the change worked.
For Shopify
- Open the winning variant under Setup › Variants.
- Copy the CSS or JS (or the redirect destination).
- Apply it to your theme:
- Online Store → Themes → Edit code → assets/base.css (or your theme’s equivalent).
- For JS, paste into a custom theme.js or use a header-injection app.
- Save and preview before publishing.
Theme and template winners (Shopify)
When the winning variant uses a theme test or a template test, the Results page shows a Make it permanent block under the rollout card (a winner with both shows both):
- Theme — Variant A wins with the theme Dawn — new design. Publish it in Shopify to make it permanent. Until then visitors are sent to it through Split Test Pro. Publish in Shopify opens your theme library and Preview opens the theme. After publishing, check that the Split Test Pro app embed is on in the new live theme — Check the app embed opens it.
- Template — Variant A wins with the template product.alternate. Assign it to all products in Shopify to make it permanent (or to the N selected products). Open products opens the products list in your Shopify admin, or the product itself when there is exactly one; choose the template under Theme template on each item.
Once the theme is published or the template assigned, visitors get it from Shopify directly and the experiment stops sending them anywhere. If the winning variant also has code, move that into the new theme as above.
Once the change is live in your theme, stop serving the rollout so your site no longer depends on the experiment.
Document the Result
Whether you declared a winner, called it inconclusive, or reverted, write down what happened:
- The hypothesis.
- The change you tested.
- The outcome (numbers + decision).
- What you learned.
This becomes your institutional knowledge. Most teams capture it in a shared Notion doc, an internal wiki, or even a Slack thread. The discipline of writing it down forces you to actually understand the result, not just react to it.
Reverting a Winner
Sometimes you ship a winner, watch it run for a week, and realize the lift didn’t hold. Or it caused an unintended downstream issue (lower AOV, higher refund rate). Don’t be precious — back it out and treat it as a learning.
The next experiment can iterate on what you learned. Reverting a winner doesn’t mean the original test was wrong; it means there’s more to the story than the test could show.
Common Mistakes
- Stopping at 90%. Single most common A/B testing mistake. The 5-percentage-point gap from 90% to 95% changes the math meaningfully — don’t shortcut it.
- Stopping after 3 days because results “look stable.” Day-of-week effects make day 3 look stable in ways day 5 doesn’t. Wait the week.
- Ending the test and not applying the change. The Declare winner button doesn’t deploy code — it ends the test and freezes the result. Rollout keeps serving the winner for a while; shipping the change is still your responsibility.
- Never ending the test. Some teams accumulate decided-but-still-running experiments forever. End the test when you’ve decided. Otherwise it keeps taking traffic and sits in the list as cognitive load.
Next Steps
- Understand the Bayesian math behind the 95% threshold: Bayesian A/B Testing: How Chance to Beat Control Works.
- Avoid the most common failure modes: A/B Testing Mistakes: Common Pitfalls and How to Avoid Them.
- Build the discipline that makes calling winners easier: A/B Testing Methodology: Principles and Best Practices.
Frequently asked questions
When should you stop an A/B test?
When three conditions hold: a variation has reached 95% probability to be best on the primary metric, each variant has at least 300 to 500 sessions, and the test has run for at least one full week. Split Test Pro also needs ten conversions on the primary goal in every variation, control included, before it names a winner.
Is 95% probability to be best the same as statistical significance?
Not exactly. A statistical significance test computes a p-value; Split Test Pro reports a Bayesian probability instead, the probability that a variant is the best given the data. It calls a winner once a variation reaches 95% probability to be best on the primary metric and every variation, control included, has at least ten conversions.
Can you stop an A/B test early if it hits 95%?
Not before it has run a full week. A test that reaches 95%, or even 99%, on day two or three may be showing a novelty effect or a day-of-week pattern, and a week of data washes those out. If the probability is still at 95% or more after day 7, you have a real winner.
What if an A/B test never reaches 95%?
Treat it as inconclusive: the change had no detectable effect at your traffic level, so don't ship the variant. If you need a definitive answer, re-run the test on a higher-traffic page. Don't ship a variant that is only probably better, because that is how false-win rates inflate over time.
Does declaring a winner change your site?
No. Declaring a winner ends the test and freezes its results; it doesn't deploy code. You can serve the winner to all targeted traffic with a rollout while you make the change permanent in your theme or site.