Declaring a Winner | Split Test Pro

The most consequential moment in an A/B test is the decision to stop. Stopping too early is the most common cause of false wins. Stopping too late is the most common cause of analysis paralysis. This guide is the rule for getting it right.

The Three Conditions

Before you click Declare winner, all three of these should be true:

1. The verdict reads “… is winning”

The verdict box on the Results tab names a variation as winning once it reaches 95% probability to be best on the primary metric. That is the threshold the Declare winner button uses: it appears in the verdict box with that verdict, as long as the test has not already ended.

95% means: based on the data so far, there’s a 95% probability this variant is genuinely the best. Stopping below 95% means accepting more than a 1-in-20 chance you’re calling a winner that isn’t.

There is also a floor under the threshold: every variation, control included, needs at least ten conversions on the primary goal before the engine will name a winner at all. Below it the verdict reads Not enough data yet, whatever the probabilities look like. A variation that clears 95% but breaches one of your guardrail goals is blocked instead, and the verdict says so.

2. Each variant has at least 300–500 sessions

Even at 95% probability, with very small samples the credible intervals are wide enough that the result isn’t trustworthy. As a baseline:

300 sessions per variant — minimum for any decision.
500 sessions per variant — comfortable for most.
1,000+ sessions per variant — for high-stakes decisions (checkout flow, pricing).

If your probability is at 95% but you only have 80 sessions per variant, wait. The math will give you a high-probability number on small samples that doesn’t reflect a real difference.

3. The experiment has run at least one full week

Day-of-week effects are real and significant. A test that runs only on weekdays may show different results than one that runs across a full week. Always run for 7+ days minimum, even if the probability hits 95% on day 2.

For low-traffic stores, plan on 3–4 weeks. For high-traffic stores, the week minimum still applies — don’t shortcut weekly cycles even if you have the volume.

When the Conditions Aren’t Met

A few common scenarios and the right call for each:

Wait. The trend matters less than the absolute number. Probability oscillates as data accumulates — you’ll see it dip and rise. Stop when it crosses your threshold and stays there for at least a day or two, not the moment it first touches.

“Probability hit 95% on day 3 with 600 sessions per variant”

Wait until at least day 7. You have the sample size, but you don’t have the calendar coverage. If by day 7 the probability is still at 95%+, you have a real winner.

“We’ve run for two weeks at 88% probability and traffic is too low to ever hit 95%”

You have a decision to make. Two paths:

Treat the experiment as inconclusive. Document that the change had no detectable effect at this traffic level. Don’t ship the variant. This is a valid result.
Re-run on a higher-traffic page if you want a definitive answer. The current data isn’t strong enough to commit to.

Don’t split the difference and ship a “probably better” variant — that’s how false-win rates inflate over time.

“Probability hit 99% in two days”

Still wait for at least one full week. Strong early signals can be novelty effects (visitors react to the change because it’s new, not because it’s better). A week of data washes out novelty.

Ending the test

When you’re ready, click Declare winner in the verdict box, or open the ⋯ menu and choose End test. Both open the same confirmation: Visitors stop entering the test and everyone sees the original page. The results so far are kept. This cannot be undone. If the results were last computed more than an hour ago, the dialog suggests refreshing them first, because the snapshot freezes what was last computed.

Confirm with End test and the experiment becomes Completed:

The Results tab shows Final snapshot. This test has ended, so its results are frozen. The numbers no longer change and the refresh button is gone.
New visitors stop being assigned to variants and see the original page.
The verdict badge reads Final results.

A completed experiment can’t be reopened. If you want to run it again, Duplicate in the ⋯ menu creates a copy you can edit and launch.

After the test

Roll out the winner

Ending a test does not change your site. On a completed test with an honest winner (95% or more, no guardrail breach), a rollout card appears under the verdict with Serve winner at 100%. While it serves, the winning variant is shown to all targeted traffic, so your visitors keep the better version while you make it permanent. The card then offers Make it permanent and Stop serving. See Winner Rollout for the details and the plan requirements.

Apply the winner permanently

Rollout is a bridge, not a destination. The winning CSS, JS or redirect lived inside the experiment’s variant; you have to move it into your site for it to outlast the experiment.

For HTML sites

Open the winning variant under Setup › Variants.
Copy the CSS, JS, or redirect URL.
Apply it to your site’s theme/template:
- CSS — add to your global stylesheet or theme’s custom CSS.
- JS — add to your site’s main script bundle, or as a <script> block in your template.
- Redirect — set a server-side 301 redirect (better SEO than a JS redirect).

Once the change is live in your site, the experiment is no longer needed — its job was to prove the change worked.

For Shopify

Open the winning variant under Setup › Variants.
Copy the CSS or JS.
Apply it to your theme:
- Online Store → Themes → Edit code → assets/base.css (or your theme’s equivalent).
- For JS, paste into a custom theme.js or use a header-injection app.
Save and preview before publishing.

Once the change is live in your theme, stop serving the rollout so your site no longer depends on the experiment.

Document the Result

Whether you declared a winner, called it inconclusive, or reverted, write down what happened:

The hypothesis.
The change you tested.
The outcome (numbers + decision).
What you learned.

This becomes your institutional knowledge. Most teams capture it in a shared Notion doc, an internal wiki, or even a Slack thread. The discipline of writing it down forces you to actually understand the result, not just react to it.

Reverting a Winner

Sometimes you ship a winner, watch it run for a week, and realize the lift didn’t hold. Or it caused an unintended downstream issue (lower AOV, higher refund rate). Don’t be precious — back it out and treat it as a learning.

The next experiment can iterate on what you learned. Reverting a winner doesn’t mean the original test was wrong; it means there’s more to the story than the test could show.

Common Mistakes

Stopping at 90%. Single most common A/B testing mistake. The 5-percentage-point gap from 90% to 95% changes the math meaningfully — don’t shortcut it.
Stopping after 3 days because results “look stable.” Day-of-week effects make day 3 look stable in ways day 5 doesn’t. Wait the week.
Ending the test and not applying the change. The Declare winner button doesn’t deploy code — it ends the test and freezes the result. Rollout keeps serving the winner for a while; shipping the change is still your responsibility.
Never ending the test. Some teams accumulate decided-but-still-running experiments forever. End the test when you’ve decided. Otherwise it keeps taking traffic and sits in the list as cognitive load.

Next Steps

Understand the Bayesian math behind the 95% threshold: Bayesian Stats Explained.
Avoid the most common failure modes: Common Mistakes.
Build the discipline that makes calling winners easier: Testing Methodology.