Declaring a Winner

The most consequential moment in an A/B test is the decision to stop. Stopping too early is the most common cause of false wins. Stopping too late is the most common cause of analysis paralysis. This guide is the rule for getting it right.

The Three Conditions

Before you click Declare Winner, all three of these should be true:

1. The leading variant is at 95%+ probability to be best

This is the threshold the Declare Winner CTA uses. The button appears in the Verdict card on the Results page once a variant crosses 95% on the primary metric.

95% means: based on the data so far, there’s a 95% probability this variant is genuinely the best. Stopping below 95% means accepting more than a 1-in-20 chance you’re calling a winner that isn’t.

2. Each variant has at least 300–500 sessions

Even at 95% probability, with very small samples the credible intervals are wide enough that the result isn’t trustworthy. As a baseline:

• 300 sessions per variant — minimum for any decision.
• 500 sessions per variant — comfortable for most.
• 1,000+ sessions per variant — for high-stakes decisions (checkout flow, pricing).

If your probability is at 95% but you only have 80 sessions per variant, wait. The math will give you a high-probability number on small samples that doesn’t reflect a real difference.

3. The experiment has run at least one full week

Day-of-week effects are real and significant. A test that runs only on weekdays may show different results than one that runs across a full week. Always run for 7+ days minimum, even if the probability hits 95% on day 2.

For low-traffic stores, plan on 3–4 weeks. For high-traffic stores, the week minimum still applies — don’t shortcut weekly cycles even if you have the volume.

When the Conditions Aren’t Met

A few common scenarios and the right call for each:

“Probability is at 87% and trending up”

Wait. The trend matters less than the absolute number. Probability oscillates as data accumulates — you’ll see it dip and rise. Stop when it crosses your threshold and stays there for at least a day or two, not the moment it first touches.

”Probability hit 95% on day 3 with 600 sessions per variant”

Wait until at least day 7. You have the sample size, but you don’t have the calendar coverage. If by day 7 the probability is still at 95%+, you have a real winner.

”We’ve run for two weeks at 88% probability and traffic is too low to ever hit 95%”

You have a decision to make. Two paths:

• Treat the experiment as inconclusive. Document that the change had no detectable effect at this traffic level. Don’t ship the variant. This is a valid result.
• Re-run on a higher-traffic page if you want a definitive answer. The current data isn’t strong enough to commit to.

Don’t split the difference and ship a “probably better” variant — that’s how false-win rates inflate over time.

”Probability hit 99% in two days”

Still wait for at least one full week. Strong early signals can be novelty effects (visitors react to the change because it’s new, not because it’s better). A week of data washes out novelty.

Clicking Declare Winner

When you’re ready, click the Declare Winner button in the Verdict card. The experiment transitions to completed status:

• Results freeze as a final snapshot — the dashboard banner reads “Final snapshot — results frozen.”
• New visitors stop being assigned to variants and see the control by default.
• The experiment moves out of the active list and into the concluded section.

There’s no intermediate confirmation step today — clicking transitions immediately. If you change your mind, you can’t unwind a completed experiment, but you can copy it as the basis for a new one.

After Declaring: Apply the Winner Permanently

Declaring a winner doesn’t apply the change to your site. The “winning” CSS or JS lived inside the experiment’s variant — you have to migrate it to your site for it to affect future visitors.

For HTML sites

1. Open the winning variant in the Variants tab.
2. Copy the CSS, JS, or redirect URL.
3. Apply it to your site’s theme/template:
   • CSS — add to your global stylesheet or theme’s custom CSS.
   • JS — add to your site’s main script bundle, or as a <script> block in your template.
   • Redirect — set a server-side 301 redirect (better SEO than a JS redirect).

Once the change is live in your site, the experiment is no longer needed — its job was to prove the change worked.

For Shopify

1. Open the winning variant in the Variants tab.
2. Copy the CSS or JS.
3. Apply it to your theme:
   • Online Store → Themes → Edit code → assets/base.css (or your theme’s equivalent).
   • For JS, paste into a custom theme.js or use a header-injection app.
4. Save and preview before publishing.

For Shopify checkout block variants, the path is different — see Shopify Checkout Blocks.

Document the Result

Whether you declared a winner, called it inconclusive, or reverted, write down what happened:

• The hypothesis.
• The change you tested.
• The outcome (numbers + decision).
• What you learned.

This becomes your institutional knowledge. Most teams capture it in a shared Notion doc, an internal wiki, or even a Slack thread. The discipline of writing it down forces you to actually understand the result, not just react to it.

Reverting a Winner

Sometimes you ship a winner, watch it run for a week, and realize the lift didn’t hold. Or it caused an unintended downstream issue (lower AOV, higher refund rate). Don’t be precious — back it out and treat it as a learning.

The next experiment can iterate on what you learned. Reverting a winner doesn’t mean the original test was wrong; it means there’s more to the story than the test could show.

Common Mistakes

• Stopping at 90%. Single most common A/B testing mistake. The 5-percentage-point gap from 90% to 95% changes the math meaningfully — don’t shortcut it.
• Stopping after 3 days because results “look stable.” Day-of-week effects make day 3 look stable in ways day 5 doesn’t. Wait the week.
• Declaring a winner and not applying the change. The Declare Winner button doesn’t deploy code — it just freezes the result. The change is your responsibility to ship.
• Never declaring a winner. Some teams accumulate completed-but-not-declared experiments forever. End the experiment when you’ve decided. Otherwise it sits in the active list as cognitive load.

Next Steps

• Understand the Bayesian math behind the 95% threshold: Bayesian Stats Explained.
• Avoid the most common failure modes: Common Mistakes.
• Build the discipline that makes calling winners easier: Testing Methodology.