Bayesian A/B Testing: How Chance to Beat Control Works
How the Bayesian engine behind Chance to beat control and the credible intervals works — the priors, the Monte Carlo sampling, and why it beats p-values.
Bayesian A/B testing answers the question product teams actually ask: how likely is it that the new variant is better than control, given the data so far? Split Test Pro uses a Bayesian engine for its results — not the frequentist p-values you might have seen in older A/B testing tools — which is why the Results tab says “73.2% chance to beat control” and not “p < 0.05.” This guide explains why, what the engine actually does under the hood, and how to read the numbers it produces.
You don’t need a stats background to use Split Test Pro. But if you want to know where those numbers come from, this is the doc.
Frequentist vs Bayesian, Briefly
Frequentist (the traditional approach):
- You compute a p-value — the probability of seeing this data if the null hypothesis (no difference between variants) were true.
- You stop the test when p < 0.05.
- You can’t peek at p-values during the test — looking inflates your false-positive rate.
- Predetermined sample size required.
Bayesian (Split Test Pro’s approach):
- You compute a posterior probability — the probability that Variant B is better than Control given the data.
- You stop the test when the probability crosses a threshold (typically 95%).
- You can check the result anytime — the math doesn’t care about peeking.
- No predetermined sample size. The data updates the model continuously.
The Bayesian framing matches how product teams actually think: “How confident are we that the new variant is better?” — not “How surprising would this data be if there were no difference?”
The Statistical Model
Split Test Pro uses two models depending on the metric type. Both are computed independently per variant.
Binary metrics — Beta-Binomial
For binary metrics (conversion happened / didn’t happen), each variant’s conversion rate is modeled as a Beta distribution:
posterior = Beta(α, β)
where α = conversions + 1
β = (sessions − conversions) + 1
The +1 on each parameter is the Beta(1, 1) prior — a uniform prior that says “before seeing any data, every conversion rate from 0% to 100% is equally plausible.” This is the standard “uninformative” prior for binary A/B testing. As data accumulates, the posterior tightens around the true rate.
Continuous metrics — Normal
For continuous metrics (revenue per session, AOV, time spent), each variant’s mean value is modeled as a Normal distribution parameterized by:
- µ = sample mean of recorded values
- σ = standard error of the mean
There’s no informative prior on the continuous side — the data drives the parameters directly. As more events arrive, the standard error shrinks and the distribution tightens around the true mean.
See Continuous Metrics for when continuous mode kicks in.
Chance to Beat Control and Probability to Be Best
The headline number on the Results tab is Chance to beat control: the probability that the best variation truly beats control on the primary metric. Behind it sits probability to be best, which the verdict and the winner rule read. Both are computed via Monte Carlo sampling:
- For each variant, draw a random sample from its posterior distribution.
- Record which variant’s sample is the largest.
- Repeat 5,000 times.
- The probability that variant X is best = (count of times X had the largest draw) / 5,000.
For a two-variant test, probability to be best equals chance to beat control — they’re the same number. For three or more variants, probability to be best sums to 100% across all variants, while chance to beat control is computed per non-control variant independently. The verdict box names a variation as winning when its probability to be best reaches 95%; the stat card and the Performance by variation table show its chance to beat control. To work out both numbers from your own visitors and conversions, use the Bayesian A/B test calculator.
Credible Intervals
A credible interval is the range of plausible values for a parameter, with a stated probability. Split Test Pro reports 95% credible intervals by default:
“We’re 95% confident the true conversion rate for Variant B is between 4.2% and 5.8%.”
This is not the same as a frequentist confidence interval. Confidence intervals make a procedural claim (“if you ran this experiment 100 times, 95 of the intervals would contain the true value”). Credible intervals make a direct claim (“there’s a 95% chance the true value is in this range, given the data”).
Credible intervals work how most people intuitively think confidence intervals work. That’s the upshot.
Reading the interval
- Wide interval (e.g., 3.0% to 7.0%) — not enough data yet. The true rate could be anywhere in there.
- Narrow interval (e.g., 4.7% to 5.1%) — data has converged. You have a high-confidence estimate.
- Two intervals don’t overlap — strong signal of a real difference.
- Two intervals overlap heavily — likely no real difference, even if the point estimates differ.
Modeled Improvement
The engine also models the improvement — the probability distribution of how much Variant B beats (or loses to) Control. On the Results tab you see it in two places: the Distributions view of the comparison card draws each variation’s posterior curve, and the line under Performance by variation reads Estimated uplift: +a% to +b% · 95% credible interval. It’s computed by sampling pairs from each variant’s posterior and computing the lift:
lift = (variant_sample - control_sample) / control_sample × 100
The result is a distribution of plausible lifts, and the 95% bounds are the interval the page shows. An interval entirely above zero with a tight spread is a strong winner. One straddling zero with a wide spread is uncertain — it needs more data.
The 95% Threshold
Split Test Pro uses 95% as the default threshold for declaring a winner. This is a convention, not a mathematical truth — you can apply different thresholds based on the stakes of the decision:
- 90% — fine for low-stakes tests. Higher false-positive rate, but for cosmetic changes that’s acceptable.
- 95% — the default. Balances false positives against waiting forever.
- 99% — for high-stakes tests (changes to checkout, pricing, or any revenue-critical flow). Lower false-positive rate, longer experiments.
The Declare winner button in the verdict box uses 95% as its trigger. You can choose to wait longer or stop earlier based on context — the threshold is your decision, not the platform’s.
Why You Can Peek (Mostly)
A signature claim of Bayesian methods is that peeking doesn’t inflate false positives the way it does with frequentist methods. This is approximately true, but with caveats:
- The math doesn’t penalize you for looking. Chance to beat control is a posterior; it updates with the data and doesn’t “spike” at random checkpoints.
- But you, the human, can fool yourself. Watching the number bounce around in real time invites confirmation bias — you’ll convince yourself it’s “going to” cross 95% any minute now and stop early.
- And the prior matters. Beta(1, 1) is uninformative, but that means very small samples can produce surprisingly extreme posteriors. Always sanity-check sample sizes.
The pragmatic rule: don’t stop until 95% AND at least 300–500 sessions per variant AND at least 7 days elapsed. That belt-and-braces combination keeps you honest.
What the Engine Doesn’t Do
Worth being explicit about the limitations:
- No multiple-testing correction. Running 10 secondary metrics on the same experiment increases your odds that at least one looks significant by chance. Treat secondaries as directional, not decisional.
- No outlier trimming. A few huge revenue events can pull a continuous mean for a long time. The math handles it via increased variance, but in practice you may want to wait longer or manually inspect outliers.
- No auto-stop. The engine never ends a test by itself. Probability hitting 95% surfaces the Declare winner button; clicking it, choosing End test or a scheduled end date is what ends the test. See Experiment Lifecycle.
- No adaptive traffic allocation. Some platforms shift traffic toward the leading variant during the test. Split Test Pro doesn’t — it uses fixed splits per the configuration.
A Note on Sample Sizes
A common question: “How many sessions do I need?” There’s no single answer — it depends on:
- The baseline conversion rate of your primary metric (lower base = need more samples).
- The lift size you care about detecting (smaller lifts = need more samples).
- The threshold (99% needs more than 95%).
There is also a hard floor built into the engine: ten conversions on the primary metric per variation, control included. Until every variation has ten, no winner is declared whatever the probabilities say, the verdict reads Not enough data yet, and the credible interval is withheld.
Beyond that floor, a rough starting point: at least 300–500 sessions per variant before the credible intervals tighten enough to matter, and at least 1,000+ sessions per variant for high-stakes decisions. Below 300, anything the dashboard says is provisional.
Next Steps
- Use the dashboard built on this engine: Results Dashboard.
- Decide when 95% is enough to act on: When to Stop an A/B Test and Declare a Winner.
- Avoid the most common ways to fool yourself with Bayesian results: A/B Testing Mistakes: Common Pitfalls and How to Avoid Them.
Frequently asked questions
What is Bayesian A/B testing?
Bayesian A/B testing computes a posterior probability: the probability that a variant is better than control, given the data collected so far. Frequentist testing computes a p-value instead, the probability of seeing the data if there were no difference between variants. Split Test Pro uses the Bayesian approach for its results.
What does chance to beat control mean?
Chance to beat control is the probability that the best variation truly beats control on the primary metric. Split Test Pro computes it with Monte Carlo sampling: it draws 5,000 samples from each variant's posterior distribution and counts how often each one comes out ahead. In a two-variant test it is the same number as probability to be best.
What is a credible interval in an A/B test?
A credible interval is the range of plausible values for a metric, with a stated probability. Split Test Pro reports 95% credible intervals, which you can read directly as a 95% chance that the true value lies in that range, given the data. A frequentist confidence interval makes a different claim, about what would happen if you repeated the experiment many times.
Can you peek at a Bayesian A/B test?
The math doesn't penalize you for looking, because chance to beat control is a posterior that updates with the data. The risk is human: watching the number move invites you to stop early. Don't stop until you have 95% probability, at least 300 to 500 sessions per variant and at least 7 days of data.
Which prior does a Bayesian A/B test use?
For binary metrics Split Test Pro uses a Beta(1, 1) prior, a uniform prior that treats every conversion rate from 0% to 100% as equally plausible before any data arrives. Continuous metrics such as Revenue per Session use a Normal model with no informative prior, so the data drives the parameters directly.