How the calculator works
The calculator uses the same Bayesian method the Split Test Pro results page uses. Instead of a p-value, it answers the question product teams actually ask: how confident are we that the new variant is better?
The model: Beta-Binomial
A visitor either converts or does not, so each version's conversion rate is modeled as a Beta distribution. With the default prior, the posterior for a version is Beta(α, β), where α = conversions + 1 and β = (visitors − conversions) + 1. Each version is modeled on its own.
The default prior: Beta(1, 1)
The +1 on each parameter is the Beta(1, 1) prior: a uniform prior that says, before seeing any data, every conversion rate from 0% to 100% is equally plausible. It is the standard uninformative prior for binary A/B testing and the one the Split Test Pro results page uses. As data accumulates, the posterior tightens around the true rate. Under Advanced you can set a different prior.
What a credible interval is
A credible interval is the range of plausible values for a conversion rate, with a stated probability. A 95% credible interval means there is a 95% chance the true value is in that range, given the data. A wide interval means there is not enough data yet. A narrow one means the data has converged.
How the Monte Carlo draws work
The probabilities are computed by Monte Carlo sampling. For each version the calculator draws 100,000 random samples from its posterior, then counts:
- Chance to beat control is the share of draws in which the variant's sample is larger than the control's.
- Probability to be best is the share of draws in which a version's sample is the largest of all.
- Uplift is (variant sample − control sample) / control sample × 100 for each draw. The figure shown is the median, with its credible interval underneath.
With one variant, probability to be best equals chance to beat control. With two or more, probability to be best adds up to 100% across all versions, while chance to beat control is worked out for each variant on its own. The draws are seeded by your inputs, so the same inputs always give the same numbers.
Reading the result
The sentence above the table follows the same thresholds as the Split Test Pro results page. A variant is winning when its probability to be best reaches 95%, and trending ahead from 80%. Control is winning at 5% or below, and trending ahead at 20% or below. Until every version has at least 10 conversions the result reads Not enough data yet, whatever the probabilities say.
An uplift interval entirely above zero is a strong winner. One that straddles zero is uncertain and needs more data.
For the full explanation of the model, read our guide to Bayesian A/B testing.