How to read your A/B test results
Learn how to analyze results, avoid common pitfalls, and make confident decisions.
Split Test Pro team · 6 min read
Start with the verdict
Open a running test and the Results tab answers the question you actually have — is it working? — before it shows you a single chart. Four stat cards sit at the top: Visitors, your primary goal, Variant uplift and Chance to beat control. Under them, the verdict box sums the test up in one line.
Watch how the verdict changes as data arrives. Early on it reads Not enough data yet: until every variation, control included, has ten conversions on the primary goal, Split Test Pro won’t name a winner however the numbers look. From there it moves through No clear leader yet and trending ahead (80% or more) to is winning — 95% or more — when a Declare winner button appears.
Read the chance, not just the rate
Split Test Pro uses Bayesian statistics, so instead of p-values you get the number you actually want: the probability that the variation truly beats control. Its confidence label climbs from Low confidence through Moderate and High to Very high confidence at 95% or more.
The comparison card puts the rates side by side, states the uplift in plain words, and shows the chance to beat control as a ring. Switch it to Distributions to see each variation’s posterior curve — wider curves mean fewer visitors and more uncertainty.
Check the credible interval
Performance by variation gives each variation a row — visitors, conversion rate with “X of Y visitors”, uplift and chance to beat control — and closes with the estimated uplift as a 95% credible interval once every variation clears the sample floor.
The interval tells you whether a difference is real. A 5.0% versus 4.5% comparison means little if the interval runs from −10% to +30%. As the test collects data, the interval narrows; a narrow one that has moved clear of zero is a confident result.
Five checks before you call it
Chance and sample
The leader is past 95%, and each variation has at least 300–500 visitors and ten conversions.
A full week
The test has run at least seven days, so day-of-week swings have evened out.
Devices and funnel
The result holds on desktop and mobile, and the lift comes from the right stage of the funnel.
Use the Devices filter to see whether a result holds on every device. A flat overall result with a strong mobile win is still actionable — you can apply the change to mobile only. And look at the other goals: a variant that lifts add to carts but lowers purchases isn’t a win.
From result to rollout
Wait for a clear verdict
Let the test reach 95% and run a full week. Refreshing more often won't make the math move faster.
Check the details
Read the credible interval, switch the Devices filter, and look at the funnel and your other goals.
Declare the winner
Click Declare winner to end the test, then make the change permanent on your site.
Want a second opinion? On workspaces with AI enabled, Generate analysis reads the results and summarises what they mean. For every panel on the page, see Reading the Results Dashboard and Declaring a Winner.
