AI Analysis

The AI Analysis card on the Results tab turns a running or finished test’s numbers into a short written read: what the data shows, what stands out, and what to do next. It is opt-in: nothing is generated until you click the button.

This guide covers what the analysis contains, when to use it, and the caveats that matter.

Generating an analysis

The card sits under the Conversion rate over time chart and the comparison card on any experiment’s Results tab, on workspaces where AI is enabled. Click Generate analysis. After a few seconds the card expands with four sections:

1. Summary: 2–3 sentences on what the data shows.
2. Key Findings: 3–5 bullet points highlighting notable patterns (winning variants, funnel drop-offs, device discrepancies).
3. Recommendation: one of three calls: Implement, Keep running or Revert, with the reasoning.
4. Estimated Impact: a monthly revenue or conversion range if you applied the winning variant, based on current traffic and conversion rates.

For Shopify experiments, the prompt includes the full funnel breakdown so the analysis can flag where in the funnel a variant changed behaviour. For HTML sites, it’s the conversion goals you’ve defined.

Once an analysis exists, the header button is replaced by Regenerate at the foot of the card. If the results have been recomputed since, a note reads Results have been updated since this analysis was generated. Click Regenerate for fresh insights.

When to use it

• Mid-experiment sanity check: your gut says the result is meaningful, but is it?
• End of experiment, before declaring a winner: an independent read on whether you’re calling the right thing.
• Communicating results to stakeholders: the structured Summary / Recommendation format drops straight into a Slack message or a report.

It’s most useful for non-statisticians. If you’re already comfortable reading credible intervals and probability distributions, you’ll often draw the same conclusions yourself.

How the Cache Works

An analysis is cached on the server per experiment for 6 hours. Opening the Results tab inside that window shows the cached text straight away, with no new generation. Regenerate forces a fresh analysis whatever the cache says.

The analysis always reads the unfiltered view of the test: every device, every visitor, the whole date range. The filter bar never changes it, which is why the card wears an All visitors tag when the Exposure filter is on Single-variant only.

A practical pattern: skip the analysis until you have meaningful data (a few hundred visitors per variation), then regenerate once a day during the experiment’s main run rather than on every page load.

What the Analysis Won’t Do

• Won’t make the decision for you. It’s a recommendation, not an authority. The “Implement” call is based on the data the prompt was given, including any caveats you’ve already considered yourself.
• Won’t catch external context. It doesn’t know about your business goals, seasonal effects, or active marketing campaigns. The recommendation is “based on this data alone.”
• Won’t flag long-tail risks. The Implement / Keep running / Revert framing is intentional and three-bucket. It doesn’t say “implement, but ramp slowly to 25% first”: that’s still your call.
• Won’t replace your stats intuition. It’s a complement to Bayesian Stats Explained, not a substitute. If the underlying numbers don’t support the recommendation, the recommendation is wrong.

Cost and Usage

Every AI feature in Split Test Pro draws from one monthly allowance of generations per workspace: 10 on Core, 200 on Pro and 500 on Agency, or 5 without a paid plan. Each fresh analysis uses one generation. A cached analysis is free, and Regenerate always uses another one. At the limit the request is refused until the allowance resets at the start of the next month. See Billing and Plans.

If you find yourself regenerating every few minutes, that’s a sign you’re peeking too aggressively at the results. See Common Mistakes.

Privacy

The prompt sent to the model includes:

• The experiment name, hypothesis, and primary metric
• Variant counts, conversion totals, and conversion rates per variant
• Funnel events (Shopify) or custom event totals (HTML)
• The target URL pattern

It does not include:

• Individual visitor IDs, sessions, or PII
• Customer data, order details, or payment information
• The actual variant CSS or JS code

If you have policies around what data can leave your domain, this is the surface to evaluate.

Common Mistakes

• Treating the analysis as final. It’s a summary. Read the actual numbers in the Results Dashboard.
• Running it too early. An analysis at 100 visitors is going to be uncertain about everything. Wait until you have meaningful data.
• Ignoring the staleness notice. If the card says the results have been updated, click Regenerate before acting on the recommendation.
• Skipping the human review. The AI is good at summary; it’s bad at “do I trust this enough to ship to all my customers.” That part is yours.

Next Steps

• Read the underlying numbers behind any AI summary: Results Dashboard.
• Check a test’s setup before it goes live: Launch checks.
• Get the framework for when an AI Implement call is enough to act on: Declaring a Winner.