Anomaly Alerts

Split Test Pro runs background checks on every active experiment to catch the kinds of problems that usually go unnoticed until they’ve cost you a week of bad data. This guide covers what the platform looks for, when each alert fires, and what to do about it.

How Detection Runs

Two layers run on a 12-hour cadence per experiment:

1. Rule-based checks — fast, deterministic checks against fixed thresholds. Always run.
2. Claude-powered anomaly review — runs every 24 hours per experiment when AI is enabled. Looks for subtler patterns: novelty effects, trend divergence, day-of-week patterns, sudden shifts.

Each detected issue produces an alert with a severity, a type, and a recommendation. Every alert is listed in the experiment’s Alerts tab, and the first time you see a new one it also appears as a banner on the Results tab, directly under the filter bar. Each alert shows a one-line message, a What to do link that opens the recommendation, its severity, an AI-detected label on the Claude ones, and a dismiss button.

The Four Rule-Based Checks

1. Sample Ratio Mismatch (SRM)

Severity: Critical

When it fires: Once the test has 1,000 or more sessions across its variants, the check compares each variant’s share of those sessions against the traffic split you configured, using a chi-square goodness-of-fit test over every variant with a weight above zero. It fires when the chance of seeing a split that far from your configuration is below 1 in 1,000 (p < 0.001). The alert reads, for example, Traffic split is 55% / 45%, not the configured 50% / 50%. A three-variant test lists all three shares. Under What to do: “Check variant assignment. If you changed the split mid-test, returning visitors keep their earlier variant, so it can lag.”

What it means: Visitors aren’t being assigned in the proportions you configured. On a 50/50 test, a 55/45 split means something is wrong with assignment — possibly a bot filter that’s hitting one variant disproportionately, a redirect loop, or a script-loading issue on one variant’s affected pages.

What to do: Don’t trust the results until you fix it. SRM is a sign your data is biased. Common causes:

• Variant CSS or JS that breaks the page enough that visitors bounce (skewing the assignment in favor of bouncers).
• A redirect variant where the destination page itself sets the experiment cookie.
• A bot or crawler hitting one URL pattern more than the other.

Pause the experiment, identify the cause, and start fresh after the fix.

2. Traffic Drop

Severity: Warning

When it fires: Once the test has at least 14 days of data, when average daily traffic in the last 7 days is less than 50% of the prior 7 days. The alert reads, for example, Traffic down 60% vs the prior 7 days.

What it means: Significantly less traffic is reaching the experiment than was expected. Often unrelated to the experiment itself — usually a campaign ending, seasonality, or a site issue.

What to do: Check whether the traffic drop is global (look at your overall analytics) or specific to the experiment’s targeted pages. If global, the experiment is fine — just expect a longer runtime. If specific, investigate whether something’s broken on the targeted pages.

3. Zero Conversions

Severity: Warning

When it fires: After more than 7 days of running, if control or the variant has zero conversions on the primary metric. The alert reads, for example, One variant has no conversions after 9 days.

What it means: Either tracking is broken for that variant, or the variant is dramatically worse than control. Both warrant a look.

What to do: First, sanity-check tracking — did the conversion event fire correctly for visitors in that variant? Open DevTools as a fresh visitor, force the variant, and trigger a conversion. If the event fires correctly and the dashboard still shows zero, something deeper is wrong — possibly the variant breaks the conversion flow entirely.

4. Test Running Too Long

Severity: Info

When it fires: After more than 60 days, if the leading variant hasn’t crossed 90% probability to be best. The alert reads, for example, No significant result after 64 days (best variant 78.0%).

What it means: This experiment isn’t going to converge. Either the effect is too small to detect with your traffic, or there’s no real effect at all.

What to do: Make a call. Either complete the experiment as inconclusive (and use the learning that the change has no detectable effect — that’s still a result), or accept that you’re investing more runtime for diminishing return. There’s no virtue in running a 90-day test that’s stuck at 78% probability.

Claude Anomaly Review

When AI is enabled, a separate analysis runs every 24 hours. It looks at the last 21 days of conversion data and flags four pattern types:

• novelty_effect — variant outperforms early but the lift decays as visitors get used to the change. Common with anything visually attention-grabbing.
• trend_divergence — variant and control conversion rates were tracking together, then suddenly diverged. Often signals a campaign-specific or audience-specific effect.
• day_pattern — variant performs differently on weekdays vs weekends, or on specific days of the week.
• sudden_shift — abrupt change in one variant’s behavior on a specific day, often correlated with an external event.

Severities for Claude alerts: info, warning, critical — matching the rule-based scale. Claude keeps each message to 90 characters and each recommendation to 160, so they read like the rule-based alerts.

How Alerts Are Delivered

Alerts are stored with the experiment and listed in its Alerts tab, after Results, Setup and Activity. The tab’s badge counts the open alerts, together with the results warnings (mixed exposure, single-variant view, low sample, out-of-date results). Each detection run updates the open alerts and removes any whose condition no longer holds, so a traffic drop that recovers clears itself. Pressing Refresh on the Results tab runs a detection pass, and any alert it raises appears without reloading the page.

Shown once, then in the Alerts tab

The first time this browser sees an alert, it appears as a banner under the filter bar on the Results tab. After that it lives only in the Alerts tab: reload the page and the banner is gone, while the badge still counts the alert. The open list shows the results warnings first, then the anomaly alerts, newest first.

There is no SMS or webhook delivery. If you need anomaly notifications outside the app, the path is to check the Results tab on a schedule.

Dismissing an Alert

Click the × on a banner or in the Alerts tab to dismiss an alert once you’ve triaged it. The dismissal is per alert and applies to everyone in the workspace: a teammate opening the same experiment will no longer see it either. Dismissed alerts move to the collapsed Dismissed (n) group at the bottom of the Alerts tab, where they stay readable. An anomaly alert can’t be brought back once dismissed; a dismissed results warning has Show again, which returns it to the open list and the Results tab.

Dismissing an alert doesn’t delete the underlying issue. While the condition persists, the next detection run keeps the alert dismissed. It is re-raised only if its severity rises, for example from a warning to critical.

What Anomaly Detection Doesn’t Do

• Doesn’t auto-pause the experiment. Even a critical SRM alert lets the experiment keep running — pausing is your call.
• Doesn’t suggest the fix. It tells you something is wrong; investigation is on you.
• Doesn’t catch experiment-design errors. Bad targeting, wrong primary metric, no hypothesis — those need launch checks, not anomaly detection.
• Doesn’t run on draft or paused experiments. Detection runs on running experiments only.

Tuning the Sensitivity

The thresholds are fixed (1,000 sessions and p < 0.001 against your configured split for SRM, 50% drop for traffic drop, 7 days for zero conversions, 60 days for too-long). They’re not configurable. If an alert is noisy for a test you understand, dismiss it; it stays dismissed unless its severity rises.

Common Mistakes

• Ignoring SRM as “probably nothing.” It’s almost always something. Treat critical alerts seriously.
• Reacting to a single Claude alert without the rule-based context. Claude flags patterns; the rule checks flag thresholds. The two together give a clearer picture than either alone.
• Dismissing without fixing. A dismissed alert about a real problem just hides the symptom, for everyone in the workspace. The data is still bad.
• Treating “info” severity as ignorable. Info-level alerts (like “test running too long”) aren’t urgent but they’re often the most actionable — they tell you something about your experimentation cadence, not just one experiment.

Next Steps

• See where alerts surface in your workflow: Results Dashboard.
• Catch problems before they trigger an SRM alert: Launch checks.
• Use AI Analysis for a higher-level read on results: AI Analysis.
• See every email Split Test Pro sends and who receives it: Notification Emails.