Split Test Pro
Beginner 8 min read

Troubleshooting

Diagnose and fix the most common Split Test Pro issues — variants not showing, conversions not tracked, results not updating, and platform-specific gotchas for both HTML and Shopify.

When Split Test Pro doesn’t behave the way you expect, the cause is almost always one of a small number of usual suspects. This guide walks through the diagnostic flow for the most common problems.

Variant Doesn’t Apply

The single most reported issue. Work through this checklist in order.

1. Confirm the script is loading

Open your site in DevTools → Network tab → reload → filter for experiment.js.

  • You see a 200 response → script is loading. Skip to step 2.
  • You see a 404 or 403 → the API key is wrong. For HTML, recopy the snippet from Settings → Script & API. For Shopify, the script lives in the Theme App Extension — see step 3.
  • You don’t see a request at all → the script tag isn’t in the page’s HTML. For HTML, verify the snippet is pasted in <head> of every page (or via your tag manager firing on All Pages). For Shopify, see step 3.

2. Confirm targeting matches the current page

The variant only activates if the current URL matches the experiment’s targeting rules.

  • Open the experiment’s Targeting tab.
  • Paste the URL you’re testing into the Preview URL field.
  • The Preview URL banner should say Match. If it says No match, your targeting doesn’t match the URL. Adjust targeting or pick a different test URL.

See URL Targeting for the exact behavior of each operator.

3. (Shopify) Confirm App Embeds is enabled

The most common Shopify-specific cause:

  1. Online Store → Themes → Customize on your live theme.
  2. App embeds icon (puzzle-piece) in the left sidebar.
  3. Find Split Test Pro — toggle should be ON.

If you recently switched themes, the new theme may have started with App embeds OFF. See Shopify Theme App Extension.

4. Check device targeting

If the experiment has device targeting enabled, you need to be on a matching device class. A mobile-targeted experiment won’t activate on a desktop browser. Check the Targeting tab.

Open DevTools → Application (Chrome) or Storage (Firefox) → Cookies. Look for splittestpro-experiment.

  • Cookie present, contains the right experiment ID → the visitor is bucketed but the variant CSS/JS isn’t applying. Skip to step 6.
  • Cookie present, but contains a different variant ID → you’re in the control. Try a fresh incognito window to get re-bucketed.
  • Cookie missing → assignment never happened. Check console for JS errors that may have interrupted the script.

6. Check for theme specificity issues

If the cookie is correct but the variant still doesn’t render visibly, theme styles are overriding your variant CSS. See Specificity Escalation in Selector Cookbook.

7. Check console for errors

DevTools → Console. Red errors that occurred before the script ran can interrupt initialization. Look especially for errors from theme scripts that load early in <head>.

Conversions Not Tracked

The variant applies fine, but the Results dashboard shows no conversions.

HTML: snippet missing on the conversion page

The snippet must be on the page where the conversion event fires (typically the thank-you / confirmation page). If the conversion happens on a different domain or a page where the snippet isn’t installed, no conversion is recorded.

Fix: install the snippet site-wide.

HTML: event key mismatch

The event you fire must match the goal’s event key exactly. trackConversion("Signup") won’t match a goal defined as signup. Case-sensitive, exact-string match.

Fix: open Settings → Conversion goals, copy the exact event key, paste it into your trackConversion call.

HTML: trackConversion not actually firing

Open DevTools → Network tab → filter for analytics. Trigger the conversion. You should see a POST to /api/html/experiment/analytics.

  • No POST → your code isn’t reaching the trackConversion call. Add a console.log to verify the line executes.
  • POST returns 4xx → the payload is malformed or the API key is wrong. Check the response body for the error.
  • POST returns 200 but Results still shows zero → see “Results not updating” below.

HTML: visitor in multiple experiments (known issue)

If the visitor is bucketed into 2+ concurrent experiments, only the first experiment’s conversion is recorded. See Custom Events for the workaround. Fix is in flight (see TASK-21).

Shopify: Web Pixel not connected

In your Shopify admin, go to Settings → Customer events. Split Test Pro should show Connected. If it doesn’t, uninstall and reinstall the app.

Shopify: customer privacy / consent

In regions where Shopify’s customer-privacy banner is active, visitors who decline analytics consent don’t get pixeled — and therefore don’t appear in any experiment results. This is expected behavior.

Results Not Updating

You’re sure data is flowing in, but the Results dashboard isn’t reflecting it.

Cache lag

Results are cached server-side and may take a few minutes to reflect new events. Click Refresh on the Results page to force a recompute. For very large datasets, refresh can take 1–2 minutes.

Stale browser state

Hard-refresh the page (Ctrl+Shift+R / Cmd+Shift+R) to bypass the browser cache. Sometimes the page itself is showing a cached older state.

Wrong workspace

If you have multiple workspaces, the dashboard is scoped to the currently-selected one. Confirm you’re in the workspace where the experiment lives.

Variant CSS Shows Briefly, Then Disappears

The variant flashes in then is replaced by the original styling. This means a theme script is reapplying styles after the variant CSS injects.

Two fixes:

  1. Use !important more aggressively in the variant CSS. The theme’s late-applied styles are usually inline (style="...") — only !important beats those.
  2. Switch to a JS variant that uses MutationObserver to re-apply the change whenever the theme overwrites it.

See JavaScript Variants for the MutationObserver pattern.

Variant Causes Layout Shift

Variant CSS that changes element sizes can cause cumulative layout shift (CLS), where the page jumps as the variant applies. Mitigations:

  • Pre-allocate space in the variant CSS (min-height, padding).
  • Avoid changing display modes (block ↔ flex ↔ grid) at runtime — those reflows are jarring.
  • For full-page redesigns, use a redirect variant instead of CSS — visitors arrive at the redesigned page directly with no shift.

Variant Works in Incognito but Not Normal Window

Cookies persist your variant assignment across visits. If you’re testing in your normal browser, you keep seeing the same variant. To verify, either:

  • Open a fresh incognito window.
  • Delete the splittestpro-experiment cookie in DevTools and reload.
  • Use the preview cookie to force a specific variant — see Screenshots and Preview.

Single-Page App: Variant Works on Initial Load Only

Split Test Pro evaluates targeting on initial page load. SPA route changes via history.pushState don’t retrigger experiments. See Single-Page Apps for the workaround.

”Connected” Indicator Stays Gray on HTML

After installing the snippet on your site, the Connected indicator in Settings → Script & API should turn green within a minute of the first pageview from your registered domain.

If it stays gray:

  1. Confirm the snippet is on your public-facing domain (not just a staging URL).
  2. Confirm the API key in the snippet matches your workspace.
  3. Visit the site once, then wait up to a minute and refresh the Settings page.
  4. If still gray, check that the registered workspace domain matches your actual domain (Settings → Workspace).

Shopify: Experiment Worked Yesterday, Not Today

Most likely cause: you published a new theme, and the new theme has App Embeds OFF.

Re-enable App Embeds on the published theme — see step 3 above. This is the single most frequent regression report.

AI Review Returns Errors or Unhelpful Output

Two common scenarios:

  • “Analysis temporarily unavailable. Try again in a few minutes.” → the underlying Claude API is rate-limited or temporarily down. Wait 5 minutes and retry.
  • The review is generic or wrong → the experiment’s data is too thin (fewer than ~100 sessions per variant) for the AI to draw meaningful conclusions. Wait until more data accumulates and request a fresh review (the cache TTL is 6 hours).

Errors in Browser Console About SplitTestPro

If you see window.SplitTestPro is undefined in the console, the script isn’t loaded yet. Common causes:

  • Script tag is at the bottom of the page (it should be in <head>).
  • Tag manager fires the script on a later trigger than All Pages.
  • A third-party blocker (ad blocker, privacy extension) is blocking the request.

Check the Network tab to confirm the script is actually loading.

When to Contact Support

If you’ve worked through this guide and the issue persists, contact support with:

  • Your workspace ID (visible in Settings → Workspace).
  • The experiment ID.
  • The exact symptom (with a URL we can reproduce on if possible).
  • Browser version and DevTools console output if you have them.

Most issues are resolvable with the workspace + experiment ID alone.

Next Steps

Ready to start testing?

Install Split Test Pro and run your first experiment today.

Install on Shopify