Frequently Asked Questions

Quick answers to the questions that come up most often. Each links to a fuller doc when there’s more to say.

Will visitors see flicker?

No. Variant CSS is injected synchronously into <head> before the browser paints, so visitors never see the original styling for a moment before the variant appears. JS variants execute in <body> and may run a fraction of a second after CSS, but well-written JS variants don’t visibly disrupt the page.

The exception: if your variant changes element sizes, you may get layout shift (CLS) as the change applies. See CSS Variants for how to avoid that.

Does this slow my site down?

The script is small, and variant CSS is applied in a single injection step — typically 1–3ms.

For visitors not in any experiment on the current page, the script does a cookie check + API fetch + “no targeted experiments” no-op — usually under 50ms total, none of it blocking visible render.

If you’re running 10+ concurrent experiments and they all target the same page, the cumulative variant injection adds up — but for most setups (1–3 experiments per page), the impact is negligible.

Can I run two experiments on the same page?

Yes, with a caveat. Two experiments on the same page but different DOM regions (e.g., one on the header, one on the footer) work fine in parallel. Two experiments on the same DOM region create interaction effects that contaminate the data.

See Running Multiple Experiments for the full breakdown.

What happens if my JS variant throws an error?

The script doesn’t wrap variant JS in try/catch. If your code throws, it can break subsequent scripts on the page for visitors in that variant.

Always test JS variants thoroughly before launching. Use null-checks before manipulating DOM elements, and wrap risky third-party calls in try/catch. See JavaScript Variants for safety patterns.

Will Split Test Pro auto-stop my experiment?

Only in two cases you set up yourself:

• A scheduled end. In the Variants step, “When should it run?” → Schedule lets you pick an End time (or No end date). When the end time passes, a running test is completed automatically. A paused test’s scheduled end does not fire. While a test is still Scheduled, Reschedule and Cancel schedule in the ⋯ menu change or drop its dates. See Scheduling.
• Autopilot’s auto-stop (Pro and Agency, opt-in in the Testing Overview). It ends a running plan experiment once it has run at least 7 days, every variant has at least 300 sessions and 15 conversions, and a variant holds 95% probability to be best on two daily checks in a row. It also stops a test that runs too long without a winner (at least 28 days, at most 60). See Autopilot.

Otherwise ending a test is manual: Declare winner on the Results tab, or End test in the ⋯ menu. See Declaring a Winner.

Does the app email me when an experiment reaches significance?

No. There is no per-experiment significance email — check the Results tab. The emails the app does send are sign-in codes, team invitations, Test Plans updates (proposed plan changes and the weekly program digest), Autopilot digests and notices, rollout reminders and post-rollout regression notices, a notice when a scheduled launch fails, and monthly session-limit warnings.

Is there an audit log of experiment changes?

Yes — the Activity tab on each experiment page (Results | Setup | Activity) is a timeline of what happened to that test and who did it: launches, pauses and resumes, schedule changes, its end, archiving, rollouts and copies. It records status changes, not individual edits to a variant’s code. See Experiment Activity.

Can I export results as CSV?

Yes. Export CSV in the experiment’s ⋯ menu downloads a per-day table: a Date column, the variation, Visitors, then the measured columns (on Shopify: added to cart, began checkout, purchases and revenue; on HTML: the primary goal), one row per day for each variation. There is no public results API; the CSV is the export.

Can I segment results?

Yes. The filter bar above the Results tab narrows every number on the page: Devices · Visitors · Exposure · Advanced · Dates. Advanced adds rules on traffic source, browser, operating system, language, sessions, landing page and UTM campaign. There is no country or geo filter.

When another experiment ran on the same visitors, the Overlap banner offers an Overlap view that splits the results by the combination of variations each visitor saw. See Segmenting Results.

How is GDPR / privacy handled?

The HTML script respects Do Not Track and Global Privacy Control by default. There’s a public API (SplitTestPro.optIn() / SplitTestPro.optOut()) for explicit consent management, and a localStorage key (stp_consent) you can integrate with your cookie-consent banner.

On Shopify, the Web Pixel respects Shopify’s customer-privacy settings automatically.

See Privacy and Consent for full details.

What’s the difference between variants and experiments?

An experiment is the test itself — the hypothesis, the targeting, the metrics. A variant is one version within the experiment — the control (Variant A) or one of the treatments (B, C, D). Each variant has its own CSS, JS, or redirect URL.

A two-variant A/B test = one experiment with two variants. A four-variant A/B/C/D test = one experiment with four variants.

Can I test on different domains under one workspace?

No. A workspace is tied to a single domain (HTML) or shop (Shopify). For multiple sites, create multiple workspaces. See Workspaces.

What happens to my data if I cancel my subscription?

Your data is preserved. Cancellation stops billing and (after the current period ends) restricts access to the Results dashboard, but the underlying data — experiments, variants, conversion history — stays in the database. Re-subscribing restores full access.

On an HTML workspace, the owner can delete everything with Settings → Danger zone → Delete workspace. See Workspaces.

How long should I run an experiment?

At least 7 days minimum to capture full weekly traffic patterns, AND until probability to be best crosses 95% on the primary metric, AND until you have at least 300–500 sessions per variant. All three.

For low-traffic sites, 3–4 weeks is realistic. For high-traffic sites, the week minimum still applies — don’t shortcut weekly cycles. See Declaring a Winner.

What if my visitor is on a single-page app?

The script evaluates targeting and applies variants on the initial page load. Client-side route changes (pushState, popstate, hash changes) re-fire page-view goals and site-wide page views, but don’t re-run targeting. See Single-Page Apps for workarounds.

Can I edit a variant while the experiment is running?

Don’t. Editing a running variant contaminates your data — visitors who see the variant before and after the edit are mixed in the same bucket but seeing different things.

If you need to fix a variant, end the experiment, fix the variant, and start a fresh one. See Experiment Lifecycle.

What metrics can I track on Shopify vs HTML?

Shopify auto-tracks the full purchase funnel: product views, cart, checkout started, payment info submitted, purchase completed. Plus continuous metrics: revenue per session, average order value. All available as primary metrics.

HTML tracks custom events you fire from your site via SplitTestPro.trackConversion(eventKey). You define the goals; the script tracks them.

See Conversion Goals for setup.

Why isn’t Variant B at 100% probability if it’s clearly winning?

Bayesian probability reflects uncertainty in the data. A variant can be obviously ahead in raw conversion rate but still only at 91% probability if:

• Sample sizes are small (so the credible intervals are wide).
• The lift is modest enough that it’s possible (just unlikely) the true rates are reversed.

100% probability would require infinite samples. 95% is the convention for “confident enough to act.” See Bayesian Stats Explained.

What are credible intervals?

The range of plausible values for a parameter, with a stated probability. “Variant B’s conversion rate has a 95% credible interval of 4.2%–5.8%” means there’s a 95% chance the true rate is in that range, given the data observed.

Different from frequentist confidence intervals — and works the way most people intuitively expect confidence intervals to work. See Bayesian Stats Explained.

What’s the difference between custom events for tracking and event activation?

Custom events for tracking (SplitTestPro.trackConversion("event_name")) record that a visitor converted on a goal. Drives results data.

Event activation uses a custom event to defer when an experiment activates for a visitor. Drives when the variant starts showing.

You can use both at once — an experiment that activates on cta_click and tracks signup as its primary conversion goal. See Custom Events and Event Activation.

Does Split Test Pro work with my CMS / framework / page builder?

Anything that lets you add a <script> tag to <head> works:

• WordPress — header injection plugin or theme customizer.
• Webflow / Squarespace / Wix — custom code / header injection in site settings.
• Next.js / Nuxt / Astro — app/layout.tsx, _app.js, or global layout <head>.
• Shopify — install the app from the App Store; no script tag needed.
• Google Tag Manager — Custom HTML tag firing on All Pages.

See Installing on an HTML Site for the platform-specific instructions.

Can I A/B test something other than visual changes?

Yes. A few less-common patterns:

• Test a JavaScript behavior with a JS variant — change the form validation, the lazy-loading threshold, the third-party SDK initialized.
• Test a different page entirely with a redirect variant — useful for full-page redesigns.
• Test only after a specific interaction with event activation — useful when the change is in a step the visitor only sees after clicking.
• Test a server-side change with a server-side test: your backend evaluates feature flags whose values come from the variant, and records exposures and conversions through the API.

The pattern: you control the JavaScript that runs for each variant. Whatever you can express in JS, you can test.

Can I test server-side, or without the pixel?

Yes, on every paid plan. Create feature flags in Settings → Feature flags and a server API key in Settings → API keys, then set an experiment’s Where does this test run? to Server-side and give each variant its flag values. Your backend evaluates the flags for each user over a REST API (or with the Node SDK), records an exposure when the user experiences the variant, and records conversions — no pixel involved. Results use the same Bayesian engine as pixel tests.

You can also mix the two: a pixel-served experiment whose variants carry flag values is a hybrid test. See Server-Side Testing.

What isn’t supported (the honest list)?

Things merchants ask about that aren’t built:

• Shopify Plus checkout block variant editor.
• SPA targeting re-evaluation — targeting and variants bind on the initial page load, not on client-side route changes.
• Holdout / mutex groups for guaranteed experiment exclusion.
• SDKs for languages other than Node — other backends call the evaluation API directly.
• Multivariate experiments.

Cross-reference with what you need before committing to a particular workflow.

How do I contact support?

In the app, open Help & Docs and choose Email support: a short form (name, email, optional subject, message) that sends your message to the team, who reply by email. You can also write to support@splittestpro.com directly. Response time is one business day.

When you get in touch, include:

• Your site domain (HTML: Settings → Workspace; Shopify: your store’s .myshopify.com domain).
• The experiment ID, if relevant.
• The exact symptom and a URL we can reproduce on, if possible.
• Browser and OS.

Next Steps

• For full coverage of any topic above, follow the cross-links to the relevant doc.
• For methodology beyond the FAQ: Testing Methodology and Common Mistakes.
• For the underlying Bayesian model: Bayesian Stats Explained.
• Have Splitty build and launch a test for you: Splitty, the AI Test Builder.