Shopify Sidekick

This is a Shopify-only doc. Sidekick is the AI assistant built into your Shopify admin. Once Split Test Pro is installed, Sidekick can look up your experiments and read their results for you, so you can ask about a test without opening the app.

What Sidekick can answer

Ask in plain words — “A/B test”, “split test”, “test” and “experiment” all work. For example:

• “Which tests are running right now?”
• “Find my homepage hero experiment.”
• “How is the free-shipping banner test doing?”
• “Does the checkout button test have a winner yet?”
• “Which of my experiments is performing best?”

For “running”, “current”, “active” or “live” questions, Sidekick lists the experiments that are currently running. For “best performing” questions, it compares each experiment’s lift and probability to win.

The two tools

Split Test Pro gives Sidekick two tools. Sidekick picks them for you; you never call them by name.

search_experiments

Finds and lists experiments. It can match part of an experiment’s name and filter by status:

Status	What it means	In the app
created	A draft, not launched yet	Draft
started	Currently running	Running
paused	Paused	Paused
completed	Ended	Completed
archived	Archived	Archived
scheduled	Queued to launch automatically at a set time	Scheduled

It returns up to 25 experiments (10 unless Sidekick asks for more). Each one comes with its status, its primary metric, total sessions, lift, probability to win and the winning variant when there is one.

get_experiment_results

Reads the full results for one experiment, usually right after a search. For each variant it reports:

• Sessions and conversions, and the conversion rate.
• For a value metric such as revenue, the mean and total value.
• Lift against the control, with its 95% credible interval.
• Probability to win.

It also reports the headline numbers the app’s Results tab leads with — the best variant’s lift, its 95% credible interval and its probability to win — plus the winner, if one has been declared, and the experiment’s revenue and conversion totals.

How to read the numbers

Sidekick is instructed to describe the numbers the way Split Test Pro does:

• Probability to win is a Bayesian probability from 0 to 100% that a variant beats the others on a metric. It is not a p-value, and Sidekick won’t call it “statistical significance.” See Bayesian Stats Explained.
• Lift is the percentage change against the control. The credible interval is the 95% range the true lift most likely falls in.
• The primary metric decides the outcome. A winner is declared only when a variant reaches at least 95% probability to win on it. If there’s no winner yet, Sidekick says so.
• Sessions are unique visitor sessions, not pageviews.
• New data can take a few minutes to appear, so a test you launched a moment ago may show no sessions yet.

Limits

• Read-only. Sidekick can search experiments and report results. It can’t create, launch, pause, edit, archive or delete an experiment, and it won’t claim that it did. To change an experiment, open Split Test Pro.
• Result links open the app. When Sidekick links to an experiment, the link opens it in Split Test Pro on its Results tab.
• Your store only. Sidekick sees the experiments in the Split Test Pro workspace connected to the store you’re signed in to.

Availability

Sidekick tools are part of the Split Test Pro Shopify app, so they’re available on Shopify workspaces with the app installed — there’s nothing to switch on. HTML workspaces don’t have Sidekick.

Next Steps

• Read every number on the Results tab: Reading the Results Dashboard.
• Understand probability to win and credible intervals: Bayesian Stats Explained.
• See how Split Test Pro runs on your storefront: Shopify Theme App Extension.