Shopify Theme A/B Testing: 3 Ways Compared, One Verdict

The three ways to split-test a Shopify theme today

TL;DR Every route keeps the second theme unpublished and decides, per visitor, which one to serve. They differ in who decides and when: Shopify's servers (Rollouts), a snippet inside your theme (Shoplift), or a script you own (cheap apps, DIY).

The Help Center is blunt about the constraint: "only one theme can be published at a time." A theme A/B test is any mechanism that gets around that sentence honestly, by showing some visitors a second, unpublished theme and keeping them on it.

For years the free way to do that was Google Optimize, which Google's support page says has been "no longer available as of September 30, 2023." The gap closed on June 5, 2026, when Shopify's changelog announced that Rollouts lets you "run an A/B test between two entirely different themes or checkout setups to find the winner." That changed the default answer, and it dated most of what ranks above this page: a 2024 tool blog and a 2021 forum thread on duplicating your theme. Here are the three routes as they stand today, with the exact clicks.

Way 1: Shopify Rollouts (native, server-side, Grow plan or higher)

TL;DR Markets > Rollouts. Rollouts exist on Basic and up; the traffic split (an "experiment") needs Grow or higher. Shopify copies your theme, serves each visitor one version, reports the numbers, and gives no verdict.

Rollouts is Shopify's own splitter. The Help Center's requirements page gives the two sentences that matter: "Rollouts are available on the Basic plan or higher." and "Experiments are available to stores on the Grow plan or higher." A launch has "no change-level traffic split"; an experiment "lets you adjust the subset of eligible visitors that changes are displayed to." The subset is the test.

The admin path, from the create-a-rollout page, read today:

1. Go to Markets > Rollouts and click Create rollout. Name it.
2. Click Add changes to your store and choose Online store theme. Two options: Edit your main theme in the theme editor, where "any customizations apply to only that rollout," or replace your main theme with another. For a whole-theme test, pick replace and select your draft theme.
3. Click Changes will publish to all visitors and drag the slider. Anything below 100% makes the rollout an experiment; 50% is the clean two-theme split.
4. Click Select launch date and time and schedule it. Launch immediately is only offered at 100% reach, and the analytics page says: "If you choose to launch a rollout immediately, then no analytics are collected for the rollout."
5. Click Add end date. Leave the end action on the default Will roll back to 0% rather than Will be applied permanently. You can apply the winner by hand afterward, and "After you apply a rollout, you can't revert the changes."

The changelog explains the mechanism: "A copy of your published theme or config is automatically created so you can continue to edit your live theme/checkout independently from your experiment." The split happens on Shopify's side before the page is sent, so the first paint is already the assigned theme. No snippet, no body-hiding, no flash of the old theme.

What Rollouts reports, per the analytics page: gross sales, average order value, orders, total sales over time, sessions over time, conversion rate over time, bounce rate, reached checkout rate, add to cart rate, and new versus returning customers. What it does not report is a significance figure or a winner. You are the statistics engine, which is what the how long to run section is for.

Three hard limits from the requirements page: "You can't apply rollouts to vintage themes." "You can't change Liquid templates as part of a rollout." "Rollouts support only online store checkouts," so headless storefronts are out. Custom staff roles need the Markets > Rollouts permission plus Online store > Themes.

The Grow price is the one number this fetch could not read cleanly. shopify.com/pricing served my request the Canadian edition (Grow CA$132 a month, CA$99 on annual billing); Google's indexed snippet of the same page, returned September 6, 2026, shows "Grow $79/mo" in USD on annual billing. The plan name is what gates the feature, so check your own plan page. The full teardown of what Rollouts can and cannot test is in the complete Rollouts breakdown.

Way 2: Shoplift (the theme-test specialist app)

TL;DR Theme tests are on Shoplift's entry plan: $99 a month, or $888 a year, with a Bayesian verdict and a one-click Apply Winner. It runs from a snippet in your theme's head and pauses if you publish page-builder pages mid-test.

If you are on Basic, or you want software to tell you whether the gap between two themes is real, Shoplift is the app built for this exact job. Its Shopify App Store listing, read today, opens with "From $99/month. Free trial available." and carries a 4.9 rating from 135 reviews.

Core is $99 a month or $888 a year; the listing's "Starting at $74 based on monthly unique visitors" is the annual price for stores up to 50,000 visitors on shoplift.ai/pricing, and the monthly $99 covers up to 100,000. Core includes "Template, Theme, & URL testing," "Bayesian predictive analysis," "Mobile vs. desktop segmentation," and "One-click Apply Winner." Advanced is $399 a month ($3,588 a year, from $299) and adds price testing in beta, country segmentation, and GA4 exports; Pro is $999 a month ($8,388 a year, from $699) and adds a success manager. All three carry a 14-day trial and are "billed every 30 days."

How the split works, from Shoplift's docs: "When you run a theme test, Shoplift splits your traffic between two themes," a control (your live theme) and a variant that is "a separate, unpublished theme." Visitors are allocated on the first page they hit, and every later action is attributed to the test. The logic lives in a snippet the docs say to keep "in the head of your theme.liquid," with "a minimal 1-2 point reduction in the overall Lighthouse score" and no external network requests. The failure case is stated just as plainly: "All tests cease to collect visitor data until the script is restored." One more line from the same docs: "Creating and deploying page builder pages while a theme test is active will pause your active theme test," so finish PageFly or GemPages work before launch. For the head-to-head with the money-testing alternative, read Shoplift vs Intelligems; for cheaper options, Shoplift alternatives.

Way 3: the cheap apps, and the DIY duplicate theme

TL;DR Theme Scientist ($9.99) and Shogun AB Testing ($39) both list theme testing. Neither listing says whether the theme is decided server-side or swapped by a script. The DIY method is free and fragile.

The App Store has a long tail of testing apps at a tenth of Shoplift's price, and two of them rank above this page for the search that brought you here. Their listings are worth reading for what they leave out.

Theme Scientist A/B Testing lists Tester at $9.99 a month, Tester+ at $19.99, and Tester Pro at $29.99 (or $83.88, $167.88, and $251.88 a year), a 14-day trial, a 4.0 rating from 13 reviews, and a launch date of March 5, 2020. The pitch is "conduct AB testing on your Shopify theme effortlessly and without code." The listing does not say how visitors are assigned or whether the variant is rendered before or after the page loads.

Shogun AB Testing, ranked first on this SERP, lists Pro at $39 a month for 3 active tests, Advanced at $119 for 5, and Unlimited at $499, with a 14-day trial, a 4.2 rating from 44 reviews, and a launch date of March 21, 2025. You "create test variants for: Pages, sections, elements, PDPs, product prices, checkout and shipping within your theme using Shopify's native editor." It also does not say whether the split is server-side.

That silence is the thing to resolve before paying. The cheap way to test themes is a script that loads on the published theme, decides the arm, then redirects to or rewrites the variant, and hides the page until the swap is done so shoppers do not see the old theme first. DebugBear's write-up of these anti-flicker snippets found they set opacity: 0 on the body while the page loads, and in its measured example, overriding those styles improved Largest Contentful Paint "from 6 seconds to 2.7 seconds." The tools it names, Optimizely, Adobe Target, Google Optimize, VWO, and Mutiny, all publish pages admitting the speed cost. A slow variant tests your snippet, not your theme. Ask the vendor whether the theme is decided before the first byte or by a script after, and install only if the answer is before.

The DIY duplicate theme is what the 2021 Shopify Community thread in these results still recommends. The Help Center's steps: go to Online Store, find your theme, click the horizontal menu button, click Duplicate; the cap is 20 themes. Edit the copy, then write a script that assigns each new visitor to A or B, sets a long-lived cookie, and sends the B group through Shopify's theme preview mechanism. Push the arm into your analytics and onto the order as a cart attribute, because Shopify's reports do not segment by theme.

It works in a demo. In production the redirect runs after the page starts loading, so you have built the flicker yourself; caches remember pages rather than cookie assignments, so B visitors get A pages; cookies expire and never follow a shopper to a second device, and every crossover drags the result toward a tie; and if the arm does not survive to the order, you have two traffic groups and no revenue comparison. Reasonable as a scrappy read below Grow with a developer on hand. Not reasonable when the decision matters.

Two more names for completeness, both in the table below. Convert is a general, client-side platform (Growth $399 a month or $299 on annual billing, 100,000 tested users a month, "Split URL/Redirect Testing" and "Default Anti-flicker" in its feature list) built for teams running a wider program. Intelligems is the money-testing tool; its pricing page today shows Unlimited at $1,279 a month billed annually with a three-month minimum, and that only makes sense if price and shipping tests are the point (Intelligems pricing breakdown).

The decision table

Every price and gate below was read off the vendor's own page on September 6, 2026. Prices move; re-check the day you buy.

Route	Who decides the theme, and when	Plan or price	Verdict engine	Best for	Watch out
Shopify Rollouts	Shopify's servers, before the page is sent	No add-on fee; experiments need Grow or higher (Help Center)	None: sessions, conversion rate, AOV, gross sales, no significance, no winner	Grow+ stores that can build the variant and read numbers	Immediate launch collects no analytics; apply is irreversible; no vintage themes or Liquid edits
Shoplift	A snippet in your theme's head, on the first page hit	Core $99/mo or $888/yr (from $74/mo annually, up to 50k visitors); 14-day trial (App Store)	Bayesian, mobile vs desktop split, one-click Apply Winner	Basic-plan stores, or anyone who wants a called result	Page-builder publishes pause the test; remove the snippet and data stops
Cheap apps (Theme Scientist, Shogun AB Testing)	Not stated on either listing	$9.99 to $29.99/mo; $39 to $499/mo; 14-day trials (App Store)	Not described on the listings	Low-stakes reads where a wrong answer costs little	Ask whether the split is server-side; body-hiding scripts cost LCP (DebugBear)
DIY duplicate theme	Your cookie and redirect, after the page starts loading	Developer time; 20-theme cap (Help Center)	None; you wire analytics and significance	Below Grow, developer on hand, directional result	Flicker, cache misassignment, cookie loss, and the SEO rules are yours
Convert	Client-side, default anti-flicker snippet	Growth $399/mo or $299 annually, 100K tested users/mo (convert.com)	Mature frequentist engine	Teams running a cross-platform program	You bring the wiring; anti-flicker costs speed

Intelligems is left out because a theme test is not what its $1,279/mo annual Unlimited plan is for. The wider tool comparison is in the full Shopify A/B testing guide.

What breaks: theme updates, app embeds, checkout, and edits mid-test

TL;DR A theme test is a frozen pair. Theme updates land as a separate draft and reach neither arm. App embeds follow theme settings, so duplicates carry them and fresh themes do not. Rollouts cannot touch Liquid templates or vintage themes, and archiving one deletes its theme changes.

Most theme tests fail on mechanics rather than design. These are the mechanics, each traced to a Help Center or vendor page read today.

Theme updates

When your theme developer ships an update, Shopify does not touch the live theme. Per the updating-themes page, you click the notification and then Add to draft themes, and the update appears as a separate unpublished theme prefixed "Updated copy of." That draft receives "any customizations made to your theme using the theme editor," including theme settings and app embeds; code edits come along only if they do not conflict, and the admin says which with "Theme added: code edits successfully included" or "code edits could not be included."

For a running test the consequence is simple: neither arm gets the update, and publishing the updated copy mid-test replaces the control with a third theme. On Rollouts, the create page warns that unended rollouts affecting the same resource must be archived before you publish a new theme, and archiving "permanently deletes its theme changes." Park the update, finish the test, publish the winner, then take the update on top.

App embeds and app blocks

App embeds are toggled in the theme editor's App embeds panel and stored with the theme. The Help Center's rule is written for a theme change but applies to any variant that is not a copy: "If you opt to install apps for your theme, and later change your published theme, then you'll need to re-activate those apps in your new theme as they won't be active by default." A duplicate carries its theme settings, so its embeds come along; a purchased or freshly built theme starts with every embed off. A variant missing your reviews widget or consent banner is testing the absence of an app, not the design. Open App embeds in the variant and compare the toggles line by line before you schedule.

Checkout

Rollouts can carry a theme change, a "checkout and accounts configuration" change, or both, and "Rollouts support only online store checkouts." Keep the checkout leg out of a theme test: when both move you cannot tell which one moved the conversion rate. Test a checkout configuration as its own rollout after the theme test settles. Staff who touch that leg need "The View and edit checkout and customer accounts permission" as well.

Edits during the test

Every edit to either theme changes what is being compared and resets the meaning of everything collected so far. Rollouts shows this structurally: launch an experiment that replaces the theme while theme-edit rollouts are active and "the theme edit rollouts become hidden until the experiment ends." Shoplift shows it operationally: page-builder pages deployed mid-test pause the test. Freeze both arms and keep a list of the fixes you are itching to make; a long list is evidence you should have tested a smaller unit.

Liquid, vintage themes, headless, and the snippet

Rollouts will not run on a vintage theme and cannot change Liquid templates, so a variant that needs a template-level Liquid change has to be built as a full draft theme and used with the replace option. Headless storefronts are not supported by Rollouts at all; there, an app is the only route. And any app-based test lives on one line in theme.liquid: a developer tidying the head, a theme update that could not merge code edits, or a Git push from a stale branch can remove it silently, and Shoplift's docs say data collection stops until it is restored. Put that check on the pre-launch list and the weekly one.

How long to run a theme test

TL;DR Fix the sample size before launch, run whole weeks, do not stop early. At a 2% baseline, detecting a 10% relative lift takes roughly 78,400 visitors per theme by Lehr's approximation, which is about seven weeks at 100,000 sessions a month and about eight months at 20,000.

This kills more theme tests than bad design does, and it is arithmetic rather than opinion. Rollouts gives you numbers and no verdict, so on that route in particular, do this before you click Create rollout.

A standard approximation for a two-arm test at 80% power and 95% confidence is Lehr's rule: visitors per arm ≈ 16 × p × (1 − p) ÷ d², where p is your baseline conversion rate and d is the absolute lift you want to detect. At a 2% baseline, a 10% relative lift means d = 0.002, so 16 × 0.02 × 0.98 ÷ 0.000004 ≈ 78,400 per theme. The table is my arithmetic on that formula, not a vendor figure.

Detectable lift (relative)	Absolute lift (d)	Visitors per theme	Store at 20k sessions/mo	Store at 100k sessions/mo
5%	0.001	~313,600	~31 months	~6 months
10%	0.002	~78,400	~8 months	~7 weeks
20%	0.004	~19,600	~2 months	~12 days (run 2 whole weeks anyway)

Both themes need the per-theme figure, so total traffic is double the middle column. Higher baselines need less: at 3%, a 10% lift needs about 51,700 per theme. Revenue-per-visitor reads need more than a conversion read at the same confidence. Run your own numbers through the significance calculator.

A theme test gets one break a section test does not: every visitor counts, because the theme covers every page. And a full redesign is one of the few changes plausibly large enough to clear a 10% to 20% bar. The Reddit thread that outranks this page offers a rule of thumb: "A/B testing is only recommended if you have ~10,000 qualified users per month." At a 2% baseline, 10,000 a month clears the 20% row in about four months and never realistically clears the 10% row. Right direction, generous number.

Then there is peeking. Evan Miller worked the numbers on stopping a test the first time the dashboard shows significance and got a real false-positive rate of "26.1%, more than five times what you probably thought the significance level was." Keeping a true 5% after ten peeks needs a reported 1.0%. His fix: "Decide on a sample size in advance and wait until the experiment is over before you start believing the 'chance of beating original' figures that the A/B testing software gives you." On Rollouts, set the end date to the day the arithmetic says and stay out of the analytics tab until it arrives.

Two more calendar rules. Run whole weeks, because Tuesday and Saturday traffic do not buy the same things. And never span an event you would not want to generalise from: a sale, a launch, Black Friday. Randomisation protects the comparison between arms; it does nothing to make a BFCM result describe March. If the table put your store in the months column, do not start a split you will abandon at week nine. Test a smaller unit, or use the low-traffic methods in the main guide.

What to measure: revenue per visitor, not conversion rate

TL;DR A theme moves conversion rate and order size at the same time. Score the test on revenue per visitor, which catches a theme that converts more people into smaller baskets.

Rollouts reports conversion rate and average order value as separate lines and never multiplies them. Do it yourself, because a whole-theme change touches discovery, cart, and upsell surfaces at once, and the two numbers routinely move in opposite directions.

Worked example. Theme A converts at 2.0% with a $60 average order: RPV $1.20. Theme B converts at 2.2% but buries your bundles and the average order slips to $53: RPV about $1.17. B wins the conversion column and loses money on every thousand visitors, and a dashboard sorted by conversion rate never flags it. The main guide's metric step makes the longer case.

Two cautions. A theme test runs for weeks across the whole store, so one $2,000 wholesale order in one arm can fake a lift: inspect the top orders in each arm before believing a result. And revenue varies far more per visitor than a yes-or-no conversion, so an RPV verdict needs more traffic than the table shows. Everything else in the report is diagnosis. Use it to explain why an arm won, never to declare that it did.

The SEO rules, from Google's own page

TL;DR No cloaking, canonical from any variant URL to the original, 302 not 301, and remove the test as soon as it ends. Rollouts serves both themes on the same URLs, which avoids most of this by construction.

Merchants make two opposite mistakes: refusing to test out of ranking fear, and building DIY redirects that drift into cloaking. Google's website-testing page, read today, sets four short rules.

• No cloaking. "Don't show one set of URLs to Googlebot, and a different set to humans. This is called cloaking, and is against our spam policies."
• Canonical, not noindex. "Use the rel="canonical" link attribute on all of your alternate URLs to indicate that the original URL is the preferred version."
• 302, not 301. "Use a 302 (temporary) redirect, not a 301 (permanent) redirect."
• End it promptly. "Remove all elements of the test as soon as possible," because "if we discover a site running an experiment for an unnecessarily long time, we may interpret this as an attempt to deceive search engines."

Rollouts and Shoplift serve the variant on your existing URLs, so there is nothing to canonicalise and no redirect to get wrong. The DIY method, which sends visitors through a preview URL, is the route where every rule is yours to enforce.

The verdict

If you are on Grow or higher, use Rollouts and do the statistics yourself. If you are on Basic, or you want the tool to call the result, pay Shoplift $99. Do not pay $9.99 for a splitter whose listing will not tell you when it decides the theme.

The honest caveat is about the unit of test rather than the tool. A whole-theme test answers one question, keep this theme or switch, and answers it slowly. When the new theme wins you learn the package won and nothing about which change did the work; when it loses, one broken template can drag down ten good changes behind a single blended number. Run one when the decision genuinely is switch or stay: a purchased theme, a redesign already committed on brand grounds, a speed rebuild you want to prove also sells. For anything narrower, test the section or template. The main Shopify A/B testing guide ranks what to test first.

The playbook in nine lines, whichever route you take:

1. Write the decision down. "If B beats A on RPV at 95% confidence, we publish B."
2. Confirm the unit. Switch-or-stay: whole theme. Anything else: section or template.
3. Do the sample-size arithmetic. If it says more than two months, change methods now.
4. Pick the route from the decision table; on a cheap app, get the server-side answer in writing.
5. Build the variant, then audit it: App embeds line by line, mobile QA, comparable load time.
6. Freeze both themes. Park theme updates, page-builder publishes, and code pushes.
7. Schedule the launch. On Rollouts, never Launch immediately.
8. Run whole weeks to the end date with acquisition steady, dashboard closed.
9. Read RPV, inspect top orders, ship, clean up. Apply the winner by hand, take the parked update, and log the result even when it is "no difference."

Disclosure. We are building StorePilot, a CRO tool for Shopify that works at the section level rather than the whole-theme level. Nothing in this article's comparison changes if you never install it; the three routes above are the three routes.

Questions merchants keep asking

Can you A/B test a theme on Shopify?

Yes, three ways. Shopify's own Rollouts feature splits visitors between your main theme and a copy from Markets > Rollouts; the split needs the Grow plan or higher (Shopify Help Center, read September 6, 2026). Shoplift runs theme tests from $99 a month per its App Store listing. Cheaper apps such as Theme Scientist ($9.99) and Shogun AB Testing ($39) list theme testing too, and a duplicate-theme split can be wired by hand.

Does Shopify have built-in A/B testing for themes?

Yes, since the June 5, 2026 changelog post that added A/B testing to Rollouts. An experiment rollout shows a treatment theme to a percentage of visitors and the control to the rest, then reports sessions, conversion rate, average order value and gross sales for each. It does not compute significance or name a winner; you judge the result yourself.

Which Shopify plan do you need to A/B test a theme with Rollouts?

Grow or higher. The Help Center's requirements page says rollouts are available on the Basic plan or higher and experiments on the Grow plan or higher. Basic can schedule and publish a theme change but cannot split traffic. Google's indexed snippet of shopify.com/pricing lists Grow at $79 a month on annual billing as of September 6, 2026.

How do I set up a theme A/B test in Shopify Rollouts?

Markets > Rollouts > Create rollout. Name it, click Add changes to your store, choose Online store theme, then edit the main theme in the theme editor or replace it with another theme. Click Changes will publish to all visitors and drag the slider below 100%, which makes it an experiment. Set a launch date and time, add an end date, and leave the end action on Will roll back to 0%. Do not pick Launch immediately: the Help Center says no analytics are collected for a rollout launched that way.

Is Shopify theme A/B testing free?

Rollouts carries no add-on fee, but experiments are gated to the Grow plan or higher. Apps are not free: Theme Scientist starts at $9.99 a month, Shogun AB Testing at $39, and Shoplift at $99, all with 14-day trials on their App Store listings. A DIY duplicate-theme split costs developer time instead.

Can you run two themes at the same time on Shopify?

Not as two published themes; the Help Center states only one theme can be published at a time. Every testing route keeps the second theme unpublished and routes a share of visitors to it: Rollouts on Shopify's side, Shoplift through a snippet in your theme, a DIY split through a cookie and the preview mechanism.

How long should a Shopify theme A/B test run?

Until it reaches the sample size you fixed before launch, in whole weeks. By Lehr's approximation at a 2% baseline conversion rate, a 10% relative lift needs roughly 78,400 visitors per theme: about seven weeks at 100,000 monthly sessions, about eight months at 20,000. Stopping the moment one theme pulls ahead inflates false positives to about 26% (Evan Miller).

Does A/B testing a theme hurt Shopify SEO?

Not if you follow Google's website-testing rules: no cloaking, rel=canonical from any variant URL to the original, 302 rather than 301 redirects, and removing the test as soon as it concludes. Google warns that a test left running for an unnecessarily long time may be read as an attempt to deceive search engines. Rollouts serves the variant on the same URLs, which sidesteps most of this.

What happens to a theme test when a theme update comes out?

Nothing automatic, which is the problem. Shopify adds the update as a separate draft named Updated copy of, carrying your theme editor customizations and app embeds. Neither theme in a running test receives it. Finish the test, publish the winner, then take the update.

Do app embeds carry over to the variant theme?

It depends how the variant was made. A duplicated theme keeps its theme settings, where app embed toggles live, so a duplicate carries them. A different theme does not: the Help Center says if you change your published theme you need to re-activate those apps in your new theme as they will not be active by default. Check the App embeds panel in the variant before launch.