Shopify SimGym: how accurate are the AI shoppers, really?

Shopify built a machine that sends hundreds of AI shoppers through your storefront and tells you which theme they preferred. Then Shopify's own researchers published, twice, how often that machine is right. The marketing and the measurement are not telling the same story, and the gap between them is where merchants are going to lose money.

This is not a takedown. SimGym is a genuinely impressive piece of engineering and the research behind it is more honest than most vendor claims you will read this year. Shopify published its confidence intervals, published its failure modes, and published the ablations showing what happens when the system is degraded. That is unusually good conduct.

The problem is that none of it reaches the merchant. The App Store listing says SimGym lets you "try new ideas without risk" and promises you will "gain the confidence of high traffic brands before launch." The Help Center's entire caveat is one sentence: "Simulations use AI shoppers that mimic real buyer behavior, but results might differ from actual buyer behavior." Meanwhile the paper says the correlation between what the robots predict and what humans actually do sits somewhere between 0.32 and 0.72, and that is on the friendliest metric available.

This page reads the primary sources and states what they support. Every number below comes from a document we fetched and quote, with the URL. Where the sources disagree with each other, we say so rather than picking the flattering one.

What SimGym actually is

TL;DR A first-party Shopify app that runs AI agents through your live storefront in real browsers, compares your active theme against a draft, and reports which one the agents added to cart from more often. Announced December 10, 2025. Open to all eligible merchants since March 11, 2026. Still in research preview.

Shopify announced SimGym in the Winter '26 Edition on December 10, 2025. The announcement's own words: "For those who want to collect early signal on their online stores, we are announcing the SimGym app in Research Preview." It describes the method as learning "from real commerce sessions across our platform" to "simulate real customer personas who can approximate what shoppers might do and feel as they land on your storefront."

Note the verb in that sentence. Approximate. It is the most accurate word Shopify has used about this product anywhere, and it appears in the launch announcement rather than in the app.

On March 11, 2026 the changelog announced that "SimGym is now available to all eligible merchants with no waitlist required." The Help Center defines it as "a Shopify app in AI Research Preview that uses AI-powered shoppers to simulate buyer behavior on your online store." As of August 11, 2026 the App Store listing still carries the line "Currently in AI Research Preview," so this remains a preview product roughly eight months after launch.

What happens when you run one

You pick your active theme and a draft theme. SimGym generates a population of shopper personas grounded in your store's own clickstream data, then runs them through both storefronts in real cloud browsers. Each agent has a goal ("you are looking for running shoes, you are ready to purchase"), a behavioural profile, and a memory of what it has already seen. It clicks, searches, browses, and either adds something to cart or does not. You get back a comparison of add-to-cart rates plus a written report on navigation and friction.

The engineering post is worth reading for scale alone. Shopify runs "up to 2,000 concurrent Chromium sessions" through Browserbase, with each session looping "10-13 times" between page state and model action, consuming "3,000 to 15,000 tokens depending on storefront complexity" per step. The inference side is "48 dedicated NVIDIA B200 GPUs" running a fixed allocation with no autoscaling. As of that post, the platform could "spawn 400,000 shopping sessions a day."

Two papers, three accuracy numbers, and why everyone quotes the wrong one

TL;DR Shopify published two SimGym papers, four months apart, on different sample sizes. Between them they contain three plausible-looking accuracy figures. They are not interchangeable, and the difference matters.

If you go looking for how accurate SimGym is, you will find people citing 69%, 73% or 77%. All three numbers are real, all three come from Shopify's own research, and all three measure different things. Here is the disambiguation, which as far as we can tell has not been published anywhere else.

The middle row is the trap. Paper A's Section 4.1 says sign alignment "rises from ~51% at 50 agents to ~73% at 700 agents" and that Pearson correlation "stabilizes around 0.65 after approximately 300 agents." Read quickly, that looks like a better accuracy result than the paper's own headline. It is not a comparable measurement at all. That analysis resamples from two runs of the same system to work out how many agents you need before the answer stops wobbling. It tells you about the simulator's reproducibility, not about its agreement with human beings.

The figure to quote is 77% alignment with 0.55 correlation, from the May 2026 paper. It is the most recent and rests on the largest ground-truth set.

The trend inside the two papers is the interesting part

Between February and May, Shopify grew the ground-truth set from 20 shops to 50, added vision to the agents, and republished. Directional alignment went up. Magnitude correlation went down.

An improved system scoring lower on magnitude while scoring higher on direction, on a sample two and a half times larger, is the ordinary signature of small-sample optimism getting corrected. The 20-shop estimate of 0.64 was flattering. With more shops the magnitude estimate settled lower even as the upgraded agents got better at calling which way the needle moved.

That pattern is a warning about the whole category. Anyone quoting a synthetic-shopper accuracy figure from a small validation set is quoting a number that will probably fall when the set grows. Shopify deserves credit for publishing the correction rather than burying it.

One more thing worth knowing about Paper A: it carries the line "We highlight that this is not the final version of our paper and more analysis will be added per revision." It was explicitly provisional. Paper B is the grown-up version.

What the confidence intervals actually say

TL;DR The 0.55 correlation has a 95% confidence interval of 0.32 to 0.72. The 77% alignment has an interval of 66% to 87%. Shopify published both. The intervals are wide because 50 shops is a small sample, and they are the honest answer to "how much can I trust this?"

Paper B is the first to publish uncertainty, and it did the work properly: a percentile bootstrap over 10,000 resamples of the 50 shops, with each shop contributing one paired observation averaged across two trials. Here is the full table.

Take the best row. A correlation of 0.55 means the simulator explains about 30% of the variance in the real add-to-cart shift. That is a real signal and it is not nothing. But the interval runs from 0.32 to 0.72, so the same evidence is consistent with the simulator explaining as little as 10% of the variance or as much as 52%. Those two worlds call for completely different behaviour from a merchant.

Now look at the bottom row, because it is the one that should worry you most. The GPT-OSS configuration's alignment interval is 47% to 71%. The lower bound is below 50%. On the evidence published, that configuration cannot be distinguished from a coin flip at 95% confidence. Hold that thought for two sections.

The third metric Shopify reported, and why it is the most honest one

Paper A carries a measure that Paper B dropped and that nobody quotes: alignment probability. Rather than scoring each shop as simply right or wrong, it assigns a value between 0 and 1 representing the Bayesian posterior probability that the agents' direction agrees with the humans', then averages. It measures not just whether the simulator was right but whether it was confident.

The paper's interpretation of the bottom row is unusually candid: an alignment probability "barely above pure uncertainty, confirms that correct predictions reflect luck rather than systematic response to UI changes."

That sentence is the most useful thing in either paper, because it names the failure mode that raw accuracy hides. A degraded system that guesses at 52% looks almost respectable next to 69% if you only read the headline. The probability metric exposes that one of those systems is reasoning and the other is flipping coins. When you read a SimGym report, the question you cannot ask the interface is which of those two you just paid for.

It was validated on add-to-cart, not on sales

TL;DR Both papers measure one outcome: add-to-cart rate. Not orders. Not revenue. Not revenue per visitor. Every accuracy claim about SimGym is a claim about add-to-cart and nothing else.

Paper B is explicit: "We use A2C rate as the primary outcome, as it directly reflects purchase intent and corresponds to a key optimization target for merchants." Paper A says the same. Search either document for revenue or orders as a validated outcome and you will not find one.

This is a defensible research choice. Add-to-cart happens far more often than purchase, so you can measure a shift in it with far less traffic, and the agents never complete a checkout anyway. But it means the honest description of SimGym is narrow: it is a system that predicts which way your add-to-cart rate will move, roughly three times in four. Anything beyond that is inference, not measurement.

How the ground truth was built, and why it is better than you would expect

The validation design deserves respect. Shopify did not compare agents against a survey or a benchmark. It found real stores that had genuinely switched themes, measured what real customers did before and after, and checked whether the agents predicted the same direction.

The confounder handling is the good part. Paper A explains that a simple before-and-after comparison is not enough because "seasonal effects, promotional periods, new store ramp-up, or pricing changes could also affect the difference." So the team filtered out shops with promotions, seasonality, merchandising changes, ramp-up periods, and pricing or assortment shifts, then validated that filtering "using double machine learning to confirm consistent treatment effect estimates." An LLM evaluator characterised each theme change from screenshots and DOM so trivial restyles could be excluded.

That is a more careful causal design than most commercial A/B testing tools apply to their own case studies.

The cohort choice you should know about

Paper B does not report on all traffic. It reports on the "skimmers" cohort, which it describes as the largest non-bouncer traffic segment, accounting for 64.5% of engaged sessions and showing roughly a 9.5% session-level add-to-cart rate.

The paper checks this choice rather than assuming it, which is good practice. Across the 50 shops, add-to-cart shifts measured on skimmers correlate at r ~ 0.75 with whole-shop traffic, and at r ~ 0.89 with whole-shop traffic once immediate bouncers are excluded. So the cohort is a fair proxy for engaged shoppers.

It is still worth naming plainly: the 77% figure describes performance on a filtered segment of a filtered set of shops, chosen for having clean, non-trivial, measurable theme changes. Your store was not filtered for any of that.

From simulated carts to real money: the attenuation nobody does

TL;DR Chain SimGym's correlation with add-to-cart (0.55) against the published correlation between add-to-cart and actual orders (about 0.50) and the implied relationship between what SimGym predicts and what you earn is roughly 0.27. That is under 8% of the variance in sales outcomes.

Here is the step that turns an interesting research result into a business decision, and it requires one outside source.

Add-to-cart is not money. The question of how well it predicts money has been studied. Jakub Linowski at GoodUI ran the analysis across real A/B tests and published it on November 30, 2023. Across 44 GoodUI experiments, the correlation between adds-to-cart and sales was R = 0.4983. A separate Conversion.com dataset of roughly 200 experiments put it at R = 0.4349. Widening to any progression metric across 119 tests gave R = 0.5085, and checkout visits against orders across 533 experiments gave a stronger R = 0.6085.

So add-to-cart explains roughly a quarter of the variance in whether orders move. This is well established and it is the reason serious CRO practice does not celebrate an add-to-cart lift.

Now chain the two links.

Run the same arithmetic across the published confidence interval and the implied relationship to orders spans roughly 0.16 to 0.36, which is between about 2.5% and 13% of the variance in sales outcomes.

Even read generously, the conclusion is the same one from either end of the interval. A simulator correlated 0.55 with a metric that is itself correlated 0.50 with sales is a screening instrument. It is not a decision instrument. Screening instruments are genuinely valuable: a cheap test that reliably eliminates bad candidates saves you the expensive test. But you do not act on a screen. You confirm it.

Which model runs your simulation, and why it might not be the one in the headline

TL;DR The 77% headline comes from a Gemini 3 Flash configuration with vision. Shopify's engineering post documents production running openai/gpt-oss-120b, text only. In the paper, that configuration scores 59% with a lower confidence bound of 47%. Shopify has not published which model serves the app today.

This is the finding that changed how we read everything else, and it needs care, because it depends on two documents published three months apart.

Shopify's engineering post of February 27, 2026 publishes the production configuration verbatim:

Our current gear: 48 dedicated NVIDIA B200 GPUs on CentML. Fixed allocation, no autoscaling. model: openai/gpt-oss-120b

The same post describes agent perception as "a representation tree of the current page," which is the accessibility-tree approach Paper A used. The words vision, visual and screenshot do not appear anywhere in that post. So as of late February 2026, production ran an open-source model, text only.

Paper B, published in May, tested exactly that pairing and reported it as the weakest configuration of the three: 59% alignment with a 95% interval of 47% to 71%, and 0.41 correlation with an interval of 0.15 to 0.62. The paper's own framing is diplomatic, noting that "GPT-OSS underperforms the Gemini 3 Flash configurations" while still "producing meaningful simulation signals." The paper also confirms the model is served internally, "on an internal cluster with 40 NVIDIA B200 GPUs," which lines up with the engineering post's description of Shopify's own hardware.

So the headline number and the documented production stack are not obviously the same system.

If you use SimGym, this is a reasonable thing to ask Shopify support directly. The difference between the top and bottom rows of that table is the difference between a useful screen and a coin flip with a written report attached.

What makes it work, and what breaks it

TL;DR Two components carry almost all the predictive power: personas grounded in your own clickstream, and session memory. Remove either and the system collapses to chance. This is useful to know because it tells you exactly which stores it will fail on.

Both papers run ablations, stripping out one component at a time. The results are unusually clear-cut.

Strip the persona and correlation falls to 0.02. Strip the memory and it falls to exactly 0.00. Those are not degraded systems, they are noise generators. The "product only" configuration even goes mildly negative at -0.14, though its interval comfortably spans zero.

Why memory matters so much

Paper B attributes the memory collapse to navigation incoherence: without memory, "70.7% of diverged agents become stuck in navigation loops." Paper A measured the same effect from the other direction. With memory, agents reached their goal 90% of the time and timed out 9.59% of the time. Without it, goal completion fell to 45% and timeouts rose to 54.93%, which Paper A calls a 5.7x increase.

An agent stuck in a loop never reaches the part of the journey where your theme design matters. It cannot tell you anything about your product page because it never got there.

Why persona grounding matters, and why you cannot fix it

Paper A tested a middle option that Paper B did not: a "generic persona" built by pooling behaviour from similar donor shops rather than extracting from the target store. It landed between the extremes at 62% alignment and 0.27 correlation, and the reason it underperformed is the single most practically useful finding in either paper.

Generic-persona agents showed the highest "Product Not Found" rate of any configuration at 43.56%, against 36.36% for properly grounded SimGym. Paper A's explanation: "the issue is not how agents search but who is searching." Donor-derived personas go looking for products that a similar store would stock but yours does not.

Keep that mechanism in mind for the merchant reviews section, because it predicts exactly what merchants complain about.

Direction and magnitude are separate skills, and Shopify accidentally proved it

Buried in Paper A's Figure 4 is the strongest single justification for the "read the direction, ignore the number" rule this article keeps repeating.

The figure plots agent-predicted against human-observed add-to-cart shifts for each configuration and fits a line. Full SimGym produces a slope of 0.57 with 15 of 20 shops in the correct quadrant. The generic-persona configuration gets exactly the same 15 of 20 shops directionally right, but with a slope of 0.18.

Two configurations, identical directional accuracy, and a threefold difference in how well they track the size of the change. Paper A's own gloss: those agents "predict the direction of behavioral shifts but not their magnitude."

Direction and magnitude are not one capability that degrades together. They are two capabilities, and the harder one is magnitude. Since magnitude is the part you would need to forecast revenue or size a lift, and direction is the part that is cheap to get right, the tool's trustworthy output is precisely the least commercially exciting one.

One caveat that Paper A volunteers and we will repeat: the scatter plots "display the best-performing simulation for each configuration," while the tables report averages across two runs. The slopes above therefore come from a favourable run. That does not undermine the comparison, since all four configurations get the same treatment, but the slope values themselves are optimistic.

The failure taxonomy Shopify published

Paper A breaks down what the agents were doing when their behaviour diverged between theme versions. These proportions are computed over the subset of agents that behaved differently across the two themes, which the paper puts at roughly 10 to 20% of the 1,200 agents per method.

Paper A argues, reasonably, that "no add-to-cart decision" is the productive category: an agent that found a product and chose not to buy is responding to presentation, trust signals and layout, which is exactly what a theme test is about.

But look at the second row. Over a third of diverging agents in the properly configured system never found a product at all. Shopify frames this as background noise. Merchants experience it as the product being broken, and they are not entirely wrong, because an agent that cannot find your catalogue is not evaluating your theme.

How many AI shoppers is enough?

TL;DR Both papers settle on 600 agents per shop. Accuracy climbs steeply to around 300 agents and then flattens. Below about 100 agents the result is close to worthless.

Both papers ran the same analysis: resample from repeated runs, see how many agents you need before the answer stops moving.

Paper A's reasoning for 600 is candid: "slightly above the plateau threshold to provide margin for session-level failures while maintaining computational feasibility." Paper B's is the same, adding "buffer for finite-sample variability and session-level failures."

Two practical consequences. First, sample size governs synthetic experiments exactly as it governs real ones, which should be reassuring and slightly deflating at once: the sample-size problem does not go away because the shoppers are fake. Second, Shopify does not publish how many agents a single merchant credit buys. If a credit buys substantially fewer than 600 agents, the published accuracy figures do not describe your run. That is the second question worth asking support.

What merchants actually report

TL;DR 2.7 out of 5 across 34 reviews, with 1 star the most common rating at 29%. The complaints are strikingly consistent, and they map almost exactly onto the failure modes Shopify measured in its own papers.

We read all 34 public App Store reviews on August 11, 2026. The aggregate rating in the listing's structured data is 2.7 with a count of 34. The distribution is unusual: it is not mediocre, it is polarised.

Reviews are a biased sample and we are not going to pretend otherwise. People with a bad experience write more often, and a research-preview product attracts people who expected a finished one. What makes these worth reading is not the score, it is that the complaints cluster into the same categories Shopify quantified in its own research.

Complaint 1: the agents shop for things you do not sell

This is the most common substantive complaint, and it is Table 6's "Product Not Found" at 36.36% showing up in the wild.

A US home goods merchant, June 15, 2026: "Did not understand my brand at all, searched for things we do not even sell, and insisted that our search was broken."

A US merchant, June 27, 2026: "I had several searching for products I don't carry, and the recommendations were entirely unhelpful for providing alternative products. This was basically a waste of a credit since I can't provide information regarding my average shopper and ideal shopper."

That second review identifies the mechanism precisely. The persona is derived from clickstream and the merchant cannot correct it. An Australian university store, April 13, 2026, hits the same wall: "our market is 95% University students who utilise this site for university merchandise, sport programs, club memberships and events and course equipment. Adding data to help fine tune the bots personas would significantly increase the effectiveness of the app."

Shopify's own ablation shows why this is not a minor annoyance. Persona grounding is the component carrying the predictive validity. When the grounding misfires on an unusual catalogue, the paper's own numbers say you are somewhere on the road from 0.55 down toward 0.02.

Complaint 2: confident false positives

Several merchants report SimGym declaring things broken that are not.

The June 15 reviewer again: "claimed our Checkout button is broken, when it is clearly not. I can go on BrowserStack and check it across multiple devices and browsers and clearly see that the Checkout button works."

A Japanese merchant, June 30, 2026, reports the same class of error, translated: the report said footer links did not navigate when multiple users clicked them, but on checking, the links went to the correct destination without any problem.

A German merchant, July 19, 2026, gives the most diagnostically useful version. Translated and summarised: SimGym appears to test against a stale snapshot rather than the current live state, so a second run re-reported problems already fixed after the first. Findings describe symptoms without causes, so each one has to be researched manually, often taking longer than the fix. Most seriously, the reviewer reports that SimGym does not distinguish real bugs from test-infrastructure artefacts, and that several reported errors turned out to be session or cookie contamination between test runs rather than actual site problems.

That last point is the sharpest criticism anyone has made of this product publicly, because it describes the tool misattributing its own harness's failures to the merchant's store. It is also entirely consistent with an architecture that runs many concurrent browser sessions against one live storefront.

Complaint 3: it does not know it is a screening tool

The most valuable review for our purposes comes from a Cyprus merchant on January 24, 2026, who did the comparison this entire article is about:

The main issue is that we have noticed very inaccurate results when comparing to our current active AB tests. [...] For any store getting enough orders to run your own AB tests, we would recommend doing that instead for accurate data.

That merchant arrived at the correct conclusion empirically. If you have the traffic to run a real test, run the real test. The simulator's value is highest exactly where real testing is hardest.

What the positive reviews say

The favourable reviews are consistent too, and they describe screening rather than deciding. A US merchant, June 28, 2026: "Very useful for identifying credible solutions to website flow, product analysis. Overall, very pleased with the problem-solving solutions provided. All suggestions have been valid so far." A UK merchant during the preview period: "Pretty cool, gave some amazing insights, the different personas are useful, so you can tailor you UI and experience to different customers."

Note what those merchants got value from: finding friction and generating ideas. Neither describes using it to decide a winner. The satisfied users are, in effect, already using it correctly.

The recurring operational gripes

Three more themes appear often enough to mention. There is no export: at least three reviewers ask for PDF or data export to share with developers. Reports come back in English regardless of store language, which two European merchants call a reason not to trust the results for their market. And the credit model is opaque, with the June 27 reviewer unable to work out "how credits are distributed (monthly?) or if you can buy more ad-hoc."

The analytics contamination problem nobody is discussing

TL;DR The agents drive your real storefront in real browsers, so they can fire the same tracking events a human would. One merchant reports SimGym creating anomalies in Google Analytics including add-to-cart and begin-checkout events. Shopify's SimGym documentation does not mention this.

This is our beat, so we are going to spend a moment on it.

A UK merchant, March 19, 2026:

Creates an anomaly in your traffic in google analytics, including conversion tracking like add to carts and beginning checkout. Very frustrating that this distorts the analytics, this should be made clear.

Take that seriously, because the architecture predicts it. The engineering post describes real Chromium sessions driving live storefronts, up to 2,000 at once, with the platform capable of 400,000 shopping sessions a day. A real browser loading your real theme will execute your real analytics tags unless something specifically prevents it. We have not been able to verify what filtering, if any, Shopify applies, and the SimGym Help Center page says nothing about analytics.

We want to be careful here: this is one merchant's report plus an architectural inference, not something we have measured ourselves or found documented. But the consequences if it is right are worth planning around regardless, because they are asymmetric. Contaminated analytics are expensive and the mitigation is nearly free.

Why this is worse than it sounds

Synthetic add-to-cart events are not randomly distributed noise. They arrive in a concentrated burst, on a specific day, aimed at whichever pages you chose to simulate, and they add cart events without ever adding an order. That has three consequences.

Your conversion rate drops artificially. Sessions and add-to-carts rise, orders do not. Any funnel report covering that window shows a phantom decline at exactly the stage you were investigating.

A concurrent real experiment gets polluted. If you are running a genuine test through Shopify Rollouts or a testing app while a simulation runs, synthetic sessions land in whichever variant the splitter assigns them to. They are unlikely to split evenly, they never purchase, and they will bias the result in a direction you cannot recover after the fact.

Attribution and audiences absorb it. Any downstream system reading cart events, including remarketing audiences and lookalike seeds, ingests behaviour from shoppers who do not exist.

That last item matters more than anything we can tell you. We are reporting a merchant's finding and an architectural inference. You can measure it directly on your own store in an afternoon, and then you will know rather than infer.

Eligibility, privacy and what it costs

TL;DR Liquid storefronts only, Shopify Network Intelligence must be on, and the store cannot be password protected. The SNI requirement carries real privacy-policy obligations that the SimGym listing does not mention.

The three eligibility requirements

From the Help Center, verbatim: your store uses a Liquid storefront (Hydrogen and headless storefronts are not supported), your store has Shopify Network Intelligence activated, and your store is not password protected.

Each excludes a real group. Headless merchants are out entirely. The password requirement means you cannot simulate a store before you launch it, which is a strange gap given that pre-launch validation is the app's pitch.

The Shopify Network Intelligence requirement is not a checkbox

This is the eligibility item worth reading twice. Shopify's SNI requirements page describes it as Shopify using "your customer data together with customer data from other merchants to generate insights that power advanced features, known as Enhanced Services."

Turning SNI on creates obligations. Per that page, merchants must include a link to Shopify's Consumer Privacy Policy in their own privacy policy and post it prominently, inform customers that Shopify processes their data from interactions across other merchants, and explain that information will be shared with Shopify and third parties that may be located in other countries. In certain US states you must provide opt-out rights for sharing or targeted advertising. In the EEA, UK and Switzerland you must obtain consent and provide a way to withdraw it.

So the honest cost of a SimGym trial for an EU merchant is not one credit. It is one credit plus a privacy policy review. Neither the App Store listing nor the SimGym Help Center page mentions this consequence, and it is the sort of thing that should be on the install screen.

What it costs

The listing says "Free to install. Additional charges may apply," with "Charges per simulation run. More details available in the app." The Help Center adds that "Each simulation you run costs one credit. During the research preview, you might be allocated free credits."

Shopify does not publish a credit price publicly. The engineering post says "the cost per merchant run is in single digits and falling," which is a statement about Shopify's own compute cost rather than the merchant price, and it is the closest thing to a figure that exists in public.

The screen-then-confirm playbook

TL;DR Use SimGym where being wrong is cheap and being fast is valuable: eliminating weak candidates before they consume real traffic. Then confirm the survivor properly. The sequencing is the whole value.

Everything above points at one conclusion. SimGym is a filter, and filters are judged on a different standard than decision tools. A filter is allowed to be wrong sometimes, as long as it is cheap and it fails in the right direction.

Here is how to sequence it.

Step 1: generate more candidates than you can afford to test

The constraint on most stores is not ideas, it is traffic. If you can only run four honest tests a year, the cost of testing a bad idea is enormous, because it is a quarter of your annual learning budget. That is the situation a cheap screen improves.

So start by deliberately producing more variants than you could ever test properly. Four product-page layouts, three navigation structures, two collection-grid densities.

Step 2: screen them, and only read the direction

Run the simulation. Then read the output with discipline: note the direction, ignore the magnitude entirely.

That is not a stylistic preference, it is what the data supports. Directional alignment is 77% with a floor of 66%. Magnitude correlation is 0.55 with a floor of 0.32. Direction is the part that carries signal. If SimGym tells you variant C produced a 14% add-to-cart lift, the honest reading of that output is "C was better," full stop. The 14% is not a forecast.

Step 3: discard the losers, not the winners

This is the step that makes the maths work in your favour, and it is worth being precise about why.

A screen at 77% directional accuracy is a poor instrument for promoting a candidate but a decent one for eliminating candidates, because you are usually eliminating several at once. Discard the three variants that lost and you are making three independent-ish calls, each around three-quarters likely to be right, on options you were never going to have the traffic to test anyway. The alternative was picking one at random.

The asymmetry is in the consequences. Wrongly discarding a good variant costs you an opportunity you would not have had time for. Wrongly shipping a bad variant costs you real revenue on live traffic. Use the tool for the cheap mistake, not the expensive one.

Step 4: confirm the survivor on real traffic

Whatever survives the screen goes into a real experiment against real humans with a real significance test. Shopify's native Rollouts will split the traffic server-side, though it will not tell you whether the result is significant. Our significance calculator will do that part, and it is also the fastest way to find out whether you have enough traffic to bother.

Judge that confirmation on revenue per visitor, not on add-to-cart. The entire reason this article spends a section on attenuation is that add-to-cart lifts do not reliably become money. If your simulator measures add-to-cart and your confirmation also measures add-to-cart, you have not confirmed anything about your business.

Step 5: keep score

Log every simulation prediction and, later, the real outcome. After a dozen paired observations you will have something nobody else has: SimGym's directional accuracy on your store, with your catalogue and your buyers.

You will not get a precise number from a dozen tests. But you will learn quickly if it is systematically wrong on your store, which is the plausible scenario given what the persona ablation and the merchant reviews both suggest about unusual catalogues. Twelve data points is enough to notice a tool that is right 30% of the time.

What it must never be used for

TL;DR Four uses are not supported by the published evidence: declaring a winner, forecasting revenue, replacing a significance test, and validating anything downstream of add-to-cart.

Declaring a winner. A 77% directional call with a 66% lower bound is not a verdict. Shopify's changelog says each simulation "recommends which theme performs better," and that recommendation carries uncertainty the interface does not show. Treat it as a vote, not a result. This is the same discipline we argue for when an AI model is involved in calling an experiment: the model can generate the candidate and describe the finding, but deterministic maths on real observations decides the winner.

Forecasting revenue. Nothing in either paper connects simulated behaviour to money. If a report gives you a projected revenue figure, that projection rests on assumptions no published validation covers.

Replacing a significance test. The agents produce a difference in add-to-cart rate between two groups of simulated shoppers. That difference has its own sampling error, which is why both papers needed 600 agents before the answer stopped moving. A simulated gap is not more certain than a real one just because it arrived faster.

Validating checkout, pricing or anything past the cart. The agents are constrained from completing purchases. Everything downstream of add-to-cart is untested by design.

There is also a category error worth naming. SimGym reports two different kinds of output: a quantitative add-to-cart comparison, and a qualitative written critique of your storefront. Only the first has been validated. The written observations are an LLM describing what an agent encountered, and the merchant reviews show that these confidently include things that are not true, from working checkout buttons reported as broken to the tool's own session contamination reported as site bugs. Read the qualitative report as a list of things to check, never as a list of things that are wrong.

What would turn this into a decision tool

TL;DR Four specific, achievable changes would move SimGym from screening to deciding. None of them requires a research breakthrough. Three are disclosure.

We would rather be useful than sceptical, so here is the concrete list of what Shopify could publish or ship that would change our recommendation. It doubles as the checklist to re-run this analysis against when the next version lands.

1. Validate against orders, not just add-to-cart. This is the only item requiring genuine new research, and it is the one that matters most. Shopify sits on the ground truth: it can see whether those 50 theme changes moved orders, not just carts. Publishing a simulated-to-orders correlation would replace the whole attenuation section above with a measured number. Until then, every revenue claim about synthetic shoppers is an extrapolation.

2. Show the confidence interval in the app. Shopify already computes uncertainty properly, with a 10,000-resample bootstrap. A merchant seeing "theme B is ahead, and the gap is within the range we would expect from noise" would make better decisions immediately. The information exists. It just stops at the paper.

3. Disclose the model and the agent budget per run. The published spread between configurations runs from 59% to 77% directional accuracy, with the weakest one unable to beat a coin flip at the bottom of its interval. A merchant cannot currently tell which they received. Two lines in the report would fix this.

4. Let merchants correct the personas. This is the most requested feature in the reviews and the papers explain exactly why it matters: persona grounding is the component carrying the predictive validity, and it is derived from clickstream with no override. The university store whose customers are 95% students, and the merchant whose agents shopped for products they do not stock, both hit the same wall. Shopify's own ablation shows what happens when the persona is wrong, and it is not a graceful degradation.

Notice that only the first is hard. The other three are disclosure decisions, and they are the difference between a research preview and an instrument a merchant can reason with.

Where we sit on this

We build StorePilot, an AI CRO agent for Shopify, so we have an obvious interest here. Two honest disclosures. First, StorePilot is in development and is not on the App Store, so nothing in this article is a suggestion to install anything of ours instead. Second, we think SimGym is directionally the right idea and we expect simulation to become a normal part of how stores get optimised.

Our disagreement is narrow and it is about presentation, not technology. Shopify's researchers did careful work, published their confidence intervals, and documented their failure modes. Then the product wrapped that in "try new ideas without risk" and a single sentence of caveat. A merchant who reads the App Store listing has no way to learn that the magnitude correlation could be as low as 0.32, or that the tool was never validated against orders.

The principle we build on is that the uncertainty belongs on the screen where the decision gets made, not in an appendix. A number without its interval is not a measurement, it is a suggestion wearing a lab coat. That applies to our own product too, and you should hold us to it when we ship.

If you take one thing from this page: SimGym is a cheap way to be wrong less often about which idea to test next. It is not a way to find out whether you were right. Those are different jobs, and only one of them can be done without real customers.

Sources

Every figure on this page was read from a primary document fetched on August 11, 2026.

• arXiv:2605.19219v1, "SimGym: A Framework for A/B Test Simulation in E-Commerce with Traffic-Grounded VLM Agents," 19 May 2026. Tables 2, 3 and 4, Section 5.3, Appendix F and G.
• arXiv:2602.01443v1, "SimGym: Traffic-Grounded Browser Agents for Offline A/B Testing in E-Commerce," 1 Feb 2026. Tables 2 to 7, Sections 3.1, 4.1 and 4.2.
• Shopify Engineering, "2,000 robots walk into a shop: Simulated A/B testing," Javier Moreno, 27 Feb 2026.
• Shopify Winter '26 Edition, 10 Dec 2025.
• Shopify changelog, 11 Mar 2026.
• Shopify Help Center: SimGym.
• Shopify Help Center: Requirements when Shopify Network Intelligence is enabled.
• Shopify App Store: SimGym listing and 34 reviews, read 11 Aug 2026.
• GoodUI, Jakub Linowski, "Do adds to cart or progression metrics correlate with sales in A/B tests?", 30 Nov 2023.

Frequently asked questions

What is Shopify SimGym?

SimGym is a first-party Shopify app that runs AI shoppers through your storefront in a real browser and reports how they behaved, chiefly whether they added to cart. Shopify's Help Center defines it as "a Shopify app in AI Research Preview that uses AI-powered shoppers to simulate buyer behavior on your online store." It was announced in the Winter '26 Edition on December 10, 2025 and opened to all eligible merchants without a waitlist on March 11, 2026 (Shopify changelog).

How accurate is SimGym?

On the measure Shopify published, SimGym got the direction of the real add-to-cart change right on 77% of 50 test shops, with a 95% confidence interval of 66% to 87% (arXiv:2605.19219v1, Table 2). Its correlation with the size of the real change was 0.55, with a confidence interval of 0.32 to 0.72. Direction is predicted well. Magnitude is predicted weakly, and the confidence interval is wide enough that the honest reading is "somewhere between loose and moderate."

Which SimGym accuracy number is the right one: 69%, 73% or 77%?

All three appear in Shopify's own research and they measure different things. 69% with 0.64 correlation is the headline of the February 2026 paper on 20 shops (arXiv:2602.01443v1, Tables 2 and 5). 73% with 0.65 comes from the same paper's agent sample-size curve, which measures self-consistency between two runs rather than agreement with humans. 77% with 0.55 is the May 2026 paper's headline on a larger 50-shop set (arXiv:2605.19219v1). The 77% figure is the most current and rests on the largest sample, so it is the one to quote.

Does SimGym predict sales or revenue?

No. Both Shopify papers validate against add-to-cart rate only. Neither reports orders, revenue or revenue per visitor. The May 2026 paper states plainly that it uses "A2C rate as the primary outcome." Add-to-cart is a proximate metric, and published meta-analysis puts its correlation with actual orders at roughly R = 0.50 across 44 A/B tests (GoodUI). So a simulator validated on add-to-cart is two steps removed from money.

Can SimGym replace a real A/B test?

No, and Shopify does not claim it can. The Help Center's own caveat is that "results might differ from actual buyer behavior." A tool that is right about direction roughly three times in four, on a metric that itself explains only about a quarter of the variance in orders, is a screening instrument. Use it to discard weak candidates cheaply, then confirm the survivors on real traffic with a real significance test.

Why do SimGym's AI shoppers search for products I don't sell?

Because it is the system's most common failure mode, and Shopify measured it. In the February 2026 paper's behavioural breakdown (Table 6), "Product Not Found" accounts for 36.36% of the agents whose behaviour diverged between theme versions, the largest single category after indecision. Multiple App Store reviewers report the same thing independently, including one who wrote that the agents "searched for things we do not even sell." The papers also show you cannot fix it yourself: persona grounding is derived from your clickstream, and the app exposes no way to edit it.

Does SimGym affect my Google Analytics data?

At least one merchant reports that it does. A March 19, 2026 App Store review states that SimGym "creates an anomaly in your traffic in google analytics, including conversion tracking like add to carts and beginning checkout" and adds that "this should be made clear." This is a real consequence of the design: the agents drive live storefronts in real browsers, so they are capable of firing the same client-side events a human would. Shopify's Help Center documentation for SimGym does not address analytics contamination. Note the timing of your runs and exclude those windows before reading any report.

How much does SimGym cost?

The App Store lists it as "Free to install" with "Charges per simulation run." The Help Center states that "Each simulation you run costs one credit" and that "During the research preview, you might be allocated free credits." Shopify does not publish a credit price on the listing, and one reviewer specifically complains they could not determine how credits are distributed or whether more can be bought ad hoc.

What are SimGym's eligibility requirements?

Three, per the Help Center: your store uses a Liquid storefront (Hydrogen and headless are not supported), your store has Shopify Network Intelligence activated, and your store is not password protected. The Liquid requirement rules out headless builds entirely, and the password requirement means you cannot test a store that is not yet public.

Does using SimGym require a privacy policy change?

Possibly, because SimGym requires Shopify Network Intelligence to be enabled, and SNI carries its own obligations. Shopify's SNI requirements page says merchants must "Include a link to Shopify's Consumer Privacy Policy in your privacy policy, and post a link to your privacy policy prominently in your store," must explain that data is shared with Shopify and third parties possibly located in other countries, must provide opt-out rights in certain US states, and must obtain consent in the EEA, UK and Switzerland. That is a compliance step, not a checkbox.

How many AI shoppers does one simulation run?

Both papers use 600 agents per shop. That number was chosen from a sample-size analysis: in the May 2026 paper, directional alignment rises from 67% at 50 agents to 73% by 300 agents with little further gain through 700, and the 10th-to-90th-percentile band narrows from 0.17 at 50 agents to 0.05 at 600. Whether the shipped app uses 600 is not documented publicly.

Is SimGym still in research preview?

Yes as of August 11, 2026. The App Store listing still reads "Currently in AI Research Preview," and the Help Center still describes it as "a Shopify app in AI Research Preview." Shopify has not published a general-availability date.

What should I actually use SimGym for?

Cheap elimination, not decisions. It is well suited to catching navigation dead ends and add-to-cart friction on a draft theme before anyone sees it, and to ranking several redesign candidates so you spend your limited real traffic on the two that survive. It is not suited to declaring a winner, sizing a lift, or justifying a change to a stakeholder on its own.