A holdout test measures a virtual try-on tool by randomly assigning some shopper sessions to see it and some not to, over the same period, and comparing the two groups. It is the only method that separates the tool's effect from everything else that changed the season, a promotion, a paid-media shift, a new product drop. This post is the method, not a number: how to set one up, how many sessions you need, and the four ways it goes wrong.

Why doesn't before-and-after work?
How does a session holdout work?
What should you measure?
How many sessions do you need?
What are the traps?
How should you read vendor numbers?
Where Spequlo fits
Questions, answered

Why doesn't before-and-after work?

Most virtual try-on results you will be shown are a comparison of conversion in the weeks before install with the weeks after. It is intuitive and it is nearly worthless, because it cannot tell you what caused the change.

Apparel conversion moves with the calendar. It moves when you run a sale, when a product sells out, when a paid campaign starts or stops, when a competitor runs a sale, when a creator posts, when the weather turns. A tool installed in the first week of November will "lift conversion" through Black Friday; one installed in the second week of January will "hurt" it. Neither reading has anything to do with the tool.

The same problem affects the shopper-level version: people who used the try-on converted at 3× the rate of people who didn't. Shoppers who choose to upload a photo of themselves are more engaged than shoppers who don't. They would have converted at a higher rate with or without the tool. That comparison measures intent, not effect.

How does a session holdout work?

The design is the same one used for any product change: randomised controlled comparison.

Define the population. Sessions that reach a product page where try-on is available. Sessions that never see a product page can't be affected and only add noise.
Assign at random. When an eligible session starts, assign it to treatment (try-on shown) or control (try-on hidden) using a random number, and record the assignment. A 90/10 split loses very little statistical power against 50/50 and keeps most shoppers on the full experience.
Keep the assignment sticky. The same visitor should stay in the same group across the test, or a shopper who saw the tool on Monday and not on Tuesday contaminates both arms. Assign by a persistent identifier — a first-party cookie or, on Shopify, the customer or cart token.
Log everything against the assignment. Every session's group, product-page views, add-to-cart, checkout, order value, and — later — returns, keyed to the order.
Run for full cycles. At least two full weeks so every weekday appears twice; longer if the numbers below say so. Do not start or stop mid-promotion.
Compare intent-to-treat. Treatment sessions versus control sessions — all of them, including treatment sessions where the shopper never opened the try-on. That is the number that answers "what happens to my store if I turn this on."

What should you measure?

Three metrics, in this order of usefulness:

Conversion rate (orders ÷ sessions) is the primary metric. It moves fastest and needs the fewest sessions.

Average order value is worth tracking but rarely the point; try-on tools are not usually designed to move basket size.

Return rate is the metric brands care most about and the hardest to measure, because returns arrive two to six weeks after the order. A holdout that reports return rate one week after the test ends is reporting nothing. Fix a return window — 30 days after the last order in the test is a sensible default — and do not read the number before it closes.

Secondary metrics that tell you why: add-to-cart rate, try-on open rate, and return reason codes split by arm. If try-on cuts returns, it should cut fit and appearance returns specifically; if it cuts "damaged" returns, something else happened.

How many sessions do you need?

This is where most tests are lost before they start. To detect a change you need enough sessions for the observed difference to be larger than random noise. The standard calculation (two-sided, 95% confidence, 80% power) for a store with a 2% baseline conversion rate:

Read that table carefully. A store doing 20,000 product-page sessions a month cannot detect a 10% conversion lift in a month — it would need eight months of 50/50 traffic. It can detect a 25% lift in about six weeks.

Three consequences:

Decide what you're testing for before you start. If a 10% lift wouldn't change your decision, don't size the test to find it.
Restrict the population to eligible product-page sessions. Counting homepage bounces in the denominator inflates the sessions needed.
Small brands should test longer, not smaller. A short test on a small store produces a number with an error bar wider than the number.

Any online A/B calculator will do the arithmetic; the inputs are your baseline rate and the smallest lift you'd act on.

What are the traps?

Sample ratio mismatch. If you set a 50/50 split and get 54/46, the randomisation is broken — usually a bot filter, a caching layer, or a redirect that only fires in one arm. Check the ratio before you read any result. A skewed ratio invalidates the test.

Novelty. Anything new on a product page gets clicked more in its first week than its fourth. A two-week test overstates a tool's durable effect. Run longer, and look at week four separately from week one.

Peeking. Checking the result daily and stopping when it looks good guarantees a false positive eventually. Set the duration from the table above and read the result once, at the end.

Contamination. A shopper who sees try-on on their phone and then buys on their laptop in the control arm counts as a control conversion caused by treatment. Sticky assignment on a logged-in identifier reduces this; you cannot eliminate it. It biases the result toward zero, so a positive result is conservative.

How should you read vendor numbers?

With the method above in hand, every vendor claim sorts into one of three bins.

"Conversion went up X% after install." Before-and-after. Not evidence.

"Shoppers who used try-on converted at X× the rate." Self-selection. Not evidence.

"In a randomised holdout across N sessions over W weeks, treatment converted at X% against Y% in control." Evidence — check N against the table and ask for the return window.

Most published figures in this category are in the first two bins. That does not mean the tools don't work; it means the number you've been shown doesn't tell you whether they do. Ask the vendor which method produced their number. If the answer is a holdout, ask for the session count.

→ See how Spequlo measures lift on your store

Where Spequlo fits

Spequlo logs every try-on against the session it happened in and the order that followed, and runs a random session holdout by default — a share of eligible sessions never see the fitting room, so every brand gets a treatment-versus-control read on its own traffic rather than a before-and-after. Return rate is reported only after a fixed post-order window closes. We built it this way because we wanted to hand brands a number we could defend, and because this is the same test they would run themselves if they had the time.

→ Turn on a holdout on your product pages

Questions, answered

Can I run a holdout on Shopify without a developer? Some try-on apps run one for you. Otherwise Shopify has no native A/B testing for app embeds; you need either the vendor's built-in split or a testing tool that can toggle an app embed per session.

Is a 10% holdout enough? For statistical power, 90/10 costs you roughly the equivalent of running a 50/50 test on a third of the traffic. It's a fair trade to keep most shoppers on the full experience, but the sessions-per-arm figures above apply to the smaller arm.

What if the vendor won't support a holdout? Run a geographic or time-alternating test as a weaker fallback, and treat the result as directional. Or pick a vendor that will.

How long until I can see return-rate results? Test duration plus your return window — typically six to ten weeks end to end. Conversion results arrive at the end of the test; return results a month later.

Should I test on all products or a subset? All eligible products, but analyse by category. Try-on quality varies by garment type, and a category-level breakdown tells you where it's earning its keep.

What's a realistic effect size? We won't quote one until our own holdout results are published. Anyone quoting you one should be able to show the method behind it.

How to Measure Whether Virtual Try-On Works: The Holdout Method (2026)

Contents