Holdout Test

A holdout test carves out a matched control group that receives no ads, then compares conversion rates against the exposed group. Measurement concepts exist because ad platforms report the results they can see, and that is not the same as the results you caused. The gap shows up whenever a channel reports growth that the P&L never receives. Knowing which question a method answers keeps you from over-reading a dashboard. Read it next to Incrementality, Control Group, A/B Test.

Decide the question, the metric and the stopping rule before launch. Change one variable per test, hold budget and audience constant, and let the test run through at least one full purchase cycle. Write the result down — including the null results, which are the ones teams repeat most often.

Check whether you have the sample size to detect the effect you care about before declaring a winner; small differences need far more data than most accounts generate in a week. Cross-check platform-reported results against a holdout or an aggregate model when the stakes are large. Why this matters commercially: It's the gold standard for measuring incrementality — real users, matched populations, real behavioral difference.

Testing at any useful rate needs supply. Xeli produces clean, single-variable variants across your catalog — same layout, one changed element — so tests are properly controlled and the production queue is never the bottleneck. In the context of testing & measurement, that means the concept stops being something a person re-applies by hand every campaign and becomes a rule the system enforces on every asset it produces.

The recurring problems are predictable: holdout groups too small for significance; contaminating the control via other channels; ending the test before the customer journey completes. Each of these is a process gap rather than a knowledge gap — which is why the fix is usually a checklist, a template or an automated rule instead of more training.