Statistical Significance

Statistical significance is the probability that the difference between test variants is real, not random — commonly measured at 95% confidence. Measurement concepts exist because ad platforms report the results they can see, and that is not the same as the results you caused. The gap shows up whenever a channel reports growth that the P&L never receives. Knowing which question a method answers keeps you from over-reading a dashboard. Read it next to A/B Test, Control Group.

Decide the question, the metric and the stopping rule before launch. Change one variable per test, hold budget and audience constant, and let the test run through at least one full purchase cycle. Write the result down — including the null results, which are the ones teams repeat most often.

Check whether you have the sample size to detect the effect you care about before declaring a winner; small differences need far more data than most accounts generate in a week. Cross-check platform-reported results against a holdout or an aggregate model when the stakes are large. Why this matters commercially: Calling a winner without significance leads to ship-and-regret cycles: variants declared winners revert to average once real volume hits.

Testing at any useful rate needs supply. Xeli produces clean, single-variable variants across your catalog — same layout, one changed element — so tests are properly controlled and the production queue is never the bottleneck. In the context of testing & measurement, that means the concept stops being something a person re-applies by hand every campaign and becomes a rule the system enforces on every asset it produces.

The recurring problems are predictable: peeking early and stopping when a variant leads; ignoring practical significance (effect size); assuming a large p-value means 'no difference'. Each of these is a process gap rather than a knowledge gap — which is why the fix is usually a checklist, a template or an automated rule instead of more training.