Holdout Test

How Holdout Test Works

A holdout test (also called a "ghost ad" test or "PSA holdout") creates a clean control group by randomly excluding a portion of your eligible audience from a marketing campaign. The holdout group either sees nothing (pure holdout) or sees a neutral public service announcement (PSA holdout — ensures the control group is exposed to the same number of ad impressions but to irrelevant content, controlling for the mere exposure effect). After the test period, you compare conversion rates between the exposed group and the holdout group. The difference, adjusted for sample size and statistical confidence, is your incremental lift rate.

Why Holdout Test Matters for B2B Marketing

Holdout test design requires careful attention to randomization, sample size, and test duration. The holdout group must be randomly selected — not selected based on any behavioral characteristic — otherwise you introduce selection bias. Sample size must be large enough to detect meaningful effect sizes: detecting a 10% lift on a 2% baseline conversion rate requires approximately 50,000 users per arm at 80% statistical power. Test duration must be long enough to capture full purchase cycle effects — for B2B with 30–90 day decision timelines, a 6–8 week holdout is typically minimum.

Holdout Test: Best Practices & Strategic Application

Holdout tests apply across B2B marketing activities: email campaign holdouts (exclude 10–20% of your list from a nurture sequence to measure email's incremental pipeline contribution); ad retargeting holdouts (withhold retargeting ads from a random 20% of your pixel audience to measure incremental conversion lift); SEO holdouts (measure organic traffic and lead volume changes in specific content clusters before and after publication); and budget holdout tests (go dark on a specific channel in a test market for 4 weeks and measure pipeline impact versus control markets).

Agency Perspective: Holdout Test in Practice

The statistical requirements for holdout tests are frequently underestimated. The most common failure is a holdout group too small to produce statistically significant results — yielding inconclusive data that can't guide decisions. Use a power calculator (G*Power, Evan Miller's A/B test calculator) to determine required sample sizes before launching. Also account for novelty effects — newly launched campaigns often show higher-than-sustainable lift in the first 2–3 weeks. Run holdouts long enough to observe steady-state performance, not initial novelty spikes. Report results with 90% or 95% confidence intervals, not just point estimates.