Your dashboard reports what happened. It cannot tell you what would have happened without the campaign, which is the number your budget depends on. So how many regions can you hold out?
There is a campaign in your account that has never been switched off. It reports a positive return, and nobody can tell you what that month would have looked like without it.
That missing month has a price. A simulation one incrementality vendor ran over 36 million marketing decisions found that acting on noisy results left the business worse off than doing nothing in 38 percent of cases. That is a vendor's own number, labelled as one here. The noise itself is structural: Lewis and Rao, in a Quarterly Journal of Economics paper covering twenty-five large field experiments and $2.8 million in ad spend, found the median confidence interval on return on investment was over 100 percentage points wide.
No dashboard closes a hundred point gap. Incrementality testing is the only route built to close it, and three designs need no vendor at all.
What Incrementality Actually Measures
Every test asks what would have happened if this had not run. You never observe that. You observe one world, the one where you spent the money, and estimate the other with a confidence interval attached. When the interval is wider than the effect you hoped to find, the test has produced a number without an answer.
The Three Designs You Can Run In-House
They cost different things and they do not prove the same claim.
Design | What it costs | Minimum volume | What it proves | What it cannot prove |
|---|---|---|---|---|
Geo holdout with matched pairs | Media spend in the suppressed markets plus analyst time. The method ships as an open source R package | 10 matched pairs, so 20 geos | The combined effect across every paid channel there, because everything stops at once | Which channel inside the set did the work, and anything at user level |
PSA, ghost ads or ghost bids | Mostly engineering time. A ghost ad needs platform cooperation to log the counterfactual impression. A PSA burns real spend on a placeholder | Set by conversion count, not geography | The effect of the exposure itself, at user level, with the least noise of the three | Anything about the budget decision. It holds out an impression, not a spend |
Pre/post with a matched control | Analyst time only. A spreadsheet for a few units, CausalImpact for a synthetic control | Set by how stable the pre-period is, not by a fixed count | Direction and rough size where you cannot withhold anything, such as a brand campaign | Causation, if parallel trends fail. The weakest of the three on identification |
Geo holdout, and how many regions you actually need
The number to fix in your head is ten. Not ten markets. Ten matched pairs, which means twenty geos.
The trimmed match design paper behind Google's geo methodology states the rule itself: with n the number of geo pairs, "we recommend n >= 10 as too few pairs may make the inference unreliable" (arXiv 2105.07060v2, Section 4.4). Its worked case settles on 24 pairs, so 48 geos, for a two week test on a $1,000,000 hold-back.
The reason is spillover. The same paper requires each geo to be large enough to be reasonably isolated from its neighbours, which "often results in only a small number of geos available for experimentation." People cross a boundary to shop, and a metro area behaves as one economy even when your spreadsheet treats it as four. Shrink the cells to get more and treatment leaks into control. Grow them and you run out of cells.
Which is why one test market against one control market is not a geo experiment. It is a matched market comparison, and the methodology says what that is worth. The paper notes that market matching of this kind "only applies to non-randomized experiments and does not meet the gold standard for causal inference." Vaver and Koehler, in Google's technical report on geo experiments, put it from the other side: that method "draws its statistical power from the number of available geos in the analysis, and therefore is not applicable for situations with only a few geos available (such as matched market tests)."
The two-market version appears in more than one guide. It cannot carry a causal claim.
What you run instead: randomise assignment across pairs, keep three periods of pretest, intervention and cooldown, and analyse the geo level series. Google publishes the method as an open source R package called GeoexperimentsResearch, and CausalImpact covers the synthetic control case. No licence.
PSA, ghost ads and ghost bids
Three terms, used as if they mean the same thing. They hold out different things.
A PSA serves a real placeholder ad to the control group. The audience still gets a paid impression, the spend is still spent, and the only difference between the arms is the creative. Useful for measuring creative, useless for deciding whether a channel deserves budget, because you paid for both arms.
A ghost ad records what would have been served to a control user without serving it. The platform has to cooperate to log that counterfactual impression, and the control group costs nothing, which is the economic argument in Johnson, Lewis and Nubbemeyer's paper on the subject (Journal of Marketing Research, December 2017). A ghost bid is the auction level version.
None of these is a self-serve setting. You cannot open Google Ads or Meta and switch on a ghost bid. Ghost ads need the platform to log an impression it deliberately did not serve, and neither platform exposes that as a control a buyer can toggle. You either build it or you buy it.
Pre/post with a matched control
This is the design for when you cannot withhold anything. A brand campaign, a price change, a launch with no control available. Compare the treated unit against a matched control before and after, and the difference between those two differences is your estimate.
Difference in differences rests on one assumption: that the two units would have moved together without the intervention. That is testable in the pre-period, and testing it is most of the job. If they tracked within a few points for twelve months and diverged after the change, you have something. If they were already drifting apart, you have two markets that were never comparable.
Synthetic control goes further with many candidates: build the weighted combination of untreated units that best reproduces the pre-period, then watch the departure. It remains the weakest of the three on identification.
The Arithmetic That Kills Most Tests
For a user level holdout the sample size comes from a two proportion test, the same formula behind every A/B calculator.
n per arm = ((z_alpha/2 + z_beta) squared x (p1(1 - p1) + p2(1 - p2))) / (p1 - p2) squared
With alpha at 0.05 two sided the first z is 1.96. With power at 0.80 the second is 0.842.
Run it on a mid-size advertiser. Baseline conversion rate 2.0 percent, and you want to detect a 20 percent relative lift, so the treatment rate is 2.4 percent.
Step | Working | Result |
|---|---|---|
Add the two z values | 1.96 + 0.842 | 2.802 |
Square that sum | 2.802 x 2.802 | 7.851 |
Variance of the baseline | 0.02 x 0.98 | 0.0196 |
Variance of the treatment | 0.024 x 0.976 | 0.023424 |
Add the two variances | 0.0196 + 0.023424 | 0.043024 |
Multiply by the squared z sum | 7.851 x 0.043024 | 0.337782 |
Square the gap between the rates | 0.004 x 0.004 | 0.000016 |
Divide | 0.337782 / 0.000016 | 21,111.4 |
Rounded up, that is 21,112 conversions per arm, or 42,224 across the test.
That figure decides your calendar. At a 2.0 percent baseline it means roughly 1,055,600 sessions per arm, so 2,111,200 across both. At 100,000 combined sessions a day that is a 21 day test. At 30,000 a day it is 70 days. At 10,000 a day it is 211 days, which is not a test, it is a business change.
Hold the baseline and halve the effect you want to detect.
Baseline | Relative lift to detect | Conversions per arm | Total |
|---|---|---|---|
2.0% | 20% | 21,112 | 42,224 |
2.0% | 10% | 80,703 | 161,406 |
5.0% | 10% | 31,240 | 62,480 |
1.0% | 20% | 42,703 | 85,406 |
Halving the effect roughly quadruples the sample, and it explains why a two week test on modest volume can only detect an enormous lift. When such a test returns nothing, it has not shown that the channel does not work. It has shown that the test could not have found anything smaller than a number you would have noticed anyway.
A null from an undersized test is uninformative, not reassuring.
The caveat that invalidates the sizing above
Read this before you size a geo holdout with that formula. Do not.
The calculation assumes users are the randomisation unit. In a geo holdout they are not. The unit is the region, so the effective sample size is the number of regions rather than the number of people inside them. Twenty million users across twenty geos is an effective sample of twenty. That is why the trimmed match paper evaluates power in pairs and needs a cross validated power analysis on pre-period data rather than a user level calculator.
If you can only isolate six geos, this design is not available to you, and no statistics software changes that.
When You Genuinely Need a Vendor
Four situations make buying the measurement the right call.
You need a cross channel read and cannot suppress twenty isolated geos. The geo design is the only one that sees every paid touchpoint at once, and it is gated on a market count most advertisers do not have.
You need continuous measurement rather than a point in time test. A holdout answers one question about one period. If your planning needs an incrementality adjusted number every month in every channel, you are buying a system rather than a test.
You need a user level read at a scale where the platform holdouts are closed to you. Google's user based Conversion Lift opened to self-serve for Search and Performance Max on 2026-10-07. It needs at least 1,000 observed conversions, a $5,000 minimum budget requirement and a compatible conversion action, with a holdout of 1 to 50 percent. Meta's version needs $5,000 of campaign spend and 500 conversions on a 1 day click, 7 day click or 1 day view setting.
Google also gives you a pre-flight signal. Experiment Power bands the setup Low at 0 to 49, Medium at 50 to 79 and High at 80 to 99, from volume, duration, traffic split and campaign choice. A Low band means do not trust a significant result if you get one.
You need the analysis defended to a board or an auditor and have nobody who can run the methodology. Knowing which output to trust, and what to say when the interval comes back wide, is a different job from running the package.
Now the other side, which is the paragraph no vendor writes for you. If you sell through one channel, your volumes clear the platform thresholds, and someone on your team can reproduce the arithmetic above, a measurement vendor is a cost you do not need yet. Run the platform lift study on your biggest channel. Run the geo holdout if you have the markets. Put the rest into media.
How This Differs From Attribution
Attribution assigns credit for what happened. A test establishes what would not have happened. Different questions, different numbers, and that is why your dashboard and your lift study will never agree while both stay correct.
Attribution models the path a converting user took. It is useful for allocation inside a channel and structurally unable to say whether the channel should exist, because the users it scores were going to be exposed either way. For that side, start with our guide to marketing attribution software, and revenue attribution covers how those models are built.
A test adds the counterfactual. It holds a group, or a set of markets, out and measures the difference. That difference is the only number in your stack produced by an experiment rather than an assumption. If you are choosing platforms for the modelling side, best attribution software compares the options, and Google Ads A/B testing covers the mechanics closest to this work.
Start with the arithmetic, not the design. If the conversions your traffic can deliver in a month land below what the formula asks for, no holdout will rescue the answer, and the honest move is to put the test budget into media instead. Which of your channels can you size today?
Frequently Asked Questions
How many regions do you need for a geo holdout?
At least ten matched pairs, so twenty geos. The trimmed match design paper behind Google's geo methodology recommends them directly: too few pairs makes the inference unreliable. Each geo has to be large enough to be reasonably isolated from its neighbours, because a smaller cell leaks treatment into control. One market against one control market is a matched comparison, not a geo experiment.
Can I set up ghost bids in Google Ads or Meta myself?
No. A ghost ad requires the platform to log the counterfactual impression for a user it chose not to serve, and neither platform exposes that as a self-serve control. What you can switch on yourself is a user based Conversion Lift study for Search or Performance Max, subject to Google's published eligibility thresholds, or the Meta equivalent. A ghost bid is a build or a purchase, never a setting.
How long does an incrementality test need to run?
Long enough to accumulate the conversions the arithmetic requires, which is a computed number for your traffic rather than a rule of weeks. Fourteen days is a floor that stops a weekly cycle masquerading as an effect, not a guarantee that the test can conclude anything. Channels with long consideration windows need more: one vendor's own guidance puts connected TV at four to six weeks.
Is a small test that shows no lift proof the channel does not work?
No. An undersized test cannot separate no effect from an effect too small for its sample, and those two findings point to opposite budget decisions. A null from a test sized to detect a 20 percent lift says there is no lift above roughly 20 percent. It says nothing about a 6 percent lift that would still pay for the media.
Size one channel before you test it
The arithmetic decides which channel is worth a test at all, and it needs the conversion count your dashboard already holds. Allable puts your search, campaign and content data in one workspace, so the number you size the test with is the number you can check afterwards. Run the calculation on your biggest channel this week.
