Run the Power Calculation Before You Spend a Dollar on Incrementality Testing

Most incrementality testing programmes fail for a reason nobody publishes: the test was never large enough to detect the lift it was looking for. A flat result then gets read as proof that the channel does nothing, when it is really proof that the experiment could not see. The sample size question comes before the test design question, not after it.

• Statistical power determines whether a test can succeed, and it is decided before launch.
• A null result from an underpowered test carries no information about the channel.
• Geo experiments and user level holdouts answer different questions and are not interchangeable.
• A test whose outcome would not change a budget is not worth running.
• Platform lift tools measure their own channel, which is a conflict worth naming.

Incrementality Testing Answers One Question That Attribution Cannot

Incrementality testing measures causation, and every attribution model measures correlation. That distinction is the whole reason the method exists. An attribution model observes that a conversion followed an ad and assigns credit; it has no way of knowing whether the conversion would have happened without the ad.

The mechanism is a randomised controlled experiment. One group is exposed to the campaign, a comparable control group is not, and the difference in outcome between them is the conversion lift. Everything else in the discipline is a variation on how to build a credible control group when you cannot randomise people directly.

The gap this exposes is usually large in one specific place. Retargeting and branded search both report excellent returns because they intercept users already heading toward purchase. When the control group converts at nearly the same rate as the exposed group, the reported return was capture, not creation. That is the finding that moves media spend, and no attribution model will ever produce it.

Most Tests Fail Because Nobody Ran the Power Calculation

Decide before launch whether the effect you care about is detectable at your conversion volume. This is the step that separates a test from an expensive coin flip, and it is skipped in the overwhelming majority of programmes.

Two definitions make it concrete. Statistical power is the probability of detecting a real effect when one exists; 80% is the working convention. The minimum detectable effect is the smallest lift your test can distinguish from noise at that power. If your minimum detectable effect is 25% and the true lift is 12%, your test will return nothing, and the channel will be blamed for a design failure.

The consequence is uncomfortable and worth stating plainly. A null result from an underpowered test is not evidence of no effect. It is the absence of evidence, and treating the two as equivalent has cut more working budgets than any bad campaign.

The Sample Size Formula That Decides Whether Incrementality Testing Is Worth Running

Use a single approximation before designing anything. For a two group comparison at 80% power and 95% confidence, the sample size per arm is roughly:

n per arm = 16 x p x (1 – p) / d squared

Here p is your baseline conversion rate and d is the absolute lift you want to detect. Two worked examples, both recalculable:

• Baseline conversion rate of 2%, target detection of a 10% relative lift. The absolute lift d is 0.002. n = 16 x 0.02 x 0.98 / 0.000004, which is about 78,400 per arm, or roughly 157,000 users in total.
• Same baseline, target detection of a 20% relative lift. d is 0.004. n falls to about 19,600 per arm, or roughly 39,000 in total.

Doubling the effect you are willing to detect cuts the sample requirement by four, because the term is squared. That single relationship should govern the entire design conversation. If your channel drives 8,000 sessions a month and you want to detect a 10% lift on a 2% baseline, the test needs about 20 months of traffic. Do not run it. Either accept a coarser minimum detectable effect, pool geographies, or reach for marketing mix modeling instead.

Running this calculation first converts a vague measurement ambition into a yes or no. It is also the fastest way to find out that an experiment your team has already scheduled cannot work. Building the underlying data foundation for this is measurement work that sits upstream of any single test, because reliable baselines are the input the formula depends on.

Geo Holdouts and User Holdouts Solve Different Problems

Pick the design from the constraint, not from preference. A user level holdout test randomises individuals inside a platform and delivers the cleanest read available, but it only works where the platform controls delivery, and it measures one channel in isolation.

A geo experiment suppresses or increases spend across matched regions and compares total business outcomes. It handles offline conversions, cross device journeys and channels with no user level control. The cost is precision: geographies differ in ways people do not, so matching quality drives the result, and you generally need at least 20 to 30 comparable regions per arm before the design holds up.

The practical rule is simple. If the question is about one platform and conversions happen online, use the user level holdout. If the question is about total business impact, offline revenue or a channel you cannot randomise, use a geo experiment and accept the wider confidence interval. Brand lift studies answer a third question entirely, about perception rather than revenue, and should not be substituted for either.

Incrementality Testing Only Pays Off When the Result Can Change a Budget

Write the decision rule before launch. State, in one sentence, what you will do at each possible outcome: if lift is above X, we increase spend by Y; if it is below Z, we reallocate. If nobody can complete that sentence, the test is a research project and should be deprioritised.

This filter kills a surprising share of proposed experiments. Testing a channel that represents 2% of media spend produces a result nobody will act on regardless of what it shows. Testing the two largest lines in the plan produces a result that reallocates real money. Sequence by budget at stake, not by ease of setup.

One conflict deserves naming. Platform native lift tools measure the incremental contribution of the platform that built them, using the data that platform controls. They are useful and they are not neutral, which is why an independent geo test on the same channel is worth running before a major reallocation.

Brand Search Is Where Incrementality Testing Stops Being Theoretical

Branded search produces the most reproduced finding in the discipline and the most misread one. Serious tests on it land somewhere between “most of this was not incremental” and “this defended revenue we would otherwise have lost”, and both conclusions get published as general truths. They are not. Each is a reading of one auction, during one window.

A Brand Lift Number Expires the Moment the Auction Changes

The share of branded clicks that are incremental is a function of who else is bidding. When you are alone in the auction, the organic listing was going to capture the click anyway and the paid click is close to pure waste. When a competitor is present, that same paid click defends revenue that would otherwise leak to them.

Run your holdout in a quiet month and brand search looks worthless. Run it during a competitor push and brand search looks essential. Neither result is wrong, and neither generalises beyond the window it measured. This is the one place where a single annual test is structurally the wrong instrument, because the underlying variable moves daily.

The Durable Output Is a Rule, Not a Percentage

Convert the finding into a policy. It has three parts: hold paid coverage when someone else is bidding on your terms, release it when nobody is, and when you do hold coverage, stop overpaying for position one.

That third part is the one most teams miss. Position two is usually enough to defend a branded query, and it lowers the paid click through rate deliberately, so a larger share of clicks flows to the organic listing you already own for free. Blended cost per branded visit falls while total branded capture stays flat. This is the point where paid search and organic stop competing for the same click and start splitting it on cost. Most accounts describe that synergy. Very few instrument it.

Running That Rule Continuously Instead of Once a Quarter

None of this survives manual execution, because auction composition changes daily and nobody checks daily. Primelis built Signal to run the policy without human intervention: it detects when you are the only bidder and pauses brand ads, tunes bids against live competitive pressure, and reactivates coverage the moment a competitor enters the auction.

The design point that matters for measurement is that it reads paid and organic together rather than judging the paid line alone. Branded traffic distribution across organic and paid becomes one number instead of two, which is the only view in which the decision to cut is safe to make. The result is what incrementality testing was supposed to deliver in the first place: brand spend that occurs only when it changes the outcome.

Incrementality Testing Earns Its Budget When the Arithmetic Comes First

Incrementality testing is the only method that separates conversions your media caused from conversions it merely witnessed. It earns its cost when the power calculation comes first, when the design matches the constraint rather than the fashion, and when the decision rule is written before the test launches.

Start this week with arithmetic rather than a platform. Take your baseline conversion rate, decide the smallest lift that would change a budget, and run the formula. If the required sample exceeds what your traffic delivers in a quarter, you have saved yourself a test that was going to fail. Bring the numbers to a 30 minute measurement review with our team of 200+ specialists, working across 300+ worldwide brands, and leave with a design that can actually resolve.