How incrementality testing shows what your ads actually cause

Incrementality testing measures how many conversions your advertising caused, as opposed to how many it was present for. It works by comparing a group that was exposed to the ads with a similar group that wasn't. The difference between the two is the incremental effect.

That difference matters because a lot of attributed conversions would have happened anyway. People who search for your brand name, people who already have items in their cart, and loyal customers who see a retargeting ad were often going to buy. Attribution credits the ad. An incrementality test asks what would have happened without it.

For any channel that takes a meaningful share of budget, that's the question that should drive how much you spend on it.

The logic behind every test

Every incrementality test has the same structure. You split a population into a test group, which can see the ads, and a control group, which can't. The split has to be random or close to it, so that the two groups would behave the same way if nothing changed. Then you run the ads for the test group only and compare outcomes.

If the test group converts more than the control group, the gap is what the ads added. Divide the spend by the incremental conversions and you get an incremental cost per acquisition, which can be well above the cost per acquisition the platform reports. Done the same way with revenue, you get incremental return on ad spend.

The hard part is making the groups comparable and making the test large enough to detect a real difference. A weak test tends to fall down on one of those two points.

Holdout tests

A holdout test keeps a random slice of your audience from seeing ads, while everyone else is eligible as usual. It's the cleanest design when you control the audience, such as a customer email list uploaded as a custom audience or an owned retargeting pool.

You randomly assign each person to test or control before the campaign starts, exclude the control group from targeting, and compare conversion rates in your own data at the end. Because you own the assignment and the outcome data, you aren't relying on the platform to grade itself.

Holdouts work well for retention, retargeting and CRM-driven campaigns. They're harder for prospecting, because you can't hold out people you haven't identified yet.

Geo tests

A geo test uses regions instead of people. Some markets keep running ads, others have them reduced or turned off, and you compare sales across the two sets of regions.

Geo tests have a few strong advantages. They don't depend on cookies, logins or user-level tracking. They can measure any channel that can be targeted by location, including TV, radio, out-of-home and search. And they're judged on your own sales or lead data, which removes platform reporting from the equation.

They also have specific risks. Regions differ in size, seasonality and competition, so matching them carefully matters. Synthetic control methods help by building a weighted combination of control regions that tracked the test regions closely before the test. People also cross borders, see ads on trips and buy online from somewhere else, which can blur the line between groups.

A good geo test plan picks regions using historical data, checks that test and control moved together in the period before the test, and fixes the duration in advance.

Conversion lift studies

Several ad platforms offer built-in lift studies. The platform randomly holds out a share of the people who would have seen your ads, then compares conversions between the exposed and held-out groups.

They're convenient. The platform handles the randomization, which is hard to do yourself at the user level, and the setup is usually quick. For large advertisers, they're a sensible first test.

Their limit is that the platform runs both the test and the measurement. The randomization is usually sound, but the outcome is measured with the platform's own conversion tracking, and anything that tracking misses is missing from the result too. Where you can, pass your own conversion data into the study or check the result against CRM or sales data for the same period.

How much volume you need

A test can only detect a lift that's large compared to the normal random variation in your conversions. If the effect you're looking for is small, or your conversion volume is low, the test will need to run for a long time or cover a large share of your audience before the result is meaningful.

Run a power calculation before you start. It takes your baseline conversion rate or weekly sales, the natural week-to-week variation, the size of the control group and the smallest lift you'd care about, and tells you how long the test needs to run. Some platform lift tools estimate this for you. For geo tests, an analyst can estimate it from historical regional data.

If the answer is that the test can't detect anything realistic in a reasonable time, don't run it. A test that ends inconclusive costs money and gives you an answer that's easy to misread. Instead, test a bigger change, like turning a channel fully off in some regions instead of trimming it, or test a channel with more volume.

Reading the results

Start with the confidence interval before the headline number. A result that says a campaign drove a certain lift should come with a range. If the range includes zero, the test didn't show that the campaign worked. That's different from showing it didn't work, and both readings deserve care.

Compare incremental results with what attribution reported. If attribution credited a channel with far more conversions than the test found, that gap is a measure of how much attribution was over-crediting it. You can use that ratio to adjust their day-to-day reporting until the next test.

Look at the test period itself. A holiday, a price change, a competitor's sale or a stock problem in one group can distort the result. Note every outside change while the test runs.

Treat each result as true for that channel, that audience, that creative and that period. A retargeting test in a slow month says little about prospecting during peak season. Incrementality needs to be measured again as those conditions change.

Making tests part of the plan

The teams that get value from incrementality testing run it on a calendar. They decide which channels carry enough budget to justify a test, run them in turn, and feed the results into budget decisions and into any marketing mix model they use. Putting that rhythm in writing is part of measurement planning, and it's the part that turns a single interesting test into better budget decisions over time.

Clean outcome data is the other requirement. Tests judged on CRM or sales data are only as reliable as that data, so reconciling it is worth doing first. That groundwork sits in our marketing data and analytics work.

Questions people ask

What is incrementality testing in marketing?

It's a controlled experiment that compares a group exposed to your ads with a comparable group that wasn't. The difference in conversions or sales between them is the incremental effect, the part of the results the ads actually caused.

What's the difference between a geo test and a conversion lift study?

A geo test splits regions into test and control and compares sales by region, using your own data. A conversion lift study splits individual users inside an ad platform and measures the difference with the platform's tracking. Geo tests work across channels and don't need user tracking. Lift studies are quicker to set up within one platform.

How long should an incrementality test run?

Long enough to reach the sample size a power calculation gives you for your conversion volume and the smallest lift you care about. Fix the duration before the test starts, include full weekly cycles, and avoid stopping early because the numbers look good.

What if my incrementality test is inconclusive?

It usually means the effect was smaller than the test could detect, or the test was too short or too small. Check the confidence interval, then consider running a larger change, a longer test or a test on a channel with more volume.