The Ins and Outs of Incrementality Testing

Delcho Stanimirov
July 18, 2024
Digital Strategy

In today’s rapidly changing advertising landscape, measuring ad effectiveness is critical.

Calculating the incremental lift and the resulting value that ads provide is a great way to accurately assess the impact of your advertising tactics, as well as improve performance and ROI.

Let’s take a closer look at what incrementality is, how incrementality testing works, and how it can benefit your marketing campaigns.

What Is Incrementality?

In digital marketing, incrementality is the measurement of events that would not have occurred without a specific marketing activity and resulted in a lift in the desired outcome (conversion, revenue, page visits, etc.)Incrementality can be used to measure ad effectiveness and identify the interaction that moves a user from passive to active.

What Is Incrementality Testing?

Incrementality testing is an increasingly popular approach for businesses to measure incremental lift and understand the true impact of their digital marketing campaigns.Incrementality can be tested by randomly segmenting audiences into two separate groups and measuring which one outperforms the other.

One group consists of users that have viewed or interacted with a particular ad in a given campaign. This is referred to as the test group.

The other group is considered the control – a group of individuals who have not been exposed to these ads. The control group is kept aside for analysis.Effectively, both groups should have similar characteristics, however, they should not overlap.

How Is Incrementality Calculated?

When measuring incrementality, the results of each group can help you determine which conversions would not have been possible without advertising.

Consider this: You have segmented your audience into Group A (the test group exposed to ads) and Group B (the control group without ads). Because both segments show the same behavior, running campaigns only for Group A will show you the impact of those ads on your conversion rates (CVR) versus a control group.

The percentage difference of the test group from the control group is also referred to as incremental lift. This can be calculated by using the following formula:(Test CVR - Control CVR) / (Test CVR) = % Incremental Sales LiftTaking the example above, if Group A has 120 installs and Group B has 100, the incrementality would be 16,7% ((120-100) / (120) = 0,1666, or 16,7%).

Why Do Advertisers Need Incrementality for Proper Measurement?

Incrementality testing is the way for advertisers to truly understand the impact of their ads and make decisions that will help improve the efficiency of their digital marketing efforts. Some of the more prominent incrementality testing benefits include:

  • Driving business innovation
  • Measuring the true ROI of advertising
  • Helping marketers make more informed decisions
  • Contributing to effective optimization

With that in mind, it’s worth noting that while there are certain similarities between incrementality testing and traditional optimization methods, there are also some differences to keep in mind.Traditional optimization is focused on attribution-based performance and actions that indicate a direct link to a conversion. Measuring incrementality by itself is not a replacement for traditional optimization methods. Instead, it works together with attribution to help brands better measure their performance.

How to Conduct an Incrementality Test

Running incrementality tests can be a complex process, but if you are familiar with regular A/B testing, you should have a good starting point.Incrementality can be measured using a variety of different ways. Two of the more popular ones are sequential testing and Facebook brand-lift testing.

Sequential Testing

Sequential testing does not require a vendor, making it an easily accessible testing method.With sequential testing, you turn a channel on or off for a specific amount of time and measure the impact on the overall baseline.With sequential testing you can:

  • Execute a testing framework that doesn’t require complex technical implementation
  • Perform incrementation testing in-house

Fully turning off a channel, however, could lead to potential revenue loss. In addition, this tactic doesn’t remove the noise of external factors (e.g. seasonality), which might impact the end results.

Facebook Brand Lift Study

Facebook Lift Studies are set up in the Facebook platform and can be used for both Facebook and Instagram campaigns. With this method, your audience is automatically split into two groups – one that is exposed to your ads and one that isn’t.Both groups are then shown a set of poll questions that you’ve selected before launching the campaigns.The test is focused on calculating the difference in performance between the test and control group, representing your Facebook advertising lift in relation to the conditions of your test.By running a Facebook brand-lift test you can:

  • Collect valuable data and compare it to previous and ongoing campaigns
  • Understand the impact of your Facebook advertising on brand awareness
  • Set benchmarks for future campaigns

Nevertheless, keep in mind that this testing method is only possible on Facebook and does not acknowledge the branding effect and the impact of other channels.

Alternative Testing Methodologies

Along with sequential testing and brand-lift studies, digital marketers can explore several other alternative testing methodologies. Here’s a quick overview.

  • Public service announcement (PSA). PSA testing splits the audience into a test group that sees brand-related ads and a control group that sees PSA ads. While PSA testing is easily accessible with most platforms, it can be expensive to execute.
  • Ghost bid testing. Unlike the brand-lift method, this framework randomizes users after the bid is won. Although ghost bid testing requires a third-party vendor, it can yield unbiased results and is highly scalable.
  • Geo-split testing. Geo-split tests are set up by selecting a subset of geographies (e.g. cities, states) to show ads to, as well as a set of geographies with similar characteristics to the test group where you won’t be showing ads (e.g. demographics, behavior). This method is highly effective for Google, though, you’ll need to keep an eye on any marketing effort changes in the test markets during the testing period.
  • Custom audience split. This method is particularly useful when you want to see what the impact of your ads is (e.g if you’re running a sale and have a big email list of people). It splits customer lists into different segments to be used on different channels to measure the incremental impact each channel has on performance (e.g. only email, email + Facebook, email + Facebook + YouTube).

Custom audience split is easy to set and can help you get the most out of remarketing, however, the audience is limited to current customers and email lists.

The Most Common Incrementality Testing Challenges

While it’s hard to argue against the benefits of measuring incremental lift, it’s worth mentioning that there might be certain unintended consequences of incrementality testing, as well.Here are four of the more common incrementality testing pitfalls to watch out for.

#1 Seasonality

When it comes to calculating an incremental lift, deciding when to start testing is vital, and seasonality is one of the most important factors to consider.Running holiday campaigns can really pay off, but when it comes to measuring incrementality, you should avoid testing during peak seasons (which may vary by brand) and try to keep the testing timeframe as equal as possible.

#2 Outliers and Overlapping Audiences

When launching incrementality tests, bear in mind that your control group should be unbiased. This is why you need to try and limit external factors, such as other digital channel partners (i.e. search, social) or offline partners (i.e. TV, print, radio).In addition, if you are testing a channel that is bottom of the funnel (e.g. Google), you should avoid making any major changes on top of funnel channels (e.g. Facebook) as this would have an impact on the bottom of the funnel.Essentially, the more awareness you’re bringing, the more your results will be skewed when you’re testing Google.Along with outliers, having overlapping audiences could alter your results, as well.For example, if you’re focusing on users interested in hip-hop music, make sure that you are not testing something else which is covering those people at the same time. The more your control group is exposed to other media that you’re running, the higher the risk of data contamination.

#3 Segment Size and Testing Duration

Deciding on the size of your segments and the timeframe of your testing can be challenging.Generally speaking, it’s recommended that you allow at least a week for your incrementality testing. The exact testing duration, however, will typically depend on your business.If you have a lot of traffic and conversions, you might be able to test a smaller number of people while still having statistically significant results and an audience that is randomized enough. On the other hand, if you have a smaller business and don’t have that many conversions, your testing period should be longer.

If you are running a test on Facebook, the platform will do the calculations for you and notify you when the percentage is significant enough.

#4 Impact on Short-Time Results

When conducting incrementality testing, it’s important to be aware that, in the short term, you might experience some decrease in results.

That said, as it shows that your ads are working and you have an impact on the results, incrementality testing can be a very beneficial framework in the long run.

What to Do With the Incrementality Testing Results?

Once you’ve collected all the data from your incrementality testing, it’s time to compare and identify the incremental lift in the KPIs that match your business goals.Understanding the relationship between the test and control groups will help you determine why there was a positive, negative, or neutral incremental lift.Keep in mind, though, that if there is a big gap between the two groups, this might be a red flag that there’s something wrong in the configuration of your testing.Based on the results, you can take further action and apply the insight to your campaigns to maximize impact, define profitability, reduce cannibalization, and define the best time for re-engagement.

Define Profitability

For most SaaS companies, the results from calculating incrementality should be compared to the customer lifetime value (LTV).In order to be profitable, the LTV needs to be higher than the number you get from dividing the cost per acquisition (CPA) by the incrementality. For instance: if your incrementality is 16,7% and your CPA is $2, your LTV should be more than $12 ($2 / 16,7% = 12).

Reduce Cannibalization

Incrementality testing is arguably one of the best ways to reduce – and even eliminate – cannibalization.The test results can be used to scale up relevant channels to stop cannibalizing organic traffic, as well as ensure your growth targets are not rewarding cannibalization.

Create the Right Re-Engagement Strategy

When it comes to re-engaging, the data gathered from the incrementality testing can help you identify the optimal post-install time to re-engage users and ensure high incremental lifts from your marketing efforts.

Spend Your Budget Wisely

The results of each incrementality test can help marketers identify how much business value is driven by a specific tactic. This, in turn, plays a crucial role in determining which media investments are most effective, allowing you to shift budgets to improve future ad performance.

Get Started with Incrementality Testing

Measuring incrementality is a great opportunity to begin to understand the positive, negative, or neutral impact an ad has on your business. Executed well, incrementality testing is a powerful tool that can give you valuable insights and ensure that you’re making good use of your time and marketing budget.

If not done right, however, incrementality can hide risks and hinder your growth. Head to our blog and learn about the challenges of incrementality testing and how to tackle them.

Delcho Stanimirov

Head of Paid Media

I lead the PPC and Analytics teams. My professional goals are happy clients and colleagues. Also I love numbers, charts, and data-driven decisions.

https://www.linkedin.com/in/delcho-stanimirov-26512b1/