What is the purpose of an incrementality test?
With ecommerce continuing its ascendency, brands are naturally looking to get their fair share of revenue, to prove the impact of performance advertising, optimize spend, and get ahead. They are much more likely to capture the growth in ecommerce by measuring appropriately and leaning into the most advanced tools. But first, let’s catch our breath with a bit of context.
Performance advertising is one of the most powerful marketing tools in any brand’s arsenal, whether it is aimed towards customer acquisition or retargeting. But building an effective customer acquisition or retargeting campaign is as much an art as it is a science. So, it’s only natural that brands want proof positive that a performance partner is worth their salt. To do this, many companies have looked to incrementality testing.
The goal of an incrementality test is to determine what portion of conversions is a result of a performance campaign specifically, or of a specific partner, rather than from general marketing efforts. The difference between the level of conversions of two groups during the test is called incremental lift and enables you to understand what impact your performance provider has on your general marketing results.
Although incrementality tests have always been considered the gold standard for performance validation, they haven’t always been easy or cheap to run. Without a dedicated analytics team to interpret the results, companies can end up spending money on a test they can’t interpret. But this has now changed.
Let’s take a deep dive into incrementality testing, see how it works and what the best way is to run one.
How do incrementality tests work?
There are a number of different ways to run an incrementality test, but they all share the same basic approach—a population split, because only this can prove sales are incremental. This is a deeply scientific approach—exactly the same method is applied to medical trials.
But, how is an incrementality test conducted in the context of retargeting? First, the retargeting provider and client define the target group. These users are then split into two groups:
Test Group - These users are delivered ads using the retargeting provider’s methodology.
Control Group - These users are not delivered ads using standard retargeting activities, and are used as a comparison.
Before conducting a test, you need to keep in mind the statistical difference between these groups. Specifically, this means that you need to eliminate the risk that the test results were obtained due to random chance. To do this, we use a P-value, which represents the probability that the results were obtained due to randomness, and variance, which tracks the randomness of the result itself.
Over the course of the campaign, it is possible to determine the difference between the test and control group. You can see some of the results below:
These results are useful because they give you a more accurate picture of your campaigns than a single method—such as attribution—is able to do in isolation.
What do you need to consider before starting an incrementality test?
On the surface, incrementality testing seems like a simple tool to validate your retargeting campaign spend. However, like anything worth doing, an incrementality test requires companies to invest significant time and resources in order to maximize its value.
Before starting an incrementality test, you should consult this checklist to ensure that your business can conduct a test that will produce reliable results.
Do you have an analytics team in place?
While your retargeting provider can give you some insights about your incrementality test, you should be able to verify the results internally.
Your team will examine the granular conversion data and validate partner data against your internal data to confirm that it is correct. Additionally, this team will need to verify that the audience split used is statistically valid. These steps help to ensure that you’re working with solid data that can be used to derive correct insights
Is your campaign large enough for an incrementality test?
Data is good, but more data is always better. If your campaign is small, or you have a limited number of transactions, an incrementality test may not provide a sufficiently large data set from which your analytics team can draw meaningful insights.
In these cases, you would be better placed using another tool to validate your retargeting strategy.
Is your campaign stable?
It can be tempting to run an incrementality test from day one, but doing so can distort the results. This can either be due to technical issues with the campaign itself or because the campaign has not scaled enough to produce statistically significant results.
Generally, we would recommend that you run a normal retargeting campaign for a limited period of time—typically at least two to three months—in order to run your incrementality test with real and stable data.
Assuming that you meet all these criteria, you can begin looking at what kind of incrementality test you want to run. If not, you can skip straight to our incrementality test alternatives chapter here.
How does RTB House conduct incrementality tests?
Yes, traditional methods of incrementality measurement are tried and tested, but do they pass the test with flying colors? The answer to the ultimate question—is this spend creating new value?---is too often delivered tentatively at best.
One such traditional method is marketing mix modelling, which can offer a wide-ranging, holistic view of long-term performance advertising strategy. However, it is slow, costly, and misses a precise granular view that could allow for real-time campaign optimization. It can be found wanting in the causation department.
The market standard for incrementality nowadays is Geo Testing. It’s simple to understand, but it’s slow, expensive, only works on large groups, is prone to interference from market noise, and provides no user-level insight.
The above methods are functional, and can be helpful in some ways, but they are far from complete.
In the recognition that brands need to get to a point where incrementality is the source of truth to what really works, RTB House deploys next-gen methods of incrementality testing that make it more accessible for advertisers, and more granular.
RTB House Ghost Ads
RTB House combines Ghost Ads with its own precise methods of measuring lift. This process involves splitting users into two groups:
A test group - who are shown several ads at a higher frequency rate.
A control group - who are shown the same ads but at a limited frequency, typically around one impression every eight days.
Our own internal data has demonstrated that the most common ideal split is 80/20. While this has a minor impact on statistical significance compared to a straight 50/50 split, it minimizes the opportunity loss of utilizing a larger control group. We mitigate any reduction in statistical significance by ensuring that the test runs for at least 21–35 days, enabling us to gather more data.
The RTB House Ghost Ads way of testing holds the following benefits:
Real-time, easy-on testing.
Granular, segment-level incrementality.
No dark DMAs (designation market area), states, or markets.
No disruptions to campaign structures.
After running the testing, we take the true incremental sales and compare them to attributed sales to determine the incrementality ratio (the percentage of attributed sales that are incremental). This gives our clients a more precise picture of, firstly, our true worth as a provider, and secondly, where to allocate budget more effectively.
| Myth Busting: Ghost Ads can’t be run during a multiple retargeting campaign - [False]. |
|---|
Ghost Ads work in both single retargeter and multiple retargeter campaigns, in fact, the more campaigns you run, the easier it is to determine incremental lift! |
Our tips for running a Ghost Ads incrementality test:
Identify high investment areas to test.
Work with partners that enable incrementality studies.
Align internal stakeholders on true performance measurement.
Deploy Deep Learning. It takes a holistic approach and doesn’t focus attention on fraud or bots.
A test-and-learn mindset is key
Whether you use an incrementality test or one of the many alternatives, the key is in how you use this data. Adopting a test-and-learn mindset will ensure that you are always looking for ways to improve your marketing campaigns, regardless of what methodology you are using to do so.
With all that said, we think it’s time we lead by example and showcase some more of our own data-led retargeting success stories. RTB House takes a proactive approach to retargeting, with a heavy emphasis on using our proprietary Deep Learning technology in order to maximize the benefits of our retargeting campaigns. Our own in-depth internal testing has ensured that we’re able to deliver significant incremental improvements.
One of the best examples of multiple retargeting success can be seen in this case study, detailing how we became Huuuge Games’ most successful retargeting provider after just three months, exceeding ROAS goals by 3-4 times.
Another example is AllBeauty, where the RTB House team was able to help the brand make the most of a spike in online interest through an extensive retargeting campaign. Over the course of the campaign, we helped boost the client’s year-on-year revenue from retargeting campaigns by 65% and the number of conversions by 45%.
You can find all of our retargeting success stories here. If you’d like to learn more about how RTB House can help your brand kickstart a retargeting campaign, or reap the benefits of a multiple retargeting campaign, talk to our team today.
