What would happen if we just stopped marketing altogether?

Understanding marketing incrementality using experiments, MMM & attribution

Marketing incrementality in marketing is much talked about, but far less acted on, as it is complex and costly to evaluate. Many CMOS understand that incrementality should be of central importance to their marketing strategy, but it is not clear to them exactly how to leverage it for greater sales impact when it comes to setting budgets.

So how can incrementality be applied to your market mix measurement to pinpoint wasted spend and focus on what really works? With businesses under pressure and marketing budgets tight, focussing on incrementality measurement is a key strategic move that guarantees every penny you spend will count.

Easy to understand, hard to quantify

Incremental sales from marketing are sales which are driven by marketing over and above the sales that could be expected to happen without that marketing. It can be thought of as the ‘extra’ sales marketing gives you, challenging the notion that all sales can be attributed to marketing. Asking the hard question about what marketing is really doing for your brand is a necessary step towards estimating the true ROI (or ROAS) of marketing and therefore to setting a reasonable level budget, optimisation of the marketing mix and also for setting realistic expectations about sales impacts.

While marketing incrementality is an intuitive idea, it is very hard to quantify in practice because it uses a counterfactual, the idea of sales levels in the absence of marketing. No brand can afford to stop marketing completely just to see how far sales might fall!

A crazy idea worth considering

The crazy idea of stopping all marketing, while not at all practical, is nonetheless useful as a thought experiment in understanding incrementality. If all marketing just stopped, a well established brand with a strong existing customer base would have a lower % fall in sales than a smaller early stage brand. This shows that established brands typically have lower marketing incrementality, at least in the short run, as their sales have a momentum all of their own. So, marketing incrementality, especially brand building, is greater over the long term.

Brands that have a well established place in the market are reaping the benefits of marketing and brand building, often over many years, which helped get them there. If these established brands stopped marketing completely, they could gradually lose market share to competitors eroding their customer base, but this would be unlikely to happen right away. So this thought experiment forces us to think about what drives sales and incrementality, and shows how it makes complete sense to think about the short term versus long term contribution of marketing.

So how exactly can incrementality be measured?

There are broadly three ways to measure marketing incrementality which we are going to dive into in detail here. Taken together these are sometimes referred to as the ‘trifecta’ of marketing measurement (more on this later!). The first and best method is using an experiment, specifically an incrementality test. The second method is using MMM (Market Mix Modelling/ Econometrics) to estimate the ‘baseline’ sales, being those driven by non-marketing factors. The third method is to incorporate incrementality into digital attribution, which can be done simplistically, for example classifying direct and brand search sales as non-incremental, or in a more advanced way by measuring the underlying likelihood of sales not driven by marketing, within a data-driven approach.

Metageni, as a custom marketing analytics provider, offers a platform for all three of these measurement solutions, and helps brands to implement them and combine them together for a complete ‘holistic’ view of marketing impact. Note that Data-Driven Attribution models do not usually include a measure of incrementality, but this is included as a unique feature of our unique predictive approach, and is also used when estimating impression driven sales from Social media, Video and Display channels.

Incrementality tests: the gold standard of marketing measurement

Incrementality Tests are seen as the Gold Standard of marketing measurement, but are used far less than other methods because they don’t scale. This scale constraint means that the best way to use them is to integrate learnings from incrementality tests into the other marketing measurement approaches.

Incrementality tests use a robust experimental approach with a control group to conclusively prove whether a particular marketing activity really drove more revenue or not. They are useful for getting a deep understanding of a channel’s contribution to overall performance without the distraction of other factors. However, due to the strict requirements of experimental design, it is not feasible to integrate multiple tests into your day to day marketing.

What exactly is an incrementality test?

To understand the incrementality test you only need to consider the classic scientific method of the control based experiment. Consider the discovery of Penicillin, the first antibiotic. Legend has it that in 1928 the Scottish scientist Dr Alexander Fleming came back from a holiday and discovered that a strange mould was preventing bacteria from growing in one of his Petri dish experiments. He discovered that when Penicillin was not present in the dishes, there was bacterial growth (the ‘control’) but when Penicillin was present, this growth was suppressed (the ‘treatment’).

Likewise with incrementality tests in marketing: you can isolate the one lever that you are testing using this ‘control and treatment’ methodology, to decisively establish the exact cause and effect mechanism at work. These experiments take the crazy idea of stopping all marketing and make it manageable by only pausing for a subgroup of their target market, and typically only pausing certain tactics, in order to get a glimpse of the true incrementality of that particular activity at that point in time.

What sorts of Incrementality Tests are used in marketing?

A/B testing: with groups of existing customers or prospects: show ad/ send email to 80% of your audience and no ad/ email to the other 20% (say). This can be harder to achieve in practice and it is much easier to control for this for website experiments rather than ad experiments but it can be done through using first party data through login IDs to target ads to different matched groups and then assess the impact.

It is worth noting that the major ad platforms, such as Google and Meta, provide experimental solutions that work in this way (‘uplift tests’). These are considered fairly robust with the limitation that they are specific just to that ad platform, so they need to be implemented and interpreted with great care.

Aware/ Not aware: when measuring marketing investments like sponsorship properties it is possible to test after the event with customers aware/ not aware of your association to get a view of the impact of the spend. This is not a true incrementality test but starts to tease out how effective this investment could be. Survey methods provide a way to discover the level of brand awareness for samples of potential customers who have seen your ads to groups who have not, to measure incremental brand awareness.

Geo testing: matching two regions with similar customer demographics. Switch on one marketing channel in one region and not the other and test the effects of the overall uplift in sales. The better the match in terms of audience and potential customer base in each region, the cleaner the geo experiment becomes with a more representative ‘control’ region.

At Metageni we like geo tests because have the huge advantage that they do not need customer IDs/ cookies to work, just a statistical evaluation of marketing and sales in given geographic areas, and so they can be applied to the wider market without risk of infringement of customer privacy, or of sample ‘contamination’ caused by imperfect targeting technologies and cross device use.

The sorts of questions incrementality tests are great at answering include:

1. What is the impact of Youtube on my paid search spend?
2. What is the impact of cutting/ doubling my Facebook spend on overall sales?
3. How much does adding TV on top of digital channels increase my overall sales?

The downside of incrementality testing is that there is a limit to how many of these questions you can ask!

We see tests as an additional piece of the analytics puzzle on top of a strong customised MTA and MMM approach as they take resource, time and dedication to the marketing measurement process to set in place. It may need dedicated head count to manage the overall process and implement the tests alongside stakeholder management to drive big marketing decisions.

The fundamental bottleneck with experiments is that they are limited by how many questions each experiment can answer, and how many experiments can be run simultaneously. Typically one ‘spot’ ad driven impact on sales can be measured at one point in time for one channel or campaign. So they just cannot scale. It is technically possible to design experiments that combine channels in different ways to measure synergistic effects, for example, but these quickly become unmanageably complex and data hungry when additional activity is added.

Practically, experiments require direct changes to the marketing itself, with different groups of potential customers exposed to different marketing treatments, and some to no marketing at all. Despite all these constraints, the rewards of adopting the gold standard approach can be great in terms of accuracy and confidence in results, with high volume businesses like booking.com and ebay both embracing the incrementality test route forward to the greatest scale they can.

The `Trifecta’ of Modern Marketing Measurement

Adding Incrementality tests on top of Multi Touch Attribution and Market Mix Modelling is now being called the ‘Trifecta’ of Modern Marketing Measurement by thought leaders in the marketing measurement space. This is because they compliment each other with no one technique able to definitively guide optimal marketing decisions. Incrementality tests provide strong and independent data points to use within both Market Mix Models, to provide some anchor constraints for the regression model and also in Multi Touch Attribution, to calibrate a truer ROI by channel.

In other words these experiments provide a validation mechanism for the strategic insights of MMM and the granular and tactical actions implied by MTA. This visualisation captures the complete The `Trifecta’ of Modern Marketing Measurement –

Incorporating incrementality into digital multi-touch attribution (MTA)

To explain how this can work in practice, let’s consider linking Incrementality tests to digital multi-touch attribution (MTA). This can be done in a few ways such as:

Simplistic Approach: Many digital marketers understand that some marketing channels are actually just stepping stones in the process of purchase. At Metageni we call these ‘navigational’ channels. The most widely used example is Paid Brand Search, where for most brands, customers search for the brand name in order to find and navigate to a site where they can purchase it, having already decided that this is what they want to do. The same can be said of Direct and to a lesser extent, Organic search.

Guiding Principle: Incrementality is typically higher further up the funnel, with the final steps of a customer journey often done to fulfil an already formed intent to purchase. So the marketer wants to understand what drove that prior intent. As they consider MTA, they should look at first touch interactions to show what marketing drives the initial engagement, and they also need to consider how to incorporate incrementality tests. Last click touchpoints are considered these least incremental, which is a key reason why so many experts advise against using last click at all. Reviewing the last click attributed sales vs other models will highlight which channels probably have lower true incremental ROI than appears in this view e.g. voucher code affiliates.

Complex Methods: It is perfectly possible to incorporate incrementality measurement into attribution but it does require specialist attention and expertise. Attribution today has changed to reflect the end of 3rd party cookies, and so divides into two parts, (i) click based multi-touch attribution which can still be done on first party data and (ii) impression based attribution which requires a statistical approach that does not require user level tracking with cookies.

At Metageni we have a method for estimating click incrementality using our predictive approach, and for impressions where statistical estimation is used and we recommend always linking these to uplift tests available in the major platforms. For example, Social media on Facebook and Instagram can be measured with Meta uplift tests can be used to calibrate and set priors for thai estimation, to increase the likelihood of more realistic results.

Incorporating incrementality into econometric Market Mix Models (MMM)

To explain how this can work in practice, let’s consider linking Incrementality Tests to Market Mix Models.This can be done in a few ways such as:

Simplistic Approach: Compare the incrementality test results with the output of a Market Mix Model to ensure alignment. If the results differ significantly, further investigation into the model’s accuracy is required, with a process of re-estimating the model until results from the model are more consistent with the results of the experiment.

Guiding Principle: Use the incrementality test as a guiding principle when selecting the most appropriate model among candidate models. The model that best aligns with the incrementality test’s results may be chosen, even if it has slightly lower statistical quality. This is similar to the simpler approach but builds the alignment more directly into the estimation process.

Complex Methods: More advanced approaches involve incorporating incrementality test results as priors or constraints in regression models, enhancing the accuracy of the modelling process. This approach effectively creates a hybrid model approach by using experiment results as data points formerly within the model estimation itself.

Incorporating Incrementality Tests into Scalable Measurement: A Case Study

Here’s how Metageni helped Awaze set up a sophisticated measurement approach using incrementality tests on top of other methodologies:

Awaze, is the largest managed vacation rentals and holiday resorts business in Europe including Cottages.com, Hoseasons, and Novasol. Our first Awaze task was to unify all their data and attribution across 3 major brands across 12 European countries.

Awaze’s key challenge was trying to understand what to spend across each brand to drive the most return in incremental revenue for the whole group, starting with more accurate attribution analysis. We built a complex comparative multi- brand attribution project aligning channel definitions, vacation property and booking categories and creating custom machine learning attribution models which could accommodate different marketing mix and consumer behaviours for each brand and market. We had to resolve significant data issues around sources, analytics and location identification and exchange rate variation.

The resulting custom solution gave Awaze the ability to evaluate marketing driven sales and ROI for any combination of the Awaze brands, sites and booking categories.

At the same time, Awaze were looking to optimise the mix of their digital media channels to optimise the customer journey. We built them a ‘What If? Simulator to forecast the revenue impacts of moving budgets between channels to drive the best return on investment from marketing.

To validate this analysis we worked with Aware to identify a representative geographic segmentation that could be used to run geo based experiments against campaigns, in order to evaluate and quantify specific hypotheses. Two regions could be targeted with different volumes of campaign activity, and compared to the countrywide picture using statistical analysis and custom charts and tables, to understand impact vs a control situation.

This analysis showed huge variations in a data driven view vs the previous last click view, averaging 12% for all channels, and much higher for some notably paid social and youtube, pointing the way to several major opportunities for incrementality test evaluation going forward.

Summary

Incrementality is a key concept for deciding which parts of your marketing are most impactful, measuring ROI and getting the right marketing mix throughout the funnel.

Incrementality testing is the gold standard for measuring incrementality and so it should be considered at all stages of a marketing team’s measurement approach. However it does not scale well, and so needs to be incorporated into the more scalable approaches of Market Mix Modelling and Multi-Touch Attribution, which can be used to estimate incrementality, but are not definitive in the way that an experiment can be.

When there are only a few marketing channels in play, incrementality tests can be set up to understand the impact of key channels. For example, adding Youtube in one country but keeping Paid Search the same vs a control region, will enable marketing teams to understand the incremental impact of video on search performance and make the appropriate decisions around marketing budgets.

As your marketing mix becomes more complex and varied, incrementality tests are incredibly useful to calibrate learning from other approaches and make sure neither Multi Touch Attribution nor Market Mix Modelling veer off course in absence of clear incremental signals, and to ensure the marketing team can ‘be less wrong’ about marketing channel performance.

Do you need help with something like this?

If so just contact us: [email protected]