Who is this presentation for?Those with an interest in:Best practice measurementPlugging the gaps we know we have in direct measurement & trackingFor describing problems with marketing measurement to non-marketers (talking to finance!)Insight into econometrics and explanation of how it works
& to gain an advantage over competitors (or to keep up with those who are doing it)- This is ‘why would you measure anything?’ not just marketing
Talking the language of financeFD’s can be heard complaining that marketers don’t speak a financial languageOne of the solutions we often try is interim metrics to avoid linking marketing to sales (cost per 000, # of friends, follower counts, column inch equivalent.).Marketing measurement is hard. So we avoid tracking to sales and instead pick ‘engagement’ or social metrics etc. This puts marketers at a disadvantage in financial conversations.
For illustration only, but even in FMCG with a relatively small number of sales drivers and where most sales drivers have good data, measurement of advertising will be very difficult.Marketing is a mid-sized driver and measurement of it will be confused by lots of other effects happening at once.
Here’s another problem that makes pulling advertising apart difficult; effects aren’t felt exactly where the advertising ran.Note that this also means if you manage to measure advertising only in the week it runs, then you’ve probably undervalued its contribution
Backing up the previous slide, this is one of the more extreme examples we’ve seen (from econometrics). TV ‘launched’ the brand and it would never have got this big otherwise.New estimates since this model suggest it would take c. 1 year for sales to return to normal
So what about what a lot of us do? Track weekly sales and try to explain why they’re moving.We’ve already seen that it’s likely to be difficult – here’s a real example. Client wanted to explain the recent uplift in performanceNote this is year on yearSolves some problems, but creates othersWe’re looking at good performance that might be because this year’s gone well, or because last year wen badly.YOY sometimes helps but:Seasonality is rarely the biggest problemIt’s very hard to fix seasonality with YOYCold sore example (it’s actually temperature, not pure seasonality)School holidays don’t line up for supermarkets
This one just created confusion. Advertising returned to normal, then sales lifted. What happened here?(we’ll come back to it and find out)
Here’s another client dashboard – extreme example. This isn’t even weather data, it’s pictures!Takes lots of time to maintain & doesn’t help.Interesting that weekly tracking often focusses on two things that from Page 4, we know aren’t the biggest drivers.We’re concentrating on what we’ve got data onSeasonality and weather.Our own activity, the market & what competitors are doing is MUCH more important.Also focus on one day, or one week won’t help to measure marketing (note data is VERY noisy at a daily level & this may become a bigger data movement than marketing.)- Legal client asked recently why they’d get no calls for half an hour and then five the next half hour. With numbers this low, it could easily just be random.
That’s not to say weekly tracking is a waste of time, but we should acknowledge what it’s good at
One part of a three part puzzle
By direct tracking I mean a direct link from marketing to sale (or a metric close to sale)Admiral ‘where did you hear about’ RotationNOT tracking all sales!Some brands can’t do itLet’s you optimise WITHIN a media channel. You can make press better but you can’t find out exactly how good press is, vs. say TV
Here’s an example of that working. Optimising and re-allocating within a tracked press campaign.Making the most of the data we’ve got. This needs to be part of everyday planning to work.(past clients doing it this way – insurance etc.)Quickly published online – available to all
Some channels can’t be tracked that way (TV etc.)Unfortunately, they’re often the expensive, bigger budget channelsAnd those spends often make direct marketing more effectiveHere’s an example of search being driven by TVWe typically find that 40—60% of brand searches are driven (short term) by off-line adsNote that I’ve used search here because it’s exactly like any other direct channel. There’s nothing special about it at all and solving problems with search is the same problems we’ve always had with direct vs. brand.We’re just concentrating on it because we’ve got more data…- Mention that for most advertisers, 90% of search clicks come from c. 20 terms (banking etc.) The ‘long tail’ is only worth having if we can get it for almost free (i.e. automatically).
Real example, trying to pick up the interactions between channelsIf you try to get to an overall model that works, there’s no way you’ll get there in one jumpMeasure directThen measure brandIf you’re happy you’ve got both right, you can try to join them togetherTrying to jump straight to the whole answer is a recipe for a very expensive failure- ‘Project Apollo’. Nielsen + P&G + $45m in 2005, to pin down all the mechanics in FMCG. Abandoned with no results.
Your brand ads generate interest, whether you’re doing direct / search ads to convert it, or not…
Unsurprising that econometrics has most taken hold where:Data is available to build the models (FMCG)The market is relatively simple (FMCG)There’s no direct tracking (FMCG!)With more computing power and more data, we’re getting better at building models in other markets though. It’s a tool that can be applied to any problem where we have data (Moneyball book)
We’re filling in the second part of the puzzle: Comparing ACROSS channels, e.g. Press vs. TV
WARC guide to econometrics follows this structure. It’s also very analytical and quite hard to read for a marketer. More by analysts, for analysts.There are very few plain English guides to econometrics for marketers (if any, I haven’t found a good one yet.)
We needDataA hypothesis to test (a question)Some econometrics skills to make sure our answers are valid.Word econometrics comes from economics. These are the tools that economists use to see if their theories about the economy hold true.
The maths is important. Without it, we could produce very dodgy analyses.But econometrics also shouldn’t ever be a ‘black box’.“If you can’t explain it to a six year old, you don’t understand it yourself”EinsteinDon’t use analysts who you don’t understand (including me.)
Note here: Who are econometricians?Usually maths / economics background. Not that many people around doing the job (still)
Break to comment that as the analyst builds the models, he’s checking these stats all the time.
Big retailer example (Sainsbury’s / Homebase / others)How their models work – store by store + loads of datae.g. 400 stores, 3 years, weekly= 62,000 data pointsThat’s why we can’t work this out by hand and need the stats!ITV at individual BARB panel member level = 6000 observations per episode.
Now we’ve combined the first part – direct tracking, with the second part – econometrics.Note that click path is just more modelling (question we can throw stats at.)We build econometric models infrequently (6-12 months), but use the results every week.Let’s see some examples…
We can build sales forecasts to convince finance to invest or to prove advertising ‘should’ word.… bringing marketing in line with other business investments. It’s not just a cost!This was a real example, that a client used to justify a roll-out of TV, based on a test (we’ll come back to tests)
Here’s the example from earlier again. If it wasn’t TV driving the sales, then what was it?
Econometrics had identified a way to track market performancec.f. real betting example. New sign ups closely track searches for ‘betting’Try this yourself!
Recap models and extend discussion – based on what we’ve just seen, there are some things that econometrics isn’t very good at:Effects that are very smallEffects that happen very slowlyDiscuss social. If it worked as a campaign and sold more product, could econometrics measure it?Building on models that have already been created – we have a good base for test and learn
Mention success criteria – Eamonn Holmes quote
Hopefully given a taster of who is using econometrics and what can be achieved. If we’ve whet your appetite, then where can you get econometrics from?
Three basic types of providers. All have advantages.The biggest question is about implementation. Models very rarely work if the analyst gives you a debrief pack and then walks away, which is why I choose to work in an agency. Advantages of being close to planning and having loads of data, outweigh potential for conflict (for me.)Warning that for the initial debrief, it’s likely to make sense in the room at the time, but you need backup. Three weeks later when you try to apply it, you’re guaranteed to need the analyst again. It won’t make nearly so much sense at that point!Who is the analysis for? You, internally? Or the planners in your ad agency? Open question…Note: It costs extra money for analysis. Even though this is the best way to plan, no agency can offer it built-in. Margins are so thin in media buying that even though this is the best way to plan, we need to pay a little more to achieve it.Still, the cost of one press ad (which might be achieving very little) to find out how all of your advertising works…
One way of following up is optimisation tools.Don’t let a tool replace your access to analysts – they rarely work without adjustment.Screenshot is a real, commercial bit of kit that you can buy, but you’ll still need analysts to help run it.Also, watch automated modelling software. It should be (much) cheaper for you to buy, because it isn’t as good.
A few points to remember when you brief / pitch
Looked at what we can do with direct. Good but not whole answerCan’t compare across channels or set budgetsEconometrics adds those bits but then…Leave on these points:Econometrics is very powerful, but it does WHAT, not WHY.It will tell you which ads work and predict the future, but it’s only a toolConsumer research does WHY. We need it to answer the questions that econometrics raises.Press doesn’t work, but could it ever?Do people like this new TV ad?Who should I be talking to?Can I launch a new product?
Summarise on blank slideTrack where you canAcknowledge what this can achieveDon’t get sucked into blaming only the factors you can get data onDefinitely don’t run only track-able mediaEconometrics makes more things track-able (and gives a lot more insight too)Use econometrics to increase understanding and to improve weekly analysis. It’s something analyse now and again, to make everyday marketing better (like a car service)Econometrics will raise questions for research. It’s not the whole answer but it has the power to move us a lot closer.Thank youQuestions / discussion
Cim & brilliant media introduction to econometrics
Maximising returns on yourcommunications investmentMarketing measurement and econometrics1st November 2011
We know that we need to measure marketing• To increase its effectiveness• To reduce risk• To justify the marketing budget
But marketing measurement is hard• Marketing doesn’t always work quickly• The effects are often not felt immediately• So we end up not being sure if it’s working at all
Week-to-week sales movements are affected by many factors other than marketing• Short-term sales movements due to advertising are difficult to pull apart from other factorsScale of weekly sales movements typically measured by an FMCG model 30-50% Promotions and discounts 30% Distribution change (+30%) 10-20% Price change (+10%) 5-10% Above the line advertising 5-6% Competitor activity 3- 5% Seasonality 1-2% Random ‘noise’ ~ 1% Weather
New Customers 0.00 50.00 100.00 150.00 200.00 250.00 01-Oct-10 05-Oct-10 09-Oct-10 13-Oct-10 17-Oct-10 21-Oct-10 25-Oct-10 29-Oct-1002-Nov-1006-Nov-10 New Customers10-Nov-1014-Nov-1018-Nov-1022-Nov-1026-Nov-10 Example of a highly impactful TV burst30-Nov-10 Immediate TV Contribution04-Dec-1008-Dec-1012-Dec-1016-Dec-1020-Dec-1024-Dec-1028-Dec-10 01-Jan-11 05-Jan-11 09-Jan-11 13-Jan-11 17-Jan-11 21-Jan-11 25-Jan-11 29-Jan-11 TV Carryover contribution02-Feb-1106-Feb-1110-Feb-1114-Feb-1118-Feb-1122-Feb-1126-Feb-1102-Mar-1106-Mar-1110-Mar-1114-Mar-1118-Mar-1122-Mar-1126-Mar-1130-Mar-1103-Apr-1107-Apr-1111-Apr-1115-Apr-1119-Apr-1123-Apr-1127-Apr-11 The effect of a TV burst can last well beyond the timing of the spots
Year on Year Change -5% -15% -10% 0% 5% 15% 20% 25% 30% 35% 10%03-Jan-1110-Jan-1117-Jan-1124-Jan-1131-Jan-1107-Feb-1114-Feb-1121-Feb-1128-Feb-1107-Mar-1114-Mar-1121-Mar-1128-Mar-1104-Apr-1111-Apr-1118-Apr-1125-Apr-1102-May-1109-May-1116-May-1123-May-1130-May-1106-Jun-1113-Jun-1120-Jun-1127-Jun-11 04-Jul-11 Year on year sales. Client was looking for the reason that sales increased from June 2011 11-Jul-11 18-Jul-11 25-Jul-1101-Aug-1108-Aug-1115-Aug-1122-Aug-1129-Aug-11 increases in sales, works occasionally Trying to identify marketing impact by looking for
Year on Year Change 100% -40% -20% 0% 20% 40% 60% 80%03-Jan-1110-Jan-1117-Jan-1124-Jan-1131-Jan-1107-Feb-1114-Feb-1121-Feb-1128-Feb-1107-Mar-1114-Mar-11 TV Ratings Year on Year Sales Sales Year on Year21-Mar-1128-Mar-1104-Apr-1111-Apr-1118-Apr-1125-Apr-1102-May-1109-May-1116-May-1123-May-1130-May-1106-Jun-1113-Jun-1120-Jun-11 Year on year sales. Client was looking for the reason that sales increased from June 201127-Jun-11 04-Jul-11 11-Jul-11 18-Jul-11 25-Jul-1101-Aug-1108-Aug-1115-Aug-1122-Aug-1129-Aug-11 Unfortunately, more often it leads to confusion. Sales increased here, after TV was returned to normal.
Weekly sales tracking that tries to pin down marketing,is rarely effective unless marketing uplifts are very large • Tracking can lead to a singular focus on trying to explain the previous week’s sales – Trying to explain away the movements that aren’t marketing is very difficult – Often we end up blaming everything on the weather A weekly retail sales tracking dashboard with a singular focus on weather (Brilliant Media client example) Total sales TY 2,534.42 3,347.83 2,321.96 2,384.24 2,320.83 2,623.11 3,429.65 Total sales LY 2,167.29 3,138.10 3,061.70 3,110.99 3,446.56 3,739.47 4,942.17 Total budget sales 2,194.87 3,357.76 3,249.18 3,275.26 3,657.56 3,955.02 5,119.55 Sales vs budget 15.5% -0.3% -28.5% -27.2% -36.5% -33.7% -33.0% Sales vs LY 16.9% 6.7% -24.2% -23.4% -32.7% -29.9% -30.6% 2011 Weather Temp 7.1°C 6.7°C 6.1°C 5.6°C 5.9°C 5.9°C 5.7°C 2010 Weather
We can achieve a lot, without complex statistics• Track what we can measure – Take care not to only spend on what we can track… • Acknowledge the issues with marketing measurement
Direct response tracking completes a part of the picture Econometrics Return on investment Budget allocation Budget setting Sales forecasting Test and learn Consumer Research Direct response tracking Brand tracking Optimisation within a marketing Brand perception channel, including: Creative tuning Colour vs. B&W Target audience Ad size Segmentation Newspaper titles Competitor benchmarks Web display placement
Direct response has problems, but it’s a good step foradvertisers with a product that’s suited to being tracked • Several mechanics allow us to track response – Bespoke numbers – Direct mail – Competitions – ‘Where did you hear about us?’ • Compares within channels only – What about brand TV, or if your brand has a memorable telephone number that you don’t want to change? – Many brands – such as FMCG - have no response mechanic
Direct response data lets us optimise within press, search or other channels with response metrics• Database technology makes this type of reporting quick and relatively easy – Tool for both agencies and clients
‘Brand’ might be driving a lot of your direct activity• True cost per acquisition is a combination of brand and direct – Most advertisers don’t analyse to this depth (yet) Large numbers of search clicks can be driven by TV (Brilliant Media client example) Base Driven by TV Organic Brand CPC Organic Cheap CPC Product 2 Organic CPC Product 1 Organic CPC 0 10,000 20,000 30,000 40,000 50,000 60,000 Number of clicks
But if you’re not careful, it can all get a little complicated
There’s a simple rule of thumb that avoids a lot of brand measurement issues• If direct channels (including search) bring in sales at a cost lower than TV, then they’re making your marketing more efficient – TV generates the interest in your product, whether you run search ads to convert it, or not• But we’ll need econometrics to find out the cost per acquisition from TV
Econometrics solves some of our direct response measurement issues
WARC case studies incorporating econometrics Number of WARC case studies referencing econometrics (to October 2011) Food 146 Retail 67 Drink and beverage 54Pharmaceutical and healthcare 43 Household and domestic 39 Toiletries and cosmetics 33 Financial services 33 Telecomms 29 Leisure and entertainment 27 Travel, transport and tourism 20 Media and publishing 20 Motor and auto 18 Govt. and non-profit 16 Wearing apparel 14 Business and industrial 5 Utilities and services 4 Tobacco 1
Econometrics adds a new set of information, that we can’t get from direct response tracking alone Econometrics Return on investment Budget allocation Budget setting Sales forecasting Test and learn Consumer Research Direct response tracking Brand tracking Optimisation within a marketing Brand perception channel, including: Creative tuning Colour vs. B&W Target audience Ad size Segmentation Newspaper titles Competitor benchmarks Web display placement
Econometrics measures and then improves the effectiveness of advertisingEconometrics uses statistical models of sales to…– Measure the effectiveness (return on investment) of past advertising campaigns– Split marketing campaigns into their individual parts (TV, radio, outdoor etc.) and measure the effectiveness of each part of the marketing Press Radio mix TV Seasonality Store Openings Actual Model 250– Forecast the effectiveness of 200 future advertising campaigns 150 Sales (£ 000s) 100 50– Use forecasts to produce a 0 more effective marketing mix -50 Week 1 Week 4 Week 7 Week 10 Week 13 Week 16 Week 19 Week 22 Week 25 Week 28 Week 31 Week 34 Week 37 Week 40 Week 43 Week 46 Week 49 Week 52 Week 55 Week 58 Week 61 Week 64 Week 67 Week 70 Week 73 Week 76 Week 79 Week 82 Week 85 Week 88 Week 91 Week 94 Week 97 Week 100 Week 103 Week 106 Week 109 Week 112 Week 115 Week 118 Week 121 Week 124 Week 127 Week 130 Week 133 Week 136 Week 139 Week 142 Week 145 Week 148 Week 151 Week 154
In marketing, econometrics usually means…• Proving the effectiveness of advertising in driving sales• Measuring return on investment (ROI)• Building a mathematical model of two to three years of historical sales data• Concentrating hardest on major above the line spends• Aiming to produce a more efficient media budget allocation
A wider definition is much more useful“Econometrics is a toolbox thathelps you to test theories about your marketing ”
What’s the output?1. Measurement of past advertising campaigns, split into the different media channels that were used. Proof that past advertising added to sales and (hopefully!) was profitable Return on investment calculations showing the individual profitability of each marketing channel2. Forecasting and improvement of future campaigns The really useful bit and why it’s worth investing in econometrics We can use the model to forecast the effectiveness of potential media schedules and then choose the one with the highest returns.
How a (standard) model actually works• The maths that goes into a model is complicated… Sti Pti 1 2 (Tti Ti ) ti Si Pi• But you really don’t need to understand it, to get a feeling for how econometrics works
Sales (£ 000s) 50 0 -50 100 150 200 250 Week 1 Week 4 Week 7 TV Week 10 Week 13 Model Week 16 Week 19 Seasonality Week 22 Week 25 Week 28 Week 31 Week 34 Week 37 Week 40 Week 43 Week 46 Actual Week 49 Week 52 Week 55 Week 58 Store Openings Week 61 Week 64 Week 67 Week 70 Week 73 Week 76 Week 79 Week 82 Week 85 Week 88 Week 91 Week 94 Week 97Week 100Week 103Week 106Week 109Week 112Week 115Week 118Week 121Week 124Week 127Week 130Week 133Week 136Week 139Week 142Week 145Week 148Week 151Week 154 Once the basic model is built, we can get a first estimate for larger marketing spends Step 3:
At every stage, diagnostic statistics tell us how well the model is working• We get a lot of information from a model 1. The sales impact of each factor that we have included (ROI) 2. How sure we are that each individual measurement is accurate (confidence) 3. How sure we are that the overall model is robustR2, t and F; diagnostic statistics that only econometricians find interesting
Sales (£ 000s) 50 0 -50 100 150 200 250 Week 1 Week 4 Week 7 Week 10 Week 13 TV Week 16 Press Week 19 Model Week 22 Week 25 Seasonality Week 28 Week 31 Week 34 Week 37 Week 40 Week 43 Week 46 Week 49 Week 52 Week 55 Week 58 Week 61 Radio Actual Week 64 Week 67 Week 70 Week 73 Week 76 Store Openings Week 79 Week 82 Week 85 Week 88 Week 91 Week 94 Week 97Week 100Week 103Week 106Week 109Week 112Week 115Week 118Week 121Week 124Week 127Week 130Week 133Week 136Week 139Week 142Week 145Week 148Week 151Week 154 The final model is a good fit for sales and includes all major marketing investments Step 4:
It’s about asking the right questions • There is a ‘standard’ econometric analysis, but modelling works much better, when we set it up from the start to answer specific questions Finance are threatening to cut the ad budget; I need to prove that advertising is profitable.Can my media mix be made [Online banking] more efficient? Do I really need[this is a ‘standard’ analysis] ‘brand’ TV, or can search and DM do Should I transfer some of the job alone? my ATL budget online? [Car insurance] [High Street banking] What will it cost to hit this sales target? How much budget do I [Automotive] need for a store re-launch? [Supermarket retail]
Finding the most effective marketing mix is more than ROI measurement• Response curves measured using econometrics forecast the effect of changing marketing budgetsExample response curves TV Press 1. The most effective marketing mix 350% Additional Sales allocates budget first to TV… 300% 250% Beyond a £400k campaign, additional TV spend generates few extra sales 200% 150% 2. …and then the remaining 100% budget to press 50% 0% 100 200 300 400 500 600 700 800 900 1000 £000s
Bringing together the three elements of marketing evaluation 1. Monitoring the market – Tracking competitor activity – Benchmarking (share of voice etc.) 2. Response tracking – Immediate indicators of consumer behaviour – Web traffic, Click through, Cost per click, Call volume, store footfall and more… – Awareness & consideration tracking 3. Modelling – Filling in the gaps that can’t be measured by direct response
The flow of campaign evaluation Post Campaign Long-termPlan Campaign Analysis analysis Flow of results Econometrics In-campaign Clicks (web response) ‘tuning’ Phone #s ‘Multiplier’ adjustments Direct sales - Adjustments to direct response Measures Market Monitoring Share of voice etc. - Click path analysis Weekly / Monthly updated Direct response tracking Cost per acquisition etc. Budget setting, forecasting Daily / Weekly updated and optimisation
Sales 0.00 200.00 400.00 600.00 800.00 1,000.00 1,200.00 1,400.00 01-Oct-10 05-Oct-10 09-Oct-10 13-Oct-10 17-Oct-10 21-Oct-10 25-Oct-10 29-Oct-1002-Nov-1006-Nov-1010-Nov-1014-Nov-1018-Nov-10 National sales revenue projections22-Nov-1026-Nov-1030-Nov-1004-Dec-1008-Dec-1012-Dec-1016-Dec-10 Sales minus Central TV burst20-Dec-1024-Dec-1028-Dec-10 01-Jan-11 05-Jan-11 09-Jan-11 Actual sales (including Central TV burst) 13-Jan-11 17-Jan-11 21-Jan-11 25-Jan-11 29-Jan-1102-Feb-1106-Feb-1110-Feb-1114-Feb-11 Projected Sales if Central burst had been run nationally18-Feb-1122-Feb-1126-Feb-1102-Mar-1106-Mar-1110-Mar-1114-Mar-1118-Mar-1122-Mar-1126-Mar-1130-Mar-1103-Apr-1107-Apr-1111-Apr-1115-Apr-1119-Apr-1123-Apr-1127-Apr-11 Sales Forecasts and Projections
Year on Year Change -5% -15% -10% 0% 5% 15% 20% 25% 30% 35% 10%03-Jan-1110-Jan-1117-Jan-1124-Jan-1131-Jan-1107-Feb-1114-Feb-1121-Feb-1128-Feb-1107-Mar-1114-Mar-1121-Mar-1128-Mar-1104-Apr-1111-Apr-1118-Apr-1125-Apr-1102-May-1109-May-1116-May-1123-May-1130-May-1106-Jun-1113-Jun-1120-Jun-1127-Jun-11 04-Jul-11 11-Jul-11 18-Jul-11 25-Jul-1101-Aug-1108-Aug-1115-Aug-1122-Aug-1129-Aug-11 Strong year on year sales performance raised the question: What was going right? (Brilliant Media client example) their recent performance had been so good A Brilliant client (with econometric models) asked why
Year on year analysis provided strong evidence that the overall market was improving• Year on year is good for evidence, but it doesn’t tell you what to track. We could only draw this chart because we already knew Google searches were important Google search activity closely matched overall sales (Brilliant Media client example) Year on Year Sales (5wk MA) Google Searches for product term (non-brand) 20% Year on Year Change 15% 10% 5% 0% -5% -10% 06-Jun-11 13-Jun-11 20-Jun-11 27-Jun-11 04-Jul-11 11-Jul-11 18-Jul-11 25-Jul-11 07-Feb-11 14-Feb-11 21-Feb-11 28-Feb-11 04-Apr-11 11-Apr-11 18-Apr-11 25-Apr-11 01-Aug-11 08-Aug-11 15-Aug-11 22-Aug-11 17-Jan-11 24-Jan-11 31-Jan-11 07-Mar-11 14-Mar-11 21-Mar-11 28-Mar-11 02-May-11 09-May-11 16-May-11 23-May-11 30-May-11
What makes a ‘good’ media test?1. Clear objectives • What, exactly, are we trying to find out?2. Designed to generate a measure • What uplift do we expect that the test might generate …? • … So what scale does the test need to be for this effect to be measurable?3. Useful negative results • If the test finds no significant uplifts, are we sure the activity doesn’t work?
Why is econometrics Important?1. Controls for external factors2. Lets us measure smaller effects3. Helps to specify an appropriate scale for the test
A ‘bad’ media test… Real world example• An advertiser wanted to find the effect of a combined TV and Radio campaign on various brand preference metrics• The campaign was run over four weeks in the Central and Granada BARB regions• Pre and Post survey dips in four cities provided the awareness data – Three test cities: Manchester, Carlisle & Birmingham – One control: Norwich• 600+ respondents in the pre-campaign dip and 700+ in the post campaign• What’s wrong with that… ?
What’s wrong with that?1. No clear objective for the campaign • ‘Run a campaign and see which metric moves’ is not an ideal starting point2. No prior knowledge of whether the test is likely to be big enough to make a difference (and so be measured) • In order to generate the 10% sales uplift that we will need to get a solid measure, should the test be run for longer? Or with more GRPs? • One control city only is very, very risky3. Post campaign measurement using a simple average vs. Control • Leaves the test exposed to unforeseen events that have a different impact in the test and control regions • Econometrics gives a much better chance of a useable result
What actually happened? The test was inconclusive…Spontaneous Awareness Awareness Metric Two test cities increased. One decreased. 68% 61% 63% 57% 55% 53% 54% 52% The two that increased were in 47% 41% different BARB regions Control also increased (by more than the test regions) Inconclusive result. Manchester Birmingham Carlisle Control Overall Test Pre Post
What actually happened? The test was inconclusive… Purchase IntentPurchase intent Purchase intent fell in two test regions and fell very 43% heavily in Birmingham 36% 33% 28% 31% 31% 31% The control region declined 23% 15% Manchester stayed at 31%, bucking the decline of 8% the control region Overall, were our test regions Manchester Birmingham Carlisle Control Overall Test better? Or worse? Pre Post
A variety of companies provide econometrics and bring different strengths to the analysis Independent Semi-Independent Media AgenciesCompletely impartial Generally impartial Risk of conflict of interestLittle internal data Strong internal data Strong internal dataSmaller (riskier) Large analytical teams Mid to large analytical teamsMay not be media Can be expensive Media measurement specialistmeasurement specialist Weak ties to planning Strong ties to planningVery weak ties to planning
Automated tools can help (and sometimes reduce analysis costs) but need a lot of care• Automated tools are seductive, but we need to be aware of their limitations• Analysis is only ever an aid to decision making
Briefing an econometric analysis1. Why do you want the work done?2. What data exists and who is responsible for it?3. When is the decision deadline that the work informs?4. How will the work inform future decisions?5. Who are your project team?6. Interim meetings7. What was your marketing designed to achieve?8. everyone who will use the results needs to be involved from the start
We haven’t mentioned consumer research, but it’s the final piece of the puzzle Econometrics Return on investment Budget allocation Budget setting Sales forecasting Test and learn Consumer Research Direct response tracking Brand tracking Optimisation within a marketing Brand perception channel, including: Creative tuning Colour vs. B&W Target audience Ad size Segmentation Newspaper titles Competitor benchmarks Web display placement
ContactNeil CharlesHead of EconometricsBrilliant Media1 City SquareLeedsLS1 2FF+44(0)113 394 0078+44 (0)7508 firstname.lastname@example.org
A particular slide catching your eye?
Clipping is a handy way to collect important slides you want to go back to later.