Disney: The Magic of Marketing Mix Analytics & Optimization
Disney Marketing ROI Case Study DMA Conference Presented by Disney & SAS October 2012
Defining A Marketing ROI Solution Reach the right audience Through the right channel Maximize Return on investment for At the right time marketing spend With the right frequency At the right price Stand-alone studies often fail to achieve long-term success—trying to implement a project instead of a process!
Presentation Agenda • Introduction – Disney Management Science & Integration – SAS – The Science Behind Marketing ROI • Case Study Overview – Project Goals & Organization – Data Management – Science Integration – Tool Development • Lessons Learned • Questions & Answers
Disney Management Science and Integration 4 employees - 2008 30 employees - 2012 • Consulting support for analytics, data and reporting needs • Technology integration for reporting and data tools • Development and management of decision science tools
SAS® Company Overview SAS® is the largest independent software vendor in the world SAS Annual Revenue 1976-2011• 2011 & 2010 Fortune Magazine: #1 Place to Work• 2011 Revenue: $2.73 billion• SAS® reinvests ~25% of annual revenue into R&D• 90 of top 100 companies on FORTUNE Global 500® use SAS®
Science Behind Marketing ROI – Modeling Marketing Effort Response Measurement Model For Each Channel vs. Variable (spend, impressions, etc.) (sales, leads, etc.) More TV effective Sales Less Radio effective Marketing Spend
Science Behind Marketing ROI – The “Right” Model Selecting the right modeling approach is critical for success! Regression / Time Series Model Econometric / Panel Model R2 = 97% R2 = 67% Sales (t) = … Sales (t) = … +0.7 * Sales (t-1) +0.2 * Sales (t-1) -0.2 * Price -1.0 * Price +0.06 * TV +0.1 * TV -0.005 * Online +0.02 * Online +… +… Heavy weight on lagged sales; sales not Less weight on lagged sales; price & responsive to price & media changes media elasticities more reasonable Better for FORECASTING Better for MEASUREMENT
Science Behind Marketing ROI – Measurement Analysts pay careful attention to data considerations and choice of models to robustly fit the data for measurement Impressions by Saturation Media Type Curves Goodwill Cable Impressions Print Ratings Ratings Model Radio Time Spend Time Model Input Model Output
Science Behind Marketing ROI – Optimization Planners leverage model output and their insights to adjust and optimize marketing plans per business constraints Impressions by Saturation Media Type Curves Goodwill Cable Impressions Print Ratings Ratings Model OptimalRadio Optimal Media Mix Flighting Radio Cable Time Spend Time Impressions Print Ratings Spend Radio Spend Time
Case Study Overview A television network is seeking decision science support to improve return on investment for the marketing of primetime television shows • How effective is our current marketing spend? • Which shows should get more marketing dollars? • Which channels are the most effective? Most efficient? • Based on current practices, where are we over-saturated?
Case Study Challenges Previous attempts to answer these questions have yielded valuable insights, but have not created sustained changes• Avoid the temptation to answer all questions with a single model• Ensure inputs into the solution are readily available and cost effective• Avoid bundling decisions that are controlled by separate teams Limited data availability prevents the network from getting accurate measures of performance for marketing efforts• Data is warehoused in multiple systems, with few connection points• Impression-level data is extremely difficult to capture, with actualized data existing in combinations of spreadsheets, e-mails, and faxes• Given the state of the data, common reports can take days to generate
Disney and SAS® Partnership Project Management 15% 15% Data Management 30% 15% Science Integration 30% 30% Tool Development 25% 40%
Project Timeline Established a separate timeline for each work stream, inclusive of milestone and reports out to key stakeholders
Data Collection Overview Data collection ultimately took four times longer than originally planned, due in large part to data quality issues• Identified over 30 potential data sources and almost 250 variables• Data sources ranged from databases, spreadsheets, e-mails, and faxes• Established weekly meetings with key stakeholders and implemented dashboards to review data collection progress• Placed an analyst in the media agency office for four weeks to speed data collection and improve understanding of the data Data collection is never really over—continue to find errors or missed opportunities even months later!
Data Collection Challenges Model database changed 17 times during a 1-year span, most often due to missing data or data collection errors Bad circulation Magazine Cume based estimate for on all publications Entertainment instead of purchased Weekly Nielsen P3 vs. C3 Duplication from Misclassified OOH SQL Errors support as Events “Week 53” Issue
Data Visualization Showing clients the relationship between impressions and costs helped to identify likely errors in the data (e.g., misclassification of spending)
Data Visualization (cont.) Exploring flights enabled us to recognize the need to model certain media types differently than others 15% 70% 15%
Data Transformation Often necessary to transform the data for measurement variables in our models to avoid creating misleading insights or recommendations Episode Promos in Promos in Air Date Calendar Week Past 7 Days Sunday S M T W R F S S Monday S M T W R F S S M Tuesday S M T W R F S S M T Wednesday S M T W R F S S M T W Thursday S M T W R F S S M T W R Friday S M T W R F S S M T W R F Saturday S M T W R F S S M T W R F S Transform to a full week
Data Handoff to Science Key milestone was the go/no-go decision on beginning the development of the measurement model NIELSEN PROMOS & MARKETING AWARENESS Program Name On-Air Promos Survey Respondents TRPs, Seconds, # of Spots Air Date Digital Aware Respondents Impressions & Clicks Start Time Cinema % Aware Impressions, Seconds Per Spot Unaided & Aided Duration National Cable Intent to View TRPs Top Box, Top 2 Box, Non Committed, Bottom Box Program Type Newspaper Impressions & Circulation Program Rating Magazine Total & Weekly Impressions Lead-in Rating Spot Cable TRPs & Impressions Competition Spot Radio TRPs OOH Impressions
Data Handoff to Science (cont.) Future iterations of the model will incorporate new data that is either unavailable right now or represents a higher level of complexity MISSING DATA MISSING COMPLEXITY DATA RECONCILIATION Network Radio On-Air Promos On-Air Promos Day-of-Week, Promo Length Synergy Cable Nielsen Digital Impressions Reach, Share, HUT, PUT Synergy Online Print Size, Placement, Inserts Emails & Newsletters National Cable Channel, # of Spots, Promo Length MODEL EXPANSION Public Relations Spot Cable & Radio # of Spots, Seconds of Promo Geo-Panel Data Local Market Ratings and Marketing Affiliate Promotions OOH # of Units, Size, Media Form On-Air Promo Precision Minute-by-Minute Ratings Digital Size, Placement, Pillar Efficiency Costs for Marketing & Promotions Social Media Facebook, Twitter, Blog Mentions
Science Integration Integration between the team managing data collection and model development is critical to the success of the projectScience Data When it doesn’t work well—each revision of the data model would delay the science timeline by 3 weeks! Critical to integrate science team with tool developers to ensure alignment with the expected input and outputs of the modelsScience Tool
Overview of Planning & Optimization Tool The tool is designed to become self-sustaining to support updates to the measurement model and to allow media plan comparisonsHistorical Data Measurement Data Model Model Model (one time) Adjustments Actualized Optimization Goals & Media Plans Model Constraints Agency Approved Recommended Media Media Plans Media Plans Plans
Optimization Goals Objective is to maximize total ratings for the premiere episodes of all shows within a marketing campaign portfolio• Provide recommended spending by channel for each show/week combination• Allow users to input constraints on total spending by show/channel/week• Define spend thresholds that reflect minimum purchase amounts for each channel• Compare optimal recommendations against manually created plans Critical to understand relationship between spend and impressions; some channels have a significant delay between purchase and delivery!
Evaluating Media Plans Ability to compare different plans by measuring the number of new households generated for each incremental unit of spendRecommended Plan: Week Cable Radio Print Outdoor Cinema(balanced by optimization) t = -5 20 N/A 20 20 20 t = -4 20 N/A 20 20 20 t = -3 20 N/A 20 20 20 t = -2 20 20 20 20 20 t = -1 20 20 20 20 20 t= 0 20 20 20 20 20Media Agency Plan: Week Cable Radio Print Outdoor Cinema(incremental opportunities) t = -5 70 N/A 80 110 10 t = -4 105 N/A 5 170 4 15 t = -3 160 N/A 5 3 5 30 t = -2 240 5 5 10 80 t = -1 355 1 75 2 25 30 125 t= 0 25 10 50 50 150 5
Key Lessons Learned Designing a Creating Clear Structured QA Requirements Process & Team Having a Test “Shadow” Environment Implementation