Data Insight Leaders Summit Barcelona 2017
•Download as PPTX, PDF•
1 like•219 views
Case Study Interactive: How To Work With Structured And Unstructured Data To Increase Customer Acquisition And Reduce Churn With Relevant Communication How can analytics improve your attribution model accuracy to highlight and transform your most successful marketing channels? How can you introduce predictive analytics to increase your customer segmentation competency? How can insights from consumer data help you to predict customer lifetime value and focus on your top customers? How can split testing consumer data help to improve your customer offering and boost retention rates?
Recommended
Recommended
More Related Content
What's hot
What's hot (13)
Similar to Data Insight Leaders Summit Barcelona 2017
Similar to Data Insight Leaders Summit Barcelona 2017 (20)
More from Harvinder Atwal
More from Harvinder Atwal (8)
Recently uploaded
Recently uploaded (20)
Data Insight Leaders Summit Barcelona 2017
- 1. Case Study Interactive: How To Work With Structured And Unstructured Data To Increase Customer Acquisition And Reduce Churn With Relevant Communication
- 2. Harvinder Atwal MoneySuperMarket.com Web dunnhumby • previous : Insight Director, Tesco Clubcard Lloyds Banking Group • previous : Senior Manager, Customer Strategy and Insight • Head of Data Strategy and Advanced Analytics @harvindersatwal British Airways • previous : Senior Operational Research Analyst {“about” : “me”} @gmail.com
- 3. 3 £1.8B SAVINGS 2016 estimate total of UK savings 1993 22M 6M MSM 14M MSE £316M 980 We started life as mortgages 2000 Adults choose to share their data with us Average monthly users 2016 Revenue 2016 Providers
- 4. How can analytics improve your attribution model accuracy to highlight and transform your most successful marketing channels? How can you introduce predictive analytics to increase your customer segmentation competency? How can insights from consumer data help you to predict customer lifetime value and focus on your top customers? How can split testing consumer data help to improve your customer offering and boost retention rates? What you wanted to know
- 5. Warning: A data-driven customer focussed strategy will not paper over cracks in operational performance or product deficiency
- 6. Unstructured data can give you important insight to prioritise
- 8. Profitably acquire customers (and acquire profitable Get
- 9. Display Ad Video Ad Social Email Search Website Physical Store TV/ Radio/ Press Outdoor $88 $132 Time Affiliates
- 10. Display Ad Video Ad Social Email Search Website Physical Store TV/ Radio/ Press Outdoor $88 $132 Time
- 11. “Last click” is still the most common approach to attribution
- 12. What are your options?
- 13. Last View Linear or Fair Share First Click Linear or Weighted Share Assumes only the “last viewed” advert, email or click counts – no earlier activities are given share of the credit. Weightings can be arbitrary and need to be constantly updated Not all interactions are equally Valuable. Not all activity can easily be counted e.g. offline Assumes only the first activity counts – no later activities are given any credit
- 14. Time decay Frequency and Recency Markov Chain Model or Bayesian Networks Positional or U model Assumes recently viewed advert, email or click counts more – earlier activities is given less share of the credit. Weightings can be arbitrary and need to be constantly updated Ignores the customer path Requires comprehensive tracking, fooled by correlations and doesn’t take into account brand equity
- 15. Determine possible influences sequentially and gather data
- 16. Add variation
- 17. Display Ad Video Ad Social Email Search Website Physical Store TV/ Radio/ Press Outdoor $88 $132 Time Measure the direct effect What is the impact of Outdoor advertising on sales?
- 18. Display Ad Video Ad Social Email Search Website Physical Store TV/ Radio/ Press Outdoor $88 $132 Time Measure the indirect effects too! Use nested models
- 19. There are many econometric techniques to measure outcomes Regression Discontinuity Design Controlled Regression Fixed Effects Regression Difference- in- Differences Instrumental Variables
- 20. Google’s Causal Impact package is great for analysing Difference-in-differences
- 21. Display Ad Video Ad Social Email Search Website Physical Store TV/ Radio/ Press Outdoor $88 $132 Time Repeatedly iterate and model. You can then apply weightings What is the impact of TV/Display, Video, Social… spend on sales?
- 22. Grow
- 23. What are all the ways you could communicate to a stadium full of customers?
- 24. What if you could walk up to ANYONE in the stadium and have a conversation knowing their individual needs and preferences?
- 25. It’s 2017, nobody should be asked how they want to be treated
- 26. Predictive modelling can help you treat different customers differently
- 27. Think beyond products, demographics and loyalty
- 28. Think actionable needs, preferences and states Borrowing Saving Risk averse Price-sensitive Brand conscious Financially Cautious Financially confident Time poor
- 29. Exercise: What are some of the actionable Customer needs, preferences and states for your organisations?
- 30. Time of day responsiveness Day of week responsiveness Device preference Marketing channel preference Offer responsiveness Help preferences Social proof/review responsiveness
- 31. Traditional propensity modelling and recommendation engine techniques can help you if you have past outcome data Customer Data Model Highest probability +
- 32. You can also think of customer history as a sequence and predict using Deep Learning Email Open Page view Product click Product click Time Customer history Sale? Future period RNN Cell RNN Cell RNN Cell RNN Cell Prediction
- 33. No Ground Truth? No Problem
- 34. Show pictures of cats Show pictures of dogs Show pictures of People (control) Test treatments at random Conversion = 5% Conversion = 3% Conversion = 3% But we’re not interested in which treatment works best on average
- 35. Find the best treatment for each customer Total Customers (100% of customers) (3% conversion) Live alone (30% of customers) (4% conversion) Don’t live alone (70% of customers) (2.6% conversion) Urban (56% of customers) (2.7% conversion) Rural (14% of customers) (2.1% conversion) Live in apartment (9% of customers) (4% conversion) Live in house (21% of customers) (4% conversion) Cat conversion = 18% Cat conversion = 1% Cat conversion = 5.4% Cat conversion = 1% Dog conversion = 2% Dog conversion = 2% Dog conversion = 2.5% Dog conversion = 7% Cat segment People segment Dog segmentCat segment Total segmented conversion = 6.5% vs 4% for best treatment on average (Cat pictures for all)
- 36. A finite number of predictive micro- segmentations can be combined to create highly personalised individual experiences
- 37. Test & Collect Model Embed Roll Out Feedback Plan Pilot test Collect Data Build Model Identify segments Adjust model to fit organisation Re-engineer business processes to support segmented execution Train organisation Incorporate segments into daily execution Provide differentiated services, products and content
- 38. Keep Retain Profitable customers longer Win Back profitable customers Eliminate unprofitable customers
- 39. Traditional techniques like RFM and Pareto-NBD omit many factors influencing Customer Lifetime Value Contribution Time Buys second product Complaint Loss Leader High Servicing costs Complaint resolution Subscription revenues
- 40. Training Features Random Forest Regression can create more accurate CLV predictions Training period Model Test period Training period Prediction period Time Product Purchases VisitsSpend Demographics Acquisition channel Complaints Future period Location Historic period Segmenting models may improve accuracy further User BehaviourShipping preferences Payment preferences Costs
- 41. Beware of survivorship bias when calculating lifetime value!
- 42. Measure cohorts not snapshots
- 44. Most Growable Customers Super Growth Customers Low Maintenance Customers Most Valuable Customers Actual Value (CLV) Low High Low High Potential Value Below Zero Customers CLV is more powerful when combined with potential value
- 45. A-B (Split) testing is an effective way to boost revenue and retention when you don’t have existing data to model
- 46. Know why you’re testing
- 47. Do not spend time AB testing small cosmetic details Simple UI changes are ineffective. Colour (changing the colour of elements on a website) +0.0% uplift Buttons (modifying website buttons) - 0.2% uplift Calls to action (changing the wording on a website to be more suggestive) - 0.3% uplift Best test categories are: Scarcity (stock pointers) +2.9% uplift Social proof (informing users of others’ behaviour) +2.3% uplift Urgency (countdown timers) +1.5% uplift Abandonment recovery (messaging to keep users on-site) +1.1% uplift Product recommendations (suggesting other products to purchase) +0.4% uplift Qubit meta-analysis of 6,700 experiments (2017)
- 50. You’re testing promotion of a new product in an email campaign What is the target variable? C) Revenue per customer B) Sales of the product A) Click-through on the email
- 51. You’re testing an outbound telesales campaign What is the unit of measurement for the target variable (sales)? A) A call C) A telesales agent B) A customer
- 52. A null hypothesis H 0 ('no effect') is tested against an alternative hypothesis H 1 ('some effect'). The results pass a test of statistical significance (P-value <0.05) in favour of H 1. What’s been shown? 1. H 0 is false. 2. H 1 is true. 3. H 0 is probably false. 4. H 1 is probably true. 5. Both (1) and (2). 6. Both (3) and (4). 7. None of the above.
- 54. Before you go anywhere near data you need to do Situational Analysis
- 55. Run experiments
- 56. …a lot of them
- 57. They are fuel for your models
- 58. Eat the elephant one bite at a time
- 59. It’s still possible to measure even if you can’t employ the gold standard of randomised control trials
- 60. Be clear about your objectives and metrics Avoid Vanity Metrics!
- 62. Beware of analytical traps
Editor's Notes
- Enough about me and MoneySuperMarket. From the research these are the questions you wanted answered.
- Last view Using this method, it’s not the click that counts but the last advert viewed
- It’s important to have a vision of what you’re trying to achieve
- Predictive Analytics is moving us from a world of having to Explicitly state our needs to having them Implicitly fulfilled
- There’s no single segmentation Use predictive analytics to understand actionable customer needs and preferences, without having to be told, so you can customise their experience
- During World War II, the statistician Abraham Wald took survivorship bias into his calculations when considering how to minimize bomber losses to enemy fire. Researchers from the Centre for Naval Analyses had conducted a study of the damage done to aircraft that had returned from missions, and had recommended that armour be added to the areas that showed the most damage. Wald noted that the study only considered the aircraft that had survived their missions—the bombers that had been shot down were not present for the damage assessment. The holes in the returning aircraft, then, represented areas where a bomber could take damage and still return home safely. Wald proposed that the Navy instead reinforce the areas where the returning aircraft were unscathed, since those were the areas that, if hit, would cause the plane to be lost
- A/B Testing Summary: In an A/B test, a change is applied to a treatment group, and its performance is compared against a control group to estimate the impact of the change. STEP 1: SELECT A PERFORMANCE METRIC It’s important to understand the metric used to evaluate the results of the test. Whether the goal is to increase sales, profit, conversion rate, etc., this should be specified at the upfront. STEP 2: SELECT THE EXPERIMENT DESIGN Matched pair - when the sample size is small and/or the data is difficult to collect, a matched pair experiment should be used. Randomized design - when the sample size is large and the data is easy to collect, then a randomized experiment should be used. Randomized experiments are very common for web-based AB tests. STEP 3: SELECT TREATMENT AND CONTROL UNITS Each individual in the test is considered a unit. The unit can be a person, store, etc. In a test, units are split into two groups, the treatment group and control group. Treatment and control units are compared against each other STEP 4: SELECT EXPERIMENTAL AND CONTROL VARIABLES Experimental variable - The experimental, or treatment, variable(s) is the variable that is different between treatment and control units. For example, if you are testing a new price point, the experimental variable would be price. Control Variables - The control variables are the variables that should remain constant between test and control groups. These variables ensure that the treatment and control groups are representative of each other and that the results will apply to the population. Control variables are used to match each treatment unit to one or more control units. STEP 5: DETERMINE TEST DURATION AND SAMPLE SIZE These two go hand in hand and contribute most directly to statistical significance. You can improve statistical significance by either increasing the sample size or test duration. Generally the duration of a test should be at least as long enough to capture a representative sample. STEP 6: RUN THE TEST AND PREPARE THE DATA Now it’s time to run the test and collect the data. Preparing the data includes filtering for the dates of the test, ensuring there are no duplicate records, removing records with incomplete data, and removing outliers. STEP 7: ANALYZE RESULTS Lift - Compare the average performance between the two groups. It can also be useful to understand the distribution of the performance of the units. Statistical Significance - Performing a t-test provides a p-value. P-values below 0.05, indicate statistically significant results. Paired t-test are used for matched pair experiments and unpaired t-test for randomized experiments. Impact Estimation - In order to provide an expected impact of broad implementation of the treatment, apply the lift calculation to the entire population.