Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The Problems and Challenges created by Observational Data

96 views

Published on

The world of market research and insights are in the process of moving from a paradigm centred on questions to one centred on observation, for example from surveys to passive data. There are several reasons for this, such as the concerns about people’s ability to describe their own motivations and the revelations of Behavioural Economics. However, the main driver of the change has been the recent growth and abundance of observational data, making its use possible as well as desirable. However, observational data brings its own problem, including those of incomplete coverage, survivor bias, perverse incentives, and problems related to underlying bias. In this presentation, Ray will outline some of the problems and challenges and propose recommendations for dealing with them.

Published in: Education
  • I pasted a website that might be helpful to you: ⇒ www.WritePaper.info ⇐ Good luck!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

The Problems and Challenges created by Observational Data

  1. 1. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection The Problems with Observa2onal Data Ray Poynter The Future Place
  2. 2. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection NewMR 2018 Sponsors Pla2num Gold Silver Communica2on
  3. 3. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Observa2onal Data •  Most big data (e.g. bank records, social media) •  Census or sample (e.g. all phones, sample with an app) •  Objec2ve or subjec2ve (e.g. receipts, ethnography) •  Structured or unstructured (e.g. phone records, images uploaded to SM) •  Behaviour or mo2va2onal (e.g. loyalty cards, facial coding) •  Naturally occurring or experiment •  Observa2onal only or with ques2ons (e.g. ad test using biometrics)
  4. 4. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Why the Change? 1.  More realisa2on of the limita2ons of asking 2.  More problems with asking – e.g. response rates and representa2on 3.  Observa2onal data more available 4.  Increased processing power 5.  From samples to census
  5. 5. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Problems?
  6. 6. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Spurious Correla2ons
  7. 7. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Underlying Bias When HRT was 1st assessed (Nurses Health Study – large observa2onal study in the USA), seemed to protect the heart. Doctors were recommended to prescribe it more widely. Women’s Health Ini2a2ve (gold standard controlled experiment) – suggested it was slightly bad for the heart. Why? Women receiving HRT were systema2cally healthier and wealthier.
  8. 8. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Missing the Real Driver 0 10 20 30 40 50 60 0 10 20 30 40 50 60 70 80 90 Time 1 Time 2 Time 3 Time 4 Time 5 Time 6 SM $ Sales $ Summer Winter Winter Ice Cream Sales
  9. 9. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection The Counterfactual 0 10 20 30 40 50 60 0 10 20 30 40 50 60 70 80 90 Time 1 Time 2 Time 3 Time 4 Time 5 Time 6 Summer Winter Winter What would have happened anyway?
  10. 10. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Measurement Error Dana Gruschwitz & Dr. Robert Schönduwe, ESRA, 2017, Lisbon, Portugal Long-standing transport study in Germany. People have been using PAPI and CAPI to capture journeys – memory based. Trial with mobiles, to automa2cally capture informa2on. Less ‘heaping’ of the distances and 2mes J But 16% fewer journeys (11% less distance, 18% fewer minutes) were recorded L Why? Phone app turned itself off when people’s phone badery reached 20%
  11. 11. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Explaining the Why
  12. 12. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Fire Hydrant Parking Fines Everywhere in upper East Side is likely to get a fine Top 2 hydrants with fines – over $55K a year between them. Why?
  13. 13. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection The Why! Discovered by Qual
  14. 14. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Observer Effects Watching / measuring behaviour can change behaviour. UK RAC study of speed cameras, 2013, found 27% reduc2on in fatal and serious collisions. Note, nobody was deliberately crashing, it was the underlying behaviour that changed. hdps://www.racfounda2on.org/media-centre/speed-camera-transparency-data
  15. 15. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Survival Bias Presiden2al Commission on Space Shudle Challenger Accident Failures at a range of temperature – assume temp does not mader
  16. 16. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Presiden2al Commission on Space Shudle Challenger Accident Failures rare at higher temperatures, but common at lower temperatures – temperature maders!
  17. 17. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Survival Bias Look at successful companies, see what they have in common, create recommenda2ons. But, aler a few years, many of the companies were failing. To measure drivers, you must look at failure and success.
  18. 18. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection U2lising Experiments Region A –  T1 sales = 100 –  T2, TV, sales = 110 –  T3, TV & Twider, sales = 130 Region B –  T1, sales 100 –  T2, Twider, sales = 110 –  T3, TV & Twider, sales = 130 Region C –  T1 sales = 100 –  T2, sales = 105 –  T3, sales = 110 The counterfactual = some growth would have happened anyway.
  19. 19. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Brent Smart, CMO IAG (Australian insurance company) – ESOMAR APAC May 2018
  20. 20. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection The Ethics of Observa2onal Data Collec2on Mis-selling?
  21. 21. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Recommenda2ons •  Talk to experts about designing and interpre2ng observa2onal data collec2on –  Weigh2ng, matching, Bayesian sta2s2cs etc •  Make predic2ons •  Iden2fy the counter factual •  Build experiments into your data collec2on •  Look for other causal paderns and explana2ons •  Add explanatory research – e.g. qual – understand the why
  22. 22. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Thank You! Follow me Twider: @RayPoynter LinkedIn: www.linkedin.com/in/raypoynter
  23. 23. The Problems with Observa2onal Data Collec2on Ray Poynter, The Future Place The Future of Data Collection Q & A Sue York NewMR Ray Poynter The Future Place

×