Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
What to Upload to SlideShare
What to Upload to SlideShare
Loading in …3
×
1 of 26

What does collecting better data mean, and how to achieve it?

0

Share

Download to read offline

Presented by Ray Poynter (NewMR & Potentiate)

Access the recording of this presentation via NewMR.org/Play-Again

Presentation Description

Ray Poynter presents a 2021 State of the Art review of the issues surrounding the collection of better data.

Ray outlines the key challenges, new initiatives, the impact of quality on decisions, and pointers to what is likely to happen in the near future.

Related Audiobooks

Free with a 30 day trial from Scribd

See all

What does collecting better data mean, and how to achieve it?

  1. 1. What does collecting ‘better data’ mean, and how to achieve it? 14 October 2021 Ray Poynter
  2. 2. Sponsors Communication
  3. 3. Agenda • The rise of Evidence-based Decision Making • The link between Better Data and ‘Good Enough’ • The link between Better Data and ‘Errors’ • Total Survey Error • Mapping Problems to Solutions • Mapping Solutions to Problems
  4. 4. “Without data, you're just another person with an opinion.” W. Edwards Deming
  5. 5. Photo by Hugo Rocha on Unsplash Photo by Nihal Demirci on Unsplash Photo by NeONBRAND on Unsplash Photo by Erik Mclean on Unsplash Good Enough
  6. 6. Improving these does not make this wagon better as a form of transport
  7. 7. Compound Errors 3 steps, A, B and C with the following errors o A with 10% error, B with 20% error, C with 50% error o Confidence = 90% * 80% * 50% = 36% Reduce the error in A by half o 95% * 80* 50% = 38% Reduce the error in C by half o 90% * 80% * 75% = 54%
  8. 8. Groves, R.; Fowler, F.; Couper, M.; Lepkowski, J.; Singer, E.; Tourangeau, R. (2009). Survey Methodology (2nd Edition). John Wiley & Sons Let’s think about TSE in terms of predicting the USA Presidential Elections 2016 and 2020 Typical model is based on 1) Who do people say they are going to vote for 2) How likely they say they are to vote 3) Weighting by demographics and the way they say they voted last time Total Survey Error
  9. 9. Total Survey Error Validity Is asking people to say which way they are going to vote a valid way of predicting the result? If I ask you to predict what you will eat on Saturday, what is the chance that it will be right? Is weighting by previous election going to work with an atypical campaign?
  10. 10. Total Survey Error Measurement Error Did people make a mistake when entering their answers? Is the scale capable of collecting the data accurately enough? Did the survey correctly display on their device, in the right language, and capture everything it should?
  11. 11. Total Survey Error Processing Error Did we spot all the bogus or flawed responses? Was the ‘likely voter’ adjustment correct? Was the weighting correct?
  12. 12. Total Survey Error Coverage Error We want a sample frame that reflects everybody who votes Online surveys reflect people who use the internet. Panel surveys reflect people who have signed up to panels. Telephone surveys reflect people with a phone who answer it.
  13. 13. Total Survey Error Sampling Error What is the risk that just by bad luck we have a sample that does not reflect the population?
  14. 14. Total Survey Error Nonresponse Error What about people who decline to take part? Busy people? Sceptical people? Evidently, many Trump supporters decline to speak to pollsters (and to other people who ring/email them).
  15. 15. Total Survey Error Adjustment Error In 2016 the weighting did not take the importance of a) not having a college education and b) being a white Christian as being important enough – both are key drivers of being pro-Trump In 2020 it looks as though one weighting error was to assume Hispanics were one group – e.g. ex- Cubans seem to be more pro-Trump
  16. 16. Applying TSE Mapping Problem to Solutions
  17. 17. Total Survey Error Validity Is asking people a direct question going to work? If not: 1. Derived answers (e.g. conjoint) 2. Projective qual 3. Observations
  18. 18. Total Survey Error Measurement Error Did people make a mistake when entering their answers? Consider 1. Avoid typing numbers, and assuming people understand percentages 2. Build redundancy or checks in to the survey 3. Probe qual answers 4. Get examples, e.g. photos or videos
  19. 19. Total Survey Error Processing Error Check for errors and bad responses in the data Recode the data to increase robustness Apply qualitative analysis methods
  20. 20. Total Survey Error Coverage Error Define the population The market? Customers? Regular customers? If you are using a panel – who are you missing? The over 70s Nat Rep? (disability, ethnicity, etc) If you are using online – who are you missing Consider multi-mode
  21. 21. Total Survey Error Sampling Error If we have a random probability sample 100 people = +/- 10% 1000 people = +/-3%
  22. 22. Total Survey Error Nonresponse Error 2 key groups 1) People who are asked take part but who decline to take part 2) People who start but do not finish This is where engagement comes into play
  23. 23. Total Survey Error Adjustment Error Dealing with errors – don’t just report what you have found Weighting the data – especially to match samples Code non-numeric data (e.g. text and images) Transcribe qual to text (enhancing the analysis)
  24. 24. Solutions Mapped to Problems Qual – when the question can’t be asked in a way that can generate numbers Gamification – reduce non-response, in some cases improve validity Multi-mode – improve coverage and reduce non-response Chatbots - reduce non-response, in some cases improve validity Video and images – reduce measurement error, increase validity Conjoint – increase validity Observational data – increase validity, reduce measurement error
  25. 25. Q & A Ray Poynter
  26. 26. Sponsors Communication

×