Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Survey Octopus Uni of Auckland cjforms

1,506 views

Published on

The Survey Octopus: An approach to teaching Total Survey Error. Seminar at the University of Auckland, New Zealand, by Caroline Jarrett.

Abstract: Although the concepts of Total Survey Error (TSE) have been widely accepted amongst survey methodologists and statisticians for many years, TSE is still not familiar to many people who commission ad-hoc surveys for business or government. A recent survey (Jablonski, 2015) suggests that even some academics teaching survey methodology at university level do not use TSE in their classes; some were not even aware of the concept.

After having many conversations with colleagues and clients where I tried to explain that a high number of responses was in itself not a guarantee of data quality, I turned to the classic Survey Lifecycle (Groves et al, 2009), with its presentation of Total Survey Error arising from errors across steps in the survey process. From this, I evolved the Survey Octopus, a way of helping non-specialists to get to grips with the issues involved in TSE and to help them to make better, and more informed, choices when deciding about how to approach a survey.

In this seminar, I will describe the Survey Octopus, show how it relates to Groves et als' model of the survey lifecycle, and explain the benefits and problems of each depiction of TSE.

References:
Groves, Robert M., Floyd J. Fowler, Mick P. Couper, James. M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2009. Survey methodology. 2nd ed, Wiley series in survey methodology. Hoboken, N.J.: Wiley.

Jablonski, Wojciech. 2015. "Survey Methodology Courses and TSE/Big Data Issues. Classroom Experiences Among University Instructors " International Total Survey Error Conference, Baltimore, Maryland.

Published in: Data & Analytics
  • Be the first to comment

Survey Octopus Uni of Auckland cjforms

  1. 1. The Survey Octopus Caroline Jarrett @cjforms Seminar, University of Auckland An approach to teaching Total Survey Error
  2. 2. Is there anyone here who… • Is a survey methodologist? • Teaches classes in survey methodology? • Uses surveys in research? • Responds to surveys? • Eats octopus? 2
  3. 3. I’m a forms specialist 3
  4. 4. Why do people answer questions? 4Caroline Jarrett @cjforms (CC) BY SA-4.0
  5. 5. People ask me about surveys “Please have a look at this survey” “How many people do I need in my sample?” “Tell me whether this is a good question” 5
  6. 6. “Please have a look at this survey?” 6 Kill survey! Kill! Kill!
  7. 7. 7
  8. 8. 8
  9. 9. Bad surveys are training us to ignore all surveys 9
  10. 10. “Please have a look at this survey?” 10 Let’s chat about a few of the issues
  11. 11. Some questions about TSE • Do you teach it? • Do you use it? 11
  12. 12. Survey Methodology Courses and TSE/Big Data Issues: Classroom Experiences Among University Instructors Wojciech Jablonski University of Lodz | Poland TSE15 | Baltimore, MD, USA | 22 September 2015
  13. 13. TSE CONCEPT 1% 36% 44% 19% In what way have you used the Total Survey Error concept during you survey methods classes? TSE is a framework of my survey methods classes I have discussed the TSE concept but it is not a framework of my survey methods classes I have not dicsussed the TSE concept during my survey methods classes Unable to say/Other answers Jablonksi, W. (2015) Survey Methodology Courses and TSE/Big Data Issues, TSE15 Baltimore
  14. 14. 14
  15. 15. Survey Methodology. Robert M Groves (ed) 15 Caroline Jarrett @cjforms (CC) BY SA-4.0
  16. 16. 16 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  17. 17. A conversation about measurement 17
  18. 18. 18
  19. 19. GDS19
  20. 20. March 2014 version of https://www.gov.uk/service-manual/measurement/user-satisfaction.html
  21. 21. 21 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  22. 22. 22 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct First I tried removing all the errors Caroline Jarrett @cjforms (CC) BY SA-4.0
  23. 23. Then I tried to approximate the technical terms with plainer ones ConstructWhat you want to ask about Measurement The questions you ask ResponseThe answers you get Edited response The answers you use Representation Who you want to ask Sampling frame The list you use to sample from Sample The ones you ask Respondents The ones who answer Post-survey adjustments The ones whose answers you can use Survey statisticThe number Caroline Jarrett @cjforms (CC) BY SA-4.0 23
  24. 24. I used this diagram to discuss satisfaction with colleagues The number The sample you use The ones who answer The sample you ask The list you use to sample from Who you want to find out from Who The answers you use The answers you get The questions you ask What you want to find out about What Caroline Jarrett @cjforms (CC) BY SA-4.0 24
  25. 25. 25 Services so good that people prefer to use them Satisfaction Overall, how satisfied were you with renewing your car tax today? Scale of 5 points from very satisfied to very dissatisfied Ratings weighted from 100% for 'very satisfied' to 0% for 'very dissatisfied' UK population People who renew car tax online Everyone who finishes the renewal People who answer the question 94% The number (Any exclusions for repeated answers or test answers?) The sample you use The ones who answer The sample you ask Everyone who finishes the renewal The list you use to sample from Who you want to find out from Who The answers you use The answers you get The questions you ask What you want to find out about What Satisfaction with “renew tax disc” Caroline Jarrett @cjforms (CC) BY SA-4.0
  26. 26. 26
  27. 27. 27 We were only asking successful users if they were satisfied We were asking users to give feedback from the Done page – a page you could only reach if you completed using the service successfully. This was a good way to make our services look really good, but a terrible way to get the information needed to improve them. We were missing out on the most important feedback – from the users who failed to complete the transaction or otherwise got stuck. We’ve changed the guidance in the Service Manual. Now we’re asking for feedback in many more places. https://designnotes.blog.gov.uk/2015/08/13/how-good-is-your-service-how-many-users-give-up/
  28. 28. 28
  29. 29. 29 Measure satisfaction across the whole service More often than not, the end of the transaction isn’t the end of the service. For example, if you claim Carer’s Allowance, the end of the transaction means you’ve finished filling in your claim. You’re still waiting for a decision. You must prompt users to give feedback at service endpoints. In the example above, this means prompting the user to give feedback when they get their benefit decision. This could be some time after they finish their transaction – the prompt could be in an email or a letter. You must also allow users to give feedback from anywhere in the service, in case anything goes wrong. And you must be able to show that you’re collecting user satisfaction data appropriately at your Service Assessment. https://www.gov.uk/service-manual/measurement/user-satisfaction.html
  30. 30. Some other chats about surveys 30
  31. 31. My ‘plain language’ diagram was OK, but not that different from many others 31
  32. 32. 32 “You need a memory hook” Christine Elgood @elgoodgames
  33. 33. 33 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
  34. 34. 34 “How many people do I need in my sample?”
  35. 35. To work that out, let’s visit the Octopus 35 Caroline Jarrett @cjforms (CC) BY SA-4.0
  36. 36. Start with how many will answer 36 Caroline Jarrett @cjforms (CC) BY SA-4.0 Fieldwork: Who answers?
  37. 37. Whether they’ll answer depends on effort 37 Questions: What are you asking about? How many questions? Caroline Jarrett @cjforms (CC) BY SA-4.0
  38. 38. And on the reward you’re offering Goals and resources: Why are you asking? Is helping you a reward in itself? Are you offering any other incentive? 38 Caroline Jarrett @cjforms (CC) BY SA-4.0
  39. 39. Then there’s the ‘Justin Bieber North Korea’ problem 39 http://www.bbc.com/news/10506482
  40. 40. If we ask ‘anyone’, we’ll have extra work here Caroline Jarrett @cjforms (CC) BY SA-4.0 Response: Whose answers can we use?
  41. 41. So it matters where we get our sample 41 Caroline Jarrett @cjforms (CC) BY SA-4.0 Sample: the list you sample from
  42. 42. 42 And now it’s easy to work out how many to ask Sample: the number of people to ask Caroline Jarrett @cjforms (CC) BY SA-4.0
  43. 43. We thought about a lot of topics to work that out Goals Sample Questions Fieldwork 43 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response
  44. 44. What about this bit? Caroline Jarrett @cjforms (CC) BY SA-4.0
  45. 45. 45 “Is this a good question?
  46. 46. 46 Caroline Jarrett @cjforms (CC) BY SA-4.0
  47. 47. 47 "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett
  48. 48. A good question gets good answers Caroline Jarrett @cjforms (CC) BY SA-4.0 Response: Is the question easy to answer?
  49. 49. Good answers help you to make decisions Caroline Jarrett @cjforms (CC) BY SA-4.0 Insight: Is the answer useful?
  50. 50. All the topics are connected Goals Sample Questions Fieldwork 50 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Insight Response
  51. 51. I tried the Survey Octopus at the Content Strategy Summit 51
  52. 52. 52
  53. 53. People want a process, too 53
  54. 54. 54 Questions Response Fieldwork Sample Goals Response Insight Questionnaires Caroline Jarrett @cjforms (CC) BY SA-4.0
  55. 55. My survey process has 6 steps 55 Caroline Jarrett @cjforms (CC) BY SA-4.0
  56. 56. A survey is a quantitative method 56
  57. 57. 57 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
  58. 58. 58 The aim of a survey is to get a number that helps you to make a decision Caroline Jarrett @cjforms (CC) BY SA-4.0
  59. 59. Some options for talking about Total Survey Error 59
  60. 60. 1. As a Survey Octopus. 60
  61. 61. 61 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
  62. 62. 2. As a more conventional diagram 62
  63. 63. The aim is to get the best number you can, within the resources you have What you want to ask about The resources you have The questions you ask The answers you get The answers you use The number Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use 63 Caroline Jarrett @cjforms (CC) BY SA-4.0
  64. 64. 3. As a link to the terms used by survey methodologists (searchable) 64
  65. 65. 65 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best number you can, within the resources you have What you want to ask about The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use The number Caroline Jarrett @cjforms (CC) BY SA-4.0 ResourcesThe resources you have
  66. 66. 4. With all the errors in it 66
  67. 67. 67 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best statistic you can, within the resources you have Measurement error Processing error Coverage error Sampling error Nonresponse error Adjustment error Caroline Jarrett @cjforms (CC) BY SA-4.0 (Lack of) Validity
  68. 68. 68 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
  69. 69. Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 69 Caroline Jarrett @cjforms (CC) BY SA-4.0

×