Survey Octopus TSE 2015 @cjforms
•Download as PPTX, PDF•
3 likes•1,385 views
Total Survey Error for non-specialists: creating better conversations. A presentation of the Survey Octopus at the TSE2015 conference in Baltimore, September 2015.
Recommended
Recommended
More Related Content
What's hot
What's hot (11)
Similar to Survey Octopus TSE 2015 @cjforms
Similar to Survey Octopus TSE 2015 @cjforms (20)
More from Caroline Jarrett
More from Caroline Jarrett (13)
Recently uploaded
Recently uploaded (20)
Survey Octopus TSE 2015 @cjforms
- 1. Total Survey Error for non-specialists Caroline Jarrett @cjforms Total Survey Error Conference Baltimore, September 2015
- 2. I’m a forms specialist 2
- 3. A conversation about measurement 3
- 4. 4
- 5. GDS5
- 6. March 2014 version of https://www.gov.uk/service-manual/measurement/user-satisfaction.html
- 7. 7 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
- 8. 8 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct First I tried removing all the errors Caroline Jarrett @cjforms (CC) BY SA-4.0
- 9. Then I tried to approximate the technical terms with plainer ones ConstructWhat you want to ask about Measurement The questions you ask ResponseThe answers you get Edited response The answers you use Representation Who you want to ask Sampling frame The list you use to sample from Sample The ones you ask Respondents The ones who answer Post-survey adjustments The ones whose answers you can use Survey statisticThe number Caroline Jarrett @cjforms (CC) BY SA-4.0 9
- 10. I used this diagram to discuss satisfaction with colleagues The number The sample you use The ones who answer The sample you ask The list you use to sample from Who you want to find out from Who The answers you use The answers you get The questions you ask What you want to find out about What Caroline Jarrett @cjforms (CC) BY SA-4.0 10
- 11. 11 Services so good that people prefer to use them Satisfaction Overall, how satisfied were you with renewing your car tax today? Scale of 5 points from very satisfied to very dissatisfied Ratings weighted from 100% for 'very satisfied' to 0% for 'very dissatisfied' UK population People who renew car tax online Everyone who finishes the renewal People who answer the question 94% The number (Any exclusions for repeated answers or test answers?) The sample you use The ones who answer The sample you ask Everyone who finishes the renewal The list you use to sample from Who you want to find out from Who The answers you use The answers you get The questions you ask What you want to find out about What Satisfaction with “renew tax disc” Caroline Jarrett @cjforms (CC) BY SA-4.0
- 12. 12
- 13. 13 We were only asking successful users if they were satisfied We were asking users to give feedback from the Done page – a page you could only reach if you completed using the service successfully. This was a good way to make our services look really good, but a terrible way to get the information needed to improve them. We were missing out on the most important feedback – from the users who failed to complete the transaction or otherwise got stuck. We’ve changed the guidance in the Service Manual. Now we’re asking for feedback in many more places. https://designnotes.blog.gov.uk/2015/08/13/how-good-is-your-service-how-many-users-give-up/
- 14. 14
- 15. 15 Measure satisfaction across the whole service More often than not, the end of the transaction isn’t the end of the service. For example, if you claim Carer’s Allowance, the end of the transaction means you’ve finished filling in your claim. You’re still waiting for a decision. You must prompt users to give feedback at service endpoints. In the example above, this means prompting the user to give feedback when they get their benefit decision. This could be some time after they finish their transaction – the prompt could be in an email or a letter. You must also allow users to give feedback from anywhere in the service, in case anything goes wrong. And you must be able to show that you’re collecting user satisfaction data appropriately at your Service Assessment. https://www.gov.uk/service-manual/measurement/user-satisfaction.html
- 16. Some other chats about surveys 16
- 17. People ask me about surveys “I think it’s best to have 5 points in my rating scale, but my boss wants to have 7. Who is right?” “What’s the best survey tool?” “How many people do we need to ask for this survey?” “Please have a look at these questions” 17 Caroline Jarrett @cjforms (CC) BY SA-4.0
- 18. My ‘plain language’ diagram was OK, but not that different from many others 18
- 19. 19 “You need a memory hook” Christine Elgood @elgoodgames
- 20. 20 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
- 21. 21 Caroline Jarrett @cjforms (CC) BY SA-4.0
- 22. 22 "Phone photography" by Petar Milošević - Own work. Licensed under CC BY-SA 3.0 via Commons - https://commons.wikimedia.org/wiki/File:Pho ne_photography.jpg#/media/File:Phone_phot ography.jpg Modified by Caroline Jarrett
- 23. You need questions that your respondents can answer accurately Questions 23 Caroline Jarrett @cjforms (CC) BY SA-4.0 Response Fieldwork Sample Goals Response Insight
- 24. I tried the Survey Octopus at the Content Strategy Summit 24
- 25. 25
- 26. People want a process, too 26
- 27. 27 Questions Response Fieldwork Sample Goals Response Insight Questionnaires Caroline Jarrett @cjforms (CC) BY SA-4.0
- 28. My survey process has 6 steps 28 Caroline Jarrett @cjforms (CC) BY SA-4.0
- 29. A survey is a quantitative method 29
- 30. 30 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
- 31. 31 The aim of a survey is to get a number that helps you to make a decision Caroline Jarrett @cjforms (CC) BY SA-4.0
- 32. Some options for talking about Total Survey Error 32
- 33. 1. As a Survey Octopus. 33
- 34. 34 To get better results from your survey, think about the Survey Octopus Caroline Jarrett @cjforms (CC) BY SA-4.0
- 35. 2. As a more conventional diagram 35
- 36. The aim is to get the best number you can, within the resources you have What you want to ask about The resources you have The questions you ask The answers you get The answers you use The number Who you want to ask The list that you sample from The sample you ask The ones who answer The ones whose answers you can use 36 Caroline Jarrett @cjforms (CC) BY SA-4.0
- 37. 3. As a link to the terms used by survey methodologists (searchable) 37
- 38. 38 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best number you can, within the resources you have What you want to ask about The questions you ask The answers you get The answers you use Who you want to ask The list you use to sample from The ones you ask The ones who answer The ones whose answers you can use The number Caroline Jarrett @cjforms (CC) BY SA-4.0 ResourcesThe resources you have
- 39. 4. With all the errors in it 39
- 40. 40 Survey statistic Post-survey adjustments Respondents Sample Sampling frame Representation Edited response Response Measurement Construct The aim is to get the best statistic you can, within the resources you have Measurement error Processing error Coverage error Sampling error Nonresponse error Adjustment error Caroline Jarrett @cjforms (CC) BY SA-4.0 (Lack of) Validity
- 41. 41 Total Survey Error diagram as presented in Groves, R. M., F. J. Fowler, M. P. Couper, J. M. Lepkowski, E. Singer and R. Tourangeau (2009). Survey methodology. Hoboken, N.J., Wiley.
- 42. Caroline Jarrett Twitter @cjforms http://www.slideshare.net/cjforms carolinej@effortmark.co.uk 42 Caroline Jarrett @cjforms (CC) BY SA-4.0
Editor's Notes
- Start of a blog post for Government Digital Service: "My new favourite form. Really" https://gds.blog.gov.uk/2015/09/22/my-new-favourite-form-really/
- A page from gov.uk showing the performance of its online tax renewal service - including a figure of 94 percent for user satisfaction. We need to understand total survey error to assess whether this figure is accurate or useful in assessing true performance.
- Excerpt from the GDS Service Design Manual page on User Satisfaction https://www.gov.uk/service-manual/measurement/user-satisfaction.html As it was in March 2014. This excerpt reads: A good service makes it possible for users to successfully complete tasks. User satisfaction helps you to measure the overall quality of the service experience. Many government transactions are mandatory for users, so they’re not naturally enjoyable, and are sometimes referred to as grudge transactions. Still, try to make them as pleasant as possible for users, who may be nervous or stressed when interacting with the government. Asking users how satisfied they are with a service can provide you with a measure of all the parts that contribute to the overall user experience, like ease of use, navigation and design. What you will be measuring Define user satisfaction as the percentage of people who answered either “very satisfied” or “satisfied” on a 5-point scale in response to this question: Q: Overall, how satisfied were you with this [eg car tax] service today? very satisfied satisfied neither satisfied or dissatisfied dissatisfied very dissatisfied
- Further excerpt from https://www.gov.uk/service-manual/measurement/user-satisfaction.html As it was in March 2014 Asking users how satisfied they are with a service can provide you with a measure of all the parts that contribute to the overall user experience, like ease of use, navigation and design. What you will be measuring Define user satisfaction as the percentage of people who answered either “very satisfied” or “satisfied” on a 5-point scale in response to this question: Q: Overall, how satisfied were you with this [eg car tax] service today? very satisfied satisfied neither satisfied or dissatisfied dissatisfied very dissatisfied How to measure user satisfaction GOV.UK will provide a user satisfaction survey at the end of your transactional service and make this data available. What happens when a user exits the transaction midway through Measure all user journeys through your transaction to better understand dropout points. If a user drops out you should try and collect some data around the quality of your service to explain why they didn’t complete the transaction. For example, ‘please tell us why are you unable to complete this transaction.’ Although this survey will not be contribute to the overall measure of user satisfaction it will help you understand service dropout points and find ways to improve transactions.
- The full survey lifecycle from a Total Survey Error perspective, showing the seven types of error
- This survey lifecycle has the technical terms used by the survey methodologists On the left: Construct Measurement Response Edited response Ending with survey statistics. On the right: Representation Sampling frame Sample Respondents Post-survey adjustments And then also ending at Survey statistics
- This slide translates the technical terms into plain language What you want to ask about = Construct The questions you ask = Measurement The answers you get = Response The answers you use = Edited response The number = Survey statistic Who you want to ask = Representation The list you use to sample from = Sampling frame The ones you ask = Sample The ones who answer = Respondents The ones whose answers you can use = Post
- https://designnotes.blog.gov.uk/2015/08/13/how-good-is-your-service-how-many-users-give-up/
- Measure satisfaction across the whole service
- Two questions from a survey: '24: Do you use a Windows or Mac computer'? '25. What is your gender'?
- Photo of a Samsung (Android) mobile with the same questions as previous slide. If you only have an Android mobile, how do you answer ‘do you use a Windows or Mac computer’ when the answer options are ‘Windows’, ‘Mac’ and ‘Both’?
- Tweet from @jberzowska: “I'm entirely charmed by with the survey Octopus from @cjforms, a charming checklist for survey planning. #cssummit” https://twitter.com/jberzowska/status/645995337522372608
- Goals: Establish your goals for the survey, leading to questions that you need answers to Sample: Decide who to ask and how many, leading to people who you will invite to answer Questions: Test the questions and build the questionnaire, leading to a questionnaire Fieldwork: Run the survey from invitation to follow-up, leading to people who actually answer Response: Clean the data, leading to answers Insight: Analyse and present the results, leading to decisions
- The Survey sits between what you want to ask about, who you want to ask, and the number
- This is a more conventional way of looking at the octopus tentacles
- This slide translates the technical terms into plain language What you want to ask about = Construct The questions you ask = Measurement The answers you get = Response The answers you use = Edited response The number = Survey statistic Who you want to ask = Representation The list you use to sample from = Sampling frame The ones you ask = Sample The ones who answer = Respondents The ones whose answers you can use = Post
- This version adds in all the types of error Between construct and measurement: (lack of) validity Between measurement and response: measurement error Between response and edited response: processing error Between representation and sampling frame: coverage error Between sampling frame and sampling: sampling error Between sample and respondents: nonresponse error Between respondents and post-survey adjustments: adjustment error