Design tipsfor surveys2012Caroline JarrettUser Interface Engineering / Rosenfeld Media Seminar
We need lots of data.     Your samples are too small.     We‟re going to do a survey.                      But, but… er, e...
We need surveys in our toolbox1. You get lots of lovely numbers2. It‟s much easier to manage what you can measure3. They‟r...
Survey = Questionnaire + Process                                   4
Agenda   Entice people to take a survey         Get users to engage with questions         Break: seminar Q & A         Ge...
Your answers to this survey areimportant for my business.       But what‟s in it for me?       And I‟m really busy today. ...
Response relies oneffort, reward, and trust                  Perceived                                        Perceived   ...
Reward, trust and effort for this invitation                                    Intrinsic reward                          ...
We‟ll offer a chance to win $100.That will get lots of responses.                But I don‟t believe                I‟ll b...
A small immediate payment works best for mail                                      Response rates100% 50%   0%            ...
„Small immediate payment‟ is difficult online                                     Accessed first page100%50% 0%           ...
Offer a meaningful,Tip 1   immediate reward                     Picture credit:      12                     Flickr -      ...
Let‟s look again at trust                            Who are you?                            Will you spam me?            ...
A well-known brand on a well-designed site                      I definitely know you                      And you won’t s...
Be trustworthy andTip 2   look trustworthy                             15
I really need theYou know that survey? I want       „recommend to ato know where they live            friend‟ question    ...
I respond to a lot of surveys     89 invitations     67 completed surveys   1387 questions               17
They wanted my opinions but it took 6 screensto get to the first opinion question                                         ...
Could the BBC trim some of these?• Gender               • What do you like about it• Region you live in   • What do you di...
Focus on the questionsTip 3   that deliver the insights        that you need the most                                     ...
Remember 1387 questions?This survey contributed 30 of them                                     21
On and on            22
On and on and on and on and on   I love John Lewis!   I shop there online and offline!   Don‟t make me feel bad because   ...
Your users may not be as interested in yourdesign decisions as you are.                                                   ...
To get a good survey, talkTip 4   to lots of users                                     25
Agenda   Entice people to take a survey         Get users to engage with questions         Break: seminar Q & A         Ge...
Back to the BBC. After I gave my initialopinions, they asked me some ratings                                           27
Most surveys I see have ratingsfrom 1 to 5            Wouldn‟t it be easier to have just two ?           7 points are more...
Likert had several different typesof question in his scalesLikert, Rensis. (1932). A Technique for the Measurement of Atti...
You can find an academic paper to supportalmost any number of points in a range• Krosnick and Presser refer to ~87 papers ...
Users don‟t care much about the number ofpoints. They care more about the questions.What‟s the differencebetween those que...
Don‟t stress about theTip 5   number of points in your        rating scale                                                ...
The BBC didn‟t stress about the number ofpoints in the rating scale. This one has 10                                      ...
This one has 5. And I never noticed at the time.                                              34
Let‟s have another lookat one of Likert‟s questions     Complex clause                                                    ...
Before- and after- for Likert‟s question?17a. The United States should join in the humanitarian work of the League of Nati...
The BBC also asked me a complex question                                           37
Let‟s split it and simplify itIt is easy to select products on this serviceIt is easy to find suppliers on this service   ...
Refine questions soTip 6   they‟re crystal clear        Present one question at a time        Use plain language        ht...
How much do you care about the economicprograms of the League of Nations?Likert, Rensis. (1932). A Technique for the Measu...
OK, so how much do you care about…                                     41
Two ways of asking the same question:One of them calls for imaginationWhat is your favourite meal?                        ...
Could we make the question more interesting?Please rate the overallinformation contenton this web site       Boring       ...
Could we make the question more interesting?Please rate the overall   Story:information content                          I...
Cambridgeshire police tried the game idea                                            45
Write interesting questionsTip 7                                           46        Picture credit: Shutterstock.com
Agenda   Entice people to take your survey         Get users to engage with questions         Break: seminar Q & A        ...
Does your household have accessto running water?Picture credits: Flickr - James Mooney; Caroline Jarrett   48
“Shared reference”:both sides interpret in the same way Reiss, E. (2000) “Practical Information Architecture: A Hands-On A...
“Shared reference”:both sides interpret in the same way                                  Service area?                    ...
Make sure that your usersTip 8   understand your questions        in the same way that you do                             ...
Let‟s look at putting the answer on the page.                                                52
Here, the „agree‟ and „disagree‟ have floated                                                53
Not hard to fix.(And let‟s delete excess instructions while we‟re about it.)                                              ...
Design the page so peopleTip 9   can answer easily        http://www.slideshare.net/cjforms/labels-and-buttons-on-forms   ...
Agenda   Entice people to take your survey         Get users to engage with questions         Break: seminar Q & A        ...
We got so much great datathat I put every detail inthis report!!!!                             57
Surveys can get you lots of lovely data                                          58
Cutting out some detail can help a lot                                         59
This graph could be good news or bad newsdepending on the question we ask of it                                           ...
This graph could be good news or bad newsdepending on the question we ask of it   How are we   doing in the 35-   44 age g...
This graph could be good news or bad newsdepending on the question we ask of it                      Disappointing results...
Think about the story theTip 10   data can tell         Perceptual Edge has a Graph Design IQ test:         http://www.per...
My best ever survey was very simple:needed data, got it, made decision                                       64
“Quality doesn‟t matter”    We are focusing our time on technicalities of    research, whereas the expectation from client...
Insight = information that provokes actionIf information doesn’t change our,or our consumers’, behaviorthen we shouldn’t b...
Know what decision you‟llTip 11   make based on this survey                                            67                 ...
We need to know oursatisfaction score             Hmm… satisfaction has many aspects.             Let‟s ask about aestheti...
Good news! We‟re rating 5 for          aesthetics, 5 for desirability          and 5 for navigation!But what aboutsatisfac...
Tip 12         Know “The Most         Crucial Question”         and don‟t         forget to ask it                        ...
Agenda   Entice people to take your survey         Get users to engage with questions         Break: seminar Q & A        ...
Remember my definition?    Survey = Questionnaire                + Process                            72
Bonus        SuccessfulTip        Survey = Questionnaire                + Process             That involves             lo...
Agenda   Entice people to take your survey         Get users to engage with questions         Break: seminar Q & A        ...
Caroline Jarretttwitter @cjformscaroline.jarrett@effortmark.co.ukSlides: http://www.slideshare.net/cjformsBlog posts and m...
Upcoming SlideShare
Loading in...5
×

Design tips for surveys UIE 2012

3,666

Published on

Ideas for extracting the maximum value from a survey that is going to happen anyway.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Published in: Design, Technology, Business
4 Comments
13 Likes
Statistics
Notes
No Downloads
Views
Total Views
3,666
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
110
Comments
4
Likes
13
Embeds 0
No embeds

No notes for slide

Design tips for surveys UIE 2012

  1. 1. Design tipsfor surveys2012Caroline JarrettUser Interface Engineering / Rosenfeld Media Seminar
  2. 2. We need lots of data. Your samples are too small. We‟re going to do a survey. But, but… er, er, … OK, you‟re the bossPlease make it one thatI‟ll enjoy answering 2
  3. 3. We need surveys in our toolbox1. You get lots of lovely numbers2. It‟s much easier to manage what you can measure3. They‟re going to do a survey anyway, so let‟s make it a good one And the more of them I do, the more I like them 3
  4. 4. Survey = Questionnaire + Process 4
  5. 5. Agenda Entice people to take a survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 5
  6. 6. Your answers to this survey areimportant for my business. But what‟s in it for me? And I‟m really busy today. 6
  7. 7. Response relies oneffort, reward, and trust Perceived Perceived effort reward TrustDiagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” 7inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method”
  8. 8. Reward, trust and effort for this invitation Intrinsic reward - but is it enough?Short is good,but do I trust Who are you?you enough to Will you spam me?believe you? 8
  9. 9. We‟ll offer a chance to win $100.That will get lots of responses. But I don‟t believe I‟ll be the winner 9
  10. 10. A small immediate payment works best for mail Response rates100% 50% 0% No incentive $300 lottery $2 bill Carlson, 1996 quoted in Dillman, D.A., Smyth, J. D. and Christian, L. M. (2009) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 10
  11. 11. „Small immediate payment‟ is difficult online Accessed first page100%50% 0% No incentive Prize draw (2x $2 via Pay Pal $2 via PayPal on $50,2x $25 prize) before survey completion Michael Bosnjak and Tracy L. Tute (2003) “Prepaid and Promised Incentives in Web Surveys : An Experiment” 11 Social Science Computer Review 2003 21: 20
  12. 12. Offer a meaningful,Tip 1 immediate reward Picture credit: 12 Flickr - Jessica Wilson (jek in the box)
  13. 13. Let‟s look again at trust Who are you? Will you spam me? 13
  14. 14. A well-known brand on a well-designed site I definitely know you And you won’t spam me 14
  15. 15. Be trustworthy andTip 2 look trustworthy 15
  16. 16. I really need theYou know that survey? I want „recommend to ato know where they live friend‟ question We have to ask them about navigation, obviouslyWhat about the new We have to ask aboutgraphics? the content! And finding us … we have to know about that Don‟t forget about the new feature 16
  17. 17. I respond to a lot of surveys 89 invitations 67 completed surveys 1387 questions 17
  18. 18. They wanted my opinions but it took 6 screensto get to the first opinion question 18
  19. 19. Could the BBC trim some of these?• Gender • What do you like about it• Region you live in • What do you dislike about it• Ethnic origin • Your main reason for visiting• Type of Internet the site today connection • And many more• Rate how it looks 19 Image credit: Flickr - Enokson
  20. 20. Focus on the questionsTip 3 that deliver the insights that you need the most 20 Image credit: Fraser Smith
  21. 21. Remember 1387 questions?This survey contributed 30 of them 21
  22. 22. On and on 22
  23. 23. On and on and on and on and on I love John Lewis! I shop there online and offline! Don‟t make me feel bad because I didn‟t read your buying guide. Just let me tell you I love you… 23
  24. 24. Your users may not be as interested in yourdesign decisions as you are. 24 “Wood for the trees” Image credit: Flickr - Paul & Aline Burland (P – A – S – T)
  25. 25. To get a good survey, talkTip 4 to lots of users 25
  26. 26. Agenda Entice people to take a survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 26
  27. 27. Back to the BBC. After I gave my initialopinions, they asked me some ratings 27
  28. 28. Most surveys I see have ratingsfrom 1 to 5 Wouldn‟t it be easier to have just two ? 7 points are more accurate! What about 1 to 10? 28
  29. 29. Likert had several different typesof question in his scalesLikert, Rensis. (1932). A Technique for the Measurement of Attitudes.Archives of Psychology, 140, 1–55. 29
  30. 30. You can find an academic paper to supportalmost any number of points in a range• Krosnick and Presser refer to ~87 papers on ranges • Krosnick, J. A. and S. Presser (2009). Question and Questionnaire Design. Handbook of Survey Research (2nd Edition) J. D. Wright and P. V. Marsden, Elsevier• If you‟d like to track down the research yourself: http://comm.stanford.edu/faculty/krosnick/ Handbook%20of%20Survey%20Research.pdf 30
  31. 31. Users don‟t care much about the number ofpoints. They care more about the questions.What‟s the differencebetween those questions?This survey is too repetitive. 31
  32. 32. Don‟t stress about theTip 5 number of points in your rating scale 32 Picture credit: Flickr - Bill Soderman (BillsoPHOTO)
  33. 33. The BBC didn‟t stress about the number ofpoints in the rating scale. This one has 10 33
  34. 34. This one has 5. And I never noticed at the time. 34
  35. 35. Let‟s have another lookat one of Likert‟s questions Complex clause A rather big commitment Double-barrelled And what are the programs anyway?Likert, Rensis. (1932). A Technique for the Measurement of Attitudes. 35Archives of Psychology, 140, 1–55.
  36. 36. Before- and after- for Likert‟s question?17a. The United States should join in the humanitarian work of the League of Nations17b. The United States should join in the economic work of the League of Nations 36
  37. 37. The BBC also asked me a complex question 37
  38. 38. Let‟s split it and simplify itIt is easy to select products on this serviceIt is easy to find suppliers on this service 38
  39. 39. Refine questions soTip 6 they‟re crystal clear Present one question at a time Use plain language http://www.plainlanguage.gov/howto/index.cfm  39
  40. 40. How much do you care about the economicprograms of the League of Nations?Likert, Rensis. (1932). A Technique for the Measurement of Attitudes.Archives of Psychology, 140, 1–55. 40
  41. 41. OK, so how much do you care about… 41
  42. 42. Two ways of asking the same question:One of them calls for imaginationWhat is your favourite meal? Imagine you are on death row and have to choose your last meal• Steak and chips • Scallops with black pudding and cream, rib eye steak with chips and a dolce latte cream sauce; stinking bishop (cheese) with 1960 port (year of my birth). Wine would have to be Chateau Lafite 1st Cru Pauillac 2000.Puleston, J. and Sleep, D (2011) “The Game Experiments” 42ESOMAR Congress
  43. 43. Could we make the question more interesting?Please rate the overallinformation contenton this web site Boring 43
  44. 44. Could we make the question more interesting?Please rate the overall Story:information content If you could wave aon this web site magic wand and make a new page for this web site, what would it do or say? Game: You‟ve asked us to improve the information content on this web site. Here are some different changes we thought of and what they would cost. If you had $10,000 to spend, which would you choose? 44
  45. 45. Cambridgeshire police tried the game idea 45
  46. 46. Write interesting questionsTip 7 46 Picture credit: Shutterstock.com
  47. 47. Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 47
  48. 48. Does your household have accessto running water?Picture credits: Flickr - James Mooney; Caroline Jarrett 48
  49. 49. “Shared reference”:both sides interpret in the same way Reiss, E. (2000) “Practical Information Architecture: A Hands-On Approach to Structuring Successful Web Sites”Picture credits: Flickr - James Mooney; Caroline Jarrett 49
  50. 50. “Shared reference”:both sides interpret in the same way Service area? 50 Photo credit: Flickr - chaptstickaddict
  51. 51. Make sure that your usersTip 8 understand your questions in the same way that you do 51
  52. 52. Let‟s look at putting the answer on the page. 52
  53. 53. Here, the „agree‟ and „disagree‟ have floated 53
  54. 54. Not hard to fix.(And let‟s delete excess instructions while we‟re about it.) 54
  55. 55. Design the page so peopleTip 9 can answer easily http://www.slideshare.net/cjforms/labels-and-buttons-on-forms 55
  56. 56. Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 56
  57. 57. We got so much great datathat I put every detail inthis report!!!! 57
  58. 58. Surveys can get you lots of lovely data 58
  59. 59. Cutting out some detail can help a lot 59
  60. 60. This graph could be good news or bad newsdepending on the question we ask of it 60
  61. 61. This graph could be good news or bad newsdepending on the question we ask of it How are we doing in the 35- 44 age group? Strong response from key target 61
  62. 62. This graph could be good news or bad newsdepending on the question we ask of it Disappointing results from seniors How are we doing with the seniors? 62
  63. 63. Think about the story theTip 10 data can tell Perceptual Edge has a Graph Design IQ test: http://www.perceptualedge.com/files/GraphDesignIQ.html 63
  64. 64. My best ever survey was very simple:needed data, got it, made decision 64
  65. 65. “Quality doesn‟t matter” We are focusing our time on technicalities of research, whereas the expectation from clients is actually in a totally different space. The clients are saying, ‘Inspire me, help me to take some transformational action’ Stan Sthanunathan Vice President Marketing Strategy & Insights The Coca-Cola Company“Research”, Oct 2009 http://www.research-live.com/4001230.articleImage credit: http://www.eltiempo.com/gente/ARTICULO-WEB- 65NEW_NOTA_INTERIOR-9736789.html
  66. 66. Insight = information that provokes actionIf information doesn’t change our,or our consumers’, behaviorthen we shouldn’t be doing it.Danny Russell,Marketing Strategy Director,BSkyB“Research”, January 2012 66http://www.research-live.com/features/who-are-you-talking-to?/4006763.article
  67. 67. Know what decision you‟llTip 11 make based on this survey 67 Image credit: Fraser Smith
  68. 68. We need to know oursatisfaction score Hmm… satisfaction has many aspects. Let‟s ask about aesthetics, desirability, navigation…. 68
  69. 69. Good news! We‟re rating 5 for aesthetics, 5 for desirability and 5 for navigation!But what aboutsatisfaction? 69
  70. 70. Tip 12 Know “The Most Crucial Question” and don‟t forget to ask it Image 70 credit: Shutterstock
  71. 71. Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 71
  72. 72. Remember my definition? Survey = Questionnaire + Process 72
  73. 73. Bonus SuccessfulTip Survey = Questionnaire + Process That involves lots of testing 73
  74. 74. Agenda Entice people to take your survey Get users to engage with questions Break: seminar Q & A Get users to answer accurately Find insight for your stakeholders Bonus tip Wrap up: seminar Q & A 74
  75. 75. Caroline Jarretttwitter @cjformscaroline.jarrett@effortmark.co.ukSlides: http://www.slideshare.net/cjformsBlog posts and more resources:http://www.rosenfeldmedia.com/books/survey-design/ 75
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×