10 tips for a better survey
Caroline Jarrett
STC Summit 2011
FORMS
SURVEYS
Caroline Jarrett
twitter @cjforms
caroline.jarrett@effortmark.co.uk
2
Survey = Questionnaire + Process
Successful
That involves
lots of testing
3
Outline Tips about questions
Tips about process
Tips about testing
4
120 characters
48 characters 5
6
Tech writing challenge:
rewrite in 140 characters
• XXX is conducting research to better understand and
meet the needs of ...
Tip 1 One introduction is enough
(and think 140 characters)
8
Try answering this question,
then tell me how you worked out your answer
9
An example of scale effects:
How many hours a day do you study?
Low scale
• ½ hour or less
• From ½ to 1 hour
• From 1 to ...
An example of scale effects:
How many hours a day do you study?
Low scale
• Very lazy
• A bit lazy
• Middling
• A bit dili...
Tip 2 Do not ask questions
about unremarkable
repetitive behaviours
12
What’s missing from this question?
13
Tip 3 Always include ‘other’ in
category questions
– Don’t know
– Prefer not to answer
– All of the above
14
Watch the order of these two questions
15
16
Tip 4 Think of your questions
as a conversation with
your respondent
17
Jarrett, C. and Gaffney, G (2008) “Forms that work...
18
Outline Tips about questions
Tips about process
Tips about testing
19
Response relies on
effort, reward, and trust
Trust
Perceived
effort
Perceived
reward
Diagram from Jarrett, C, and Gaffney,...
How are response, effort and trust
working here?
21
In mail surveys,
a token incentive
works better than
no incentive or a
prize draw
22
Tip 5 Try to find an incentive
that gives an immediate
reward
23
24
25
26
27
Tip 6 One good incentive:
allowing the respondents
to express their views
28
Write down your answers to this questionnaire
1. How many user surveys have you run?
NONE 1 to 5 6 to 10 more than 10
2. W...
Now try it as an interview.
In pairs: use this is a script for an interview
1. How many user surveys have you run?
NONE 1 ...
Now tell each other a story about your
experience relevant to these questions
1. How many user surveys have you run?
NONE ...
Now let’s share data and stories
• What numbers have we collected?
• What open answers?
• What stories?
32
Tip 7 Interview some users
to find out what
they want to tell you
33
Did you answer the same question
that I asked?
1. How many user surveys have you run?
34
Does your household have access
to running water?
Picture credits: James Mooney, Caroline Jarrett 35
“Shared reference”:
both sides interpret in the same way
Service area?
36
Tip 8 Ask users for stories to
explore vocabulary and
concerns
Quesenbery, W and Brooks, K (2010)
“Storytelling in User Ex...
© Caroline Jarrett and Effortmark Ltd
If you use this cartoon in another context,
please credit Caroline Jarrett and Effor...
What recommendation will you make
based on your results?
• What decision will you make?
• What happens if …
– The users al...
Tip 9 Try writing your
presentation first
(before the questions)
Natalie Webb, conversation at UPA2010
40
Outline Tips about questions
Tips about process
Tips about testing
41
“If you don’t have time to do the pilot,
you don’t have time to do the study”
• Oppenheim, A. P. (1992) “Questionnaire Des...
Tip 10 Run a pilot
43
Caroline Jarrett
Twitter: @cjforms
Slideshare: cjforms
caroline.jarrett@effortmark.co.uk
I’m a consultant, hire me:
Consul...
Upcoming SlideShare
Loading in …5
×

10 tips for a better survey at STC2011

4,191 views

Published on

Ten tips for surveys: on questions, process, and testing your survey.
Books mentioned are listed here: http://rosenfeldmedia.com/uxzeitgeist/lists/cjforms/10-tips-for-a-better-survey-stc2011

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Published in: Design, Technology, Business
1 Comment
6 Likes
Statistics
Notes
No Downloads
Views
Total views
4,191
On SlideShare
0
From Embeds
0
Number of Embeds
804
Actions
Shares
0
Downloads
45
Comments
1
Likes
6
Embeds 0
No embeds

No notes for slide

10 tips for a better survey at STC2011

  1. 1. 10 tips for a better survey Caroline Jarrett STC Summit 2011 FORMS SURVEYS
  2. 2. Caroline Jarrett twitter @cjforms caroline.jarrett@effortmark.co.uk 2
  3. 3. Survey = Questionnaire + Process Successful That involves lots of testing 3
  4. 4. Outline Tips about questions Tips about process Tips about testing 4
  5. 5. 120 characters 48 characters 5
  6. 6. 6
  7. 7. Tech writing challenge: rewrite in 140 characters • XXX is conducting research to better understand and meet the needs of the users of their site. Would you like to participate? • Responses are anonymous, view our privacy policy 7
  8. 8. Tip 1 One introduction is enough (and think 140 characters) 8
  9. 9. Try answering this question, then tell me how you worked out your answer 9
  10. 10. An example of scale effects: How many hours a day do you study? Low scale • ½ hour or less • From ½ to 1 hour • From 1 to 1 ½ hours • From 2 to 2 ½ hours • More than 2 ½ hours High scale • 2 ½ hours or less • From 2 ½ to 3 hours • From 3 ½ to 4 hours • From 4 to 4 ½ hours • More than 4 ½ hours 10Schuman, H. and Presser, S. (1996) “Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context” 70% 30% 29% 71%
  11. 11. An example of scale effects: How many hours a day do you study? Low scale • Very lazy • A bit lazy • Middling • A bit diligent • Very diligent High scale • Very lazy • A bit lazy • Middling • A bit diligent • Very diligent 11Schuman, H. and Presser, S. (1996) “Questions and Answers in Attitude Surveys: Experiments on Question Form, Wording, and Context” 70% 30% 29% 71%
  12. 12. Tip 2 Do not ask questions about unremarkable repetitive behaviours 12
  13. 13. What’s missing from this question? 13
  14. 14. Tip 3 Always include ‘other’ in category questions – Don’t know – Prefer not to answer – All of the above 14
  15. 15. Watch the order of these two questions 15
  16. 16. 16
  17. 17. Tip 4 Think of your questions as a conversation with your respondent 17 Jarrett, C. and Gaffney, G (2008) “Forms that work: Designing web forms for usability” Janice (Ginny) Redish (2007) “Letting go of the words: Writing web content that works”
  18. 18. 18
  19. 19. Outline Tips about questions Tips about process Tips about testing 19
  20. 20. Response relies on effort, reward, and trust Trust Perceived effort Perceived reward Diagram from Jarrett, C, and Gaffney, G (2008) “Forms that work: Designing web forms for usability” inspired by Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” 20
  21. 21. How are response, effort and trust working here? 21
  22. 22. In mail surveys, a token incentive works better than no incentive or a prize draw 22
  23. 23. Tip 5 Try to find an incentive that gives an immediate reward 23
  24. 24. 24
  25. 25. 25
  26. 26. 26
  27. 27. 27
  28. 28. Tip 6 One good incentive: allowing the respondents to express their views 28
  29. 29. Write down your answers to this questionnaire 1. How many user surveys have you run? NONE 1 to 5 6 to 10 more than 10 2. What is your top tip for a better user survey, based on experience of writing or answering? __________________________________ __________________________________ 29
  30. 30. Now try it as an interview. In pairs: use this is a script for an interview 1. How many user surveys have you run? NONE 1 to 5 6 to 10 more than 10 2. What is your top tip for a better user survey, based on experience of writing or answering? __________________________________ __________________________________ 30
  31. 31. Now tell each other a story about your experience relevant to these questions 1. How many user surveys have you run? NONE 1 to 5 6 to 10 more than 10 2. What is your top tip for a better user survey, based on experience of writing or answering? __________________________________ __________________________________ 31
  32. 32. Now let’s share data and stories • What numbers have we collected? • What open answers? • What stories? 32
  33. 33. Tip 7 Interview some users to find out what they want to tell you 33
  34. 34. Did you answer the same question that I asked? 1. How many user surveys have you run? 34
  35. 35. Does your household have access to running water? Picture credits: James Mooney, Caroline Jarrett 35
  36. 36. “Shared reference”: both sides interpret in the same way Service area? 36
  37. 37. Tip 8 Ask users for stories to explore vocabulary and concerns Quesenbery, W and Brooks, K (2010) “Storytelling in User Experience Design” 37
  38. 38. © Caroline Jarrett and Effortmark Ltd If you use this cartoon in another context, please credit Caroline Jarrett and Effortmark Ltd. 38
  39. 39. What recommendation will you make based on your results? • What decision will you make? • What happens if … – The users all agree with your gut feel? – The users all disagree with your gut feel? – The users don’t care either way? 39
  40. 40. Tip 9 Try writing your presentation first (before the questions) Natalie Webb, conversation at UPA2010 40
  41. 41. Outline Tips about questions Tips about process Tips about testing 41
  42. 42. “If you don’t have time to do the pilot, you don’t have time to do the study” • Oppenheim, A. P. (1992) “Questionnaire Design, Interviewing and Attitude Measurement “ – And in each subsequent edition • Dillman, D.A. (2000) “Internet, Mail and Mixed Mode Surveys: The Tailored Design Method” – And in each subsequent edition • Gillham, B (2000) “Developing a Questionnaire” – And in the second edition • Jarrett, C (2010) “I wish I’d run a pilot” 42
  43. 43. Tip 10 Run a pilot 43
  44. 44. Caroline Jarrett Twitter: @cjforms Slideshare: cjforms caroline.jarrett@effortmark.co.uk I’m a consultant, hire me: Consultancy: www.effortmark.co.uk Training: www.usabilitythatworks.com Free stuff: Forms advice: www.formsthatwork.com Editing: www.editingthatworks.com Columns: www.uxmatters.com “Good Questions” www.usabilitynews.com “Caroline’s Corner” 44

×