Surveys are an efficient way to gather information from users. They can range from simple to complex and use closed or open-ended questions. Online survey tools make administration and analysis easier. Good survey design involves determining goals, sample size, question wording free of bias, logical flow, and testing. Analysis requires looking at trends in raw data. Reporting should include an executive summary and presentation of data. Examples show poor question wording can introduce bias or be convoluted.

1. Surveys
Christopher S. LaRoche
MIT, Usability Consultant
Northeastern University, Senior Lecturer
c.laroche@neu.edu
Copyright © 2012 - 2014 - Christopher S. LaRoche & Katherine Wahl

2. Survey Overview
• Surveys are an efficient way to gather a high volume of
information and specific data from your users
• Surveys are multi-faceted. You can create simple surveys
and get massive amounts of data; but to get statistically
significant information you often have to parse your
demographics and question wording (Goodman, p. 327)
• Surveys can range from simple to incredibly complex
2

3. Survey Overview
• Most questions are closed (multiple choice) vs. open-
ended (open text box)
• Try to strip out subjectivity and bias. (Goodman, p. 331-
343)
3

4. Online tools
• Online tools make surveys much easier and efficient
• One of the most popular online survey services:
www.surveymonkey.com
• Not only do online survey sites allow you to conduct the
survey online, they provide tools to help you analyze the
data
4

5. Types of Surveys
• There are several different kinds of surveys:
– Profile, when you want to get a snapshot of the
composition of your current user population
– Satisfaction, when you want to find out what what
works well and what doesn’t
– Value, when you want to know what people find
important (Goodman, p. 328)
5

6. When a survey is a good choice
• A survey is a good choice when you want to get answers
to specific questions by reaching a large group of users
• It’s not a good idea to use a survey when you have broad
questions about your users or want to understand
comprehensive workflow
• A survey is a good method when you want quantitative
[numbers] data, not qualitative [text] data
6

7. Disadvantages of surveys
• Quality of survey data can be uneven, as users will
sometimes skip questions or respond quickly with out
fully considering questions
• Questions MUST be well-written and free of bias
• Analysis can be time consuming and complex
• Users don’t always report truthfully in surveys
7

8. Getting started
• Determine the goals for your survey. What do you want
to learn?
• Make sure your stakeholders know what they will and
will not learn from the survey
• Clearly define the profiles of the survey participants and
how you will reach them
8

9. Sample size
• How many participants should you have for your survey?
This is one of the hardest questions to answer and, as
with many user research related questions, the answer is
‘it depends’
• You want to make sure you are covering the audiences
for your project and that they are all represented in your
sample group. This means your sample size will vary
• If you don’t get enough responses, you need to report
that even though it may invalidate the survey
9

10. 10
Closed questions
• Closed questions are those where users pick the
response from a list
• You want users to pick a specific answer or answers from
a list of possibilities. You need to make sure the list of
possible answers is complete and that there is no overlap
among the answers so users will not get confused.
(Goodman, p. 333)
• Always include an “other” option in your list and include
a text box for participants to write in what that option is.
That way, the list is really complete!

11. 11
Open-ended questions
• Open-ended questions are those where participants
write their responses as free form text
• These yield more complete and complex data but they
are difficult and time consuming to analyze
• Participants may be likely to skip them if they are in a
hurry
• Some of the data may be difficult to interpret if the
answer uses unfamiliar terminology

12. 12
Open-ended questions
• Include at least one open-ended question at the end of
the survey, even if it’s “Do you have any thoughts,
comments, or suggestions?” This gives participants a
nice ‘release valve’ to give any feedback they wanted to
share but didn’t see covered in the survey

13. Writing questions
• Keep questions positive (avoid negative statements).
Negative questions are more difficult to understand
(Goodman, 336)
• Keep each question one question – do not have multiple
questions in one. This may involve breaking one question
into multiple questions to get at what you want to know
13

14. Writing questions
• Be consistent in how questions are formatted
• Use Likert scales of 1 – 5 or 1 – 7. Make sure the scale is
consistent throughout the survey
• Include an opt-out option so that participants can
indicate that the question does not apply to them
• Also limit required questions to not distract users who
might prefer to skip questions
14

15. Writing Questions
• Revise, revise, and edit, edit questions over and over to
get any potential leading or subjective wording removed
• This usually takes many versions of questions to actually
come up with a final version. Your own bias and
subjectivity is often quite subtle, especially to you!
• This is the most difficult part of the process (Goodman,
p. 331-341)
15

16. Constructing the survey
• Once you have your survey questions, it’s time to think
about how they will flow in a survey
• If you are asking for demographic information in the
survey, include it at the end (the only exception to this
rule is if you are using it to determine other questions)
16

17. Survey logic
• Survey logic can help you guide participants through the
survey and answer sets of questions based on answers to
previous questions
• You can use ‘skip logic’ to have participants skip over
sections of the survey that are not relevant to them
17

18. Survey logic
• If you find yourself writing questions like “If you
answered ‘no’ to the previous question, skip to question
6,” using logic is a good idea!
• Most online survey tools include logic as part of their
paid packages, not the free ones
• Test your survey logic thoroughly to make sure it works
the way you intended. Having faulty logic can ruin your
survey data!
18

19. Organizing the survey
• Make sure your questions flow logically. Ask the more
general questions up front and the more specific,
detailed questions in the middle. The final questions
should be more general and summary in nature
• Make sure your survey is not too long
• A general rule is to limit the survey to 20 questions
19

20. Testing the survey
• Once your survey is constructed, test it!
Have a few people try the survey to make sure the
instructions are clear and the logic works
• If you are sending an invitation to participants via email,
make sure the link to the survey is correct and working
20

21. Launching the survey
• If you are sending your survey via email, think about
when you are going to send it out. Sending a survey on
Thursday afternoon means it may be buried under a long
list of other emails. Having a survey go out first thing on
Monday morning means it will be at the top of the
potential participant’s inbox
21

22. Closing the survey
• You want to leave the survey open long enough to give
participants enough time to complete it. Usually five
business days to a full week is sufficient
• If you find that your response rate is low, you may want
to send out a reminder to participants who haven’t
completed the survey
22

23. Incentives
• Sometimes, offering potential participants a chance to
win a prize after completing the survey will help boost
response rates. You could also provide a small incentive
to all participants
• Usually participants opt in to the incentive by providing
an email address after completing the survey
• Be sure to complete the drawing and send out the
incentive as soon as the survey closes
23

24. Analysis
• You want to analyze the data as soon as you can.
It’s much easier to analyze while the data is fresh
• Some online survey tools also provide components that
help you count and compare the data to see trends
• Be careful with the conclusions you draw! Using
numbers as evidence does not mean a rigorous process
produced them (Goodman, p. 359)
24

25. Analysis
• Look at the raw data first. You want to scan the data to
look for obvious trends or errors
• Consider missing data (from questions skipped or ‘no
answer’ being selected). It can be as important as
completed data
• Look for trends and consistent themes
25

26. Reporting
• Draft an executive summary. If your audience only reads
this portion of the report, you want to make sure this
includes the most important points
• In the more detailed body of the report, include your
goals, methods, and the participant profile(s)
• Summarizing the data you’ve collected in tables and
graphs will help your readers quickly scan the
information
26

27. Examples of Poor Survey Questions
• What is wrong with this question?
27
• Question is leading (‘the right technological depth’)
• If this question is used/kept, should be a Likert scale (1-5 or 1-7 choices)
• Wording is poor in the last option (“Ok” and “Bad/No Comments”)

28. Examples of Poor Survey Questions
• What is wrong with this question?
28
• Question is leading (‘that take away American jobs’) and for this type of
(Likert) scale, questions should be written in a manner such as: ‘American
should buy imported automobiles’ or ‘Buying imported automobiles takes
away American jobs’
• Follows a sort of Likert scale (1-5 choices), but should be strongly agree,
agree, neutral, disagree, strongly disagree

29. Examples of Poor Survey Questions
• What is wrong with this question?
29
• Question is fine, but the way they used the various mechanism to choose the
option is convoluted. You should have one drop-down list box asking the sex
of your child (Male or Female) and another drop-down list box to choose
their age (0-15). That would be only two drop-down list boxes and be much
easier to read and use much less space

30. Contacting Me
Email me at c.laroche@neu.edu with any questions
30

31. Bibliography
Baty, Steve. (2009). User Research for Personas and Other Audience Models. UX
Matters, User Dialogs, April 2009. http://www.uxmatters.com
Courage, Catherine & Kathy Baxter. (2005). Understanding Your Users: A Practical
Guide to User Requirements Methods, Tools, and Techniques. Amsterdam,
Netherlands: Morgan Kaufmann/Elsevier.
Goodman, Elizabeth, Mike Kuniavsky, and Andrea Moed. (2012). Observing the User
Experience: A Practitioner’s Guide to User Research - Second Edition. San Francisco,
California: Elsevier/Morgan Kaufmann.
Morville, Peter & Louis Rosenfeld. (2006). Information Architecture for the World
Wide Web – Third Edition. Sebastopol, California: O’Reilly Media, Inc.
Nielsen, Jakob. (1993) Usability Engineering. San Francisco, California: Morgan
Kaufmann.
Nielsen, Jakob. (1993) Usability Engineering. San Francisco, California: Morgan
Kaufmann.
31

32. Bibliography
Pearrow, Mark. (2007). Web Usability Handbook, Second Edition. Boston,
Massachusetts: Charles River Media.
Schumacher, Robert. Editor (2009). Handbook of Global User Research. Amsterdam,
Netherlands: Elsevier/Morgan Kaufmann.
Wilson, Chauncey. (2009). User Experience – Re-mastered. Amsterdam:
Netherlands: Elsevier/Morgan Kaufmann.
Wilson, Chauncey. (2007). “Designing useful and usable Questionnaires: You Can’t
Just ‘throw a Questionnaire together.’” Interactions May/June 2007: pp. 48-49, 63.
32