The good the bad and the ugly: how to design an effective survey
1. The Good, the Bad, and the Ugly:
How to design an effective survey
Workshop 26.02.2013
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
2. The Good, the Bad, and the Ugly:
How to design an effective survey
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
3. Workshop overview
10.00-10.15 Introduction to survey design
10.15-11.15 How to design an effective survey
Writing good questions
11.15-11.30 Morning tea
11.30-12.30 Tips and tricks
How to maximise response rate
Getting the most out of your data
12.30-1.00 Q&A and Close
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
4. Introduction
So you want to create a survey...
What is the purpose of the survey?
• Needs assessment
• Impact assessment
• Knowledge assessment
• Client satisfaction / program feedback
• Program evaluation
What survey mode will you use?
• Telephone, online, paper, interview
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
5. When to use a survey
To collect information from a large number of
people or records
To collect this information relatively inexpensively
To say something about the population from which
the sample was drawn
When you do not need very in-depth information
When you want the data collection procedure to
be as standardised as possible
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
6. Some advantages of surveys
Anonymity
Asking sensitive questions
Standardised data to monitor trends
Quick to administer
Simple data analysis (%, frequencies)
Data can be presented as graphs/charts
Multiple distribution channels (online,
paper, social networking sites)
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
7. Some limitations of surveys
Limited explanations for the data
Comments open to a range of
interpretations
Don‟t know why questions were skipped
or why survey was partially completed
No opportunity to explain questions that
people don‟t understand
Can‟t guarantee the survey was even
received or completed by the right person
Basic literacy levels needed
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
8. What makes a ‘good’ survey?
Have you ever created a survey and felt
disappointed with the data you collected?
Have you ever been asked to answer a survey and
felt irritated with the questions asked?
A good survey considers both your needs for data
and the capabilities and experience of the
respondent.
Collects complete, accurate, and reliable data.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
9. What is a ‘bad’ survey?
Too wordy
Language is hard to understand
Bad timing - e.g. respondents tired at the end of a session
No „Other‟ response category
Questions that seem irrelevant to the topic
Repetitive
Boring – e.g. same structure throughout, no images
Does not consider the capabilities of the respondents
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
10. Is a long survey a bad survey?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
11. HOW TO DESIGN AN
EFFECTIVE SURVEY
IN 3 EASY STEPS!
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
12. Step 1.
Establish your survey goals!
Ask yourself:
Why am I creating this survey?
What do I hope to learn or
accomplish?
How do I plan on using the data
that I collect?
How will the data influence my
decisions?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
13. Step 2.
Create a ‘dummy table’
To ensure all objectives are being addressed and all
questions address objectives. Consider data analysis
before, during and after survey development.
Areas of inquiry Questions
Demographics
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
14. A dummy table …
• Ensures you can interpret data
• Ensures each question will collect the
information you need
• Ensures you leave nothing of importance out
• Ensures you don‟t collect more information
than you need
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
15. Example: Health Care Experience of the
LGBTI* Community in WA
Activity – Find questions related to:
Area of inquiry Questions
LGBTI health issues
Demographics and lifestyle
Current experience of healthcare
services
How and where to reorient current
health services
*LGBTI – lesbian, gay, bisexual, transgender, intersex
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
16. Answers
Area of inquiry Questions
LGBTI health issues 10,11,24
Demographics and lifestyle 1,2,3,4,5,6,7,8,9,15,16
Current experience of healthcare 12,13,14,20,21,22,23
services
How and where to reorient current 17,18,19,25
health services
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
17. Step 3.
Write good questions
Try to use a range of question types:
Open-ended questions
• To explore issues
• Open to a variety of responses (comment boxes)
Closed-ended questions
• Dichotomous
• Likert scale
• Guttman scale
• Multiple choice (one response, multiple responses)
• Rank order
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
18. Example - Dichotomous
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
19. Example – Likert Scale
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
20. Example - Guttman Scale
Agreement with any one item implies agreement with the lower-order
items.
E.g. are you willing to permit _______ to...
- live in your country?
- live in your community?
- live next door to you?
- marry your child?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
21. Example – Multiple Choice (one response)
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
22. Example – Multiple Choice
(multi-response)
Evaluation of Health Promotion to design an effective survey
How
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
382
23. Example – Rank Order
Rank the following methods of receiving a positive STI
diagnosis in order of preference from highest to lowest
(1=your most preferred method; 5=your least preferred
method)
• SMS______
• Phone call ______
• In person ______
• Email ______
• Letter ______
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
24. Developing a Client Satisfaction
Survey
Example: The Wheatbelt Resilience Program
Included a series of community workshops
A survey was developed to find out how useful the
workshops were in terms of increasing awareness of the
Wheatbelt Resilience Program; understanding of suicide
and suicidal behaviours; and opportunities to help with this
issue in the community
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
25. Can you identify
any ways in which
this feedback sheet
could be improved?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
26. The questionnaire
was adapted to
capture more
specific information
and to also gauge
satisfaction with the
workshop. The
pre-questionnaire
was on a separate
document to the
post-workshop
evaluation.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
27. Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
28. WRITING GOOD QUESTIONS
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
29. What do you want to know?
What do you need to know?
Don‟t waste the opportunity to collect good quality
data by asking vague questions
Be economical with the number of questions but
make sure you collect all the information required
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
30. Activity: Work in small groups
Re-write these questions and include closed
ended responses:
(a) “How often do you participate in physical
activities on average?”
(b) “When were you last physically active?”
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
31. What to avoid
Double-barreled questions
Biased/leading questions
Questions with a double negative
Loaded questions
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
32. Example: double-barreled question
Please indicate how much you agree or
disagree with each of the following statements
about our service:
I feel welcomed by staff and other youth at the
centre
Example: leading question
Negotiating condom use with a sexual
partner is difficult. Did this session help
you feel more confident to negotiate
condom use successfully?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
33. Example: double negative
How do you feel about the following statement?
We should not reduce investment in sexual health
services.
Strongly Agree Agree
Disagree Strongly Disagree
Example: Loaded questions
Which of the following services most
urgently requires improvement?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
34. What to avoid
Questions which could create a response set
i.e. the tendency for a respondent to answer later questions
the same way they answered earlier questions
Technical language/jargon
Use simple, common words
Emotive language
Do you approve of the government‟s oppressive immigration
policy?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
35. What to avoid
Using “and”, “or”, “but” in the same question
Long, complex, multi-part questions
Questions which contain a negative and a positive
If you were in a foreign country and did not speak the language but knew
someone who did would you…
a. Let the person that spoke the language speak for you Huh?!
b. Try to learn the language on your own
c. Not try to learn the language or let the person that you knew that spoke
the language speak for you.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
36. What to avoid
Response categories that are not exhaustive
What is your current age?
10 to 20
21 to 30 I‟m 41!
31 to 40
Response categories that are not mutually exclusive
What is your current age?
10 or less
10 to 20 I‟m 30!
20 to 30
30 or greater
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
37. Avoid fuzzy words in the question
Those words which are open to different
interpretation by the respondents.
• Have you ever had any major injuries during the past year?
Avoid words like: important, major, better, regular,
often, occasional (subjective interpretation).
Best to give specific time & details
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
38. Irrelevant questions
Questions which are irrelevant to the survey
objectives
• E.g. do you really need to know someone‟s income?
Questions which are irrelevant to the respondent
• e.g. Don‟t ask nurses to compare the effectiveness of different
patient management techniques if they have no experience of the
alternatives.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
39. Presuming knowledge
Don‟t assume people either know or have
appropriate experience to answer the
question. What is
bullying?
• Have you been bullied this year? What is a
• How many standard drinks have you drunk in the standard
drink?
last week?
• How many cigarettes have you smoked in the
I don‟t
last week?
smoke!
Use filter questions or provide more
information to avoid these problems
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
40. Presuming knowledge
Hypothetical or predictive questions
• Would you like to try marijuana?
• Avoid - many people might like to try anything once.
Generally avoid:
• Would you like to...
• What would you do if..?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
41. Personal questions
Be sensitive!
May need to preface personal questions to give
respondent „permission‟ to respond truthfully
Consider the structure of response choices
• people don‟t like to consider themselves at extreme ends of a
scale
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
42. Activity: Design a brief survey to
evaluate this workshop
Workshop objectives
• Increase participants’ knowledge and skills in effective survey design
• Assess participants’ satisfaction with the SiREN training workshop
Areas of inquiry Questions
Demographics
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
43. MORNING
TEA
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
44. TIPS AND TRICKS
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
45. Critically assess your survey
questions
Is the respondent‟s understanding of response
categories likely to be influenced by more than
words?
The vaguer the question and answer categories,
the greater potential for error / poor data
Is the survey information being collected in more
than one mode? E.g. phone, interview, email
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
46. Critically assess your survey
questions
Is changing a question acceptable to the survey
sponsor?
Is the question necessary?
Will it improve the validity of your study if you do
ask more than one question?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
47. Critically assess your survey
questions
How easy is it to get an accurate answer?
Can people accurately recall and report past
behaviours?
Will the respondent be willing to reveal the
requested information?
Will the respondent feel threatened or defensive
by the question?
Will the respondent feel motivated to answer each
question?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
48. Tips and tricks
Insert dummy questions
• These may be irrelevant to the research to avoid respondents
guessing desirable answers
Reverse the wording of some survey items
• This technique can avoid creating response sets
I always enjoy going to work
I dread going to work
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
49. Tips and tricks
The order of questions may influence responses
• Put interesting questions first
• Sensitive questions towards end
• Demographic information – location of these questions depends
on importance
• Similar themes together
Put instructions for questions before the question
not at the beginning of the survey
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
50. Tips and tricks
Be specific
Example: How can we improve our service?
This is too vague and may elicit single responses about many
different aspects of the service without much guidance to help you
decide which ones are important.
Instead, consider asking respondents to rank from 1-5 the quality of
different aspects of the service
What claims will you use the data to support?
Be honest and consider if this is realistic or credible. This technique
helps you to clarify the wording of questions
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
51. HOW TO MAXIMISE RESPONSE
RATE
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
52. Aim to get a 65% response rate
That is: 65% of your sample complete and
return a questionnaire
Non-respondents may bias your responses
Need to consider response rates when selecting
sample size
Need to consider attrition
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
53. How to maximise response rate
Involve your target group in design if possible
Pre-test!
Ask questions that people can and will answer
Uplift, motivate and encourage the respondents to
give information
Avoid bad survey questions
Allow sufficient time to complete survey
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
54. How to increase responses
Plan multiple contacts
• Pre-notification (tell people you are planning to survey)
• Reminders for non-responders
Consider incentives
• Individual incentives more effective than prize draws e.g. incentive
given for getting others to complete the survey
• Dependent on resources
Obtain ethics approval for your survey
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
55. A note about survey ethics…
Considerations may include:
Information and consent forms
Confidentiality and anonymity
Samples which include minors
How to ask questions about potentially illegal activities
Length of survey / burden on respondents
Format of survey - e.g. acceptability, culturally appropriate
Please contact siren@curtin.edu.au for more advice about survey ethics
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
56. GETTING THE MOST OUT OF
YOUR SURVEY DATA
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
57. Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
58. Planning and evaluation cycle
Needs
assessment Survey
data
Use different
Re-planning Program types of survey
planning data as part of a
continuous
Survey
data
program
Survey
data
planning and
evaluation cycle
Evaluation
Implementation
Survey
data
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
59. How will you use the data?
Use survey data to inform the program planning cycle
Adopt a mindset of continuous program improvement
After implementing the survey, critically evaluate the
quality and completeness of the data collected
• Were any questions ambiguous?
• Were all the questions necessary?
• Did any questions appear to cause confusion? Ask yourself what
• Did you get some unexpected responses? will you change?
• Were the data useful?
• What decisions could the data inform?
• Are there still gaps in the data?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
60. Getting the most out of your data
Don‟t just file the survey data away until the next activity
report for your funding provider is due!
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
61. Health promotion short course –
example survey data
How confident do you Pre course Post course
11 11
feel to undertake a needs
assessment to plan a 3
5
2
health promotion 0
program? Not Confident Very Not Confident Very
confident confident confident Confident
How confident do you feel to Pre course Post course
10 9
contribute to the design and 7
implementation of an 3 3
evaluation of a health 0
promotion initiative? Not Confident Very Not Confident Very
confident Confident confident confident
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
62. Health promotion short course –
example survey data
Pre course Post course
How confident do you feel to 13
12
work collaboratively with
your community and with 4
3
other organisations?
0 0
Not Confident Very Not Confident Very
confident Confident confident confident
What could you conclude from these data?
What decisions about the course would you make as a
result of these data?
Are these decisions valid? What other data might you want
to collect?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
63. Any questions?
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
64. FOLLOW UP SESSION
Curtin University is a trademark of Curtin University of Technology
CRICOS Provider Code 00301J
65. Follow up session
SURVEY CLINIC
Tuesday 12th March
2013, 10am – 12 noon
Curtin University (Room TBA)
In two weeks‟ time!
Aim to put what you have learned into practice
Either:
1. Critically review a survey you have used previously OR
2. Design a new survey using what you have learned.
Bring your surveys and any questions to the survey clinic
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
66. Workshop evaluation
Please draw around
your hand (or both
hands) and provide
some feedback on
today‟s workshop.
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
67. THANKS FOR COMING!
Please don’t forget to complete the workshop evaluation!
Thank you
Curtin University is a trademark of Curtin University of Technology How to design an effective survey
CRICOS Provider Code 00301J
Editor's Notes
This questionnaire was given to participants at the beginning of the workshop when they completed the ‘Before presentation’ section. The ‘After presentation’ section was completed immediately after the presentation. While this is a good way to gather process data you should not give participants the questions on the same sheet pre and post presentation. Usually for process it is enough to collect data at post presentation and ask participants to reflect on how the presentation has made them think about changes to their knowledge, attitudes etc. If you do want to collect at pre and post presentation see the Wheatbelt Resiliency Program document for an example as to how to do this. The questions are very broad. For example ‘I can identify factors associated with suicide.’ It is better to relate back to perceived knowledge. When conducting workshops it is important to collect information about the workshop – e.g. was the workshop useful, well facilitated etc. These types of questions help with the planning of subsequent workshops. Questions can be asked using a Likert scale. Avoid too many open ended questions, however it is always useful to ask what participants liked and what could be improved.