3. 3
Objectives
To introduce key aspects of
questionnaire design including
question style, response formats,
layout, and pilot testing
To consider implementation
issues (such as sampling)
To demonstrate the importance of
rigour in planning, developing, and
implementing research questionnaires
7. 7
What is a survey?
A “standardised” stimulus
A measuring instrument
8. 8
What is a survey?
A way of converting
“fuzzy stuff”
into hard data for analysis
9. 9
Purposes of survey research
Information gathering &
describing
e.g., polls, attitudes, demographics
Theory-building & testing
Explanatory, e.g., why?
Predictive, e.g., what is likely to
happen?
Often survey research does some
of both.
10. 10
Research purposes
Descriptive
Collects basic descriptive
data/statistics e.g., consumer
profiles…(age, gender)
Explanatory
Examine underlying data patterns
Linked to a hypothesis/research
objective
14. 14
1. Formulate
Generic
Questionnaire
2. Expand the
Questionnaire
Based on
study objectives
Turn into
separate
sections
Question styles
& types
3. Finalise
Questionnaire
-Pre-test/pilot test
-Several drafts needed
Placement
& Funnel Qs
Questionnaire planning/design
15. 15
Formulate generic questionnaire
Turn objectives into sections of
the survey
Ensure all questions relate to
research objectives
For explanatory objectives or
hypotheses ensure both
dependent and independent
variables exist
16. 16
Cover letter / Ethics statement
Outline details of research project
Purpose
What's involved?
Explain any risks/costs/rewards
Contact details
Human Ethics approval #
How is consent given/not give?
How to return?
Can choose not to continue anytime
17. 17
Instructions
Provides consistency - helps to
ensure standard conditions across
different administrations
Explain how to do the survey in a
user-friendly manner
Example:
Life Effectiveness Questionnaire
19. 19
Screening
Does the participant qualify for the
survey? (esp. for internet surveys)
Ask screening questions first, rather
than later
Use branching if there are
conditional questions
20. 20
Flow and structure
Logical order of questions
(use sections)
Use funnel questions to move
respondents through survey
Start off with easy to answer and
engaging questions
More controversial questions in
middle section
Personal questions – start or end?
21. 21
Survey design principles
Jenkins and Dillman (1995)
1. Use the visual elements of
brightness, color, shape, and
location in a consistent manner to
define the desired navigational
path for respondents to follow
when answering the
questionnaire.
22. 22
Survey design principles
Jenkins and Dillman (1995)
2. When established format
conventions are changed in the
midst of a questionnaire use
prominent visual guides to redirect
respondents.
23. 23
Survey design principles
Jenkins and Dillman (1995)
3. Place directions where they are
to be used and where they can be
seen.
4. Present information in a manner
that does not require respondents
to connect information from
separate locations in order to
comprehend it.
27. 27
Open-ended question examples
What are the main issues you are
currently facing in your life?
How many hours did you spend
studying this week? _________
30. 30
Dichotomous
Simple Yes / No response e.g.,
Excluding this trip, have you
visited Canberra in the previous
five years?
__ Yes __ No
31. 31
Multichotomous
Choice one of several answers e.g.
How many hours did you spend
studying this week?
__ less than 5 hours
__ > 5 to 10 hours
__ > 10 to 20 hours
__ more than 20 hours
32. 32
Frequency scale
Over the past month, how often have
you argued with your intimate
partner?
1. All the time
2. Fairly often
3. Occasionally
4. Never
5. Doesn’t apply to me at the moment
33. 33
The list (Multiple response)
Provides a list of answers for
respondents to choose from e.g.,
Tick any words or phrases that
describe your perception of
Canberra as a travel destination:
__ Exciting __ Important
__ Boring __ Enjoyable
__ Interesting __ Historical
34. 34
Ranking
Helps to measure the relative
importance of several items
Rank the importance of these
reasons for taking a holiday to
Canberra (from 1 (most) to 4
(least)):
__ to visit friends and relatives
__ for business
__ for educational purposes
35. 35
Likert Scale
Assesses differences of perceptions
and allows measurement and
comparison of these differences
Indicate your degree of agreement
with this statement:
“I am an adventurous person.”
(circle the best response for you)
36. 36
Graphical rating scale
How would you rate your enjoyment
of the movie you just saw?
Mark with a cross (X)
not enjoyable very enjoyable
37. 37
Semantic differential
What is your view of smoking?
Tick to show your opinion.
Bad ___:___:___:___:___:___:___ Good
Strong ___:___:___:___:___:___:___ Weak
Masculine ___:___:___:___:___:___:___ Feminine
Unattractive ___:___:___:___:___:___:___ Attractive
Passive ___:___:___:___:___:___:___ Active
40. General aim:
Maximise sensitivity (i.e. more options)
Maximise reliability (i.e. less options)
How many measurement options?
Minimum = 2
Average = 3 to 7
Maximum = 10?
Scale of measurement guidelines
41. FEELING ABOUT SOMETHING
EXTREMELY POSITIVE EXTREMELY NEGATIVE
2-Categories
GOOD NOT GOOD
3-Categories
GOOD FAIR POOR
4-Categories
VERY GOOD GOOD FAIR POOR
5-Categories
EXCELLENT VERY GOOD GOOD FAIR POOR
42. Watch out for too many or
too few responses
“Capital punishment should be
reintroduced for serious crimes”
1 = Agree 2 = Disagree
1 = Very, Very Strongly Agree 7 = Slightly Disagree
2 = Very Strongly Agree 8 = Disagree
3 = Strongly Agree 9 = Strongly Disagree
4 = Agree 10 = V. Strongly Disagree
5 = Slightly Agree 11 = V, V Strongly Disagree
6 = Neutral
43. 43
Wording questions
Does the question focus directly on
the issue or topic to be
measured?
(If not, rewrite.)
Is the question stated as briefly as it
can be?
(If too long, restate it more briefly.)
44. 44
Wording questions
Is the question expressed as clearly
and simply as it can be? I
(f the meaning won’t be clear to
every respondent, restructure the
question.)
Use only core vocabulary - words
and phrases people use in casual
speech
45. 45
Wording questions
Limit the vocabulary so the least
sophisticated respondent would
be familiar with the words
Use simple sentences where
possible and complex sentences
only when actually required
Use two or more short, simple
sentences rather than one
compound or complex sentences
46. 46
Finalise questionnaire draft
Length
Try to keep them as short as
possible
Only ask questions that relate to
objectives
Tricks? Font size/double sided
photocopying/numbering sections
47. 47
Pre-testing and pilot testing
Pre-test – try out on convenient
others & revise
Pilot test – try out on a small
sample from the target
population & revise
Be assertive and interactive about
seeking feedback – ask
questions & observe
“The customer is always right.”
48. 48
Maximising response rate
Layout and design is key
Respondent’s level of interest
Colour of paper
Accompanying letter / introduction
Mail surveys - self-addressed
stamped return envelope
Rewards
Reminders or follow up calls
53. 53
Example 4
Environmental issues have become
increasingly important in choosing
hotels. Are environmental
considerations an important factor
when deciding on your choice of
hotel accommodation?
____ Yes ____ No
54. 54
Example 5
What information sources did you
use to locate your restaurant for
today’s meal?
(please tick appropriate spaces)
____ Yellow pages
____ Internet
____ Word of mouth
55. Pre-test & revise
Pre-test items and ask for feedback
Revise:
items which don’t apply to everybody
redundancy
skewed response items
misinterpreted items
non-completed items
Reconsider ordering & layout
56. Survey format checklist
Introduction/covering letter or verbal
introducation
e.g. Who are you? Are you bona fide?
Purpose of survey? Ethical approval? How
results will be used? Confidentiality?
Further info? Complaints?
Instructions
Sets the “mind frame”, but be aware few
people will read it without good prompting
and being easy-to-read
Group like questions together
Consider order effects, habituation, fatigue,
switching between response formats
57. Survey format
Font type / size, number of pages,
margins, double vs. single-siding,
colour, etc.
Demographics - usually beginning or
end; only use relevant questions
Space for comments?
Ending – say thanks!
Pre-test & revise/refine
59. 59
Comparison of
Data Collection Methods
Alreck and Settle (1995:32)
Personal Telephone Mail
Data collection costs High Medium Low
Data collection time required Medium Low High
Sample size for a given budget Small Medium Large
Data quantity per subject High Medium Low
Reaches widely dispersed sample No Maybe Yes
Reaches special locations Yes Maybe No
Interaction with respondents Yes Yes No
Degree of interviewer bias High Medium None
Severity of non-response Low Low High
Presentation of visual stimuli Yes No Maybe
Fieldworker training required Yes Yes No
62. 62
Why sample?
Why sampling rather than a
census?
Sampling reduces:
Cost, time, sample size and defines
the research
If the sample is representative,
allows inferences to be drawn
concerning the total population
63. What is sampling?
“Sampling is the process of
selecting units (e.g., people,
organizations) from a
population of interest so that by
studying the sample we may
fairly generalize our results
back to the population from
which they were chosen.”
- Trochim, 2002
64. 64
Sampling frame
The set of participants from which
the sample is drawn
Examples:
Electoral Rolls
Membership Lists (organisations,
graduates association)
Telephone Book
Members of Specific Groups or Clubs
(Fishing, Ramblers)
Households or post codes
65. Representativeness of sample
depends on:
Adequacy of sampling frame
Selection strategy
Adequacy of sample size
Response rate – both the % &
representativeness of people in
sample who actually complete
survey
Note: It is better to have a small,
good sample than a large, poor
sample.
67. Male-female relations
Shere Hite ‘doyenne of sex polls’
Media furors & worldwide attention
127-item questionnaire about
marriage & relations between sexes
4500 USA women, 14 to 85 years
Society and men need to change to
improve lives of women
68. Some of Hite’s findings....
70% married for 5 years having affairs...
(usually more for ‘emotional closeness’ than
sex)
76% did not feel guilty
87% had a closer female friend than husband
98% wanted “basic changes” to love
relationships
only 13% married for 2+years were still in love
84% were emotionally unsatisfied
95% reported emotional & psychological
harassment from their men
69. Some of the critical
comments....
“She goes in with prejudice & comes
out with a statistic.”
“The survey often seems merely to
provide an occasion for the author’s
own male-bashing diatribes.”
“Hite uses statistics to bolster her
opinion that American women are
justifiably fed up with American men.”
70. Response rate & Selection bias - 1
100,000 questionnaires
Sent to a variety of women’s
groups
- feminist organisations, church
groups, garden clubs, etc.
4,500 replied
(4.5% return rate)
71. “We get pretty nervous if
respondents in our survey go
under 70%. Respondents to
surveys differ from
nonrespondents in one important
way: they go to the trouble of filling
out what in this case was a very
long, complicated, and personal
questionnaire.”
- Regina Herzog, University of Michigan Institute for
Response rate & Selection bias - 2
73. 73
Random/probability sampling
Each unit has an equal (and
perhaps known) chance or
probability of selection in the
sample
Selection occurs entirely by random
chance
Often called representative
sampling
74. 74
Simple random sampling
Everyone in the target population
has an equal chance of selection
Useful if clear study area or
population is identified
Similar to a lottery
List of names are assigned #s and
randomly select #s of respondents
Randomly select # through table of
random #s or by computer
75. 75
Systematic random sampling
Selecting without first numbering
Respondents (units) selected from a
list/file.
Useful when survey population is
similar e.g. List of Students, List of
Package Tourists
Select sample at regular intervals
from the population e.g., every 5th
person on a list
76. 76
Systematic random sampling
Cannot do 1 in every 5
As then 4 people out of 5 stand no
chance of being selected
Select a random starting point
between 1 and 5
77. 77
Stratified random sampling
Sub-divide population into strata
(e.g., by gender, age, or location)
Then random selection from within
each stratum
Improves representativeness
e.g., Telephone interviews using
post-code strata
78. 78
Non-random/Non-probability
Also called purposive or
judgemental sampling
Useful for exploratory research and
case study research
Able to get large sample size quickly
and useful when can’t find a
sample frame
80. 80
Convenience sampling
Sampling is by convenience rather
than randomly
Due to time/financial constraints
e.g. surveying all those at a tourist
attraction over one weekend
81. 81
Purposive sampling
Respondents selected for a
particular purpose e.g., because
they may be “typical” respondents
e.g., select sample of tourists aged
40-60 as this is the typical age
group of visitors to Canberra
e.g., Frequent flyers to contact
regarding service quality in an
airline setting
82. 82
Snowballing
Useful for difficult to access
populations e.g., illegal
immigratnts, drug users
Respondents recommend other
respondents
e.g., in studying ecstasy users, gain trust of a
few potential respondents and ask them to
recommend the researcher to other
potential respondents
83. Summary of sampling
strategy
Identify target population and
sampling frame
Selection sampling method
Calculate required sample size
Maximise return rate
84. 84
Task
A research project's aim is –
“To identify the behaviour and
attitudes of UC students with
regard to its computing services”.
What is the research population?
How might you get hold of a sample
frame?
What sampling technique would you
use?
