Survey design workshop

Survey Design Workshop Inter-University Research Workshop Program Dr. James Neill Centre for Applied Psychology University of Canberra 11 February, 2009

Outline

Workshop Objectives
Introductions
Logins & Resources
Research methods
Questionnaire design
Implementation
Conclusion & Evaluation

Workshop Objectives

To examine issues surrounding questionnaire design including question style, response formats, layout, and pilot testing
To consider implementation issues (such as sampling)
To demonstrate the importance of rigour in planning, developing, and implementing research questionnaires

Learning Outcomes

Understanding common methods for the design and implementation of survey-based research and the pros and cons of each method (e.g., f2f interview, mail survey, internet survey).
Understanding the importance of a rigorous, step-by-step process for the development of instrumentation
Planning, drafting, and/or revising of an initial draft (pilot) survey.

Resources

Survey Design Workshop Notes (Wikiversity)
Readings

Books about designing surveys and survey research... Look for texts like these in your libraries.

Introductions Introductions

What is a survey? A “standardised” stimulus A measuring instrument

What is a survey? A way of converting “fuzzy stuff” into hard data for analysis

Purposes of survey research

Information gathering & describing

e.g., polls, attitudes, demographics

Theory-building & testing

Explanatory, e.g., why?
Predictive, e.g., what is likely to happen?

Often survey research does some of both.

Types of Surveys (Research Purposes) Descriptive

Collects basic descriptive data/statistics e.g., consumer profiles…(age, gender)

Explanatory

Examine underlying data patterns
Linked to a hypothesis/research objective

Types of Surveys (Research Purposes) Predictive

What happens if…
Useful for marketing or assessing consumer behavior
Honours-MA-Ph.D survey research

Research Paradigms Phenomenological Positivist

Usually small # of participants
More about opinions & experiences
Focused on ethnographic style of research
Usually associated with qualitative data

Deductive
Seeks to explain causal relationships between variables
Usually uses quantitative data
Follows a structured approach or methodology

Quantitative Research Methods

Advantages

Larger sample sizes  greater confidence in results
Results can be representative of target population
Data can be readily summarised using computers

Quantitative Research Methods

Disadvantages

Impersonal (don’t get background context)
Large samples needed
Samples may not be representative
Cost and time involved

Types of Questionnaires Self - administered Interview - administered Postal questionnaire Delivery and collection questionnaire Telephone survey Face to face structured interview Web-based

Alreck and Settle (1995:26)

Questionnaire Planning/Design 1. Formulate Generic Questionnaire 2. Expand the Questionnaire Based on study objectives Turn into separate sections Question styles & types 3. Finalise Questionnaire -Pre-test/pilot test -Several drafts needed Placement & Funnel Qs

Formulate Generic Questionnaire

Turn objectives into sections of the survey
Ensure all questions relate to research objectives
For explanatory objectives or hypotheses ensure both dependent and independent variables exist

Cover Letter / Ethics Statement Outline details of research project

Purpose
What's involved?
Explain any risks/costs/rewards
Contact details
Human Ethics approval #
How is consent given/not give?
How to return?
Can choose not to continue anytime

Instructions

Provides consistency - helps to ensure standard conditions across different administrations
Explain how to do the survey in a user-friendly manner
Example: Life Effectiveness Questionnaire

Expanding the Survey

Screening

Does the participant qualify for the survey? (esp. for internet surveys)
Ask screening questions first, rather than later
Use branching if there are conditional questions

Flow/Structure

Logical order of questions (use sections)
Use funnel questions to move respondents through survey
Start off with easy to answer and engaging questions
More controversial questions in middle section

Personal Information

Put personal information at end of survey (if required), otherwise it may put off respondents
Consider response format e.g, Income in categories or ranges

Personal Information

More likely to respond to personal questions for anonymous or mail surveys as opposed to face to face or telephone
Show cards for responses may help for face to face interviews

Types of Questions Be able to justify and defend your choices ...

Types of Questions

Open-ended Questions

Rich information can be gathered
Useful for descriptive, exploratory work
Difficult and subjective to analyse
Time consuming

Closed-ended Questions

Important information may be lost forever
Useful for hypothesis testing
Easy and objective to analyse
Time-efficient

Open-ended Question Examples

What are the main issues you are currently facing in your life?
How many hours did you spend studying this week? _________

Closed-ended Question Types

Dichotomous questions
Multichotomous questions
The list (multiple response)
Ranking
Likert Scale
Graphical Scale
Semantic Differential
Non-verbal (Idiographic)

Dichotomous Simple Yes / No response e.g., Excluding this trip, have you visited Canberra in the previous five years? __ Yes __ No

Multichotomous More than two possible answers e.g., What type of attractions in your current trip to Canberra most appeal to you? __ historic buildings __ museum/art galleries __ parks and gardens

Multichotomous How many hours did you spend studying this week? __ less than 5 hours __ > 5 to 10 hours __ > 10 to 20 hours __ more than 20 hours

Frequency Scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment

The List (Multiple Response) Provides a list of answers for respondents to choose from e.g., Tick any words or phrases that describe your perception of Canberra as a travel destination: __ Exciting __ Important __ Boring __ Enjoyable __ Interesting __ Historical

The List (Muliple-Response) What are the main issues you are currently facing in your life? (tick all that apply) __ financial __ physical/health __ academic __ employment/unemployment __ intimate relations __ social relations __ other (please specify)

Ranking Helps to measure the relative importance of several items Rank the importance of these reasons for taking a holiday to Canberra (from 1 (most) to 4 (least)): __ to visit friends and relatives __ for business __ for educational purposes __ for holiday/ sightseeing

Likert Scale Assesses differences of perceptions and allows measurement and comparison of these differences Indicate your degree of agreement with this statement: “ I am an adventurous person.” (circle the best response for you)

Graphical Rating Scale How would you rate your enjoyment of the movie you just saw? Mark with a cross (X) not enjoyable very enjoyable

Semantic Differential What is your view of smoking? Tick to show your opinion. Bad ___:___:___:___:___:___:___ Good Strong ___:___:___:___:___:___:___ Weak Masculine ___:___:___:___:___:___:___ Feminine Unattractive ___:___:___:___:___:___:___ Attractive Passive ___:___:___:___:___:___:___ Active

Non-verbal (Idiographic) Scale Point to the face that shows how you feel about what happened to the toy.

Sensitivity & Reliability

Scale should be sensitive yet reliable.
Watch out for too few or too many options

General aim: Maximise sensitivity (i.e. more options) Maximise reliability (i.e. less options) How many measurement options?

Minimum = 2
Average = 3 to 7
Maximum = 10?

Scale of Measurement Guidelines

FEELING ABOUT SOMETHING EXTREMELY POSITIVE EXTREMELY NEGATIVE 2-Categories GOOD NOT GOOD 3-Categories GOOD FAIR POOR 4-Categories VERY GOOD GOOD FAIR POOR 5-Categories EXCELLENT VERY GOOD GOOD FAIR POOR

Watch out for too many or too few responses “ Capital punishment should be reintroduced for serious crimes” 1 = Agree 2 = Disagree 1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral

Wording Questions

Does the question focus directly on the issue or topic to be measured? (If not, rewrite.)
Is the question stated as briefly as it can be? (If too long, restate it more briefly.)

Wording Questions

Is the question expressed as clearly and simply as it can be? I (f the meaning won’t be clear to every respondent, restructure the question.)
Use only core vocabulary - words and phrases people use in casual speech

Wording Questions

Limit the vocabulary so the least sophisticated respondent would be familiar with the words
Use simple sentences where possible and complex sentences only when actually required
Use two or more short, simple sentences rather than one compound or complex sentences

Finalise Questionnaire Draft

Questions need to be exhaustive and mutually exclusive

Include ‘other (please specify)’
Ensure categories do not overlap

Finalise Questionnaire Draft

Length

Try to keep them as short as possible
Only ask questions that relate to objectives
Tricks? Font size/double sided photocopying/numbering sections

Pre-testing and Pilot testing

Pre-test – try out on convenient others & revise
Pilot test – try out on a small sample from the target population & revise
Be assertive and interactive about seeking feedback – ask questions & observe
“The customer is always right.”

Maximising Response Rate

Layout and design is key
Respondent’s level of interest
Colour of paper
Accompanying letter / introduction
Mail surveys - self-addressed stamped return envelope
Rewards
Reminders or follow up calls

Examples

Examine the examples
What is wrong with the questions, if anything?

Examples How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over

Examples Are you satisfied with your marriage and your job? __________________________

Examples You didn’t think the food was very good, did you? _____ Yes _____ No

Examples Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No

Examples What information sources did you use to locate your restaurant for today’s meal? (please tick appropriate spaces) ____ yellow pages ____ Internet ____ word of mouth

Examples

Examine questionnaire examples
Examine structure, design issues and question styles
Note cover page and details provided

Pre-test & Revise

Pre-test items and ask for feedback
Revise:

items which don’t apply to everybody
redundancy
skewed response items
misinterpreted items
non-completed items

Reconsider ordering & layout

Survey Format Checklist

Introduction/covering letter or verbal introducation

e.g. Who are you? Are you bona fide? Purpose of survey? Ethical approval? How results will be used? Confidentiality? Further info? Complaints?

Instructions

Sets the “mind frame”, but be aware few people will read it without good prompting and being easy-to-read

Group like questions together
Consider order effects, habituation, fatigue, switching between response formats

Survey Format

Font type / size, number of pages, margins, double vs. single-siding, colour, etc.
Demographics - single section, usually at beginning or end of questionnaire, only use relevant questions
Space for comments?
Ending the questionnaire – say thanks!
Pre-test the questionnaire & revise/refine

LUNCH BREAK

Survey Design Critique In pairs, look through the example questionnaires, and highlight aspects which:

could be improved
are particularly good
you would like to ask about

Implementing Surveys

Research Proposal

What are your study parameters?
When will you carry out the research?
How long will data collection last?
Who is your target population?
What are your time and cost constraints?
What research method will be used?

Types of Questionnaires Self - administered Interview - administered Postal questionnaire Delivery and collection questionnaire Telephone survey Face to face structured interview Web-based

Comparison of Data Collection Methods Alreck and Settle (1995:32)

Sampling

Sampling Terminology

What is Sampling?
Sampling Techniques
Example: Shere Hite’s Sex Survey
Summary of Sampling Strategy

Population
Sampling Frame
Sample
Representativeness

What is a sample?

A proportion of the research or survey population
Sample of a population being all ‘known cases’

Why sample?

Why sampling rather than a census?
Sampling reduces:

Cost, time, sample size and defines the research
If the sample is representative, allows inferences to be drawn concerning the total population

What is sampling? “ Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” - Trochim, 2002

To generate a sample you need a sampling frame to select your sample from
List or database of all ‘known’ cases in a research population or study area (e.g., Canberra residents, Students at the University of Canberra, those staying in 5 star hotels in Sydney)

Sampling frame

The set of participants from which the sample is drawn
Examples:

Electoral Rolls
Membership Lists (organisations, graduates association)
Telephone Book
Members of Specific Groups or Clubs (Fishing, Ramblers)
Households or post codes

Sampling frame

What is the scope of your study? e.g.,

Local residents of Canberra over 18 or Canberra Households?
5 Star hotel guests in last 5 years, 12 months, 2 weeks?

Sampling frame

Difficulties in identifying research population or finding a reasonable sampling frame e.g., ecstasy users
How accurate/up to date are lists or sample frames? Up to date? Are they missing members of the research population? e.g., Ph books
These issues impact on sampling and data collection techniques

Representativeness of sample depends on:

Adequacy of sampling frame
Selection strategy
Adequacy of sample size
Response rate – both the % & representativeness of people in sample who actually complete survey
Note: It is better to have a small, good sample than a large, poor sample.

Sampling Example: Shere Hite ‘American Sexology’

Male-Female Relations

Shere Hite ‘doyenne of sex polls’
Media furors & worldwide attention
127-item questionnaire about marriage & relations between sexes
4500 USA women, 14 to 85 years
Society and men need to change to improve lives of women

Some of Hite’s findings....

70% married for 5 years having affairs...

(usually more for ‘emotional closeness’ than sex)

76% did not feel guilty
87% had a closer female friend than husband
98% wanted “basic changes” to love relationships
only 13% married for 2+years were still in love
84% were emotionally unsatisfied
95% reported emotional & psychological harassment from their men

Some of the critical comments....

“She goes in with prejudice & comes out with a statistic.”
“The survey often seems merely to provide an occasion for the author’s own male-bashing diatribes.”
“Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.”

Response rate & Selection bias - 1 100,000 questionnaires Sent to a variety of women’s groups - feminist organisations, church groups, garden clubs, etc. 4,500 replied (4.5% return rate)

“ We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.” - Regina Herzog, University of Michigan Institute for Social Research Response rate & Selection bias - 2

Sampling Techniques

Probability (random) sampling

Random
Systematic
Cluster

Multi-Stage Cluster

Non-probability sampling

Quota
Convenience
Snowball

Random/probability sampling

Each unit has an equal (and perhaps known) chance or probability of selection in the sample
Selection occurs entirely by random chance
Often called representative sampling

Simple random sampling

Everyone in the target population has an equal chance of selection
Useful if clear study area or population is identified
Similar to a lottery
List of names are assigned #s and randomly select #s of respondents
Randomly select # through table of random #s or by computer

Systematic random sampling

Selecting without first numbering
Respondents (units) selected from a list/file.
Useful when survey population is similar e.g. List of Students, List of Package Tourists
Select sample at regular intervals from the population e.g., every 5 th person on a list

Systematic random sampling

Cannot do 1 in every 5
As then 4 people out of 5 stand no chance of being selected
Select a random starting point between 1 and 5

Stratified Random Sampling

Sub-divide population into strata (e.g., by gender, age, or location)
Then random selection from within each stratum
Improves representativeness
e.g., Telephone interviews using post-code strata

Non-Random/Non-Probability

Also called purposive or judgemental sampling
Useful for exploratory research and case study research
Able to get large sample size quickly and useful when can’t find a sample frame

Non-Random/Non-Probability

Make assumptions and maybe generalisations from your data, but not on statistical grounds
Limitations include potential bias and applicability

Convenience Sampling

Sampling is by convenience rather than randomly
Due to time/financial constraints
e.g. surveying all those at a tourist attraction over one weekend

Purposive Sampling

Respondents selected for a particular purpose e.g., because they may be “typical” respondents
e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra
e.g., Frequent flyers to contact regarding service quality in an airline setting

Summary of sampling strategy

Identify target population and sampling frame
Selection sampling method
Calculate required sample size
Maximise return rate

Task A research project's aim is – “To identify the behaviour and attitudes of UC students with regard to its computing services”.

What is the research population?
How might you get hold of a sample frame?
What sampling technique would you use?

Confidence Levels / Margins of Error

Relates to representativeness of a sample of a target population - to what extent can we be confident about the results?
Gives the estimated range of values into which we expect other samples to fall say 95% of the time
Social sciences = at least 90% or above, preferably 95%

Example 95% Confidence Level Graphs

Example 95% Confidence Level Table

Confidence Level Example Survey of visitors to Canberra between June 1 - August 31, 2004 = 10,000 visitors Want to work with 95% confidence level and 3% margin of error How many to survey? Answer: Sample size = 964 people Q: What are your favourite attractions?

Following Results + / - 3% error @ 95% confidence level

Confidence Intervals / Margins of Error

One time out of 20, we expect the answers may be greater than +/- 3%

Establish the confidence level, margin of error you want to work with
Identify the number of surveys to be done
Once you have completed that number, do not do any additional surveys

HOWEVER….

Sometimes we do surveys and we do not know how many will be returned until later, as with postal surveys
Thus you have to calculate the margin of error afterwards….

Count up # of returned surveys
Identify target population and confidence level
Calculate margin of error

All this assumes you know your target population and can get a sample frame

If not, a non-random sampling technique is best

Also consider whether you want to report results for sub-groups – the margins of error will be wider

Consider stratified random sampling

Evaluation

Please complete the workshop evaluations

References

Alreck, P. L. & Settle, R. B. (2003). The survey research handbook (3rd ed.). Chicago: McGraw-Hill/Irwin.
See also Readings

Acknowledgements This is an adaptation and extension of presention slides originally developed by (and with permission from) Dr Brent Ritchie , currently at the School of Tourism at The University of Queensland.

Open Office Impress