Survey Design: Introduction & Overview Dr. James Neill Centre for Applied Psychology University of Canberra 28 September, 2009

Outline Research process Questionnaire design Types of questions Response formats Sampling

Research process

Questionnaire design

Types of questions

Response formats

Sampling

Objectives To introduce key aspects of questionnaire design including question style, response formats, layout, and pilot testing To consider implementation issues (such as sampling) To demonstrate the importance of rigour in planning, developing, and implementing research questionnaires

To introduce key aspects of questionnaire design including question style, response formats, layout, and pilot testing

To consider implementation issues (such as sampling)

To demonstrate the importance of rigour in planning, developing, and implementing research questionnaires

Resources Survey Design (Wikiversity) Readings (Wikiversity)

Survey Design (Wikiversity)

Readings (Wikiversity)

Books Look for books about surveys design and survey research in libraries.

Research process

What is a survey? A “standardised” stimulus A measuring instrument

What is a survey? A way of converting “fuzzy stuff” into hard data for analysis

Purposes of survey research Information gathering & describing e.g., polls, attitudes, demographics Theory-building & testing Explanatory, e.g., why? Predictive, e.g., what is likely to happen? Often survey research does some of both.

Information gathering & describing

e.g., polls, attitudes, demographics

Theory-building & testing

Explanatory, e.g., why?

Predictive, e.g., what is likely to happen?

Often survey research does some of both.

Research purposes Descriptive Collects basic descriptive data/statistics e.g., consumer profiles…(age, gender) Explanatory Examine underlying data patterns Linked to a hypothesis/research objective

Collects basic descriptive data/statistics e.g., consumer profiles…(age, gender)

Examine underlying data patterns

Linked to a hypothesis/research objective

Research purposes Predictive What happens if… Useful for marketing or assessing consumer behavior Honours-MA-Ph.D survey research

What happens if…

Useful for marketing or assessing consumer behavior

Honours-MA-Ph.D survey research

Types of Questionnaires Self - administered Interview - administered Postal questionnaire Delivery and collection questionnaire Telephone survey Face to face structured interview Web-based

Designing a survey

Questionnaire planning/design 1. Formulate Generic Questionnaire 2. Expand the Questionnaire Based on study objectives Turn into separate sections Question styles & types 3. Finalise Questionnaire -Pre-test/pilot test -Several drafts needed Placement & Funnel Qs

Formulate generic questionnaire Turn objectives into sections of the survey Ensure all questions relate to research objectives For explanatory objectives or hypotheses ensure both dependent and independent variables exist

Turn objectives into sections of the survey

Ensure all questions relate to research objectives

For explanatory objectives or hypotheses ensure both dependent and independent variables exist

Cover letter / Ethics statement Outline details of research project Purpose What's involved? Explain any risks/costs/rewards Contact details Human Ethics approval # How is consent given/not give? How to return? Can choose not to continue anytime

Purpose

What's involved?

Explain any risks/costs/rewards

Contact details

Human Ethics approval #

How is consent given/not give?

How to return?

Can choose not to continue anytime

Instructions Provides consistency - helps to ensure standard conditions across different administrations Explain how to do the survey in a user-friendly manner Example: Life Effectiveness Questionnaire

Provides consistency - helps to ensure standard conditions across different administrations

Explain how to do the survey in a user-friendly manner

Example: Life Effectiveness Questionnaire

Expanding the survey

Screening Does the participant qualify for the survey? (esp. for internet surveys) Ask screening questions first, rather than later Use branching if there are conditional questions

Does the participant qualify for the survey? (esp. for internet surveys)

Ask screening questions first, rather than later

Use branching if there are conditional questions

Flow and structure Logical order of questions (use sections) Use funnel questions to move respondents through survey Start off with easy to answer and engaging questions More controversial questions in middle section Personal questions – start or end?

Logical order of questions (use sections)

Use funnel questions to move respondents through survey

Start off with easy to answer and engaging questions

More controversial questions in middle section

Personal questions – start or end?

Survey design principles Jenkins and Dillman (1995) 1. Use the visual elements of brightness, color, shape, and location in a consistent manner to define the desired navigational path for respondents to follow when answering the questionnaire.

Survey design principles Jenkins and Dillman (1995) 2. When established format conventions are changed in the midst of a questionnaire use prominent visual guides to redirect respondents.

Survey design principles Jenkins and Dillman (1995) 3. Place directions where they are to be used and where they can be seen. 4. Present information in a manner that does not require respondents to connect information from separate locations in order to comprehend it.

Types of questions

Types of questions Be able to justify and defend your choices ...

Open-ended questions Rich information can be gathered Useful for descriptive, exploratory work Difficult and subjective to analyse Time consuming

Rich information can be gathered

Useful for descriptive, exploratory work

Difficult and subjective to analyse

Time consuming

Open-ended question examples What are the main issues you are currently facing in your life? How many hours did you spend studying this week? _________

What are the main issues you are currently facing in your life?

How many hours did you spend studying this week? _________

Closed-ended questions Important information may be lost forever Useful for hypothesis testing Easy and objective to analyse Time-efficient

Important information may be lost forever

Useful for hypothesis testing

Easy and objective to analyse

Time-efficient

Closed-ended question types Dichotomous questions Multichotomous questions The list (multiple response) Ranking Likert Scale Graphical Scale Semantic Differential Non-verbal (Idiographic)

Dichotomous questions

Multichotomous questions

The list (multiple response)

Ranking

Likert Scale

Graphical Scale

Semantic Differential

Non-verbal (Idiographic)

Dichotomous Simple Yes / No response e.g., Excluding this trip, have you visited Canberra in the previous five years? __ Yes __ No

Multichotomous Choice one of several answers e.g. How many hours did you spend studying this week? __ less than 5 hours __ > 5 to 10 hours __ > 10 to 20 hours __ more than 20 hours

Frequency scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment

The list (Multiple response) Provides a list of answers for respondents to choose from e.g., Tick any words or phrases that describe your perception of Canberra as a travel destination: __ Exciting __ Important __ Boring __ Enjoyable __ Interesting __ Historical

Ranking Helps to measure the relative importance of several items Rank the importance of these reasons for taking a holiday to Canberra (from 1 (most) to 4 (least)): __ to visit friends and relatives __ for business __ for educational purposes __ for holiday/ sightseeing

Likert Scale Assesses differences of perceptions and allows measurement and comparison of these differences Indicate your degree of agreement with this statement: “ I am an adventurous person.” (circle the best response for you)

Graphical rating scale How would you rate your enjoyment of the movie you just saw? Mark with a cross (X) not enjoyable very enjoyable

Semantic differential What is your view of smoking? Tick to show your opinion. Bad ___:___:___:___:___:___:___ Good Strong ___:___:___:___:___:___:___ Weak Masculine ___:___:___:___:___:___:___ Feminine Unattractive ___:___:___:___:___:___:___ Attractive Passive ___:___:___:___:___:___:___ Active

Non-verbal (Idiographic) Scale Point to the face that shows how you feel about what happened to the toy.

Sensitivity & reliability Scale should be sensitive yet reliable. Watch out for too few or too many options

Scale should be sensitive yet reliable.

Watch out for too few or too many options

General aim: Maximise sensitivity (i.e. more options) Maximise reliability (i.e. less options) How many measurement options? Minimum = 2 Average = 3 to 7 Maximum = 10? Scale of measurement guidelines

Minimum = 2

Average = 3 to 7

Maximum = 10?

FEELING ABOUT SOMETHING EXTREMELY POSITIVE EXTREMELY NEGATIVE 2-Categories GOOD NOT GOOD 3-Categories GOOD FAIR POOR 4-Categories VERY GOOD GOOD FAIR POOR 5-Categories EXCELLENT VERY GOOD GOOD FAIR POOR

Watch out for too many or too few responses “ Capital punishment should be reintroduced for serious crimes” 1 = Agree 2 = Disagree 1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral

Wording questions Does the question focus directly on the issue or topic to be measured? (If not, rewrite.) Is the question stated as briefly as it can be? (If too long, restate it more briefly.)

Does the question focus directly on the issue or topic to be measured? (If not, rewrite.)

Is the question stated as briefly as it can be? (If too long, restate it more briefly.)

Wording questions Is the question expressed as clearly and simply as it can be? I (f the meaning won’t be clear to every respondent, restructure the question.) Use only core vocabulary - words and phrases people use in casual speech

Is the question expressed as clearly and simply as it can be? I (f the meaning won’t be clear to every respondent, restructure the question.)

Use only core vocabulary - words and phrases people use in casual speech

Wording questions Limit the vocabulary so the least sophisticated respondent would be familiar with the words Use simple sentences where possible and complex sentences only when actually required Use two or more short, simple sentences rather than one compound or complex sentences

Limit the vocabulary so the least sophisticated respondent would be familiar with the words

Use simple sentences where possible and complex sentences only when actually required

Use two or more short, simple sentences rather than one compound or complex sentences

Finalise questionnaire draft Length Try to keep them as short as possible Only ask questions that relate to objectives Tricks? Font size/double sided photocopying/numbering sections

Length

Try to keep them as short as possible

Only ask questions that relate to objectives

Tricks? Font size/double sided photocopying/numbering sections

Pre-testing and pilot testing Pre-test – try out on convenient others & revise Pilot test – try out on a small sample from the target population & revise Be assertive and interactive about seeking feedback – ask questions & observe “The customer is always right.”

Pre-test – try out on convenient others & revise

Pilot test – try out on a small sample from the target population & revise

Be assertive and interactive about seeking feedback – ask questions & observe

“The customer is always right.”

Maximising response rate Layout and design is key Respondent’s level of interest Colour of paper Accompanying letter / introduction Mail surveys - self-addressed stamped return envelope Rewards Reminders or follow up calls

Layout and design is key

Respondent’s level of interest

Colour of paper

Accompanying letter / introduction

Mail surveys - self-addressed stamped return envelope

Rewards

Reminders or follow up calls

Examples Examine the examples What is wrong with the questions, if anything?

Examine the examples

What is wrong with the questions, if anything?

Example 1 How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over

Example 2 Are you satisfied with your marriage and your job? __________________________

Example 3 You didn’t think the food was very good, did you? _____ Yes _____ No

Example 4 Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No

Example 5 What information sources did you use to locate your restaurant for today’s meal? (please tick appropriate spaces) ____ Yellow pages ____ Internet ____ Word of mouth

Pre-test & revise Pre-test items and ask for feedback Revise: items which don’t apply to everybody redundancy skewed response items misinterpreted items non-completed items Reconsider ordering & layout

Pre-test items and ask for feedback

Revise:

items which don’t apply to everybody

redundancy

skewed response items

misinterpreted items

non-completed items

Reconsider ordering & layout

Survey format checklist Introduction/covering letter or verbal introducation e.g. Who are you? Are you bona fide? Purpose of survey? Ethical approval? How results will be used? Confidentiality? Further info? Complaints? Instructions Sets the “mind frame”, but be aware few people will read it without good prompting and being easy-to-read Group like questions together Consider order effects, habituation, fatigue, switching between response formats

Introduction/covering letter or verbal introducation

e.g. Who are you? Are you bona fide? Purpose of survey? Ethical approval? How results will be used? Confidentiality? Further info? Complaints?

Instructions

Sets the “mind frame”, but be aware few people will read it without good prompting and being easy-to-read

Group like questions together

Consider order effects, habituation, fatigue, switching between response formats

Survey format Font type / size, number of pages, margins, double vs. single-siding, colour, etc. Demographics - usually beginning or end; only use relevant questions Space for comments? Ending – say thanks! Pre-test & revise/refine

Font type / size, number of pages, margins, double vs. single-siding, colour, etc.

Demographics - usually beginning or end; only use relevant questions

Space for comments?

Ending – say thanks!

Pre-test & revise/refine

Implementing surveys

Comparison of Data Collection Methods Alreck and Settle (1995:32)

Sampling

Sampling terminology Population Sampling Frame Sample Representativeness

Population

Sampling Frame

Sample

Representativeness

Why sample? Why sampling rather than a census? Sampling reduces: Cost, time, sample size and defines the research If the sample is representative, allows inferences to be drawn concerning the total population

Why sampling rather than a census?

Sampling reduces:

Cost, time, sample size and defines the research

If the sample is representative, allows inferences to be drawn concerning the total population

What is sampling? “ Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” - Trochim, 2002

Sampling frame The set of participants from which the sample is drawn Examples: Electoral Rolls Membership Lists (organisations, graduates association) Telephone Book Members of Specific Groups or Clubs (Fishing, Ramblers) Households or post codes

The set of participants from which the sample is drawn

Examples:

Electoral Rolls

Membership Lists (organisations, graduates association)

Telephone Book

Members of Specific Groups or Clubs (Fishing, Ramblers)

Households or post codes

Representativeness of sample depends on: Adequacy of sampling frame Selection strategy Adequacy of sample size Response rate – both the % & representativeness of people in sample who actually complete survey Note: It is better to have a small, good sample than a large, poor sample.

Adequacy of sampling frame

Selection strategy

Adequacy of sample size

Response rate – both the % & representativeness of people in sample who actually complete survey

Note: It is better to have a small, good sample than a large, poor sample.

Sampling example: Shere Hite ‘American Sexology’

Male-female relations Shere Hite ‘doyenne of sex polls’ Media furors & worldwide attention 127-item questionnaire about marriage & relations between sexes 4500 USA women, 14 to 85 years Society and men need to change to improve lives of women

Shere Hite ‘doyenne of sex polls’

Media furors & worldwide attention

127-item questionnaire about marriage & relations between sexes

4500 USA women, 14 to 85 years

Society and men need to change to improve lives of women

Some of Hite’s findings.... 70% married for 5 years having affairs... (usually more for ‘emotional closeness’ than sex) 76% did not feel guilty 87% had a closer female friend than husband 98% wanted “basic changes” to love relationships only 13% married for 2+years were still in love 84% were emotionally unsatisfied 95% reported emotional & psychological harassment from their men

70% married for 5 years having affairs...

76% did not feel guilty

87% had a closer female friend than husband

98% wanted “basic changes” to love relationships

only 13% married for 2+years were still in love

84% were emotionally unsatisfied

95% reported emotional & psychological harassment from their men

Some of the critical comments.... “She goes in with prejudice & comes out with a statistic.” “The survey often seems merely to provide an occasion for the author’s own male-bashing diatribes.” “Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.”

“She goes in with prejudice & comes out with a statistic.”

“The survey often seems merely to provide an occasion for the author’s own male-bashing diatribes.”

“Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.”

Response rate & Selection bias - 1 100,000 questionnaires Sent to a variety of women’s groups - feminist organisations, church groups, garden clubs, etc. 4,500 replied (4.5% return rate)

“ We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.” - Regina Herzog, University of Michigan Institute for Social Research Response rate & Selection bias - 2

Sampling techniques Probability (random) sampling Random Systematic Cluster Multi-Stage Cluster Non-probability sampling Quota Convenience Snowball

Probability (random) sampling

Random

Systematic

Cluster

Multi-Stage Cluster

Non-probability sampling

Quota

Convenience

Snowball

Random/probability sampling Each unit has an equal (and perhaps known) chance or probability of selection in the sample Selection occurs entirely by random chance Often called representative sampling

Each unit has an equal (and perhaps known) chance or probability of selection in the sample

Selection occurs entirely by random chance

Often called representative sampling

Simple random sampling Everyone in the target population has an equal chance of selection Useful if clear study area or population is identified Similar to a lottery List of names are assigned #s and randomly select #s of respondents Randomly select # through table of random #s or by computer

Everyone in the target population has an equal chance of selection

Useful if clear study area or population is identified

Similar to a lottery

List of names are assigned #s and randomly select #s of respondents

Randomly select # through table of random #s or by computer

Systematic random sampling Selecting without first numbering Respondents (units) selected from a list/file. Useful when survey population is similar e.g. List of Students, List of Package Tourists Select sample at regular intervals from the population e.g., every 5 th person on a list

Selecting without first numbering

Respondents (units) selected from a list/file.

Useful when survey population is similar e.g. List of Students, List of Package Tourists

Select sample at regular intervals from the population e.g., every 5 th person on a list

Systematic random sampling Cannot do 1 in every 5 As then 4 people out of 5 stand no chance of being selected Select a random starting point between 1 and 5

Cannot do 1 in every 5

As then 4 people out of 5 stand no chance of being selected

Select a random starting point between 1 and 5

Stratified random sampling Sub-divide population into strata (e.g., by gender, age, or location) Then random selection from within each stratum Improves representativeness e.g., Telephone interviews using post-code strata

Sub-divide population into strata (e.g., by gender, age, or location)

Then random selection from within each stratum

Improves representativeness

e.g., Telephone interviews using post-code strata

Non-random/Non-probability Also called purposive or judgemental sampling Useful for exploratory research and case study research Able to get large sample size quickly and useful when can’t find a sample frame

Also called purposive or judgemental sampling

Useful for exploratory research and case study research

Able to get large sample size quickly and useful when can’t find a sample frame

Non-random/Non-probability Make assumptions and maybe generalisations from your data, but not on statistical grounds Limitations include potential bias and applicability

Make assumptions and maybe generalisations from your data, but not on statistical grounds

Limitations include potential bias and applicability

Convenience sampling Sampling is by convenience rather than randomly Due to time/financial constraints e.g. surveying all those at a tourist attraction over one weekend

Sampling is by convenience rather than randomly

Due to time/financial constraints

e.g. surveying all those at a tourist attraction over one weekend

Purposive sampling Respondents selected for a particular purpose e.g., because they may be “typical” respondents e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra e.g., Frequent flyers to contact regarding service quality in an airline setting

Respondents selected for a particular purpose e.g., because they may be “typical” respondents

e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra

e.g., Frequent flyers to contact regarding service quality in an airline setting

Snowballing Useful for difficult to access populations e.g., illegal immigratnts, drug users Respondents recommend other respondents e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents

Useful for difficult to access populations e.g., illegal immigratnts, drug users

Respondents recommend other respondents

e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents

Summary of sampling strategy Identify target population and sampling frame Selection sampling method Calculate required sample size Maximise return rate

Identify target population and sampling frame

Selection sampling method

Calculate required sample size

Maximise return rate

Task A research project's aim is – “To identify the behaviour and attitudes of UC students with regard to its computing services”. What is the research population? How might you get hold of a sample frame? What sampling technique would you use?

What is the research population?

How might you get hold of a sample frame?

What sampling technique would you use?