Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Survey Design: Introduction & Overview Dr. James Neill Centre for Applied Psychology University of Canberra 28 September, ...
Outline <ul><li>Research process
Questionnaire design
Types of questions
Response formats
Sampling </li></ul>
Objectives <ul><li>To introduce key aspects of  questionnaire design  including question style, response formats, layout, ...
To consider  implementation issues  (such as sampling)
To demonstrate the importance of  rigour  in planning, developing, and implementing research questionnaires </li></ul>
Resources <ul><li>Survey Design   (Wikiversity)
Readings  (Wikiversity) </li></ul>
Books Look for books about  surveys design and survey research in  libraries.
Research process
What is a survey? A “standardised” stimulus A measuring instrument
What is a survey? A way of converting  “fuzzy  stuff”  into hard data for analysis
Purposes of survey research <ul><li>Information gathering & describing </li><ul><li>e.g., polls, attitudes, demographics <...
Predictive, e.g., what is likely to happen? </li></ul><li>Often survey research does some of both. </li></ul>
Research purposes Descriptive <ul><li>Collects basic descriptive data/statistics e.g., consumer profiles…(age, gender) </l...
Linked to a hypothesis/research objective </li></ul>
Research purposes Predictive <ul><li>What happens if…
Useful for marketing or assessing consumer behavior
Honours-MA-Ph.D survey research </li></ul>
Types of Questionnaires Self - administered Interview - administered Postal questionnaire Delivery and collection question...
Designing a survey
Questionnaire planning/design 1. Formulate Generic  Questionnaire 2. Expand the Questionnaire Based on  study objectives  ...
Formulate generic questionnaire <ul><li>Turn objectives into sections of the survey
Ensure all questions relate to research objectives
For explanatory objectives or hypotheses ensure both dependent and independent variables exist </li></ul>
Cover letter / Ethics statement Outline details of research project <ul><li>Purpose
What's involved?
Explain any risks/costs/rewards
Contact details
Human Ethics approval #
How is consent given/not give?
How to return?
Can choose not to continue anytime </li></ul>
Instructions <ul><li>Provides consistency - helps to ensure standard conditions across different administrations
Explain how to do the survey in a user-friendly manner
Example:  Life Effectiveness Questionnaire </li></ul>
Expanding the survey
Screening <ul><li>Does the participant qualify for the survey? (esp. for internet surveys)
Ask screening questions first, rather than later
Use branching if there are conditional questions </li></ul>
Flow and structure <ul><li>Logical order of questions  (use sections)
Use funnel questions to move respondents through survey
Start off with easy to answer and engaging questions
More controversial questions in middle section
Personal questions – start or end? </li></ul>
Survey design principles Jenkins and Dillman (1995) 1. Use the visual elements of brightness, color, shape, and location i...
Survey design principles Jenkins and Dillman (1995) 2. When established format conventions are changed in the midst of a q...
Survey design principles Jenkins and Dillman (1995) 3. Place directions where they are to be used and where they can be se...
Types of questions
Types of questions Be able to justify and defend your  choices ...
Open-ended questions <ul><li>Rich information can be gathered
Useful for descriptive, exploratory work
Difficult and subjective to analyse
Time consuming </li></ul>
Open-ended question examples <ul><li>What are the main issues you are currently facing in your life?
How many hours did you spend studying this week? _________ </li></ul>
Closed-ended questions <ul><li>Important information may be lost forever
Useful for hypothesis testing
Easy and objective to analyse
Time-efficient </li></ul>
Closed-ended question types <ul><li>Dichotomous questions
Multichotomous questions
The list (multiple response)
Ranking
Likert Scale
Graphical Scale
Semantic Differential
Non-verbal (Idiographic) </li></ul>
Dichotomous Simple Yes / No response e.g.,  Excluding this trip, have you visited Canberra in the previous five years? __ ...
Multichotomous Choice one of several answers e.g. How many hours did you spend studying this week? __   less than 5 hours ...
Frequency scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often...
The list (Multiple response) Provides a list of answers for respondents to choose from e.g., Tick any words or phrases tha...
Ranking Helps to measure the relative importance of several items Rank the importance of these reasons for taking a holida...
Likert Scale Assesses differences of perceptions and allows measurement and comparison of these differences Indicate your ...
Graphical rating scale How would you rate your enjoyment of the movie you just saw?  Mark with a cross (X) not enjoyable  ...
Semantic differential What is your view of smoking?  Tick to show your opinion. Bad  ___:___:___:___:___:___:___  Good Str...
Non-verbal (Idiographic) Scale Point to the face that shows how you feel about what happened to the toy.
Sensitivity & reliability <ul><li>Scale should be sensitive yet reliable.
Watch out for too few or too many options </li></ul>
General aim:  Maximise sensitivity  (i.e. more options) Maximise reliability  (i.e. less options) How many measurement opt...
Average = 3 to 7
Upcoming SlideShare
Loading in …5
×

Survey Design: Introduction & Overview

14,431 views

Published on

Describes the nuts and bolts of designing a survey for research in the social sciences.

Published in: Education, Technology, Business
  • I've tried several times to download this presentation to my computer. Unfortunately, I end up with a zip file that I am unable to compress. Could you send this document to me, as a Microsoft PowerPoint file? My email address is DrOMalone@msn.com. Many thanks!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Survey Design: Introduction & Overview

  1. 1. Survey Design: Introduction & Overview Dr. James Neill Centre for Applied Psychology University of Canberra 28 September, 2009
  2. 2. Outline <ul><li>Research process
  3. 3. Questionnaire design
  4. 4. Types of questions
  5. 5. Response formats
  6. 6. Sampling </li></ul>
  7. 7. Objectives <ul><li>To introduce key aspects of questionnaire design including question style, response formats, layout, and pilot testing
  8. 8. To consider implementation issues (such as sampling)
  9. 9. To demonstrate the importance of rigour in planning, developing, and implementing research questionnaires </li></ul>
  10. 10. Resources <ul><li>Survey Design (Wikiversity)
  11. 11. Readings (Wikiversity) </li></ul>
  12. 12. Books Look for books about surveys design and survey research in libraries.
  13. 13. Research process
  14. 14. What is a survey? A “standardised” stimulus A measuring instrument
  15. 15. What is a survey? A way of converting “fuzzy stuff” into hard data for analysis
  16. 16. Purposes of survey research <ul><li>Information gathering & describing </li><ul><li>e.g., polls, attitudes, demographics </li></ul><li>Theory-building & testing </li><ul><li>Explanatory, e.g., why?
  17. 17. Predictive, e.g., what is likely to happen? </li></ul><li>Often survey research does some of both. </li></ul>
  18. 18. Research purposes Descriptive <ul><li>Collects basic descriptive data/statistics e.g., consumer profiles…(age, gender) </li></ul>Explanatory <ul><li>Examine underlying data patterns
  19. 19. Linked to a hypothesis/research objective </li></ul>
  20. 20. Research purposes Predictive <ul><li>What happens if…
  21. 21. Useful for marketing or assessing consumer behavior
  22. 22. Honours-MA-Ph.D survey research </li></ul>
  23. 23. Types of Questionnaires Self - administered Interview - administered Postal questionnaire Delivery and collection questionnaire Telephone survey Face to face structured interview Web-based
  24. 24. Designing a survey
  25. 25. Questionnaire planning/design 1. Formulate Generic Questionnaire 2. Expand the Questionnaire Based on study objectives Turn into separate sections Question styles & types 3. Finalise Questionnaire -Pre-test/pilot test -Several drafts needed Placement & Funnel Qs
  26. 26. Formulate generic questionnaire <ul><li>Turn objectives into sections of the survey
  27. 27. Ensure all questions relate to research objectives
  28. 28. For explanatory objectives or hypotheses ensure both dependent and independent variables exist </li></ul>
  29. 29. Cover letter / Ethics statement Outline details of research project <ul><li>Purpose
  30. 30. What's involved?
  31. 31. Explain any risks/costs/rewards
  32. 32. Contact details
  33. 33. Human Ethics approval #
  34. 34. How is consent given/not give?
  35. 35. How to return?
  36. 36. Can choose not to continue anytime </li></ul>
  37. 37. Instructions <ul><li>Provides consistency - helps to ensure standard conditions across different administrations
  38. 38. Explain how to do the survey in a user-friendly manner
  39. 39. Example: Life Effectiveness Questionnaire </li></ul>
  40. 40. Expanding the survey
  41. 41. Screening <ul><li>Does the participant qualify for the survey? (esp. for internet surveys)
  42. 42. Ask screening questions first, rather than later
  43. 43. Use branching if there are conditional questions </li></ul>
  44. 44. Flow and structure <ul><li>Logical order of questions (use sections)
  45. 45. Use funnel questions to move respondents through survey
  46. 46. Start off with easy to answer and engaging questions
  47. 47. More controversial questions in middle section
  48. 48. Personal questions – start or end? </li></ul>
  49. 49. Survey design principles Jenkins and Dillman (1995) 1. Use the visual elements of brightness, color, shape, and location in a consistent manner to define the desired navigational path for respondents to follow when answering the questionnaire.
  50. 50. Survey design principles Jenkins and Dillman (1995) 2. When established format conventions are changed in the midst of a questionnaire use prominent visual guides to redirect respondents.
  51. 51. Survey design principles Jenkins and Dillman (1995) 3. Place directions where they are to be used and where they can be seen. 4. Present information in a manner that does not require respondents to connect information from separate locations in order to comprehend it.
  52. 52. Types of questions
  53. 53. Types of questions Be able to justify and defend your choices ...
  54. 54. Open-ended questions <ul><li>Rich information can be gathered
  55. 55. Useful for descriptive, exploratory work
  56. 56. Difficult and subjective to analyse
  57. 57. Time consuming </li></ul>
  58. 58. Open-ended question examples <ul><li>What are the main issues you are currently facing in your life?
  59. 59. How many hours did you spend studying this week? _________ </li></ul>
  60. 60. Closed-ended questions <ul><li>Important information may be lost forever
  61. 61. Useful for hypothesis testing
  62. 62. Easy and objective to analyse
  63. 63. Time-efficient </li></ul>
  64. 64. Closed-ended question types <ul><li>Dichotomous questions
  65. 65. Multichotomous questions
  66. 66. The list (multiple response)
  67. 67. Ranking
  68. 68. Likert Scale
  69. 69. Graphical Scale
  70. 70. Semantic Differential
  71. 71. Non-verbal (Idiographic) </li></ul>
  72. 72. Dichotomous Simple Yes / No response e.g., Excluding this trip, have you visited Canberra in the previous five years? __ Yes __ No
  73. 73. Multichotomous Choice one of several answers e.g. How many hours did you spend studying this week? __ less than 5 hours __ > 5 to 10 hours __ > 10 to 20 hours __ more than 20 hours
  74. 74. Frequency scale Over the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment
  75. 75. The list (Multiple response) Provides a list of answers for respondents to choose from e.g., Tick any words or phrases that describe your perception of Canberra as a travel destination: __ Exciting __ Important __ Boring __ Enjoyable __ Interesting __ Historical
  76. 76. Ranking Helps to measure the relative importance of several items Rank the importance of these reasons for taking a holiday to Canberra (from 1 (most) to 4 (least)): __ to visit friends and relatives __ for business __ for educational purposes __ for holiday/ sightseeing
  77. 77. Likert Scale Assesses differences of perceptions and allows measurement and comparison of these differences Indicate your degree of agreement with this statement: “ I am an adventurous person.” (circle the best response for you)
  78. 78. Graphical rating scale How would you rate your enjoyment of the movie you just saw? Mark with a cross (X) not enjoyable very enjoyable
  79. 79. Semantic differential What is your view of smoking? Tick to show your opinion. Bad ___:___:___:___:___:___:___ Good Strong ___:___:___:___:___:___:___ Weak Masculine ___:___:___:___:___:___:___ Feminine Unattractive ___:___:___:___:___:___:___ Attractive Passive ___:___:___:___:___:___:___ Active
  80. 80. Non-verbal (Idiographic) Scale Point to the face that shows how you feel about what happened to the toy.
  81. 81. Sensitivity & reliability <ul><li>Scale should be sensitive yet reliable.
  82. 82. Watch out for too few or too many options </li></ul>
  83. 83. General aim: Maximise sensitivity (i.e. more options) Maximise reliability (i.e. less options) How many measurement options? <ul><li>Minimum = 2
  84. 84. Average = 3 to 7
  85. 85. Maximum = 10? </li></ul>Scale of measurement guidelines
  86. 86. FEELING ABOUT SOMETHING EXTREMELY POSITIVE EXTREMELY NEGATIVE 2-Categories GOOD NOT GOOD 3-Categories GOOD FAIR POOR 4-Categories VERY GOOD GOOD FAIR POOR 5-Categories EXCELLENT VERY GOOD GOOD FAIR POOR
  87. 87. Watch out for too many or too few responses “ Capital punishment should be reintroduced for serious crimes” 1 = Agree 2 = Disagree 1 = Very, Very Strongly Agree 7 = Slightly Disagree 2 = Very Strongly Agree 8 = Disagree 3 = Strongly Agree 9 = Strongly Disagree 4 = Agree 10 = V. Strongly Disagree 5 = Slightly Agree 11 = V, V Strongly Disagree 6 = Neutral
  88. 88. Wording questions <ul><li>Does the question focus directly on the issue or topic to be measured? (If not, rewrite.)
  89. 89. Is the question stated as briefly as it can be? (If too long, restate it more briefly.) </li></ul>
  90. 90. Wording questions <ul><li>Is the question expressed as clearly and simply as it can be? I (f the meaning won’t be clear to every respondent, restructure the question.)
  91. 91. Use only core vocabulary - words and phrases people use in casual speech </li></ul>
  92. 92. Wording questions <ul><li>Limit the vocabulary so the least sophisticated respondent would be familiar with the words
  93. 93. Use simple sentences where possible and complex sentences only when actually required
  94. 94. Use two or more short, simple sentences rather than one compound or complex sentences </li></ul>
  95. 95. Finalise questionnaire draft <ul><li>Length </li><ul><li>Try to keep them as short as possible
  96. 96. Only ask questions that relate to objectives
  97. 97. Tricks? Font size/double sided photocopying/numbering sections </li></ul></ul>
  98. 98. Pre-testing and pilot testing <ul><li>Pre-test – try out on convenient others & revise
  99. 99. Pilot test – try out on a small sample from the target population & revise
  100. 100. Be assertive and interactive about seeking feedback – ask questions & observe
  101. 101. “The customer is always right.” </li></ul>
  102. 102. Maximising response rate <ul><li>Layout and design is key
  103. 103. Respondent’s level of interest
  104. 104. Colour of paper
  105. 105. Accompanying letter / introduction
  106. 106. Mail surveys - self-addressed stamped return envelope
  107. 107. Rewards
  108. 108. Reminders or follow up calls </li></ul>
  109. 109. Examples <ul><li>Examine the examples
  110. 110. What is wrong with the questions, if anything? </li></ul>
  111. 111. Example 1 How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over
  112. 112. Example 2 Are you satisfied with your marriage and your job? __________________________
  113. 113. Example 3 You didn’t think the food was very good, did you? _____ Yes _____ No
  114. 114. Example 4 Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No
  115. 115. Example 5 What information sources did you use to locate your restaurant for today’s meal? (please tick appropriate spaces) ____ Yellow pages ____ Internet ____ Word of mouth
  116. 116. Pre-test & revise <ul><li>Pre-test items and ask for feedback
  117. 117. Revise: </li></ul><ul><ul><li>items which don’t apply to everybody
  118. 118. redundancy
  119. 119. skewed response items
  120. 120. misinterpreted items
  121. 121. non-completed items </li></ul></ul><ul><li>Reconsider ordering & layout </li></ul>
  122. 122. Survey format checklist <ul><li>Introduction/covering letter or verbal introducation </li><ul><li>e.g. Who are you? Are you bona fide? Purpose of survey? Ethical approval? How results will be used? Confidentiality? Further info? Complaints? </li></ul><li>Instructions </li><ul><li>Sets the “mind frame”, but be aware few people will read it without good prompting and being easy-to-read </li></ul><li>Group like questions together
  123. 123. Consider order effects, habituation, fatigue, switching between response formats </li></ul>
  124. 124. Survey format <ul><li>Font type / size, number of pages, margins, double vs. single-siding, colour, etc.
  125. 125. Demographics - usually beginning or end; only use relevant questions
  126. 126. Space for comments?
  127. 127. Ending – say thanks!
  128. 128. Pre-test & revise/refine </li></ul>
  129. 129. Implementing surveys
  130. 130. Comparison of Data Collection Methods Alreck and Settle (1995:32)
  131. 131. Sampling
  132. 132. Sampling terminology <ul><li>Population
  133. 133. Sampling Frame
  134. 134. Sample
  135. 135. Representativeness </li></ul>
  136. 136. Why sample? <ul><li>Why sampling rather than a census?
  137. 137. Sampling reduces: </li><ul><li>Cost, time, sample size and defines the research
  138. 138. If the sample is representative, allows inferences to be drawn concerning the total population </li></ul></ul>
  139. 139. What is sampling? “ Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our results back to the population from which they were chosen.” - Trochim, 2002
  140. 140. Sampling frame <ul><li>The set of participants from which the sample is drawn
  141. 141. Examples: </li><ul><ul><li>Electoral Rolls
  142. 142. Membership Lists (organisations, graduates association)
  143. 143. Telephone Book
  144. 144. Members of Specific Groups or Clubs (Fishing, Ramblers)
  145. 145. Households or post codes </li></ul></ul></ul>
  146. 146. Representativeness of sample depends on: <ul><li>Adequacy of sampling frame
  147. 147. Selection strategy
  148. 148. Adequacy of sample size
  149. 149. Response rate – both the % & representativeness of people in sample who actually complete survey
  150. 150. Note: It is better to have a small, good sample than a large, poor sample. </li></ul>
  151. 151. Sampling example: Shere Hite ‘American Sexology’
  152. 152. Male-female relations <ul><li>Shere Hite ‘doyenne of sex polls’
  153. 153. Media furors & worldwide attention
  154. 154. 127-item questionnaire about marriage & relations between sexes
  155. 155. 4500 USA women, 14 to 85 years
  156. 156. Society and men need to change to improve lives of women </li></ul>
  157. 157. Some of Hite’s findings.... <ul><li>70% married for 5 years having affairs... </li></ul>(usually more for ‘emotional closeness’ than sex) <ul><li>76% did not feel guilty
  158. 158. 87% had a closer female friend than husband
  159. 159. 98% wanted “basic changes” to love relationships
  160. 160. only 13% married for 2+years were still in love
  161. 161. 84% were emotionally unsatisfied
  162. 162. 95% reported emotional & psychological harassment from their men </li></ul>
  163. 163. Some of the critical comments.... <ul><li>“She goes in with prejudice & comes out with a statistic.”
  164. 164. “The survey often seems merely to provide an occasion for the author’s own male-bashing diatribes.”
  165. 165. “Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men.” </li></ul>
  166. 166. Response rate & Selection bias - 1 100,000 questionnaires Sent to a variety of women’s groups - feminist organisations, church groups, garden clubs, etc. 4,500 replied (4.5% return rate)
  167. 167. “ We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.” - Regina Herzog, University of Michigan Institute for Social Research Response rate & Selection bias - 2
  168. 168. Sampling techniques <ul><li>Probability (random) sampling </li><ul><li>Random
  169. 169. Systematic
  170. 170. Cluster </li><ul><li>Multi-Stage Cluster </li></ul></ul><li>Non-probability sampling </li><ul><li>Quota
  171. 171. Convenience
  172. 172. Snowball </li></ul></ul>
  173. 173. Random/probability sampling <ul><li>Each unit has an equal (and perhaps known) chance or probability of selection in the sample
  174. 174. Selection occurs entirely by random chance
  175. 175. Often called representative sampling </li></ul>
  176. 176. Simple random sampling <ul><li>Everyone in the target population has an equal chance of selection
  177. 177. Useful if clear study area or population is identified
  178. 178. Similar to a lottery
  179. 179. List of names are assigned #s and randomly select #s of respondents
  180. 180. Randomly select # through table of random #s or by computer </li></ul>
  181. 181. Systematic random sampling <ul><li>Selecting without first numbering
  182. 182. Respondents (units) selected from a list/file.
  183. 183. Useful when survey population is similar e.g. List of Students, List of Package Tourists
  184. 184. Select sample at regular intervals from the population e.g., every 5 th person on a list </li></ul>
  185. 185. Systematic random sampling <ul><li>Cannot do 1 in every 5
  186. 186. As then 4 people out of 5 stand no chance of being selected
  187. 187. Select a random starting point between 1 and 5 </li></ul>
  188. 188. Stratified random sampling <ul><li>Sub-divide population into strata (e.g., by gender, age, or location)
  189. 189. Then random selection from within each stratum
  190. 190. Improves representativeness
  191. 191. e.g., Telephone interviews using post-code strata </li></ul>
  192. 192. Non-random/Non-probability <ul><li>Also called purposive or judgemental sampling
  193. 193. Useful for exploratory research and case study research
  194. 194. Able to get large sample size quickly and useful when can’t find a sample frame </li></ul>
  195. 195. Non-random/Non-probability <ul><li>Make assumptions and maybe generalisations from your data, but not on statistical grounds
  196. 196. Limitations include potential bias and applicability </li></ul>
  197. 197. Convenience sampling <ul><li>Sampling is by convenience rather than randomly
  198. 198. Due to time/financial constraints
  199. 199. e.g. surveying all those at a tourist attraction over one weekend </li></ul>
  200. 200. Purposive sampling <ul><li>Respondents selected for a particular purpose e.g., because they may be “typical” respondents
  201. 201. e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra
  202. 202. e.g., Frequent flyers to contact regarding service quality in an airline setting </li></ul>
  203. 203. Snowballing <ul><li>Useful for difficult to access populations e.g., illegal immigratnts, drug users
  204. 204. Respondents recommend other respondents
  205. 205. e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents </li></ul>
  206. 206. Summary of sampling strategy <ul><li>Identify target population and sampling frame
  207. 207. Selection sampling method
  208. 208. Calculate required sample size
  209. 209. Maximise return rate </li></ul>
  210. 210. Task A research project's aim is – “To identify the behaviour and attitudes of UC students with regard to its computing services”. <ul><li>What is the research population?
  211. 211. How might you get hold of a sample frame?
  212. 212. What sampling technique would you use? </li></ul>

×