Survey design-workshop-1234170716539145-3[1]


Published on


Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Survey Design Workshop University of Canberra, ACT, Australia James T. Neill This workshop was previously presented by Dr. Brent Ritchi e, 2008 who is now at the School of Tourism at The University of Queensland – he kindly gave me a copy of his slides, which have been adapted and expanded each year since. Image sources: Questionnaires are by James Neill (License: Public domain) Scissors are by Gracenotes - - (License: - Creative Commons by SA 2.5) Further info:
  • Image: James Neill, from Flickr, cc-by-a
  • Image source: License: GFDL
  • Groves et al. (2004)
  • Image source: License: GFDL
  • Image source: License: GFDL
  • Image source: License: GFDL
  • Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia,
  • Image source:
  • Image source: Unknown.
  • Image source: Unknown.
  • Image source: Life Effectiveness Questionaire (Neill, 2009)
  • Image sources: Questionnaires are by James Neill (License: Public domain), based on the Life Effectiveness Questionnaire
  • Image sources: Questionnaires are by James Neill (License: Public domain), based on the Life Effectiveness Questionnaire
  • Image sources: Questionnaires are by James Neill (License: Public domain), based on the Life Effectiveness Questionnaire
  • Image source:
  • e.g., skewed response items
  • Imag sourcese:
  • Social desirability Acquiescence or Yea- and Nay-saying - tendency to agree or disagree with everything, use reversed items to control Self-serving bias - tendency to enhance self Order effects - routine, fatigue
  • Image: This example actually has many elements of a well-structued/designed survey – what are they?
  • Image source:,M1
  • See alos: The Power of Survey Design By Giuseppe Iarossi,M1
  • Image source: Unknown.
  • Nominal/Category - measures identify categories e.g., sex, ethnicity. Ordinal - relative ordering of responses e.g., rankings in an exam Interval - scores stand in a quantitative relationship to one another, adjacent scores are separated by an equal interval Ratio - like interval but with a true zero value e.g., height, speed
  • Discrete data: finite options (e.g., labels) Continuous data: infinite options (e.g., cms) Discrete data is generally only whole numbers, whilst continuous data can have many decimals Discrete: nominal, ordinal, interval Continuous: ratio
  • Image source: Unknown.
  • Image source:L.N Fowler & Co. c. 1870.
  • Image: Cropped version of CC-by-A by Beat -
  • Image source:L.N Fowler & Co. c. 1870.
  • Image source:L.N Fowler & Co. c. 1870.
  • Image source:L.N Fowler & Co. c. 1870.
  • Image source: Unknown.
  • Image source: Questionnaire by Tuppus License: Creative Commons Attribution 2.0
  • eg. Which of the following statements best describes your reasons for taking a holiday to Canberra? (please tick one only) Ž to visit friends and relatives Ž for business Ž for educational purposes Ž for holiday/ sightseeing
  • eg. Which of the following statements best describes your reasons for taking a holiday to Canberra? (please tick one only) Ž to visit friends and relatives Ž for business Ž for educational purposes Ž for holiday/ sightseeing
  • Consider number of points (avoid over ~10) Consider direction Consider layout
  • Consider number of points (avoid over ~10) Consider direction Consider layout
  • Image source: Unknown.
  • Image source:
  • Image source: Unknown.
  • Population - set of all individuals having some common characteristic, e.g., Australians Sampling Frame – subset of the population from which the sample is actually drawn – e.g., White pages Sample – set of people who actually contribute data to – e.g., Every 1000 th person in the white pages who answers the phone and responds Representativeness – How similar is the sample to the population with regard to the constructs of interest?
  • Probability sampling - each member of population has a specific probability of being chosen. Random Sampling - everyone in population has an equal chance of being selected. Systematic Sampling - e.g., every 10 th student ID number Stratified Random Sampling - population divided into strata, then random sampling from within each stratum (e.g., an equal number of males/females are selected) Cluster Sampling - identify ‘clusters’ of individuals & sample from these (e.g., 1 person per household) Multi-Stage Cluster Sampling – (e.g., 1 person per selected household per selected suburb) Non-probability sampling - arbitrary, sample not representative of population Quota Sampling - e.g., 50% psychology students, 30% economics students, 20% law students Convenience Sampling - “take them where you find them” method e.g., at shopping mall Snowball Sampling - ask each respondent if they know someone else suitable for survey e.g., studying drug-users.
  • Sometimes called file sampling
  • Image sources: Unknown.
  • Regina Herzog, University of Michigan Institute for Social Research To learn more about Shere Hite’s research, visit her website:
  • Regina Herzog, University of Michigan Institute for Social Research To learn more about Shere Hite’s research, visit her website:
  • Image source: Image license: GFDL 1.2+
  • Image source:s Unknown. Paradigm (e.g., assumptions, focus, collection method) Personal researcher bias (conscious & unconscious) Sampling (e.g., representativeness of sample) Non-sampling (e.g, non-response, inaccurate response due to unreliable measurements, misunderstanding, social desirability, faking bad, researcher expectancy, administrative error)
  • Source:
  • Image source: Questionnaire by Tuppus License: Creative Commons Attribution 2.0
  • Survey design-workshop-1234170716539145-3[1]

    1. 1. Survey Design Workshop Inter-University Research Workshop Program Dr. James Neill Centre for Applied Psychology University of Canberra 1 February, 2011
    2. 2. Outline• Objectives• Introductions• Logins & Resources• Research methods• Questionnaire design• Levels of measurement• Sampling• Evaluation 2
    3. 3. Objective 1Understand the importance of a rigorous, step-by-step process in planning, developing & implementing research questionnaires 3
    4. 4. Objective 2Consider the pros and cons of common methods for survey administration2. Face-to-face interview3. Telephone survey4. Mail survey5. Internet/mobile survey 4
    5. 5. Objective 3Examine nuts & bolts of questionnaire design e.g.,2 Question style,3 Response formats,4 Layout, and5 Pre- and pilot testing 5
    6. 6. Objective 4Consider implementation issues2 Sampling methods3 Sample size 6
    7. 7. Objective 5Critically review example surveys.2 Existing examples3 Student in-progress exampleswith a view towards planning, drafting, and/or revising of an initial draft (pilot) survey. 7
    8. 8. Resources• Survey Design Workshop Notes (Wikiversity)• Readings• Books about surveys design and survey research (check library) 8
    9. 9. Introductions Introductions 9
    10. 10. Types of Research (Research Methods)There are 3 main research methods:2.Experimental3.Quasi-experimental4.Non-experimental 10
    11. 11. Types of Research - ExperimentalCharacterised by:• Random assignment• Control over extraneous variables 11
    12. 12. Types of Research - Quasi-experimentalCharacterised by:•Non-random assignment•Control over some extraneousvariables•Groups are “naturally occuring” 12
    13. 13. Types of Research - Non-experimentalCharacterised by:•No “groups” or “conditions” arecreated or used (i.e., no fullexperimental or quasi-experimentalgroups)•Minimal control over extraneousvariables 13
    14. 14. Survey Research Characteristics •Surveys are widely used in non-experimental social science research. •Often use interviews or questionnaires. •Involve real-world samples. •Often quantitative, but can be qualitative. 14
    15. 15. History of Survey Research•Survey research methodologywas initially developed in the1940s – 1960s.•Since the 1980s, theories andprinciples evolved to create aunified perspective on the design,conduct, and evaluation ofsurveys. 15
    16. 16. 8 Survey Research Characteristics Backstrom & Hursh-César (1981, pp. 3-4) 1Systematic: follows a specific set of rules, a formal and orderly logic of operations 2Impartial: selects units of the population without prejudice or preference 3Representative: includes units that together are representative of the problem under study and the population affected by it 16
    17. 17. 8 Survey Research Characteristics Backstrom & Hursh-César (1981, pp. 3-4) 4Theory-based: operations are guided by relevant principles of human behaviour and by mathematical laws of probability (chance). 5Quantitative: assigns numerical values to nonnumerical character 6Self-monitoring: procedures can be designed in ways that reveal any unplanned and unwanted distortions (biases) that may occur 17
    18. 18. 8 Survey Research Characteristics Backstrom & Hursh-César (1981, pp. 3-4) 7Contemporary: it is current, more than historical, fact-finding 8Replicable: other people using the same methods in the same ways can get essentially the same results 18
    19. 19. Advantages of Survey-Based Research• Ecological validity• Access to wide range of participants• Potentially large amounts of data• May be more ethical (than experiments) 19
    20. 20. Disadvantages of Survey-Based Research• Lack of control → less internal validity• Data may be superficial• Can be costly to obtain representative data• Self-report data only• Potentially low compliance rates 20
    21. 21. Example SurveysUnit Satisfaction SurveyCommunity Library Use 21
    22. 22. The research process 1. Establish need for info/ research 2. Problem6. Reporting definition/ Hypotheses 5. Data 3. Research analysis design 4. Sampling/ Data collection 22
    23. 23. The research process 23
    24. 24. Survey construction: Overview1What is a survey?2Types of questionnaires3Questionnaire development4Writing questions5Types of questions6Response formats R LOM7Survey formatting 24
    25. 25. What is a survey?•A standardised stimulus•A measuring instrument•A way of converting fuzzypsychological stuffinto hard datafor analysis 25
    26. 26. Types of surveys Types of surveys Self - Interview -administered administeredPostal Delivered and TelephoneFace to face collected structured interview Web-based 26
    27. 27. Questionnaire development1. Formulate 2. Expand generic thequestionnair questionnair Turneinto Question e Draft qs & separate order & response sections based funnel qs formats onstudy objectives.4. Finalise 3. Pre-test,questionnair pilot test, e & redraft& implement 27
    28. 28. Formulate Generic Questionnaire • Turn objectives into sections of the survey • Ensure all questions relate to research objectives • For explanatory objectives or hypotheses ensure both dependent and independent variables exist 28
    29. 29. Cover Letter & Ethics Statement Introduction or cover letter: • Outline details of research project and allow for ethical informed consent. • Few will read it without good prompting and being easy-to-read 29
    30. 30. Instructions• Provides consistency - helps to ensure standard conditions across different administrations• Explain how to do the survey in a user-friendly manner• Example: Life Effectiveness Questionnaire 30
    31. 31. Instructions: Example 31
    32. 32. Cover letter / ethics statement checklistOutline details of research project e.g.,:• Who are you? Are you bona fide?• Purpose of survey?• Whats involved?• Explain any risks/costs/rewards• How will results be used?• Human ethics approval #• How is consent given / not given?• Voluntary - can choose not to continue anytime• More info: Complaints, how to obtain results, 32
    33. 33. Layout•Font (type, size)•No. of pages•Margins•Double vs. single-siding•Colour, etc. 33
    34. 34. LayoutDemographics• single section, usually at beginning or end of questionnaire• only include relevant questions 34
    35. 35. Layout•Space for comments?•Indicate the end•Say thanks! 35
    36. 36. Flow/Structure• Logical order of questions (use sections)• Ask screening questions first, rather than later. Does the participant qualify for the survey? (esp. for internet surveys)• Use funneling/branching questions to move respondents through survey• Start off with easy to answer and engaging questions• More controversial questions in middle36
    37. 37. Expanding the Survey 37
    38. 38. Personal Information• Generally, researchers put personal information at beginning of survey (if required). However, this may put off respondents, so also consider uncluding at towards end.• Consider response format e.g, Income in categories or ranges 38
    39. 39. Personal Information• More likely to respond to personal questions for anonymous or mail surveys as opposed to face to face or telephone• Show cards for responses may help for face to face interviews 39
    40. 40. Pre- & pilot-testing•Pre-test items on convenient others -ask for feedback•Revise items e.g.,–Which don’t apply to everybody–Are redundant–Are misunderstood–Are non-completed• Reconsider ordering & layout•Pilot test on a small sample from the 40
    41. 41. Types of Questions Be able to justify and defend your choices... 41
    42. 42. Writing questions - Dos1 Define target constructs2 Check related research & questionnaires3 Draft items (for important, fuzzy constructs aim to have multiple indicators)4 Pre-test & revise 42
    43. 43. Writing questions - Dos• Focus directly on topic/issue• Be clear• Be brief• Avoid big words• Use simple and correct grammar 43
    44. 44. Writing questions – DontsInapplicable – must apply to all respondentsOver-demanding – e.g., recall of detail or time-consuming, unnecessary questioningAmbiguous – meaning must be clear to all respondentsDouble negative – e.g., Do you not disapprove of tax reforms? 44
    45. 45. Writing questions – DontsDouble-barrelled - e.g., “Do you think speed limits should be lowered for cars & trucks?”Leading - e.g., “don’t you see some danger in the new policy?”Loaded – e.g., “Do you advocate a lower speed limit to save human lives?” vs “Does traffic safety require a lower speed limit?” 45
    46. 46. Response biases• Social desirability• Acquiescence – yea- and nay-saying• Self-serving bias• Order effects 46
    47. 47. Demand characteristicsInterview• High demand characteristics• Can elicit richer informationQuestionnaire• Lower demand characteristics• Information may be less rich 47
    48. 48. 48
    49. 49. Accuracy of recalldecreases over time 49
    50. 50. Objective questions• A verifiably true answer (i.e., factual information) exists for each unit.• The question could be accurately answered by an observer.e.g.,How times during 2009 did you visit a G.P.? ______ 50
    51. 51. Subjective questions•Asks about fuzzy personal perceptions.•There is no “true”, factual answer.•Many possible answers per unit.•Cant be accurately answered by an observer. e.g.,Think about the visits you made to a G.P. during 2010. How well did you understand the medical advice you received? perfectly very well reasonably poorly not at all 51
    52. 52. Objective vs. subjective questions• Both types of questions may be appropriate; depends on the purpose of the study.• One criticism of this distinction: There is no such thing as “objective” and that all responses are subjective. 52
    53. 53. Open-ended Questions• Rich information can be gathered• Useful for descriptive, exploratory work• Difficult and subjective to analyse• Time consuming 53
    54. 54. Open-ended questions• Rich information can be gathered• Useful for descriptive, exploratory work• Difficult and subjective to analyse• Time consuming 54
    55. 55. Open-ended questions: ExamplesWhat are the main issues you are currently facing in your life?How many hours did you spend doing university study last week? _________ 55
    56. 56. Closed-ended questions• Important information may be lost forever• Useful for hypothesis testing• Easy and objective to analyse• Time efficient 56
    57. 57. Levels ofMeasurement =Type of DataStevens (1946) 57
    58. 58. Levels of measurement•Nominal / Categorical•Ordinal•Interval•Ratio 58
    59. 59. Discrete vs. continuous Discrete ---------- Continuous ___________ 59
    60. 60. Each level has the properties of the preceding levels, plus something more! 60
    61. 61. Categorical / nominal•Conveys a category label•(Arbitrary) assignment of #s tocategories e.g. Gender•No useful information, except aslabels 61
    62. 62. Categorical / nomimal example: Phrenological labels 62
    63. 63. Ordinal / ranked scale•Conveys order, but not distance e.g. in a race, 1st, 2nd, 3rd, etc. or ranking of favourites or preferences 63
    64. 64. Ordinal / ranked example: Ranked importanceRank the following aspects of the university according to what is most important to you (1 = most important through to 5 = least important)t Quality of the teaching and educationQ Quality of the social lifeQ Quality of the campusQ Quality of the administrationQ Quality of the universitys reputation64
    65. 65. Interval scale•Conveys order & distance•0 is arbitrary e.g., temperature (degrees C)•Usually treat as continuous for > 5intervals 65
    66. 66. Interval example: 8 point Likert scale 66
    67. 67. Ratio scale•Conveys order & distance•Continuous, with a meaningful 0point e.g. height, age, weight, time, number of times an event has occurred•Ratio statements can be made e.g. X is twice as old (or high or heavy) as Y 67
    68. 68. Ratio scale: Time 68
    69. 69. Why do levels of measurement matter?Different analytical procedures are used for different levels of data. More powerful statistics can be applied to higher levels 69
    70. 70. Levels of measurement: Revision question Fill in all cellsLevel Prop-erties Examples Descriptive Statistics GraphsNominal / CategoricalOrdinal / RankIntervalRatio 70
    71. 71. Closed-ended rating scales1.Dichotomous2.Multichotomous3.Verbal frquency scale4.The list (multiple response)5.Ranking6.Likert scale7.Graphical rating scale8.Semantic differential 71
    72. 72. Dichotomous2 response options e.g.,Excluding this trip, have you visited Canberra in the previous five years? __ Yes __ NoProvides the simplest type of quantification 72
    73. 73. MultichotomousHow many hours did you spend doing university study last week?__ less than 5 hours__ > 5 to 10 hours__ > 10 to 20 hours__ more than 20 hours 73
    74. 74. MultichotomousMore than two possible answers e.g.,What type of attractions in your current trip to Canberra most appeal to you?__ historic buildings__ museum/art galleries__ parks and gardens 74
    75. 75. Verbal frequency scaleOver the past month, how often have you argued with your intimate partner?1. All the time2. Fairly often3. Occasionally4. Never5. Doesn’t apply to me at the moment 75
    76. 76. The list (multiple response)Provides a list of answers for respondents to choose from e.g.,Tick any words or phrases that describe your perception of Canberra as a travel destination:__ Exciting __ Important__ Boring __ Enjoyable__ Interesting __ Historical 76
    77. 77. The list (multiple-response)What are the main issues that you are currently facing in your life? (tick all that apply)__ financial__ physical / health__ academic__ employment / unemployment__ relationships__ other (please specify) 77
    78. 78. RankingHelps to measure the relative importance of several itemsRank the importance of these reasons for visiting Canberra (from 1 (most) to 4 (least)):__ to visit friends and relatives__ for business__ for educational purposes 78
    79. 79. Likert scaleMeasures strength of feeling or perception.Indicate your degree of agreement with this statement:“I am an adventurous person.” (circle the best response for you) 1 2 3 4 5strongly disagree neutral agree stronglydisagree agree 1 2 3 4 5strongly agree neutral disagree stronglyagree disagree 79
    80. 80. Graphical rating scaleHow would you rate your enjoyment of the movie you just saw? Mark with a cross (X)not enjoyable very enjoyable 80
    81. 81. Semantic differentialWhat is your view of smoking?Tick to show your opinion.Bad ___:___:___:___:___:___:___ GoodStrong ___:___:___:___:___:___:___ WeakMasculine ___:___:___:___:___:___:___ FeminineUnattractive ___:___:___:___:___:___:___ AttractivePassive ___:___:___:___:___:___:___ Active 81
    82. 82. Non-verbal scalePoint to the face that shows how you feel about what happened to the toy.Also called an idiographic scale. 82
    83. 83. Verbal frequency scaleOver the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment 83
    84. 84. Sensitivity & reliability•Scale should be sensitive yetreliable.•Watch out for too few or toomany options. 84
    85. 85. Number of response options?How many response options?•Minimum = 2•Average = 3 to 9•Maximum = 10?Basic guide: 7 +/- 2 85
    87. 87. Watch out for too many or too few response options “Capital punishment should be reintroduced for serious crimes” 1 = Agree 2 = Disagree1 = Very, Very Strongly Agree 7 = Slightly Disagree2 = Very Strongly Agree 8 = Disagree3 = Strongly Agree 9 = Strongly Disagree4 = Agree 10 = V. Strongly Disagree5 = Slightly Agree 11 = V, V Strongly Disagree6 = Neutral 87
    88. 88. Example: How couldthis question be improved?How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over 88
    89. 89. Example: How could this question be improved?Are you satisfied with your marriage and your job? __________________________ 89
    90. 90. Example: How could this question be improved?You didn’t think the food was very good, did you?_____ Yes _____ No 90
    91. 91. Example: How could this question be improved?Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No 91
    92. 92. Example: How could this question be improved?What information sources did you use to locate your restaurant for today’s meal? (please tick appropriate spaces) ____ yellow pages ____ Internet ____ word of mouth 92
    93. 93. Comparison of data collection methods Personal Telephone MailData collection costs High Medium LowData collection time required Medium Low HighSample size for a given budget Small Medium LargeData quantity per subject High Medium LowReaches widely dispersed sample No Maybe YesReaches special locations Yes Maybe NoInteraction with respondents Yes Yes NoDegree of interviewer bias High Medium NoneSeverity of non-response Low Low HighPresentation of visual stimuli Yes No MaybeFieldworker training required Yes Yes No Alreck and Settle (1995:32) 93Alreck & Settle (1995; 32)
    94. 94. Finalise Questionnaire Draft• Questions need to be exhaustive and mutually exclusive – Include ‘other (please specify)’ – Ensure categories do not overlap 94
    95. 95. Finalise Questionnaire Draft• Length – Try to keep them as short as possible – Only ask questions that relate to objectives – Tricks? Font size/double sided photocopying/numbering sections 95
    96. 96. Maximising Response Rate• Layout and design is key• Respondent’s level of interest• Colour of paper• Accompanying letter / introduction• Mail surveys - self-addressed stamped return envelope• Rewards• Reminders or follow up calls96
    97. 97. Pre-testing and Pilot testing• Pre-test – try out on convenient others & revise• Pilot test – try out on a small sample from the target population & revise• Be assertive and interactive about seeking feedback – ask questions & observe 97
    98. 98. Pre-test & Revise•Pre-test items and ask for feedback•Revise:–items which don’t apply to everybody–redundancy–skewed response items–misinterpreted items–non-completed items•Reconsider ordering & layout
    99. 99. LUNCH BREAK 99
    100. 100. Survey Design CritiqueIn pairs, look through the example questionnaires, and highlight aspects which:• could be improved• are particularly good• you would like to ask about• 100
    101. 101. Examples• Examine questionnaire examples• Examine structure, design issues and question styles• Note cover page and details provided 101
    102. 102. Sampling 102
    103. 103. Sampling: OverviewSampling terminologyWhat is sampling?Why sample?Sampling methodsExample: Shere Hite’s survey 103
    104. 104. Sampling terminology• Target population – To whom you wish to generalise• Sampling frame – Those who have a chance to be selected• Sample – Those who were selected and responsed• Representativeness – The extent to which the sample is a good 104
    105. 105. What is sampling?“Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our Picture 2 results back to the population from which they were chosen.” 105
    106. 106. Why sample?• Reduces cost, time, sample size etc.• If the sample is representative, the sample data allows inferences to be drawn about the total population. 106
    107. 107. Representativeness of a sample depends on:• Adequacy of sampling frame• Sampling method• Adequacy of sample size• Response rate – both the % & representativeness of people in sample who actually complete surveyIt is better to have a small, representative sample than a large, 107
    108. 108. Sampling methodsProbability sampling• Random• Systematic• Cluster – Multi-Stage ClusterNon-probability sampling–Quota–Convenience–Snowball 108
    109. 109. Random/probability sampling• Each unit has an equal chance of selection• Selection occurs entirely by random chance• Also called representative sampling 109
    110. 110. Simple random sampling• Everyone in the target population has an equal chance of selection• Useful if clear study area or population is identified• Similar to a lottery: – List of names are assigned #s and randomly select #s of respondents – Randomly select # through table of random #s or by computer 110
    111. 111. Systematic random sampling• Selecting without first numbering• Respondents (units) selected from a list/file.• Useful when survey population is similar e.g. List of students• Select sample at regular intervals from the population e.g., every 5th person on a list, starting at a random number between 1 and 5 111
    112. 112. Stratified random sampling• Sub-divide population into strata (e.g., by gender, age, or location)• Then random selection from within each stratum• Improves representativeness• e.g., Telephone interviews using post-code strata 112
    113. 113. Non-random / non-probability• Also called purposive or judgemental sampling• Useful for exploratory research and case study research• Able to get large sample size quickly• Limitations include potential bias and non-representativeness 113
    114. 114. Convenience sampling• Sampling is by convenience rather than randomly• Due to time/financial constraints• e.g. surveying all those at a tourist attraction over one weekend 114
    115. 115. Purposive samplingRespondents selected for a particular purpose e.g., because they may be “typical” respondents• e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra• e.g., Frequent flyers to contact regarding service quality in an airline setting 115
    116. 116. Snowball sampling• Useful for difficult to access populations e.g., illegal immigratnts, drug users• Respondents recommend other respondents• e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents 116
    117. 117. Sampling process1Identify target population andsampling frame2Select sampling method3Calculate sample size for desiredpower.4Maximise return rate 117
    118. 118. Summary of sampling strategy•Identify target population andsampling frame•Selection sampling method•Calculate required sample size•Maximise return rate
    119. 119. Sampling Example: Shere Hite‘American Sexology’ 119
    120. 120. Hites survey of Americanmale-female relations (early 1980s)•Shere Hite ‘doyenne of sex polls’•Media furors & worldwide attention•127-item questionnaire aboutmarriage & relations between sexes•Sample: 4500 USA women, 14 to 85years•Conclusion: Society and men need tochange to improve lives of women 120
    121. 121. Some of Hite’s findings about American women....•Only 13% married for 2+years were still in love•70% married for 5+ years were having affairs... – usually more for emotional closeness’ than sex – 76% of these women did not feel guilty•87% had a closer female friend than husband•98% wanted “basic changes” to loverelationships•84% were emotionally unsatisfied•95% reported emotional & psychologicalharassment from their men 121
    122. 122. Some of the critical comments.... The survey often seems merely toShe goes in with provide anprejudice & occasioncomes for the author’sout with a own male-bashingstatistic. diatribes. Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men. 122
    123. 123. Hites response rate & selection bias•100,000 questionnaires were sentto a variety of women’s groups(feminist organisations, church groups,garden clubs etc.)•4,500 replied(4.5% return rate) 123
    124. 124. Hites response rate & selection bias“We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.” 124
    125. 125. Lessons from Hites male- female relations survey1Sample size – its not how big, its how representative2Objectivity – watch out for manipulating the survey questions and results interpretation to suit your personal conjectures 125
    126. 126. Measurement Error 126
    127. 127. Measurement error•Observed score = true score + measurement error•Measurement error = systematic error + random error•Any deviation from the truevalue caused by themeasurement procedure. 127
    128. 128. Sources of measurement error Picture 5 Non-sampling Sampling (e.g., unreliable (e.g., non-rep. samp or invalid tests) Personal bias (e.g., researcher fParadigm(e.g., Western focus on individualism) 128
    129. 129. To minimise measurement error Use well designed measures: • Multiple indicators • Sensitive to target constructs • Clear wording on questions/instructions 129
    130. 130. To minimise measurement error Reduce demand effects: • Train interviewers • Use standard protocol 130
    131. 131. To minimise measurement error Maximise response rate: • Pre-survey contact • Minimise length / time / hassle • Offer rewards / incentives • Coloured paper • Call backs / reminders 131
    132. 132. To minimise measurement error Ensure administrative accuracy • Set up efficient coding, with well- labelled variables • Check data 132
    133. 133. Summary of sampling strategy•Identify target population andsampling frame•Selection sampling method•Calculate required sample size•Maximise return rate
    134. 134. Sampling TaskA research projects aim is – “To identify the behaviour and attitudes of UC students with regard to its computing services”.• What is the research population?• How might you get hold of a sample frame?• What sampling technique would 134
    135. 135. Confidence Levels / Margins of Error• Relates to representativeness of a sample of a target population - to what extent can we be confident about the results?• Gives the estimated range of values into which we expect other samples to fall say 95% of the time• Social sciences = at least 90% or above, preferably 95% 135
    136. 136. Example 95%Confidence Level Graphs 136
    137. 137. Example 95%Confidence Level Table 137
    138. 138. Confidence Level ExampleSurvey of visitors to Canberra between June 1 - August 31, 2004 = 10,000 visitorsWant to work with 95% confidence level and 3% margin of errorHow many to survey?Answer: Sample size = 964 peopleQ: What are your favourite attractions? 138
    139. 139. Following Results+ / - 3% error @ 95% confidence level 139
    140. 140. Confidence Intervals / Margins of Error• One time out of 20, we expect the answers may be greater than +/- 3% – Establish the confidence level, margin of error you want to work with – Identify the number of surveys to be done – Once you have completed that number, do not do any additional surveysHOWEVER…. 140
    141. 141. Confidence Intervals / Margins of Error• Sometimes we do surveys and we do not know how many will be returned until later, as with postal surveys• Thus you have to calculate the margin of error afterwards…. – Count up # of returned surveys – Identify target population and confidence level 141
    142. 142. Confidence Intervals / Margins of Error• All this assumes you know your target population and can get a sample frame – If not, a non-random sampling technique is best• Also consider whether you want to report results for sub-groups – the margins of error will be wider 142
    143. 143. Summary - 11 Survey research has developed into a popular research method since the 1940s.2 A survey is a standardised stimulus designed to convert fuzzy psychological phenomenon into hard data. 143
    144. 144. Summary - 23 Survey development - types of questions and response formats.4 Sampling - probability & non-prob.5 Levels of measurement & parametric / non-parametric stats6 Ethical considerations7 Sources of measurement error 144
    145. 145. Evaluation• Please complete the workshop evaluations 145
    146. 146. ReferencesAlreck, P. & Settle, R. (1995). The survey research handbook (2nd ed.). New York: Irwin.Stevens, S.S. (1946). On the theory of scales of measurement. Science, 103, 677-680.Trochim, W. M. K. (2006). Sampling. In Research Methods Knowledge Base.Wikipedia (2009). Shere Hite - Methodology.• 146
    147. 147. Open Office Impress● This presentation was made usingOpen Office Impress.● Free and open source software.●