• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Survey design-workshop-1234170716539145-3[1]
 

Survey design-workshop-1234170716539145-3[1]

on

  • 503 views

survey

survey

Statistics

Views

Total Views
503
Views on SlideShare
503
Embed Views
0

Actions

Likes
0
Downloads
32
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Survey Design Workshop University of Canberra, ACT, Australia James T. Neill This workshop was previously presented by Dr. Brent Ritchi e, 2008 who is now at the School of Tourism at The University of Queensland – he kindly gave me a copy of his slides, which have been adapted and expanded each year since. Image sources: Questionnaires are by James Neill (License: Public domain) Scissors are by Gracenotes - http://commons.wikimedia.org/wiki/File:Edit-cut-mod.svg - (License: - Creative Commons by SA 2.5) Further info: http://en.wikiversity.org/wiki/Survey_design/Workshop
  • Image: James Neill, from Flickr, cc-by-a
  • Image source: http://en.wikiversity.org/wiki/File:Nuvola_apps_edu_science.svg License: GFDL
  • Groves et al. (2004)
  • Image source: http://commons.wikimedia.org/wiki/File:Crystal_Clear_action_edit_add.png License: GFDL
  • Image source: http://commons.wikimedia.org/wiki/File:Crystal_Clear_action_edit_remove.png License: GFDL
  • Image source: http://commons.wikimedia.org/wiki/File:Hyperlink-internet-search.svg License: GFDL
  • Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
  • Image source: http://www.socialresearchmethods.net/kb/Assets/images/hourglas.gif
  • Image source: Unknown.
  • Image source: Unknown.
  • Image source: Life Effectiveness Questionaire (Neill, 2009)
  • Image sources: Questionnaires are by James Neill (License: Public domain), based on the Life Effectiveness Questionnaire
  • Image sources: Questionnaires are by James Neill (License: Public domain), based on the Life Effectiveness Questionnaire
  • Image sources: Questionnaires are by James Neill (License: Public domain), based on the Life Effectiveness Questionnaire
  • Image source: http://www.flickr.com/photos/peretzpup/3059447579/
  • e.g., skewed response items
  • Imag sourcese: http://commons.wikimedia.org/wiki/File:Aiga_information_.svg http://www.flickr.com/photos/laffy4k/404298099/
  • Social desirability Acquiescence or Yea- and Nay-saying - tendency to agree or disagree with everything, use reversed items to control Self-serving bias - tendency to enhance self Order effects - routine, fatigue
  • Image: http://commons.wikimedia.org/wiki/File:Aiga_information_.svg This example actually has many elements of a well-structued/designed survey – what are they?
  • Image source: http://books.google.com/books?id=E-8XHVsqoeUC&pg=PA58&lpg=PA58&dq=survey+design+types+of+questions+objective+subjective&source=bl&ots=fwFJdFznwn&sig=7Sil7P1uq4j6ctHzw3oKqliUhB4&hl=en&ei=KBSqSd6NJpm0sQPI3pzlDw&sa=X&oi=book_result&resnum=2&ct=result#PPA59,M1
  • See alos: The Power of Survey Design By Giuseppe Iarossi http://books.google.com/books?id=E-8XHVsqoeUC&pg=PA58&lpg=PA58&dq=survey+design+types+of+questions+objective+subjective&source=bl&ots=fwFJdFznwn&sig=7Sil7P1uq4j6ctHzw3oKqliUhB4&hl=en&ei=KBSqSd6NJpm0sQPI3pzlDw&sa=X&oi=book_result&resnum=2&ct=result#PPA59,M1
  • Image source: Unknown.
  • Nominal/Category - measures identify categories e.g., sex, ethnicity. Ordinal - relative ordering of responses e.g., rankings in an exam Interval - scores stand in a quantitative relationship to one another, adjacent scores are separated by an equal interval Ratio - like interval but with a true zero value e.g., height, speed
  • Discrete data: finite options (e.g., labels) Continuous data: infinite options (e.g., cms) Discrete data is generally only whole numbers, whilst continuous data can have many decimals Discrete: nominal, ordinal, interval Continuous: ratio
  • Image source: Unknown.
  • Image source:L.N Fowler & Co. c. 1870.
  • Image: Cropped version of http://www.flickr.com/photos/beatkueng/1350250361/?addedcomment=1#comment72157605326099631 CC-by-A by Beat - http://www.flickr.com/photos/beatkueng/
  • Image source:L.N Fowler & Co. c. 1870.
  • Image source:L.N Fowler & Co. c. 1870.
  • Image source:L.N Fowler & Co. c. 1870.
  • Image source: Unknown.
  • Image source: Questionnaire by Tuppus License: Creative Commons Attribution 2.0
  • eg. Which of the following statements best describes your reasons for taking a holiday to Canberra? (please tick one only) Ž to visit friends and relatives Ž for business Ž for educational purposes Ž for holiday/ sightseeing
  • eg. Which of the following statements best describes your reasons for taking a holiday to Canberra? (please tick one only) Ž to visit friends and relatives Ž for business Ž for educational purposes Ž for holiday/ sightseeing
  • Consider number of points (avoid over ~10) Consider direction Consider layout
  • Consider number of points (avoid over ~10) Consider direction Consider layout
  • Image source: Unknown.
  • Image source: http://commons.wikimedia.org/wiki/File:Marbles_canicas.PNG
  • Image source: Unknown.
  • Population - set of all individuals having some common characteristic, e.g., Australians Sampling Frame – subset of the population from which the sample is actually drawn – e.g., White pages Sample – set of people who actually contribute data to – e.g., Every 1000 th person in the white pages who answers the phone and responds Representativeness – How similar is the sample to the population with regard to the constructs of interest?
  • Probability sampling - each member of population has a specific probability of being chosen. Random Sampling - everyone in population has an equal chance of being selected. Systematic Sampling - e.g., every 10 th student ID number Stratified Random Sampling - population divided into strata, then random sampling from within each stratum (e.g., an equal number of males/females are selected) Cluster Sampling - identify ‘clusters’ of individuals & sample from these (e.g., 1 person per household) Multi-Stage Cluster Sampling – (e.g., 1 person per selected household per selected suburb) Non-probability sampling - arbitrary, sample not representative of population Quota Sampling - e.g., 50% psychology students, 30% economics students, 20% law students Convenience Sampling - “take them where you find them” method e.g., at shopping mall Snowball Sampling - ask each respondent if they know someone else suitable for survey e.g., studying drug-users.
  • Sometimes called file sampling
  • Image sources: Unknown.
  • Regina Herzog, University of Michigan Institute for Social Research To learn more about Shere Hite’s research, visit her website: http://www.hite-research.com
  • Regina Herzog, University of Michigan Institute for Social Research To learn more about Shere Hite’s research, visit her website: http://www.hite-research.com
  • Image source: http://commons.wikimedia.org/wiki/File:Noise_effect.svg Image license: GFDL 1.2+
  • Image source:s Unknown. Paradigm (e.g., assumptions, focus, collection method) Personal researcher bias (conscious & unconscious) Sampling (e.g., representativeness of sample) Non-sampling (e.g, non-response, inaccurate response due to unreliable measurements, misunderstanding, social desirability, faking bad, researcher expectancy, administrative error)
  • Source: http://en.wikipedia.org/wiki/File:Marginoferror95.PNG
  • Image source: Questionnaire by Tuppus License: Creative Commons Attribution 2.0

Survey design-workshop-1234170716539145-3[1] Survey design-workshop-1234170716539145-3[1] Presentation Transcript

  • Survey Design Workshop Inter-University Research Workshop Program Dr. James Neill Centre for Applied Psychology University of Canberra 1 February, 2011
  • Outline• Objectives• Introductions• Logins & Resources• Research methods• Questionnaire design• Levels of measurement• Sampling• Evaluation 2
  • Objective 1Understand the importance of a rigorous, step-by-step process in planning, developing & implementing research questionnaires 3
  • Objective 2Consider the pros and cons of common methods for survey administration2. Face-to-face interview3. Telephone survey4. Mail survey5. Internet/mobile survey 4
  • Objective 3Examine nuts & bolts of questionnaire design e.g.,2 Question style,3 Response formats,4 Layout, and5 Pre- and pilot testing 5
  • Objective 4Consider implementation issues2 Sampling methods3 Sample size 6
  • Objective 5Critically review example surveys.2 Existing examples3 Student in-progress exampleswith a view towards planning, drafting, and/or revising of an initial draft (pilot) survey. 7
  • Resources• Survey Design Workshop Notes (Wikiversity)• Readings• Books about surveys design and survey research (check library) 8
  • Introductions Introductions 9
  • Types of Research (Research Methods)There are 3 main research methods:2.Experimental3.Quasi-experimental4.Non-experimental 10
  • Types of Research - ExperimentalCharacterised by:• Random assignment• Control over extraneous variables 11
  • Types of Research - Quasi-experimentalCharacterised by:•Non-random assignment•Control over some extraneousvariables•Groups are “naturally occuring” 12
  • Types of Research - Non-experimentalCharacterised by:•No “groups” or “conditions” arecreated or used (i.e., no fullexperimental or quasi-experimentalgroups)•Minimal control over extraneousvariables 13
  • Survey Research Characteristics •Surveys are widely used in non-experimental social science research. •Often use interviews or questionnaires. •Involve real-world samples. •Often quantitative, but can be qualitative. 14
  • History of Survey Research•Survey research methodologywas initially developed in the1940s – 1960s.•Since the 1980s, theories andprinciples evolved to create aunified perspective on the design,conduct, and evaluation ofsurveys. 15
  • 8 Survey Research Characteristics Backstrom & Hursh-César (1981, pp. 3-4) 1Systematic: follows a specific set of rules, a formal and orderly logic of operations 2Impartial: selects units of the population without prejudice or preference 3Representative: includes units that together are representative of the problem under study and the population affected by it 16
  • 8 Survey Research Characteristics Backstrom & Hursh-César (1981, pp. 3-4) 4Theory-based: operations are guided by relevant principles of human behaviour and by mathematical laws of probability (chance). 5Quantitative: assigns numerical values to nonnumerical character 6Self-monitoring: procedures can be designed in ways that reveal any unplanned and unwanted distortions (biases) that may occur 17
  • 8 Survey Research Characteristics Backstrom & Hursh-César (1981, pp. 3-4) 7Contemporary: it is current, more than historical, fact-finding 8Replicable: other people using the same methods in the same ways can get essentially the same results 18
  • Advantages of Survey-Based Research• Ecological validity• Access to wide range of participants• Potentially large amounts of data• May be more ethical (than experiments) 19
  • Disadvantages of Survey-Based Research• Lack of control → less internal validity• Data may be superficial• Can be costly to obtain representative data• Self-report data only• Potentially low compliance rates 20
  • Example SurveysUnit Satisfaction SurveyCommunity Library Use 21
  • The research process 1. Establish need for info/ research 2. Problem6. Reporting definition/ Hypotheses 5. Data 3. Research analysis design 4. Sampling/ Data collection 22
  • The research process 23
  • Survey construction: Overview1What is a survey?2Types of questionnaires3Questionnaire development4Writing questions5Types of questions6Response formats R LOM7Survey formatting 24
  • What is a survey?•A standardised stimulus•A measuring instrument•A way of converting fuzzypsychological stuffinto hard datafor analysis 25
  • Types of surveys Types of surveys Self - Interview -administered administeredPostal Delivered and TelephoneFace to face collected structured interview Web-based 26
  • Questionnaire development1. Formulate 2. Expand generic thequestionnair questionnair Turneinto Question e Draft qs & separate order & response sections based funnel qs formats onstudy objectives.4. Finalise 3. Pre-test,questionnair pilot test, e & redraft& implement 27
  • Formulate Generic Questionnaire • Turn objectives into sections of the survey • Ensure all questions relate to research objectives • For explanatory objectives or hypotheses ensure both dependent and independent variables exist 28
  • Cover Letter & Ethics Statement Introduction or cover letter: • Outline details of research project and allow for ethical informed consent. • Few will read it without good prompting and being easy-to-read 29
  • Instructions• Provides consistency - helps to ensure standard conditions across different administrations• Explain how to do the survey in a user-friendly manner• Example: Life Effectiveness Questionnaire 30
  • Instructions: Example 31
  • Cover letter / ethics statement checklistOutline details of research project e.g.,:• Who are you? Are you bona fide?• Purpose of survey?• Whats involved?• Explain any risks/costs/rewards• How will results be used?• Human ethics approval #• How is consent given / not given?• Voluntary - can choose not to continue anytime• More info: Complaints, how to obtain results, 32
  • Layout•Font (type, size)•No. of pages•Margins•Double vs. single-siding•Colour, etc. 33
  • LayoutDemographics• single section, usually at beginning or end of questionnaire• only include relevant questions 34
  • Layout•Space for comments?•Indicate the end•Say thanks! 35
  • Flow/Structure• Logical order of questions (use sections)• Ask screening questions first, rather than later. Does the participant qualify for the survey? (esp. for internet surveys)• Use funneling/branching questions to move respondents through survey• Start off with easy to answer and engaging questions• More controversial questions in middle36
  • Expanding the Survey 37
  • Personal Information• Generally, researchers put personal information at beginning of survey (if required). However, this may put off respondents, so also consider uncluding at towards end.• Consider response format e.g, Income in categories or ranges 38
  • Personal Information• More likely to respond to personal questions for anonymous or mail surveys as opposed to face to face or telephone• Show cards for responses may help for face to face interviews 39
  • Pre- & pilot-testing•Pre-test items on convenient others -ask for feedback•Revise items e.g.,–Which don’t apply to everybody–Are redundant–Are misunderstood–Are non-completed• Reconsider ordering & layout•Pilot test on a small sample from the 40
  • Types of Questions Be able to justify and defend your choices... 41
  • Writing questions - Dos1 Define target constructs2 Check related research & questionnaires3 Draft items (for important, fuzzy constructs aim to have multiple indicators)4 Pre-test & revise 42
  • Writing questions - Dos• Focus directly on topic/issue• Be clear• Be brief• Avoid big words• Use simple and correct grammar 43
  • Writing questions – DontsInapplicable – must apply to all respondentsOver-demanding – e.g., recall of detail or time-consuming, unnecessary questioningAmbiguous – meaning must be clear to all respondentsDouble negative – e.g., Do you not disapprove of tax reforms? 44
  • Writing questions – DontsDouble-barrelled - e.g., “Do you think speed limits should be lowered for cars & trucks?”Leading - e.g., “don’t you see some danger in the new policy?”Loaded – e.g., “Do you advocate a lower speed limit to save human lives?” vs “Does traffic safety require a lower speed limit?” 45
  • Response biases• Social desirability• Acquiescence – yea- and nay-saying• Self-serving bias• Order effects 46
  • Demand characteristicsInterview• High demand characteristics• Can elicit richer informationQuestionnaire• Lower demand characteristics• Information may be less rich 47
  • 48
  • Accuracy of recalldecreases over time 49
  • Objective questions• A verifiably true answer (i.e., factual information) exists for each unit.• The question could be accurately answered by an observer.e.g.,How times during 2009 did you visit a G.P.? ______ 50
  • Subjective questions•Asks about fuzzy personal perceptions.•There is no “true”, factual answer.•Many possible answers per unit.•Cant be accurately answered by an observer. e.g.,Think about the visits you made to a G.P. during 2010. How well did you understand the medical advice you received? perfectly very well reasonably poorly not at all 51
  • Objective vs. subjective questions• Both types of questions may be appropriate; depends on the purpose of the study.• One criticism of this distinction: There is no such thing as “objective” and that all responses are subjective. 52
  • Open-ended Questions• Rich information can be gathered• Useful for descriptive, exploratory work• Difficult and subjective to analyse• Time consuming 53
  • Open-ended questions• Rich information can be gathered• Useful for descriptive, exploratory work• Difficult and subjective to analyse• Time consuming 54
  • Open-ended questions: ExamplesWhat are the main issues you are currently facing in your life?How many hours did you spend doing university study last week? _________ 55
  • Closed-ended questions• Important information may be lost forever• Useful for hypothesis testing• Easy and objective to analyse• Time efficient 56
  • Levels ofMeasurement =Type of DataStevens (1946) 57
  • Levels of measurement•Nominal / Categorical•Ordinal•Interval•Ratio 58
  • Discrete vs. continuous Discrete ---------- Continuous ___________ 59
  • Each level has the properties of the preceding levels, plus something more! 60
  • Categorical / nominal•Conveys a category label•(Arbitrary) assignment of #s tocategories e.g. Gender•No useful information, except aslabels 61
  • Categorical / nomimal example: Phrenological labels 62
  • Ordinal / ranked scale•Conveys order, but not distance e.g. in a race, 1st, 2nd, 3rd, etc. or ranking of favourites or preferences 63
  • Ordinal / ranked example: Ranked importanceRank the following aspects of the university according to what is most important to you (1 = most important through to 5 = least important)t Quality of the teaching and educationQ Quality of the social lifeQ Quality of the campusQ Quality of the administrationQ Quality of the universitys reputation64
  • Interval scale•Conveys order & distance•0 is arbitrary e.g., temperature (degrees C)•Usually treat as continuous for > 5intervals 65
  • Interval example: 8 point Likert scale 66
  • Ratio scale•Conveys order & distance•Continuous, with a meaningful 0point e.g. height, age, weight, time, number of times an event has occurred•Ratio statements can be made e.g. X is twice as old (or high or heavy) as Y 67
  • Ratio scale: Time 68
  • Why do levels of measurement matter?Different analytical procedures are used for different levels of data. More powerful statistics can be applied to higher levels 69
  • Levels of measurement: Revision question Fill in all cellsLevel Prop-erties Examples Descriptive Statistics GraphsNominal / CategoricalOrdinal / RankIntervalRatio 70
  • Closed-ended rating scales1.Dichotomous2.Multichotomous3.Verbal frquency scale4.The list (multiple response)5.Ranking6.Likert scale7.Graphical rating scale8.Semantic differential 71
  • Dichotomous2 response options e.g.,Excluding this trip, have you visited Canberra in the previous five years? __ Yes __ NoProvides the simplest type of quantification 72
  • MultichotomousHow many hours did you spend doing university study last week?__ less than 5 hours__ > 5 to 10 hours__ > 10 to 20 hours__ more than 20 hours 73
  • MultichotomousMore than two possible answers e.g.,What type of attractions in your current trip to Canberra most appeal to you?__ historic buildings__ museum/art galleries__ parks and gardens 74
  • Verbal frequency scaleOver the past month, how often have you argued with your intimate partner?1. All the time2. Fairly often3. Occasionally4. Never5. Doesn’t apply to me at the moment 75
  • The list (multiple response)Provides a list of answers for respondents to choose from e.g.,Tick any words or phrases that describe your perception of Canberra as a travel destination:__ Exciting __ Important__ Boring __ Enjoyable__ Interesting __ Historical 76
  • The list (multiple-response)What are the main issues that you are currently facing in your life? (tick all that apply)__ financial__ physical / health__ academic__ employment / unemployment__ relationships__ other (please specify) 77
  • RankingHelps to measure the relative importance of several itemsRank the importance of these reasons for visiting Canberra (from 1 (most) to 4 (least)):__ to visit friends and relatives__ for business__ for educational purposes 78
  • Likert scaleMeasures strength of feeling or perception.Indicate your degree of agreement with this statement:“I am an adventurous person.” (circle the best response for you) 1 2 3 4 5strongly disagree neutral agree stronglydisagree agree 1 2 3 4 5strongly agree neutral disagree stronglyagree disagree 79
  • Graphical rating scaleHow would you rate your enjoyment of the movie you just saw? Mark with a cross (X)not enjoyable very enjoyable 80
  • Semantic differentialWhat is your view of smoking?Tick to show your opinion.Bad ___:___:___:___:___:___:___ GoodStrong ___:___:___:___:___:___:___ WeakMasculine ___:___:___:___:___:___:___ FeminineUnattractive ___:___:___:___:___:___:___ AttractivePassive ___:___:___:___:___:___:___ Active 81
  • Non-verbal scalePoint to the face that shows how you feel about what happened to the toy.Also called an idiographic scale. 82
  • Verbal frequency scaleOver the past month, how often have you argued with your intimate partner? 1. All the time 2. Fairly often 3. Occasionally 4. Never 5. Doesn’t apply to me at the moment 83
  • Sensitivity & reliability•Scale should be sensitive yetreliable.•Watch out for too few or toomany options. 84
  • Number of response options?How many response options?•Minimum = 2•Average = 3 to 9•Maximum = 10?Basic guide: 7 +/- 2 85
  • Number of response options? Likert scale example AGREEMENT ABOUT SOMETHING 2-Categories DISAGREE AGREE 3-Categories DISAGREE NEUTRAL AGREE 4-CategoriesSTRONGLY MILDLY MILDLY STRONGLYDISAGREE DISAGREE AGREE AGREE 5-CategoriesSTRONGLY MILDLY MILDLY STRONGLYDISAGREE DISAGREE NEUTRAL AGREE AGREE 86
  • Watch out for too many or too few response options “Capital punishment should be reintroduced for serious crimes” 1 = Agree 2 = Disagree1 = Very, Very Strongly Agree 7 = Slightly Disagree2 = Very Strongly Agree 8 = Disagree3 = Strongly Agree 9 = Strongly Disagree4 = Agree 10 = V. Strongly Disagree5 = Slightly Agree 11 = V, V Strongly Disagree6 = Neutral 87
  • Example: How couldthis question be improved?How old are you? ___ 18-20 ___ 20-22 ___ 22-30 ___ 30 and over 88
  • Example: How could this question be improved?Are you satisfied with your marriage and your job? __________________________ 89
  • Example: How could this question be improved?You didn’t think the food was very good, did you?_____ Yes _____ No 90
  • Example: How could this question be improved?Environmental issues have become increasingly important in choosing hotels. Are environmental considerations an important factor when deciding on your choice of hotel accommodation? ____ Yes ____ No 91
  • Example: How could this question be improved?What information sources did you use to locate your restaurant for today’s meal? (please tick appropriate spaces) ____ yellow pages ____ Internet ____ word of mouth 92
  • Comparison of data collection methods Personal Telephone MailData collection costs High Medium LowData collection time required Medium Low HighSample size for a given budget Small Medium LargeData quantity per subject High Medium LowReaches widely dispersed sample No Maybe YesReaches special locations Yes Maybe NoInteraction with respondents Yes Yes NoDegree of interviewer bias High Medium NoneSeverity of non-response Low Low HighPresentation of visual stimuli Yes No MaybeFieldworker training required Yes Yes No Alreck and Settle (1995:32) 93Alreck & Settle (1995; 32)
  • Finalise Questionnaire Draft• Questions need to be exhaustive and mutually exclusive – Include ‘other (please specify)’ – Ensure categories do not overlap 94
  • Finalise Questionnaire Draft• Length – Try to keep them as short as possible – Only ask questions that relate to objectives – Tricks? Font size/double sided photocopying/numbering sections 95
  • Maximising Response Rate• Layout and design is key• Respondent’s level of interest• Colour of paper• Accompanying letter / introduction• Mail surveys - self-addressed stamped return envelope• Rewards• Reminders or follow up calls96
  • Pre-testing and Pilot testing• Pre-test – try out on convenient others & revise• Pilot test – try out on a small sample from the target population & revise• Be assertive and interactive about seeking feedback – ask questions & observe 97
  • Pre-test & Revise•Pre-test items and ask for feedback•Revise:–items which don’t apply to everybody–redundancy–skewed response items–misinterpreted items–non-completed items•Reconsider ordering & layout
  • LUNCH BREAK 99
  • Survey Design CritiqueIn pairs, look through the example questionnaires, and highlight aspects which:• could be improved• are particularly good• you would like to ask about• 100
  • Examples• Examine questionnaire examples• Examine structure, design issues and question styles• Note cover page and details provided 101
  • Sampling 102
  • Sampling: OverviewSampling terminologyWhat is sampling?Why sample?Sampling methodsExample: Shere Hite’s survey 103
  • Sampling terminology• Target population – To whom you wish to generalise• Sampling frame – Those who have a chance to be selected• Sample – Those who were selected and responsed• Representativeness – The extent to which the sample is a good 104
  • What is sampling?“Sampling is the process of selecting units (e.g., people, organizations) from a population of interest so that by studying the sample we may fairly generalize our Picture 2 results back to the population from which they were chosen.” 105
  • Why sample?• Reduces cost, time, sample size etc.• If the sample is representative, the sample data allows inferences to be drawn about the total population. 106
  • Representativeness of a sample depends on:• Adequacy of sampling frame• Sampling method• Adequacy of sample size• Response rate – both the % & representativeness of people in sample who actually complete surveyIt is better to have a small, representative sample than a large, 107
  • Sampling methodsProbability sampling• Random• Systematic• Cluster – Multi-Stage ClusterNon-probability sampling–Quota–Convenience–Snowball 108
  • Random/probability sampling• Each unit has an equal chance of selection• Selection occurs entirely by random chance• Also called representative sampling 109
  • Simple random sampling• Everyone in the target population has an equal chance of selection• Useful if clear study area or population is identified• Similar to a lottery: – List of names are assigned #s and randomly select #s of respondents – Randomly select # through table of random #s or by computer 110
  • Systematic random sampling• Selecting without first numbering• Respondents (units) selected from a list/file.• Useful when survey population is similar e.g. List of students• Select sample at regular intervals from the population e.g., every 5th person on a list, starting at a random number between 1 and 5 111
  • Stratified random sampling• Sub-divide population into strata (e.g., by gender, age, or location)• Then random selection from within each stratum• Improves representativeness• e.g., Telephone interviews using post-code strata 112
  • Non-random / non-probability• Also called purposive or judgemental sampling• Useful for exploratory research and case study research• Able to get large sample size quickly• Limitations include potential bias and non-representativeness 113
  • Convenience sampling• Sampling is by convenience rather than randomly• Due to time/financial constraints• e.g. surveying all those at a tourist attraction over one weekend 114
  • Purposive samplingRespondents selected for a particular purpose e.g., because they may be “typical” respondents• e.g., select sample of tourists aged 40-60 as this is the typical age group of visitors to Canberra• e.g., Frequent flyers to contact regarding service quality in an airline setting 115
  • Snowball sampling• Useful for difficult to access populations e.g., illegal immigratnts, drug users• Respondents recommend other respondents• e.g., in studying ecstasy users, gain trust of a few potential respondents and ask them to recommend the researcher to other potential respondents 116
  • Sampling process1Identify target population andsampling frame2Select sampling method3Calculate sample size for desiredpower.4Maximise return rate 117
  • Summary of sampling strategy•Identify target population andsampling frame•Selection sampling method•Calculate required sample size•Maximise return rate
  • Sampling Example: Shere Hite‘American Sexology’ 119
  • Hites survey of Americanmale-female relations (early 1980s)•Shere Hite ‘doyenne of sex polls’•Media furors & worldwide attention•127-item questionnaire aboutmarriage & relations between sexes•Sample: 4500 USA women, 14 to 85years•Conclusion: Society and men need tochange to improve lives of women 120
  • Some of Hite’s findings about American women....•Only 13% married for 2+years were still in love•70% married for 5+ years were having affairs... – usually more for emotional closeness’ than sex – 76% of these women did not feel guilty•87% had a closer female friend than husband•98% wanted “basic changes” to loverelationships•84% were emotionally unsatisfied•95% reported emotional & psychologicalharassment from their men 121
  • Some of the critical comments.... The survey often seems merely toShe goes in with provide anprejudice & occasioncomes for the author’sout with a own male-bashingstatistic. diatribes. Hite uses statistics to bolster her opinion that American women are justifiably fed up with American men. 122
  • Hites response rate & selection bias•100,000 questionnaires were sentto a variety of women’s groups(feminist organisations, church groups,garden clubs etc.)•4,500 replied(4.5% return rate) 123
  • Hites response rate & selection bias“We get pretty nervous if respondents in our survey go under 70%. Respondents to surveys differ from nonrespondents in one important way: they go to the trouble of filling out what in this case was a very long, complicated, and personal questionnaire.” 124
  • Lessons from Hites male- female relations survey1Sample size – its not how big, its how representative2Objectivity – watch out for manipulating the survey questions and results interpretation to suit your personal conjectures 125
  • Measurement Error 126
  • Measurement error•Observed score = true score + measurement error•Measurement error = systematic error + random error•Any deviation from the truevalue caused by themeasurement procedure. 127
  • Sources of measurement error Picture 5 Non-sampling Sampling (e.g., unreliable (e.g., non-rep. samp or invalid tests) Personal bias (e.g., researcher fParadigm(e.g., Western focus on individualism) 128
  • To minimise measurement error Use well designed measures: • Multiple indicators • Sensitive to target constructs • Clear wording on questions/instructions 129
  • To minimise measurement error Reduce demand effects: • Train interviewers • Use standard protocol 130
  • To minimise measurement error Maximise response rate: • Pre-survey contact • Minimise length / time / hassle • Offer rewards / incentives • Coloured paper • Call backs / reminders 131
  • To minimise measurement error Ensure administrative accuracy • Set up efficient coding, with well- labelled variables • Check data 132
  • Summary of sampling strategy•Identify target population andsampling frame•Selection sampling method•Calculate required sample size•Maximise return rate
  • Sampling TaskA research projects aim is – “To identify the behaviour and attitudes of UC students with regard to its computing services”.• What is the research population?• How might you get hold of a sample frame?• What sampling technique would 134
  • Confidence Levels / Margins of Error• Relates to representativeness of a sample of a target population - to what extent can we be confident about the results?• Gives the estimated range of values into which we expect other samples to fall say 95% of the time• Social sciences = at least 90% or above, preferably 95% 135
  • Example 95%Confidence Level Graphs 136
  • Example 95%Confidence Level Table 137
  • Confidence Level ExampleSurvey of visitors to Canberra between June 1 - August 31, 2004 = 10,000 visitorsWant to work with 95% confidence level and 3% margin of errorHow many to survey?Answer: Sample size = 964 peopleQ: What are your favourite attractions? 138
  • Following Results+ / - 3% error @ 95% confidence level 139
  • Confidence Intervals / Margins of Error• One time out of 20, we expect the answers may be greater than +/- 3% – Establish the confidence level, margin of error you want to work with – Identify the number of surveys to be done – Once you have completed that number, do not do any additional surveysHOWEVER…. 140
  • Confidence Intervals / Margins of Error• Sometimes we do surveys and we do not know how many will be returned until later, as with postal surveys• Thus you have to calculate the margin of error afterwards…. – Count up # of returned surveys – Identify target population and confidence level 141
  • Confidence Intervals / Margins of Error• All this assumes you know your target population and can get a sample frame – If not, a non-random sampling technique is best• Also consider whether you want to report results for sub-groups – the margins of error will be wider 142
  • Summary - 11 Survey research has developed into a popular research method since the 1940s.2 A survey is a standardised stimulus designed to convert fuzzy psychological phenomenon into hard data. 143
  • Summary - 23 Survey development - types of questions and response formats.4 Sampling - probability & non-prob.5 Levels of measurement & parametric / non-parametric stats6 Ethical considerations7 Sources of measurement error 144
  • Evaluation• Please complete the workshop evaluations 145
  • ReferencesAlreck, P. & Settle, R. (1995). The survey research handbook (2nd ed.). New York: Irwin.Stevens, S.S. (1946). On the theory of scales of measurement. Science, 103, 677-680.Trochim, W. M. K. (2006). Sampling. In Research Methods Knowledge Base.Wikipedia (2009). Shere Hite - Methodology.• 146
  • Open Office Impress● This presentation was made usingOpen Office Impress.● Free and open source software.http://www.openoffice.org/product/impress.html●