Review of "Survey Research Methods & Design in Psychology"

14,073 views

Published on

Reviews the 150 hour, third year psychology unit which examined survey research methods, with an emphasis on the second-half of the unit on MLR, ANOVA, power, and effect size.

0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
14,073
On SlideShare
0
From Embeds
0
Number of Embeds
248
Actions
Shares
0
Downloads
870
Comments
0
Likes
19
Embeds 0
No embeds

No notes for slide
  • 7126/6667 Survey Research & Design in Psychology Semester 1, 2011, University of Canberra, ACT, Australia James T. Neill Home page: http://ucspace.canberra.edu.au/display/7126 Lecture page: http://ucspace.canberra.edu.au/display/7126/Lecture+-+Review Notes page: http://en.wikiversity.org/wiki/Survey_research_methods_and_design_in_psychology Image name: Recycle symbol Taiwan.svg Image source: http://commons.wikimedia.org/wiki/File:Recycle_symbol_Taiwan.svg Image author:Bryan Derksen http://commons.wikimedia.org/wiki/User:Bryan_Derksen License : Public domain Description: Reviews this semester-long (150-hour), third year undergraduate psychology research unit which focused on survey research methods and survey design. This lecture emphasises the second-half of the unit's content on MLR, ANOVA, significance testing, power, and effect size, as well as providing advice about the lab report and final exam assessment items.
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain Image source: http://commons.wikimedia.org/wiki/File:Autoroute_icone.svg License: CC-BY-A 2.5 Author: http://commons.wikimedia.org/wiki/User:Doodledoo
  • http://en.wikiversity.org/wiki/Survey_research_and_design_in_psychology/Overview
  • Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
  • Image source: Unknwn
  • Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
  • Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Compare models across groups % variance explained No. of factors Item loadings
  • Reversing a scale e.g., IM = mean(item1,item2,item3) EM = mean (item4,item5,item6) M = IM + (8 – EM) 1 2 3 4 5 6 7 7 6 5 4 3 2 1
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: James Neill, Creative Commons Attribution-Share Alike 2.5 Australia, http://creativecommons.org/licenses/by-sa/2.5/au/
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Image source: http://commons.wikimedia.org/wiki/File:Information_icon4.svg License: Public domain
  • Review of "Survey Research Methods & Design in Psychology"

    1. 1. Lecture 12 Survey Research & Design in Psychology James Neill, 2011 Survey Research & Design in Psychology: Review & Summary
    2. 2. Overview <ul><li>Review </li></ul><ul><ul><li>Unit aims and outcomes
    3. 3. Research process
    4. 4. Survey design
    5. 5. MLR, ANOVA, ES & Power
    6. 6. What type of analysis? (decision tree) </li></ul></ul><ul><li>Assessment </li><ul><li>Lab report
    7. 7. Final exam </li></ul><li>Evaluation & feedback </li></ul>
    8. 8. Unit aims & outcomes
    9. 9. Aims & outcomes <ul><li>Knowledge & skills for conducting ethical , well-designed , survey-based research in psychology. </li></ul>How confident are you that could conduct a good quality survey-based research study? For 4 th year Honours? In the work-place?
    10. 10. Aims & outcomes <ul><li>Theory & practice of survey-based research, incl.: </li></ul><ul><ul><li>Research questions / hypotheses
    11. 11. Survey design
    12. 12. Sampling
    13. 13. Interpreting & communicating results </li></ul></ul>
    14. 14. Aims & outcomes <ul><li>Use of SPSS for: </li></ul><ul><ul><li>Data entry
    15. 15. Correlations
    16. 16. Factor analysis
    17. 17. Qualitative analysis
    18. 18. Reliability
    19. 19. MLR
    20. 20. Advanced ANOVA </li></ul></ul>
    21. 21. Research process
    22. 22. Research process 1. Establish need for info/ research 2. Problem definition/ Hypotheses 3. Research design 4. Sampling/ Data collection 5. Data analysis 6. Reporting
    23. 23. Funnel model
    24. 24. Survey design
    25. 25. Survey design <ul><li>Operationalisation of fuzzy concepts
    26. 26. Question types (objective/subjective, open/closed)
    27. 27. Response formats (e.g., dichotomous, frequency, Likert, multiple response, idiographic)
    28. 28. Levels of measurement (LOM)
    29. 29. Reliability & validity
    30. 30. Sampling (probability/non-probability)
    31. 31. Modes of administration (self-report (f2f, mail, web), interview (f2f/phone)) </li></ul>
    32. 32. Items should measure different aspects of the latent (underlying) construct Latent Construct Measured Construct
    33. 33. Latent Construct Poor items will create ‘brown sludge’ (noisy (unreliable) measure) Measured Construct
    34. 34. Describing data
    35. 35. Describing data <ul><li>Data screening (out of range, invalid etc.)
    36. 36. Discrete data: </li><ul><li>Frequencies & %s </li></ul><li>Continuous data: 4 moments of a normal distribution </li><ul><li>Central tendency
    37. 37. Dispersion
    38. 38. Skewness
    39. 39. Kurtosis </li></ul></ul>
    40. 40. Visualisation of data <ul><li>Aids interpretation of descriptives and tests of differences or relationships.
    41. 41. Univariate : </li><ul><li>histogram, bar graph, error-bar graph </li></ul><li>Bivariate : </li><ul><li>scatterplot, clustered bar graph </li></ul><li>Multivariate : </li><ul><li>Venn diagrams, multiple line graph, 3-d scatterplot </li></ul></ul>
    42. 42. Software for data visualisation (graphing) <ul><li>Statistical packages </li></ul><ul><ul><li>e.g., SPSS Graphs or via Analyses </li></ul></ul><ul><li>Spreadsheet packages </li></ul><ul><ul><li>e.g., MS Excel </li></ul></ul><ul><li>Word-processors </li></ul><ul><ul><li>e.g., MS Word – Insert – Object – Micrograph Graph Chart </li></ul></ul>
    43. 43. Correlations
    44. 44. Correlations <ul><li>Strength & direction of bivariate linear relations
    45. 45. Building block for FA & MLR
    46. 46. Non-parametric correlations (e.g., Point bi-serial, Phi/Cramer's V )
    47. 47. Scatterplots – watch out for: </li><ul><li>Outliers
    48. 48. Non-linearity
    49. 49. Limited range </li></ul><li>Caution with causal interpretation </li></ul>
    50. 50. Factor analysis
    51. 51. Factor analysis <ul><li>Purpose </li><ul><li>Data reduction
    52. 52. Developing reliable & valid measures of fuzzy construct </li></ul><li>Assumptions </li><ul><li>Linear relations
    53. 53. Sample size (min. 5 cases: 1 variable) </li></ul></ul>
    54. 54. Factor analysis <ul><li>Extraction </li><ul><li>PC vs. PAF </li></ul><li>Rotation method </li><ul><li>Varimax vs. Oblimin </li></ul><li>Number of factors </li><ul><li>Kaiser’s criterion
    55. 55. Scree plot
    56. 56. Theoretical structure
    57. 57. Interpretation of factor loadings </li></ul></ul>
    58. 58. Factor analysis <ul><li>Decide how many factors
    59. 59. Iteratively eliminate items </li><ul><li>Communality > .5?
    60. 60. Primary loadings > .5?
    61. 61. Cross-loadings < .3?
    62. 62. Diff. btw primary & cross-loadings > .2?
    63. 63. Sufficient items per factor
    64. 64. Face validity </li></ul><li>Correlations between factors
    65. 65. Compare model across groups </li></ul>
    66. 66. Reliabilities & composite scores <ul><li>Internal reliability : For each factor, calculate Cronbach’ s  : </li><ul><li>> .8 = very good
    67. 67. > .7 = good
    68. 68. > .6 = OK </li></ul><li>Composite scores </li><ul><li>Unit-weighting
    69. 69. Regression-weighting </li></ul></ul>
    70. 70. Qualitative analysis
    71. 71. Qualitative analysis <ul><li>Purposes </li><ul><li>Pilot study, exploratory research
    72. 72. Theory testing; Data reduction (synthesise meaning)
    73. 73. Validity-testing </li></ul><li>Methods </li><ul><li>Quantitative (Content analysis, Multiple response analysis, Graph (e.g., bar graph)
    74. 74. Qualitative (Thematic analysis) </li></ul></ul>
    75. 75. Inferential statistical decision making tree
    76. 76. Statistical decision tree <ul><li>Establish a research question and/or hypothesis </li><ul><li>Differences or relationships?
    77. 77. No. of IVs and DVs
    78. 78. Identify levels of measurement </li></ul><li>Use a statistical decision tree to identify an appropriate analysis
    79. 79. See Inferential statistics decision-making tree </li></ul>
    80. 80. Multiple linear regression
    81. 81. Multiple linear regression <ul><li>L inear regression formula </li></ul>Y hat = ax + b Y = ax + b + e <ul><li>Proportion of variance in a DV explained by one or more IVs </li><ul><li>R , R 2 , Adjusted R 2
    82. 82. F (significance of R )
    83. 83. Change in R 2 (Hierarchical) </li></ul></ul>
    84. 84. Multiple linear regression <ul><li>Assumptions : </li><ul><li>LOM </li><ul><li>Continuous DV
    85. 85. Dichotomous or continuous IVs </li></ul><li>Normality (helps satisfy other assumptions)
    86. 86. Linearity
    87. 87. Homoscedasticity
    88. 88. Multicollinearity
    89. 89. Multivariate outliers </li></ul></ul>
    90. 90. Multiple linear regression <ul><li>Methods </li><ul><li>Standard / Direct
    91. 91. Hierarchical
    92. 92. Forward
    93. 93. Backward
    94. 94. Stepwise </li></ul></ul>
    95. 95. <ul><li>Overall hypothesis : </li><ul><li>(Null) That the IVs do not explain variance in the DV (i.e., that R is 0) </li></ul><li>+ one hypothesis per predictor : </li><ul><li>(Null) That each IV is not a significant predictor of variance in the DV (i.e., that t for each predictor is less than the critical value) </li></ul></ul>Multiple linear regression
    96. 96. <ul><li>Consider/interpret : </li><ul><li>Direction of each predictor
    97. 97. Size of each predictor (β - standardised)
    98. 98. When IVs are correlated, interpret zero-order vs. partial correlations
    99. 99. sr 2 = % variance in DV explained by each IV </li></ul><li>Can use Venn or path diagrams to depict relationships between variables </li></ul>Multiple linear regression
    100. 100. ANOVA
    101. 101. ANOVA <ul><li>Extension of t -test
    102. 102. Like MLR , ANOVA involves: </li><ul><li>One continuous DV (although ANOVA can handle multiple DVs)
    103. 103. One or more IVs </li></ul><li>Unlike MLR , in ANOVA: </li><ul><li>Interactions are automatically tested
    104. 104. IVs must be categorical
    105. 105. Significant results may indicate need for followup or post-hoc tests </li></ul></ul>
    106. 106. Types of ANOVA <ul><li>1-way ANOVA
    107. 107. 1-way repeated measures ANOVA
    108. 108. 2-way factorial ANOVA
    109. 109. Mixed design ANOVA (Split-plot ANOVA)
    110. 110. ANCOVA
    111. 111. MANOVA </li></ul>
    112. 112. ANOVA <ul><li>Assumptions </li><ul><li>Cell size > 20 (Ideal)
    113. 113. Normally distributed DVs ( for each level of the IVs )
    114. 114. Homogeneity of variance (b/w subjects)
    115. 115. Sphericity (w/in subjects) </li></ul><li>Follow-up tests: </li><ul><li>Post-hoc tests
    116. 116. Planned comparisons </li></ul><li>Effect sizes : η 2 , η p 2 and Cohen’s d </li></ul>
    117. 117. Power, effect size, significance testing, publication bias & academic integrity
    118. 118. Significance testing Significance testing has dominated psychology, but is problematic, mainly because: <ul><ul><li>Results are dichotomous (sig. or not), which doesn't help us to understand the size of effect.
    119. 119. Sig. test results are influenced by power esp. if particularly high or low. </li></ul></ul>
    120. 120. Power & effect sizes Power and effect sizes have been neglected. Therefore: <ul><ul><li>Calculate the power of studies (prospectively &/or retrospectively)
    121. 121. Report ESs & CIs to complement inferential statistics </li></ul></ul><ul><ul><ul><li>r, r 2 , R , R 2
    122. 122. η 2 , η p 2 , d </li></ul></ul></ul>
    123. 123. Publication bias & academic integrity Publication bias (low power; favouritism of sig. findings; funnel plots) Academic integrity - “Integrity is doing the right thing, especially when no one is watching”.
    124. 124. Lab report
    125. 125. Lab report - Tips <ul><li>Check the lab report guidelines and the marking criteria
    126. 126. Use example HD lab report and sample write-ups as guides
    127. 127. Demonstrate your knowledge via independent thinking/work
    128. 128. Tell a story </li></ul>
    129. 129. Lab report - Structure <ul><li>Cover Sheet
    130. 130. Title Page
    131. 131. Abstract
    132. 132. Body
    133. 133. References
    134. 134. Appendices </li></ul>
    135. 135. Lab report - Introduction <ul><li>Explain the rationale – what is the problem to be solved?
    136. 136. No waffle – cut to the chase – only review literature or argument relevant to the RQs / hypotheses
    137. 137. State clear RQs and/or hypotheses – One per test/analysis/effect </li></ul>
    138. 138. Lab report - Method <ul><li>Well-organised (like a recipe)
    139. 139. Present relevant details efficiently (avoid getting bogged down in extraneous detail) </li></ul>
    140. 140. Lab report - Method <ul><li>Sections: </li></ul><ul><ul><li>Participants
    141. 141. Materials or Instrumentation
    142. 142. Procedure </li></ul></ul><ul><li>Replicable? A naïve reader must be able to replicate the study </li></ul>
    143. 143. Lab report – Results <ul><li>Data screening
    144. 144. Consider LOM assumptions
    145. 145. Caution in use of overall scores </li></ul>Overall Score not valid Overall Score valid 1 3 2 1 3 2
    146. 146. Lab report – Results (EFA) <ul><li>EFA : </li><ul><li>PC/PAF?
    147. 147. Varimax/Oblimin?
    148. 148. 2-6 factors?
    149. 149. 5-30 items removed?
    150. 150. 50%-60% of variance? </li></ul><li>Table of factor loadings and communalities
    151. 151. Correlations between factors
    152. 152. Internal consistency for each factor </li></ul>
    153. 153. Lab report – Results (MLR) <ul><li>Example MLR : Hierarchical: </li><ul><li>DV = Campus Satisfaction
    154. 154. Step 1 </li><ul><li>IV1 = Gender (M / F) </li></ul><li>Step 2 </li><ul><li>IV1 = Planning TM (Continuous)
    155. 155. IV2 = Time wasting TM (Continuous) </li></ul><li>R 2 , Adjusted R 2 , Change in R 2 </li></ul><li>Table of correlations and regression coefficients </li></ul>
    156. 156. Lab report – Results (ANOVA) <ul><li>ANOVA : e.g., 2 x (3) Mixed ANOVA </li><ul><li>Between-subjects IV: Enrolment Status (FT / PT)
    157. 157. Within-subjects DV: Satisfaction (Educational / Social / Campus) </li></ul><li>Table of cell and marginal descriptives ( M , SD , Sk, Kurt) + Graph
    158. 158. Effect sizes ( η p 2 , d ) </li></ul>
    159. 159. Lab report – Results (Qualitative) <ul><li>Example Qualitative analysis : Least or most satisfactory aspects of UC </li><ul><li>Research question?
    160. 160. Qualitative or quantitative approach?
    161. 161. Table of main themes (names, description, example quotes, frequency/%)
    162. 162. Bar graph?
    163. 163. Thick description?
    164. 164. Sub-samples? </li></ul></ul>
    165. 165. Lab report - Discussion <ul><li>Provide insight about the results
    166. 166. Draw conclusions about the RQs & hypotheses in light of the results.
    167. 167. Discuss key strengths & limitations of the study. ( Balanced criticism )
    168. 168. Draw out implications and recommendations </li></ul>
    169. 169. Lab report - Discussion <ul><li>Offer specific, practical recommendations e.g., </li></ul><ul><ul><li>Theory : What are the implications for the theory/rationale upon which the study was based?
    170. 170. Methods : How could the research design (e.g., instrumentation) be improved?
    171. 171. Practice : Implications for students and universities e.g., for improving satisfaction? </li></ul></ul>
    172. 172. Lab report – Appendices <ul><li>Optional : Include appendices where relevant and referred to in the body text. Appendices may not be consulted by a reader, so if its important make sure key content is covered in the text .
    173. 173. Use for content which would break the flow , but which is relevant to understanding the study e.g., the EFA correlation matrix. </li></ul>
    174. 174. Lab report – Appendices <ul><li>APA style not necessary.
    175. 175. Use headings (e.g., Appendix A, B, C etc.) and possibly titles e.g., </li></ul>Appendix A: Bivariate correlations amongst the university student satisfaction items
    176. 176. Lab report - Submission <ul><li>Insert & complete the Cover Sheet
    177. 177. Submit ONE DOCUMENT containing the coversheet, lab report and appendices
    178. 178. NO EXTRA ATTACHMENTS
    179. 179. Upload via the Moodle drop-box
    180. 180. You can re-submit up until the due date (after that, resubmission will incur late penalties) </li></ul>
    181. 181. Lab report - Marking <ul><li>Lab report marks should be returned before the exam .
    182. 182. Keep an eye on your student email and Moodle Announcements . </li></ul>
    183. 183. Final exam
    184. 184. Final exam <ul><li>Computer-based , supervised
    185. 185. Style, structure and content will be similar to the quizzes
    186. 186. ~ 120 multi-choice questions in 180 mins (plenty of time).
    187. 187. Practice exams are available </li></ul>
    188. 188. Final exam <ul><li>Permitted materials </li></ul><ul><ul><li>Non-programmable calculator
    189. 189. Non-annotated foreign language dictionary </li></ul></ul><ul><li>Marks and feedback </li></ul><ul><ul><li>Upon submitting, you will receive your exam mark and feedback. </li></ul></ul>
    190. 190. Evaluation & feedback
    191. 191. Evaluation & feedback: USS Unit Satisfaction Survey <ul><li>Available now on OSIS – but you may wish to wait until after assessment results
    192. 192. UC takes these results very seriously – ~40% response rate </li></ul>
    193. 193. Possible issues & topics Please contribute your honest feedback: <ul><li>Lectures?
    194. 194. Tutorials?
    195. 195. Textbooks?
    196. 196. Assessment?
    197. 197. Website(s)?
    198. 198. Software – SPSS?
    199. 199. Workload? </li></ul>
    200. 200. Open Office Impress <ul><li>This presentation was made using Open Office Impress.
    201. 201. Free and open source software.
    202. 202. http://www.openoffice.org/product/impress.html </li></ul>

    ×