Research Design in Clinical Psychology


Published on

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Research Design in Clinical Psychology

  1. 1. Research Design in Clinical Psychology Lecture 2 Reliability, Validity, and Artifact/Bias (Chapters 2-4 in Kazdin)
  2. 2. Internal Validity • To what extent can the intervention, and not other factors, account for study results • History • Maturation • Testing (Practice) • Instrumentation • Statistical Regression • Selection Bias • Attrition – these can occur across all groups or only to select groups • Diffusion/Imitation of treatment • Special treatment/reaction of controls
  3. 3. External Validity • To what extent can the study results be generalized to other samples with different characteristics than the study sample
  4. 4. External Validity: Examples 1 • Sample characteristics • Differences b/n study sample and other samples • Include age, gender, culture, education • Stimulus characteristics and settings • Extension across characteristics of the study • Includes setting, experimenter, materials/apparatus • Ecological validity? • Reactivity of experimental arrangements • Awareness of being in a study may affect behavior • Are responses relevant to those who are not in a study • Multiple-treatment interference • Subjects receives multiple interventions • Relevant to those who did not receive other interventions?
  5. 5. External Validity: Examples 1 • Novelty • Is effect due to newness • Reactivity of assessment • Similar to experimental arrangements, but focuses on awareness of what the measures are tapping • Test sensitization • Does pre-testing or the test itself (in the case of post-test sensitization) alter subject experience and responses • Timing of Measurement • When assessments are given could alter results
  6. 6. Parsimony • Least complex explanations first • Frequently, a threat to internal validity is most parsimonious • When considering limitations in both internal and external validity, however, parsimony suggests that “findings are the best statement of a relationship, unless there are clear reasons to think otherwise.
  7. 7. Why internal validity precedes external validity • Can’t have ExtVal, without IntVal • It would be like asking, “gee can we generalize these results we have no confidence in to a wide variety of individuals?” • One can still have important findings that elucidate basic principles without much ExtVal. • A lack of generalization of a finding across samples may be very important and can span other research
  8. 8. Construct Validity • What is the causal agent and conceptual basis underlying an effect (what is the intervention and why did it lead to change?) • In clinical research methods, CV is different from in test construction where CV is the extent that a measure captures a construct of interest.
  9. 9. Threats to construct validity • Contact time • Placebo effects • Nonblind (single and double) designs • Single operations and narrow stimulus sampling – Is the effect due to the selected IV – Does other aspects of the intervention have an effect beyond the aspect identified by the experimenter • Expectancies • Cues and Demand characteristics • Others exist based on conceptual relevance
  10. 10. Statistical Conclusion Validity AKA Data Evaluation Validity • Refers to the facets of the quantitative evaluation that influence the conclusions reached about experimental conditions and effects (to what extent are “real” effects demonstrated and interpreted) • Does one understand the stats used • Has one done stats correctly
  11. 11. Rejecting the Null Hypothesis I • Alpha = probability of rejecting the null when you shouldn’t (Type 1 error) • Saying groups are different when in reality are same • Beta = probability of accepting the null when you shouldn’t (Type 2 error) • Saying groups are same when in reality are different • Power = probability of correctly accepting or rejecting the null
  12. 12. Rejecting the Null Hypothesis II • Standard deviation = variability around mean – sqrt [(each observation – mean)2 / (N -1)] – SS/df • Effect size = Beyond significance, it is the magnitude of difference b/n groups • Expressed in terms of SD units » ES = (M1 – M2) / SD
  13. 13. Threats to statistical conclusion validity • Low power • Inability to detect real differences • Variability in the procedures • Subjects in same group get different treatment • Subject heterogeneity • Subjects in same group differ on potentially confounding variables • Unreliability of measures • Multiple comparisons/error rates • Experiment-wise error
  14. 14. Experimental Precision • Controlling vs holding constant • Tradeoffs
  15. 15. Sources of Artifacts and Bias
  16. 16. Rationale, scripts, & procedures • Imprecision in carrying out procedures • Experimenter may not have clear a protocol – Loose protocol effect • Experimenter may ignore protocol • Strategies for overcoming include • developing a clear protocol • actually testing the protocol out ahead of time • Get feedback from participants • Train experimenters together • Document all deviations
  17. 17. Experimenter Expectancy • Experimenters’ knowledge of study hypotheses may bias study implementation • Strategies for overcoming include • Keep experimenters naïve to purpose • Double blind procedures for group assignment – Success of these strategies can be empirically reviewed
  18. 18. Sample selection • Convenience samples and Volunteers – Do these sample differ in ways that affect the results or their generalizability • May select certain types of convenience or volunteer samples to answer particular questions • Attrition – Difference in # of dropouts across groups could have huge effect on results – Reminders and Commitment tactics (backloading $$$) – Select those unlikely to drop-out? – Plan ahead from past research
  19. 19. Other sources • Experimenter characteristics • Situational and context cues • Demand characteristics – clues on how to respond • Subject roles (pg 95 in book) • Data recording and analysis