Published on

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  •     2 June 2008
  • 2 June 2008
  • 2 June 2008
  • 2 June 2008
  • 2 June 2008
  • 2 June 2008
  • Within the one meta-analysis, can include studies based on any combination of statistical analysis (e.g., t-tests, ANOVA, multiple regression, correlation, odds-ratio, chi-square, etc). However, you have to convert each of these to a common “effect size” metric. 2 June 2008
  • 2 June 2008
  • Download

    1. 1. NCRM Research Methods Festival University of Oxford Dept of Education, University of Oxford
    2. 2. <ul><li>Traditionally, education researchers collect and analyse their own data (referred to as primary data). Secondary data analysis is based on data collected by someone else (or, perhaps, re-analysis of your own published data). There are at least four logical perspectives to this issue: </li></ul><ul><li>1. Meta-analysis -- systematic, quantitative review of published research in a particular field, the focus of this presentation. </li></ul><ul><li>2. Systematic review -- systematic, qualitative review of published research in a particular field </li></ul><ul><li>3. Secondary Data Analyses -- using large (typically public) databases </li></ul><ul><li>4. Re-analyses of published studies -- often in ways critical of the original study. </li></ul>
    3. 3. <ul><li>Systematic synthesis of various studies on a particular research question </li></ul><ul><ul><li>Do boys or girls have higher self-concepts? </li></ul></ul><ul><li>Collect all studies relevant to a topic </li></ul><ul><ul><li>Find all published journal articles on the topic </li></ul></ul><ul><li>An effect size (the ‘dependent variable’) is calculated for each outcome </li></ul><ul><ul><li>Determine the size/direction of gender difference for each study </li></ul></ul><ul><li>“ Content analysis” </li></ul><ul><ul><li>code characteristics of the study; age, setting, ethnicity, self-concept domain (math, physical, social), etc. </li></ul></ul><ul><li>Effect sizes with similar features are grouped together and compared; tests moderator variables </li></ul><ul><ul><li>Do gender differences vary with age, setting, ethnicity, self-concept, domain, etc. </li></ul></ul>
    4. 4. <ul><li>Coding: the process of extracting the information from the literature included in the meta-analysis. Involves noting the characteristics of the studies in relation to a priori variables of interest (qualitative) </li></ul><ul><li>Effect size: the numerical outcome to be analysed in a meta-analysis; a summary statistic of the data in each study included in the meta-analysis (quantitative) </li></ul><ul><li>Summarise effect sizes: central tendency, variability, relations to study characteristics (quantitative) </li></ul>
    5. 5. <ul><li>Compared to traditional literature reviews: </li></ul><ul><ul><li>(1) there is a definite methodology employed in the research analysis; and  </li></ul></ul><ul><ul><li>(2) the results of the included studies are quantified to a standard metric thus allowing for statistical techniques for further analysis. </li></ul></ul><ul><li>Therefore less biased and more replicable </li></ul>
    6. 6. <ul><li>Increased power: increases the chance of detecting a true treatment effect </li></ul><ul><li>Improved precision: with more information than a single study, the treatment effect estimate is improved </li></ul><ul><li>When study-to-study variation in results (which is typical) can evaluate differences in relation to study characteristics. Can delve into research questions not explored by the individual studies </li></ul><ul><li>Easy to interpret summary statistics (useful if communicating findings to a non-academic audience) </li></ul>
    7. 7. <ul><li>The essence of good science is replicable and generalisable results. </li></ul><ul><li>Do we get the same answer to important research questions when we run the study again? </li></ul><ul><li>The primary aims of meta-analysis is to test the generalisability of results across a set of studies designed to answer the same research question. </li></ul><ul><li>Are the results consistent? If not, what are the differences in the studies that explain the lack of consistency? </li></ul>
    8. 8. <ul><li>Meta-analysis is an increasingly popular tool for summarising research findings; literature review method of choice in many disciplines </li></ul><ul><li>Widely-cited. If there is a good meta-analysis relevant to your study, you have to cite it </li></ul><ul><li>Relied upon by policymakers </li></ul><ul><li>Important that we understand the method, whether we conduct or consume meta-analytic research </li></ul><ul><li>Should be one of the topics covered in all introductory research methodology courses </li></ul>
    9. 9. <ul><li>There exists a critical mass of comparable studies designed to address a common research question. </li></ul><ul><li>Data are presented in a form that allows the meta-analyst to compute an effect size for each study. </li></ul><ul><li>Characteristics of each study are described in sufficient detail to allow meta-analysts to compare characteristics of different studies and to judge the quality of each study. </li></ul>
    10. 10. The number of meta-analyses is increasing at a rapid rate.
    11. 11.
    12. 12. Psychology: Citations Psychology: Articles
    13. 13. <ul><li>Amato, P. R., & Keith, B. (1991). Parental divorce and the well-being of children: A meta-analysis . Psychological Bulletin, 110, 26-46. Times Cited: 471 </li></ul><ul><li>Linn, M. C., & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis . Child Development, 56, 1479-1498. Times Cited: 570 </li></ul><ul><li>Johnson, D. W., & et al (1981). Effects of cooperative, competitive, and individualistic goal structures on achievement: A meta-analysis . Psychological Bulletin, 89, 47-62. Times Cited: 426 </li></ul><ul><li>Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review . Personnel Psychology, 44, 703-742 Times Cited: 387 </li></ul><ul><li>Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis . Psychological Bulletin, 104, 53-69. Times Cited: 316 </li></ul><ul><li>Iaffaldano, M. T., & Muchinsky, P. M. (1985). Job satisfaction and job performance: A meta-analysis . Psychological Bulletin, 97, 251-273. Times Cited: 263. </li></ul>
    14. 14. <ul><li>De Wolff, M., & van IJzendoorn, M. H. (1997). Sensitivity and attachment: A meta-analysis on parental antecedents of infant attachment . Child Development, 68, 571-591. Times Cited: 340 </li></ul><ul><li>Wellman, H. M., Cross, D., & Watson, J. (2001). Meta-analysis of theory-of-mind development: The truth about false belief . Child Development, 72, 655-684. Times Cited: 276 </li></ul><ul><li>Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups . Review of Educational Research, 64, 1-35. Times Cited: 235 </li></ul><ul><li>Hansen, W. B. (1992). School-based substance abuse prevention: A review of the state of the art in curriculum, 1980-1990 . Health Education Research, 7, 403-430. Times Cited: 207 </li></ul><ul><li>Kulik, J. A., Kulik, C-L., Cohen, P. A. (1980). Effectiveness of Computer-Based College Teaching: A Meta-Analysis of Findings. Review of Educational Research, 50, 525-544. Times Cited: 198. </li></ul>
    15. 15. <ul><li>Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A meta-analysis of past research with recommendations for modifications and future research . Journal of Consumer Research, 15, 325-343. Times Cited: 515 </li></ul><ul><li>Jackson, S. E., & Schuler, R. S. (1985). A meta-analysis and conceptual critique of research on role ambiguity and role conflict in work settings . Organizational Behavior and Human Decision Processes, 36, 16-78. Times Cited: 401 </li></ul><ul><li>Tornatzky Lg, Klein Kj. (1994). Innovation characteristics and innovation adoption-implementation - A meta-analysis of findings . IEEE Transactions On Engineering Management, 29, 28-4. Times Cited: 269. </li></ul><ul><li>Lowe KB, Kroeck KG, Sivasubramaniam N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7 , 385-425. Times Cited: 203. </li></ul><ul><li>Churchill GA, Ford NM, Hartley SW, et al. (1985). Title: The determinants of salesperson performance - A meta-analysis . Journal Of Marketing Research, 22, 103-118. Times Cited: 189. </li></ul>
    16. 16. <ul><li>Jadad AR, Moore RA, Carroll D, et al. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17, 1-12. Times Cited:2008 </li></ul><ul><li>Boushey Cj, Beresford Saa, Omenn Gs, Et . Al. (1995). A quantitative assessment of plasma homocysteine as a risk factor for vascular-disease - Probable benefits of increasing folic-acid intakes. JAMA-journal Of The American Medical Assoc, 274, 1049-1057. Times Cited: 2,128 </li></ul><ul><li>Alberti W, Anderson G, Bartolucci A, et al. (1995). Chemotherapy in non-small-cell lung-cancer - A metaanalysis using updated data on individual patients from 52 randomized clinical-trials. British Medical Journal, 311, 899-909. Times Cited:1,591 </li></ul><ul><li>Block G, Patterson B, Subar A (1992). Fruit, vegetables, and cancer prevention - A review of the epidemiologic evidence. Nutrition And Cancer-an International Journal, 18, 1-29. Times Cited: 1,422 </li></ul>
    17. 17. <ul><li>Gene Glass coined the phrase meta-analysis in classic study of the effects of psychotherapy. Because most individual studies had small sample sizes, the effects typically were not statistically significant. </li></ul><ul><ul><li>Results of 375 controlled evaluations of psychotherapy and counselling were coded and integrated statistically. The findings provide convincing evidence of the efficacy of psychotherapy. </li></ul></ul><ul><ul><li>On the average, the typical therapy client is better off than 75% of untreated individuals. </li></ul></ul><ul><ul><li>Few important differences in effectiveness could be established among many quite different types of psychotherapy (e.g., behavioral and non-behavioral). </li></ul></ul>ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)
    18. 18.
    19. 19. <ul><li>Need to have explicit inclusion and exclusion criteria </li></ul><ul><li>The broader the research domain, the more detailed they tend to become </li></ul><ul><li>Refine criteria as you interact with the literature </li></ul><ul><li>Components of a detailed criteria </li></ul><ul><ul><ul><li>distinguishing features </li></ul></ul></ul><ul><ul><ul><li>research respondents </li></ul></ul></ul><ul><ul><ul><li>key variables </li></ul></ul></ul><ul><ul><ul><li>research methods </li></ul></ul></ul><ul><ul><ul><li>cultural and linguistic range </li></ul></ul></ul><ul><ul><ul><li>time frame </li></ul></ul></ul><ul><ul><ul><li>publication types </li></ul></ul></ul>
    20. 20. <ul><li>Search electronic databases (e.g., ISI, Psychological Abstracts, Expanded Academic ASAP, Social Sciences Index, PsycINFO, and ERIC) </li></ul><ul><li>Examine the reference lists of included studies to find other relevant studies </li></ul><ul><li>If including unpublished data, email researchers in your discipline, take advantage of Listservs, and search Dissertation Abstracts International </li></ul>
    21. 21. <ul><li>Random selection of papers coded by both coders </li></ul><ul><li>Meet to compare code sheets </li></ul><ul><li>Where there is discrepancy, discuss to reach agreement </li></ul><ul><li>Amend code materials/definitions in code book if necessary </li></ul><ul><li>May need to do several rounds of piloting, each time using different papers </li></ul>
    22. 22. <ul><li>__ Study ID </li></ul><ul><li>_ _ Year of publication </li></ul><ul><li>__ Publication type (1-5) </li></ul><ul><li>__ Geographical region (1-7) </li></ul><ul><li>_ _ _ _ Total sample size </li></ul><ul><li>_ _ _ Total number of males </li></ul><ul><li>_ _ _ Total number of females </li></ul>Code Sheet Code Book/manual ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) <ul><li>Publication type (1-5) </li></ul><ul><li>Journal article </li></ul><ul><li>Book/book chapter </li></ul><ul><li>Thesis or doctoral dissertation </li></ul><ul><li>Technical report </li></ul><ul><li>Conference paper </li></ul>1 99 2 1 87 41 46
    23. 23. <ul><li>The effect size makes meta-analysis possible </li></ul><ul><ul><li>It is the “dependent variable” </li></ul></ul><ul><ul><li>It standardizes findings across studies such that they can be directly compared </li></ul></ul><ul><li>Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds-ratio), but must </li></ul><ul><ul><li>be comparable across studies (generally requires standardization) </li></ul></ul><ul><ul><li>represent the magnitude and direction of the relationship of interest </li></ul></ul><ul><ul><li>be independent of sample size </li></ul></ul>
    24. 24. Means and standard deviations Correlations P-values F -statistics d t -statistics “ other” test statistics Almost all test statistics can be transformed into an standardized effect size “d” ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)
    25. 25. <ul><li>Lipsey & Wilson (2001) present many formulae for calculating effect sizes from different information </li></ul><ul><li>All effect sizes in a single meta-analysis must be in a common metric, typically based on the “natural” metric given research in the area. E.g.: </li></ul><ul><ul><li>Standardized mean difference </li></ul></ul><ul><ul><li>Odds-ratio </li></ul></ul><ul><ul><li>Correlation coefficient </li></ul></ul>
    26. 26. Each study is one line in the data base Effect size Duration Sample sizes Reliability of the instrument Variance of the effect size
    27. 27. <ul><li>There are various ways of analysing meta-analytic data </li></ul><ul><li>Three main methods based on different statistical assumptions: </li></ul><ul><ul><li>Fixed effects models </li></ul></ul><ul><ul><li>Random effects models </li></ul></ul><ul><ul><li>Multilevel models </li></ul></ul><ul><li>These will be discussed in the afternoon workshop </li></ul>
    28. 28. <ul><li>Meta-analysis is a method for synthesising and analysing the research literature on a particular topic </li></ul><ul><li>The essence of good science is replicable and generalisable results. </li></ul><ul><li>Increasingly sophisticated </li></ul><ul><li>Continuously evolving </li></ul><ul><li>For more information about the meta-analysis training courses that we offer, please see </li></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.