Powerpoint Presentation
Upcoming SlideShare
Loading in...5

Powerpoint Presentation






Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  •     2 June 2008
  • 2 June 2008
  • 2 June 2008
  • 2 June 2008
  • 2 June 2008
  • 2 June 2008
  • Within the one meta-analysis, can include studies based on any combination of statistical analysis (e.g., t-tests, ANOVA, multiple regression, correlation, odds-ratio, chi-square, etc). However, you have to convert each of these to a common “effect size” metric. 2 June 2008
  • 2 June 2008

Powerpoint Presentation Powerpoint Presentation Presentation Transcript

  • NCRM Research Methods Festival University of Oxford Dept of Education, University of Oxford
    • Traditionally, social science researchers collect and analyse their own data (referred to as primary data). Secondary data analysis is based on data collected by someone else (or, perhaps, re-analysis of your own published data). There are at least four logical perspectives to this issue:
    • 1. Meta-analysis -- systematic, quantitative review of published research in a particular field, the focus of this presentation.
    • 2. Systematic review -- systematic, qualitative review of published research in a particular field
    • 3. Secondary Data Analyses -- using large (typically public) databases
    • 4. Re-analyses of published studies -- often in ways critical of the original study.
    • Systematic synthesis of various studies on a particular research question
      • Do boys or girls have higher self-concepts?
    • Collect all studies relevant to a topic
      • Find all published journal articles on the topic
    • An effect size (the ‘dependent variable’) is calculated for each outcome
      • Determine the size/direction of gender difference for each study
    • “ Content analysis”
      • code characteristics of the study; age, setting, ethnicity, self-concept domain (math, physical, social), etc.
    • Effect sizes with similar features are grouped together and compared; tests moderator variables
      • Do gender differences vary with age, setting, ethnicity, self-concept, domain, etc.
    • Coding: the process of extracting the information from the literature included in the meta-analysis. Involves noting the characteristics of the studies in relation to a priori variables of interest (qualitative)
    • Effect size: the numerical outcome to be analysed in a meta-analysis; a summary statistic of the data in each study included in the meta-analysis (quantitative)
    • Summarise effect sizes: central tendency, variability, relations to study characteristics (quantitative)
    • Compared to traditional literature reviews:
      • (1) there is a definite methodology employed in the research analysis; and 
      • (2) the results of the included studies are quantified to a standard metric thus allowing for statistical techniques for further analysis.
    • Therefore less biased and more replicable
    • Increased power: increases the chance of detecting a true treatment effect
    • Improved precision: with more information than a single study, the treatment effect estimate is improved
    • When study-to-study variation in results (which is typical) can evaluate differences in relation to study characteristics. Can delve into research questions not explored by the individual studies
    • Easy to interpret summary statistics (useful if communicating findings to a non-academic audience)
    • The essence of good science is replicable and generalisable results.
    • Do we get the same answer to important research questions when we run the study again?
    • The primary aims of meta-analysis is to test the generalisability of results across a set of studies designed to answer the same research question.
    • Are the results consistent? If not, what are the differences in the studies that explain the lack of consistency?
    • Meta-analysis is an increasingly popular tool for summarising research findings; literature review method of choice in many disciplines
    • Widely-cited. If there is a good meta-analysis relevant to your study, you have to cite it
    • Relied upon by policymakers
    • Important that we understand the method, whether we conduct or consume meta-analytic research
    • Should be one of the topics covered in all introductory research methodology courses
    • There exists a critical mass of comparable studies designed to address a common research question.
    • Data are presented in a form that allows the meta-analyst to compute an effect size for each study.
    • Characteristics of each study are described in sufficient detail to allow meta-analysts to compare characteristics of different studies and to judge the quality of each study.
  • The number of meta-analyses is increasing at a rapid rate.
  • Psychology: Citations Psychology: Articles
    • Amato, P. R., & Keith, B. (1991). Parental divorce and the well-being of children: A meta-analysis . Psychological Bulletin, 110, 26-46. Times Cited: 471
    • Linn, M. C., & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis . Child Development, 56, 1479-1498. Times Cited: 570
    • Johnson, D. W., & et al (1981). Effects of cooperative, competitive, and individualistic goal structures on achievement: A meta-analysis . Psychological Bulletin, 89, 47-62. Times Cited: 426
    • Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review . Personnel Psychology, 44, 703-742 Times Cited: 387
    • Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis . Psychological Bulletin, 104, 53-69. Times Cited: 316
    • Iaffaldano, M. T., & Muchinsky, P. M. (1985). Job satisfaction and job performance: A meta-analysis . Psychological Bulletin, 97, 251-273. Times Cited: 263.
    • De Wolff, M., & van IJzendoorn, M. H. (1997). Sensitivity and attachment: A meta-analysis on parental antecedents of infant attachment . Child Development, 68, 571-591. Times Cited: 340
    • Wellman, H. M., Cross, D., & Watson, J. (2001). Meta-analysis of theory-of-mind development: The truth about false belief . Child Development, 72, 655-684. Times Cited: 276
    • Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups . Review of Educational Research, 64, 1-35. Times Cited: 235
    • Hansen, W. B. (1992). School-based substance abuse prevention: A review of the state of the art in curriculum, 1980-1990 . Health Education Research, 7, 403-430. Times Cited: 207
    • Kulik, J. A., Kulik, C-L., Cohen, P. A. (1980). Effectiveness of Computer-Based College Teaching: A Meta-Analysis of Findings. Review of Educational Research, 50, 525-544. Times Cited: 198.
    • Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A meta-analysis of past research with recommendations for modifications and future research . Journal of Consumer Research, 15, 325-343. Times Cited: 515
    • Jackson, S. E., & Schuler, R. S. (1985). A meta-analysis and conceptual critique of research on role ambiguity and role conflict in work settings . Organizational Behavior and Human Decision Processes, 36, 16-78. Times Cited: 401
    • Tornatzky Lg, Klein Kj. (1994). Innovation characteristics and innovation adoption-implementation - A meta-analysis of findings . IEEE Transactions On Engineering Management, 29, 28-4. Times Cited: 269.
    • Lowe KB, Kroeck KG, Sivasubramaniam N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7 , 385-425. Times Cited: 203.
    • Churchill GA, Ford NM, Hartley SW, et al. (1985). Title: The determinants of salesperson performance - A meta-analysis . Journal Of Marketing Research, 22, 103-118. Times Cited: 189.
    • Jadad AR, Moore RA, Carroll D, et al. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17, 1-12. Times Cited:2008
    • Boushey Cj, Beresford Saa, Omenn Gs, Et . Al. (1995). A quantitative assessment of plasma homocysteine as a risk factor for vascular-disease - Probable benefits of increasing folic-acid intakes. JAMA-journal Of The American Medical Assoc, 274, 1049-1057. Times Cited: 2,128
    • Alberti W, Anderson G, Bartolucci A, et al. (1995). Chemotherapy in non-small-cell lung-cancer - A metaanalysis using updated data on individual patients from 52 randomized clinical-trials. British Medical Journal, 311, 899-909. Times Cited:1,591
    • Block G, Patterson B, Subar A (1992). Fruit, vegetables, and cancer prevention - A review of the epidemiologic evidence. Nutrition And Cancer-an International Journal, 18, 1-29. Times Cited: 1,422
    • Gene Glass coined the phrase meta-analysis in classic study of the effects of psychotherapy. Because most individual studies had small sample sizes, the effects typically were not statistically significant.
      • Results of 375 controlled evaluations of psychotherapy and counselling were coded and integrated statistically. The findings provide convincing evidence of the efficacy of psychotherapy.
      • On the average, the typical therapy client is better off than 75% of untreated individuals.
      • Few important differences in effectiveness could be established among many quite different types of psychotherapy (e.g., behavioral and non-behavioral).
    ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)
    • Need to have explicit inclusion and exclusion criteria
    • The broader the research domain, the more detailed they tend to become
    • Refine criteria as you interact with the literature
    • Components of a detailed criteria
        • distinguishing features
        • research respondents
        • key variables
        • research methods
        • cultural and linguistic range
        • time frame
        • publication types
    • Search electronic databases (e.g., ISI, Psychological Abstracts, Expanded Academic ASAP, Social Sciences Index, PsycINFO, and ERIC)
    • Examine the reference lists of included studies to find other relevant studies
    • If including unpublished data, email researchers in your discipline, take advantage of Listservs, and search Dissertation Abstracts International
    • Random selection of papers coded by both coders
    • Meet to compare code sheets
    • Where there is discrepancy, discuss to reach agreement
    • Amend code materials/definitions in code book if necessary
    • May need to do several rounds of piloting, each time using different papers
    • __ Study ID
    • _ _ Year of publication
    • __ Publication type (1-5)
    • __ Geographical region (1-7)
    • _ _ _ _ Total sample size
    • _ _ _ Total number of males
    • _ _ _ Total number of females
    Code Sheet Code Book/manual ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)
    • Publication type (1-5)
    • Journal article
    • Book/book chapter
    • Thesis or doctoral dissertation
    • Technical report
    • Conference paper
    1 99 2 1 87 41 46
    • The effect size makes meta-analysis possible
      • It is the “dependent variable”
      • It standardizes findings across studies such that they can be directly compared
    • Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds-ratio), but must
      • be comparable across studies (generally requires standardization)
      • represent the magnitude and direction of the relationship of interest
      • be independent of sample size
    • Represents a standardized group contrast on an inherently continuous measure
    • Uses the pooled standard deviation (some situations use control group standard deviation)
    • Commonly called “d”
    In a gender difference study, the effect size might be: In an intervention study with experimental and control groups, the effect size might be:
  • Means and standard deviations Correlations P-values F -statistics d t -statistics “ other” test statistics Almost all test statistics can be transformed into an standardized effect size “d” ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) Lipsey & Wilson (2001) present formulae for calculating effect sizes from different information
  • Each study is one line in the data base Effect size Duration Sample sizes Reliability of the instrument Variance of the effect size
    • There are various ways of analysing meta-analytic data
    • Three main methods based on different statistical assumptions:
      • Fixed effects models
      • Random effects models
      • Multilevel models
    • These will be discussed in the afternoon workshop
    • Meta-analysis is a method for synthesising and analysing the research literature on a particular topic
    • The essence of good science is replicable and generalisable results.
    • Increasingly sophisticated
    • Continuously evolving
    • For more information about the meta-analysis training courses that we offer, please see http://education.ox.ac.uk/research/resgroup/self/training.php