NCRM Research Methods Festival University of Oxford Dept of Education, University of Oxford
Traditionally, social science researchers collect and analyse their own data (referred to as primary data). Secondary data analysis is based on data collected by someone else (or, perhaps, re-analysis of your own published data). There are at least four logical perspectives to this issue:
1. Meta-analysis -- systematic, quantitative review of published research in a particular field, the focus of this presentation.
2. Systematic review -- systematic, qualitative review of published research in a particular field
3. Secondary Data Analyses -- using large (typically public) databases
4. Re-analyses of published studies -- often in ways critical of the original study.
Systematic synthesis of various studies on a particular research question
Do boys or girls have higher self-concepts?
Collect all studies relevant to a topic
Find all published journal articles on the topic
An effect size (the ‘dependent variable’) is calculated for each outcome
Determine the size/direction of gender difference for each study
“ Content analysis”
code characteristics of the study; age, setting, ethnicity, self-concept domain (math, physical, social), etc.
Effect sizes with similar features are grouped together and compared; tests moderator variables
Do gender differences vary with age, setting, ethnicity, self-concept, domain, etc.
Coding: the process of extracting the information from the literature included in the meta-analysis. Involves noting the characteristics of the studies in relation to a priori variables of interest (qualitative)
Effect size: the numerical outcome to be analysed in a meta-analysis; a summary statistic of the data in each study included in the meta-analysis (quantitative)
Summarise effect sizes: central tendency, variability, relations to study characteristics (quantitative)
Compared to traditional literature reviews:
(1) there is a definite methodology employed in the research analysis; and
(2) the results of the included studies are quantified to a standard metric thus allowing for statistical techniques for further analysis.
Therefore less biased and more replicable
Increased power: increases the chance of detecting a true treatment effect
Improved precision: with more information than a single study, the treatment effect estimate is improved
When study-to-study variation in results (which is typical) can evaluate differences in relation to study characteristics. Can delve into research questions not explored by the individual studies
Easy to interpret summary statistics (useful if communicating findings to a non-academic audience)
The essence of good science is replicable and generalisable results.
Do we get the same answer to important research questions when we run the study again?
The primary aims of meta-analysis is to test the generalisability of results across a set of studies designed to answer the same research question.
Are the results consistent? If not, what are the differences in the studies that explain the lack of consistency?
Meta-analysis is an increasingly popular tool for summarising research findings; literature review method of choice in many disciplines
Widely-cited. If there is a good meta-analysis relevant to your study, you have to cite it
Relied upon by policymakers
Important that we understand the method, whether we conduct or consume meta-analytic research
Should be one of the topics covered in all introductory research methodology courses
There exists a critical mass of comparable studies designed to address a common research question.
Data are presented in a form that allows the meta-analyst to compute an effect size for each study.
Characteristics of each study are described in sufficient detail to allow meta-analysts to compare characteristics of different studies and to judge the quality of each study.
The number of meta-analyses is increasing at a rapid rate.
Psychology: Citations Psychology: Articles
Amato, P. R., & Keith, B. (1991). Parental divorce and the well-being of children: A meta-analysis . Psychological Bulletin, 110, 26-46. Times Cited: 471
Linn, M. C., & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis . Child Development, 56, 1479-1498. Times Cited: 570
Johnson, D. W., & et al (1981). Effects of cooperative, competitive, and individualistic goal structures on achievement: A meta-analysis . Psychological Bulletin, 89, 47-62. Times Cited: 426
Tett, R. P., Jackson, D. N., & Rothstein, M. (1991). Personality measures as predictors of job performance: A meta-analytic review . Personnel Psychology, 44, 703-742 Times Cited: 387
Hyde, J. S., & Linn, M. C. (1988). Gender differences in verbal ability: A meta-analysis . Psychological Bulletin, 104, 53-69. Times Cited: 316
Iaffaldano, M. T., & Muchinsky, P. M. (1985). Job satisfaction and job performance: A meta-analysis . Psychological Bulletin, 97, 251-273. Times Cited: 263.
De Wolff, M., & van IJzendoorn, M. H. (1997). Sensitivity and attachment: A meta-analysis on parental antecedents of infant attachment . Child Development, 68, 571-591. Times Cited: 340
Wellman, H. M., Cross, D., & Watson, J. (2001). Meta-analysis of theory-of-mind development: The truth about false belief . Child Development, 72, 655-684. Times Cited: 276
Cohen, E. G. (1994). Restructuring the classroom: Conditions for productive small groups . Review of Educational Research, 64, 1-35. Times Cited: 235
Hansen, W. B. (1992). School-based substance abuse prevention: A review of the state of the art in curriculum, 1980-1990 . Health Education Research, 7, 403-430. Times Cited: 207
Kulik, J. A., Kulik, C-L., Cohen, P. A. (1980). Effectiveness of Computer-Based College Teaching: A Meta-Analysis of Findings. Review of Educational Research, 50, 525-544. Times Cited: 198.
Sheppard, B. H., Hartwick, J., & Warshaw, P. R. (1988). The theory of reasoned action: A meta-analysis of past research with recommendations for modifications and future research . Journal of Consumer Research, 15, 325-343. Times Cited: 515
Jackson, S. E., & Schuler, R. S. (1985). A meta-analysis and conceptual critique of research on role ambiguity and role conflict in work settings . Organizational Behavior and Human Decision Processes, 36, 16-78. Times Cited: 401
Tornatzky Lg, Klein Kj. (1994). Innovation characteristics and innovation adoption-implementation - A meta-analysis of findings . IEEE Transactions On Engineering Management, 29, 28-4. Times Cited: 269.
Lowe KB, Kroeck KG, Sivasubramaniam N. (1996). Effectiveness correlates of transformational and transactional leadership: A meta-analytic review of the MLQ literature. Leadership Quarterly, 7 , 385-425. Times Cited: 203.
Churchill GA, Ford NM, Hartley SW, et al. (1985). Title: The determinants of salesperson performance - A meta-analysis . Journal Of Marketing Research, 22, 103-118. Times Cited: 189.
Jadad AR, Moore RA, Carroll D, et al. (1996). Assessing the quality of reports of randomized clinical trials: Is blinding necessary? Controlled Clinical Trials, 17, 1-12. Times Cited:2008
Boushey Cj, Beresford Saa, Omenn Gs, Et . Al. (1995). A quantitative assessment of plasma homocysteine as a risk factor for vascular-disease - Probable benefits of increasing folic-acid intakes. JAMA-journal Of The American Medical Assoc, 274, 1049-1057. Times Cited: 2,128
Alberti W, Anderson G, Bartolucci A, et al. (1995). Chemotherapy in non-small-cell lung-cancer - A metaanalysis using updated data on individual patients from 52 randomized clinical-trials. British Medical Journal, 311, 899-909. Times Cited:1,591
Block G, Patterson B, Subar A (1992). Fruit, vegetables, and cancer prevention - A review of the epidemiologic evidence. Nutrition And Cancer-an International Journal, 18, 1-29. Times Cited: 1,422
Gene Glass coined the phrase meta-analysis in classic study of the effects of psychotherapy. Because most individual studies had small sample sizes, the effects typically were not statistically significant.
Results of 375 controlled evaluations of psychotherapy and counselling were coded and integrated statistically. The findings provide convincing evidence of the efficacy of psychotherapy.
On the average, the typical therapy client is better off than 75% of untreated individuals.
Few important differences in effectiveness could be established among many quite different types of psychotherapy (e.g., behavioral and non-behavioral).
ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)
Need to have explicit inclusion and exclusion criteria
The broader the research domain, the more detailed they tend to become
Refine criteria as you interact with the literature
Components of a detailed criteria
cultural and linguistic range
Search electronic databases (e.g., ISI, Psychological Abstracts, Expanded Academic ASAP, Social Sciences Index, PsycINFO, and ERIC)
Examine the reference lists of included studies to find other relevant studies
If including unpublished data, email researchers in your discipline, take advantage of Listservs, and search Dissertation Abstracts International
Random selection of papers coded by both coders
Meet to compare code sheets
Where there is discrepancy, discuss to reach agreement
Amend code materials/definitions in code book if necessary
May need to do several rounds of piloting, each time using different papers
__ Study ID
_ _ Year of publication
__ Publication type (1-5)
__ Geographical region (1-7)
_ _ _ _ Total sample size
_ _ _ Total number of males
_ _ _ Total number of females
Code Sheet Code Book/manual ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg)
Publication type (1-5)
Thesis or doctoral dissertation
1 99 2 1 87 41 46
The effect size makes meta-analysis possible
It is the “dependent variable”
It standardizes findings across studies such that they can be directly compared
Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds-ratio), but must
be comparable across studies (generally requires standardization)
represent the magnitude and direction of the relationship of interest
be independent of sample size
Represents a standardized group contrast on an inherently continuous measure
Uses the pooled standard deviation (some situations use control group standard deviation)
Commonly called “d”
In a gender difference study, the effect size might be: In an intervention study with experimental and control groups, the effect size might be:
Means and standard deviations Correlations P-values F -statistics d t -statistics “ other” test statistics Almost all test statistics can be transformed into an standardized effect size “d” ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) Lipsey & Wilson (2001) present formulae for calculating effect sizes from different information
Each study is one line in the data base Effect size Duration Sample sizes Reliability of the instrument Variance of the effect size
There are various ways of analysing meta-analytic data
Three main methods based on different statistical assumptions:
Fixed effects models
Random effects models
These will be discussed in the afternoon workshop
Meta-analysis is a method for synthesising and analysing the research literature on a particular topic
The essence of good science is replicable and generalisable results.
For more information about the meta-analysis training courses that we offer, please see http://education.ox.ac.uk/research/resgroup/self/training.php