This lecture explains why systematic reviews were catapulted to the peak of the study designs. Its advantages and limitations are discussed. The measures to overcome the limitations are also discussed.

1. Systematic Reviews
At The Peak Of Research Designs:
What Catapulted Them There?
Nemencio A. Nicodemus Jr., MD, FPCP
Professor, UP College of Medicine

3. Catapult - verb
“Move suddenly or at great speed
as though hurled by a catapult”
- Oxford Dictionaries

4. Scientists Have Been
Thinking About How
to Integrate Studies
for a Long Time

5. James Lind, English naval surgeon
(18th Century)
“...it became requisite to
exhibit a full and
impartial view of what
had hitherto been
published on the scurvy
... by which the sources
of these mistakes may
be detected”

6. 1932: R. A. Fisher
Statistical Methods for Research Workers. London: Oliver & Boyd
“…it sometimes happens
that although few or [no
statistical tests] can be
claimed individually as
significant, yet the
aggregate gives an
impression that the
probabilities are lower
than would have been
obtained by chance.”

7. In response to calls from the 'evidence
movement' to organise knowledge into
a useable and reliable format

8. History of Systematic Reviews
• Archie Cochrane's
seminal paper
'Effectiveness and
Efficiency' (1972)
urged health
practitioners to
practice evidence
based medicine
Cochrane AL (1972) Effectiveness and efficiency: random reflections on health services.
London: Royal Society of Medicine Press.

9. GV Glass, 1975
• Coined the term ‘meta analysis’ to refer to
critical appraisal and synthesis of research
findings in a systematic manner
Glass GV, Smith ML (1979) Meta-analysis of research on the relationship of class-size and achievement.
Educational Evaluation and Policy Analysis 1: 2-16

10. Early 1980s
• A group of health service researchers in
Oxford began a programme of systematic
reviews on the effectiveness of health care
interventions.

11. The Cochrane Collaboration opened its
centre in Oxford in 1992

12. Cochrane Collaboration

13. Number of Scientific Articles Per Year

14. Number of Systematic Reviews in Cochrane
Collaboration

16. Systematic Reviews in Pubmed
(by year)
1975
6227
1985
11999
1995
48873
2005
80861
2015
83743

17. 2,083,884 systematic reviews
As of January 9, 2016

18. Systematic
Reviews
Narrative
Reviews

19. Relationship of Review Studies

20. Traditional synthesis procedures
typically ...
• Rely on a convenience sample of studies
• Use decision rules that are not transparent
– e.g., rarely state a priori what constitutes
“good” evidence
• Rely on statistical significance to gauge
what a study “says”

21. What is wrong with narrative reviews?
Different experts may perform a narrative
review on the same question and come to
different conclusions.
• Sometimes this is because they review
different sets of studies
• Even when the same studies are
reviewed, the process of integrating them
in a narrative review is subjective

22. Systematic Reviews
Use a more systematic approach to
data collection
Have more transparent and explicit
procedures,– thus have less potential
for hidden assumptions or biases to
drive results
Focus on the magnitude of effects
rather than statistical significance
Result in conclusions that are less
bound by context than conclusions
that arise from individual studies

23. Why are systematic reviews better?
• They are more objective and more replicable
• They are able to deal efficiently with large
amounts of information
• They can systematically examine variations
in treatment effectiveness or relationship
strength
• They can examine differences due to study
methods, types of participants, settings, etc.

25. In systematic reviews, can
different researchers review the same
question and come to different conclusions?

26. What makes systematic reviews
different from narrative reviews?
• Variability in conclusion can be
systematically explained:
– due to moderator variables (populations,
treatments, settings, methods, designs)
– systematic reviews help us understand how
these influence results.

27. Why are systematic reviews better?
• Studies with different strengths and
weaknesses can be used to rule out each
others’ alternative explanations.
– Similar results from studies with different
designs lead to greater confidence in
conclusions.

28. Conceptual Issues
• Attempts to combine results of studies that
vary (to some degree) in their methods,
treatments, samples, outcome measures

29. Conceptual Issues
• Parallel problems in studies of individuals
(there is no “average” person)

30. Conceptual Issues
• Given variation in primary research
– What should be included in a synthesis?
– How should we synthesize results?

31. The goal of a systematic review
• To limit bias in the identification, evaluation
and synthesis of the body of relevant
studies that address a specific research
question

32. GiGo Principle

33. Standards for Systematic Reviews
• There are good
frameworks and
standards for
systematic reviews
– Cochrane
Collaboration
Handbook
– Campbell Collaboration
policy briefs

34. Standards for Systematic Reviews
• PRISMA (Principles
of Reporting
Information on
Systematic reviews
and Meta-Analysis)

35. Not All Evidence Is Created
Equally
• Sampling: Larger samples are more likely to
estimate true population values and result in
narrower confidence intervals than small
samples.
• Research methods: There are two models
of a research method hierarchy.
– A hierarchy of strength of evidence for treatment
decisions
– The Oxford Center for Evidence-Based Medicine
hierarchy

36. The Oxford Center for Evidence-
Based Medicine Hierarchy
Level of Evidence
Randomized controlled trials 1.
a. Systematic review of RCTs
b. Individual RCTs
c. All-or-none studies
Cohort studies 2.
a. Systematic review of RCTs
b. Individual cohort studies
c. Outcomes research
Case-control studies 3.
a. Systematic reviews of case-control studies
b. Individual case-control studies
Case series / Case study 4.
a. Includes poorly designed cohort and case-
control studies
Anecdotal evidence 5.
a. Animal research
b. Bench research
c. Unpublished clinical observations
(Adapted with permission from http://www.cebm.net/index.aspx?o=1025. Oxford Center for Evidence-
Based Medicine--Levels of Evidence (March 2009). Accessed May 10, 2010.)

37. What catapulted systematic reviews
at the peak of research designs?
The rapid growth of high-quality clinical
trials
The imperative to integrate studies and
practice evidence-based medicine
The methodologic rigor in doing a
systematic review
The high standards for conducting and the
principles for reporting systematic reviews