The Importance of Systematic Reviews

The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org
Howard White
CEO, Campbell Collaboration
@washeval @c2update @HowardNWhite
The importance of systematic reviews

The Campbell Collaboration www.campbellcollaboration.org

The seven piece study
But these are observational data,
which don’t control for selection
bias (people who eat more than five
portions a day are wealthy,
educated, health fanatics)
The five piece study
This is a systematic review, using data from
16 high-quality studies (observational data
but analysis controls for confounders)

WHAT IS GOING ON
HERE?

Errors in hypothesis testing
H0 correct H0 false
Don’t reject H0 No error Type II error
‘false negative’
Reject H0 Type I error
‘false positive’
No error

Hypothesis testing: if H0 true
H0: ß = 0
Don’t reject H0
5% chance
that when
H0 true get a
sample
leads you to
reject H0

H0 true H0 false
= 5%
No error

Hypothesis testing HA correct
0
Incorrectly reject
HA approx 40%
of the time
Power = 1 – type II error
HA

H0 true H0 false
Maybe 20% but often
40-60%
= 5%
No error

The horrifying truth about hypothesis
testing
• If the ‘null hypothesis’ is correct (null = no programme
impact) then we will correctly agree with the null 95% of
the time (we are wrong 5% of the time)
• But if the null hypothesis is wrong (the programme works)
then we probably incorrectly conclude the programme
doesn’t work 40-60% of the time!!!

Implications
• An under-powered RCT is no better than tossing a coin at
determining if a successful programme is working so
• Power, power, power
• We also need replicate ‘unsuccessful’ programmes
• And we really REALLY need to do systematic reviews with
meta-analysis….

Pooling evidence
1 = no effect
Intervention worksIntervention is harmful
So pooling data
allows us to
overcome the
high risk of
Type II error
And goal scoring
can be very
misleading
(DON’T do it)

A real life example
30-50% reduction
in mortality
Corticosteriod for
women about to
deliver
prematurely

So let’s look at the cancer graph again …

Eggs

So not much gives you cancer…

Source: Li-Xuan Sang et al. Consumption of coffee associated with reduced risk of liver
cancer: a meta-analysis BMC Gastroenterol. 2013; 13: 34 doi: 10.1186/1471-230X-13-34
COFFEE AND LIVER CANCERBut coffee
is good for
you

And this really matters… growing number of studies
show most things don’t work.
• Education: 90 interventions evaluated in RCTs by IES - 90%
had weak or no positive effects.
• Employment/training: Department of Labor-commissioned
RCTs 75% weak or no positive effects
• Business: Over 13,000 RCTs of new products/strategies
conducted by Google and Microsoft, 80- 90% no significant
effects.
But are these sufficiently powered???
Need to combine the evidence

There’s more to a systematic review than meta-analysis
- Systematic search
- Systematic screening
- Systematic coding
- Systematic synthesis
- Systematic presentation of results
Not being systematic introduces bias – as we shall see

What the evidence synthesis process should look like
Source: Julia Littell: Campbell Systematic Reviews:
Evidence for Implementation and Impact, GIC Dublin 2015

What the evidence synthesis process actually looks like
Over-emphasis on
significant findings
Studies with significant findings are 2-3 times
more likely to be published
Selective presentation of results

An example: the treatment of results from a single study of parent
training (PT) versus multi-systemic training (MST) (a branded
programme)
RCT assigning 43 abusive or neglectful families to either:
• Parent training: group sessions discussing parenting
techniques
• Multi-systemic therapy: individual family treatment tackling
multiple issues, e.g expectation re. child behaviour, child
management, emotional support, parental behaviour
change
Study looked at 30 outcomes on individual and child
functioning, stress etc.

Outcome reporting and confirmation bias in action
9 out of 14
reviews report
just one outcome
from the paper,
favouring MST

And how reviewers summarized the Brunk et al.
paper…
No difference

What the systematic review says
Out of home placement: no difference
Delinquency: no difference
Family cohesion: no
difference

Systematic reviews rebalance the evidence
pyramid
Narrative
reviews
More than 100 narrative
reviews most stating
MST is more effect than
alternatives
MST is not
consistently
better or worse
than
alternatives

So drop narrative reviews in favour of systematic
reviews to rebalance the evidence pyramid
More than 100 narrative
reviews most stating
MST is more effect than
alternatives
MST is not
consistently
better or worse
than
alternatives

And in some areas, SRs have already made a
difference
Crime and justice
1970s “Nothing works”
Analysis of 231 studies:
“With few exceptions, the rehabilitative efforts that have
been reported so far have no appreciable effect on
recidivism”
Lipton, Martinson and Wilks ‘The Effectiveness of
Correctional Treatment, 1975
Abandonment of rehabilitation in US
and other countries (in prison at least
get an incarceration effect)
But ….

The analysis was goal scoring:
and there is no mixed evidence, only
poorly synthesized evidence. Meta-
analysis gave clear results
though in fact 48% of studies found significant
positive effects
showed ‘mixed evidence’ for all categories of intervention

Rehabilitation works
Review of 9 meta-analyses found ALL had positive average treatment effect
Source: Lipsey and
Wilson (1993)
And prison no better than non- custodial
sentences… or possibly worse
RCTs & 1 natural experiment
PSM
Source: Villletaz et al., 2013

Which means that…
• Prison is at best no better and possibly worse than non-custodial
sentences
• Each additional year of prison increases recidivism by 3-4 per cent
Source: Petrosino et al. Scared Straight and Other Juvenile Awareness
Programs for Preventing Juvenile Delinquency: A Systematic Review
Campbell Systematic Reviews 2013:5
‘Shock
approaches’
such as boot
camps and
scared
straight are
unsuccessful
and even
harmful

And this evidence is being used
“We must use sound,
research-based,
rehabilitation programmes
for offenders so they do
not re-offend.”

Similar story in policing
1997 Sherman et al. Preventing Crime: What works, what doesn’t,
reviewing over 500 crime prevention programmes
UK £250 million ‘Crime reduction programme’
Source: Braga et al. Hot spots policing effects on crime.
Campbell Systematic Reviews 2012:8
For example
hotspot
policing

Reviews can be used to answer both first
generation questions (does it work) and second
generation (design) questions.
Examples of looking at design questions

First generation: payment for environmental services
Effects of PES on forest cover change rate due to deforestation
ra -rc
−0.010 −0.005 0.000 0.005 0.010
Costa Rica PSA
Costa Rica PSA
Costa Rica PSA
Mexico PSAH
Mexico MBCF
1998−2005
1997−2000
2000−2005
2004−2006
2000−2009
Arriagada et al. 2011
Pfaff et al. 2008
Robalino et al. 2008
Alix−Garcia et al. 2012
Honey−Roses et al. 2011
Random effects mean
Predictive interval
I2
= 67.9%, t2
= 0 (0), t = 0
Tiny effect
0.3 % reduction in
deforestation
i.e. after 10 years 97% of
land for which payments
received would still have
been forested in absence
of the programme

What works by type of programme for teenage pregnancy
Source: Scher et al. Interventions Intended to Reduce
Pregnancy-Related Outcomes Among Adolescents. Campbell
Review 2006:12

Impact of welfare for work schemes by administration
• Source: Smedslund et al Work Programs for Welfare recipients. Campbell Review 2006:9
0.00 0.20 0.40 0.60 0.80 1.00 1.20
Employment
Earnings
Welfare payments
Welfare proportion
Bush Clinton Reagan 2nd Reagan 1st Carter Ford Johnson

Second generation: conditional cash transfers (CCTs)
• Mexico: Progressa
launched 1996
(renamed
Oportunidades)
• Brazil: 12 million
families by 2010Cash payment on
conditions:
• Education with 80%
attendance and
maintaining certain grade
• Health: Ante-natal care,
child immunization
Targeted both geographically
and by means test
Design questions:
• Do conditions
matter?
• Timing, nature and
size of payment
• Who to give it to

Meta-analysis allows us to get at design features,
for example …
CCTs have a larger effect on enrolment rates
• Secondary than primary
• The larger the transfer
• The less frequent the transfer
• If conditions include achievement not just attendance
And…

Conditionality works
Children 60% more likely to be in school with conditionality
which is monitored and enforced compared to no
conditions
But we need a lot of primary studies
to exploit heterogeneity

Water, supply and sanitation interventions
4 main types of intervention:
– Water supply improvement: source or point-of-use
– Water quality: water treatment/protection at source or
point-of-use (households)
– Sanitation: provision of facilities (improved latrines,
sewer connection)
– Hygiene: soap, hygiene education
Usual outcome variable is child diarrhoea

The Campbell Collaboration www.campbellcollaboration.orgwww.3ieimpact.orgAuthor name
44
65 studies (71 interventions) included in meta-analysis
11 studies from searches met the inclusion criteria
54 studies from previous reviews met the inclusion
criteria
Review against inclusion criteria
Abstract review of 278 papers, with full text copies
obtained for 68 of these
Full text copies obtained of all 110 studies
Search strategy
Title review of 19,233 papers identified from
searches of databases, organisations and
communication with researchers
110 studies identified from the bibliographies
previous reviews

45
Effectiveness results pooled (outcome = child diarrhoea)
NOTE: Weights are from random effects analysis
Water supply interventions
Subtotal
Water quality interventions
Subtotal
Sanitation interventions
Subtotal
Hygiene interventions
Subtotal
Multiple interventions
Subtotal
ID
Study
0.98 (0.89, 1.06)
0.58 (0.50, 0.67)
0.63 (0.43, 0.93)
0.69 (0.61, 0.77)
0.62 (0.46, 0.83)
ES (95% CI)
0.98 (0.89, 1.06)
0.58 (0.50, 0.67)
0.63 (0.43, 0.93)
0.69 (0.61, 0.77)
0.62 (0.46, 0.83)
ES (95% CI)
Ratio favours intervention
1.1 .5 .75 1 2
BUT
Evidence
largely from
trials not actual
projects
And hints of
weak
sustainability

Sustainability 1: less impact over longer periods
NOTE: Weights are from random effects analysis
Water supply (12 months or more)
Subtotal
Water quality (under 12 months)
Subtotal
Water quality (12 months or more)
Subtotal
Sanitation (12 months or more)
Subtotal
Hygiene (under 12 months)
Subtotal
Hygiene (12 months or more)
Subtotal
Multiple (under 12 months)
Subtotal
Multiple (12 months or more)
Subtotal
ID
Study
0.82 (0.71, 0.96)
0.56 (0.47, 0.66)
0.81 (0.67, 0.97)
0.64 (0.37, 1.10)
0.72 (0.60, 0.86)
0.67 (0.49, 0.91)
0.41 (0.23, 0.74)
0.77 (0.70, 0.85)
ES (95% CI)
0.82 (0.71, 0.96)
0.56 (0.47, 0.66)
0.81 (0.67, 0.97)
0.64 (0.37, 1.10)
0.72 (0.60, 0.86)
0.67 (0.49, 0.91)
0.41 (0.23, 0.74)
0.77 (0.70, 0.85)
ES (95% CI)
Ratio favours intervention
1.1 .5 .75 1 2

Sustainability 2: low compliance after a while
• Ceramic filter provision in Cambodia; 3 years later only 31% households
were still using the filters (Brown et al, 2007)
• Pasteurisation in Kenya; 4 years later only 30% continued to pasteurise
their water (Iijima et al, 2001)
• Programme promoting POU water disinfectant in Guatemala 1 year
later; repeated use among only 5% of households from original trials
(Luby et al, 2008).
• Water filters in Bolivia; compliance 67%; assessment made 4 months
after trial ended (Clasen et al, 2006)

The Campbell Collaboration www.campbellcollaboration.orgwww.3ieimpact.orgAuthor name
Sustainability 3: lack of WTP
In Kenya, access to free chlorine increased uptake to over 60
percent, whereas coupons for even a 50 percent discount had a
minimal effect

– Point of use water treatment has
large health effect (community-
level doesn’t)
– But challenge is to ensure
sustained proper use
– Any water supply intervention not
taking into account this demand
element should be questioned
So the systematic review tells us that:

A quick word on the Campbell Collaboration
• Coordinating groups (CGs) for
– Crime and Justice
– Education
– International development
– Social welfare
• CGs manage editorial process
– Three stage process: title, protocol, review
– Any team can submit proposed title
• All published in Campbell Library, managed by Secretariat in Oslo

In summary
• Rigorous evidence matters
• High quality systematic reviews sort out what is
rigorous and what is not
• And synthesize the evidence in policy-relevant ways
• - telling us what works and why
• Make them and use them!!

The Campbell Collaboration www.campbellcollaboration.orgThe Campbell Collaboration www.campbellcollaboration.org
Thank you
Visit our website
www.campbellcollaboration.org
Follow us on Twitter @C2update & Facebook

The Importance of Systematic Reviews

Recommended

Recommended

More Related Content

Similar to The Importance of Systematic Reviews

Similar to The Importance of Systematic Reviews (17)

More from Washington Evaluators

More from Washington Evaluators (20)

Recently uploaded

Recently uploaded (20)

The Importance of Systematic Reviews