Presentation to school leaders on systematic review

1. Characteristics of systematic review vs literature review
In a traditional Literature Review conducted by experts:
“..there may be a lack of clarity about the:
• perspective and conceptual framework
• inclusion criteria for evidence;
• nature of the search for evidence;
• sort of evidence that is thus being considered;
• quality and relevance appraisal of that evidence;
• method of synthesis of evidence;
• use of evidence other than research;
• basis for their expertise” (Gough et al,2012)
“..being systematic
does not mean being all
embracing and if
exclusion/exclusion
criteria are carefully
selected then the
search can be kept
manageable. What is
important is to retain a
sense of the search, of
the decisions made on
the way and the source
of ideas.” (Badger,
2000)
Systematic Review
•developed as a solution to these problems
•criteria must be set for including or excluding studies
•search strategy has to be made explicit
•studies included need to be coded and analysed
•Reviewers decide methods for coding & analysis
•should include meta-analysis, where possible
•all these protocols are published

2. In what ways can systematic reviews vary?
The spectrum of approaches to conducting a systematic review (Pearson, p. 511)
Evidence Based Medicine
Aggregative (a deductive method)
‘add up’ findings from similar studies
answer tightly specified questions
use quantitative methods
test theory using empirical observations
(Gough,2012)
Interpretative approach
Configurative (inductive method)
more open questions
use qualitative data
more iterative methods
interpret specific examples of things
address Qns about experiences
generate and explore theory
(Gough,2012)
Many reviews include some aggregation and some configuration
Question needs a clear meaning
‘Do students in classes where there is a classroom teaching assistant get higher or
lower scores on test scores?’ aggregative
‘How can we conceptualise the way that the presence of classroom assistants changes
relationships between students and teachers and between teachers in class?’
configurative

3. Possible benefits to our school of Systematic Reviews?
Is it safe, as school leaders, to rely on a lone expert voice?
Systematic reviews help avoid complacency so we do more good than harm
“If advice as apparently innocuous and ‘theoretically sound’ as recommending a
baby’s sleeping position can be lethal, there is clearly no room for complacency
among professionals about their potential for harming those whom they purport to
help.”(Chalmers,2005)
Should our teenagers at risk of offending visit a Prison ?
Systematic reviews evaluate the impact of well intentioned interventions
“..would [a reader] really want [their son] exposed to a ‘Scared Straight’ programme
based on the conclusions of many poorly controlled before and after observational
studies suggesting that it would be helpful, rather than the systematic review of the
controlled trials suggesting that the programme increases subsequent involvement in
crime?” (Chalmers, 2005 citing Petrosino et al, 2003).
Can we trust that findings of educational research identify what will work for us?
Systematic reviews use scientific methods to discern causal relationships
“Much [educational research] has used experimental or quasi-experimental method, but
the results do not suggest (to say the least) that simple causal relations can be found”
(Hammersly,2005)
Campbell , cited by Pearson (2007), advocated experimental approach because of its
strength in discerning causal relationships between phenomena .

4. References
Slide 1
Gough D, Oliver S, Thomas J (2012) Introducing Systematic Reviews, p 6-7
Badger D, Nursten J, Williams P, Woodward M (2000) Should all Literature reviews be
systematic? Evaluation and Research in Education, Vol. 14, No 3&4, p 227
Campbell Collaboration website: examples of systematic reviews and 27/09/2014
http://www.campbellcollaboration.org/what_is_a_systematic_review/index.php
Slide 2
Pearson, M. (2007) “Systematic reviews in social policy: to go forward, do we first need
to look back?” . In Evidence and Policy 3 (4) p 511
Gough D, Oliver S, Thomas J (2012) Introducing Systematic Reviews, p 8-9
Slide 3
Chalmers, I. (2005) 'If evidence-informed policy works in practice, does it matter if it
doesn’t work in theory?', in Evidence and Policy 1 (2) pp227-242.
Chalmers, I. (2005) 'If evidence-informed policy works in practice, does it matter if it
doesn’t work in theory?', in Evidence and Policy 1 (2) p 237
Petrosino et al (May 2013) Scared Straight and Other Juvenile Awareness Programs for
Preventing Juvenile Delinquency: A Systematic Review p 7
Hammersley M (2005) “Is the evidence-based practice movement doing more good than
harm? Reflections on Iain Chalmers' case for research-based policy making and
practice” Evidence and Policy 1 (1) p 90
Pearson, M. (2007) “Systematic reviews in social policy: to go forward, do we first need
to look back?” . In Evidence and Policy 3 (4) p 507