Presentation by David Gough, Director, EPPI-Centre, UCL Institute of Education, at the event on Governing better through evidence-informed policy making, 26-27 June 2017. The event was organised by the OECD Directorate for Public Governance in cooperation with the European Commission’s Joint Research Centre (JRC), the Campbell Collaboration and the International Network for Government Science Advice (INGSA). For further information please see http://www.oecd.org/gov/evidence-informed-policy-making.htm

1. (1)
Justifiable Evidence Claims:
evidence standards to inform
decision making
David Gough
Governing better through evidence-informed
policy making OECD, Paris, 27th June 2017
Email: david.gough@ucl.ac.uk
Twitter: @ProfDavidGough
EPPI-Centre, Social Science Research Unit,
UCL Institute of Education, University College London
http://eppi.ioe.ac.uk

2. (2)
OECD and EC long term supporters
of evidence use in policy & practice
• 2005 OECD international comparisons of
use of research in educational policy
• 2007 OECD book on evidence and
education
• OECD Governance of Complex Education
Systems
• 2010-2013 EC DGEAC funding of the
Evidence in Education in Policy and
Practice in Europe (EIPPEE)

3. (3)
Evidence standard 1: Systematic
reviews
– not just any available study!
• Similar to primary research:
– Same expectations of rigor and transparency
– Same range of questions and assumptions
– Methods of review reflect methods and
assumptions of primary studies
– May include mixed methods
• A higher level of analysis:
– Data usually from pre-existing studies rather than
new primary data
– More justifiable evidence claims than from
individual studies alone

4. (4)
Evidence standard 2: Justifiable evidence
claims based on technique and relevance
Evidence claim
• Depends on research question. For e.g.: that an
intervention has a certain type and size of effect
Justification for claim: Research method
Sub-dimensions:
• Methods standards achieved
• Suitability of study design/method
• Relevance to study focus
- not only based on the technical execution of method

5. (5)
Evidence standard 3: Justifiable
evidence claims – 3 dimensions
Evidence claim:
• Depends on research question. For e.g.:
that an intervention has a certain type and
size of effect
Justification for claim: 3 Dimensions
• Review method: Standard+Suitability+Relevance
• Included studies: Standard+Suitability+Relevance
• Evidence produced: Nature + Extent

6. (6)
Dimension 1: Review method
Sub-dimensions:
a) Methods standards achieved: how well was review
undertaken methodologically?
b) Suitability of the review method: Is this an appropriate
method for answering this review question?
c) Relevance to review focus: Is the way that the review is
conducted appropriate for the specific focus of the
review question?
Example of components of appraisal instruments to assess Dimension 1:
AMSTAR: (a) Methodological quality
ROBIS: (a) Concerns with the review process; Risk of bias; (c) Relevance

7. (7)
Dimension 2: Included studies
Sub-dimensions:
a) Methods standards achieved: how well were the
‘included studies’ undertaken methodologically?
b) Suitability of the research methods of the ‘included
studies’: Were these appropriate methods for
answering this review question?
c) Relevance to review focus: Is the way that the
‘included studies’ were conducted appropriate for the
specific focus of the review question?
Example of components of appraisal instruments to assess Dimension 2:
GRADE: (a) Study design; (a&b) Study limitations; (c) Indirectness
CERQUAL: (a&b) Methodological limitations; (c) Relevance

8. (8)
Dimension 3: Evidence produced by
the review
Sub-dimensions:
a) Nature of the included studies: qualities of the
evidence when considered together (such as
heterogeneity, statistical independence).
b) Extent of evidence from the included studies:
further to issues of quality maybe issues of the
extent of evidence available to address the
review question.
Example of components of appraisal instruments to assess Dimension 3:
GRADE: (a&b) Inconsistency; Imprecision; Publication bias; Magnitude of
treatment effect; Impact of confounders; Dose /response
CERQUAL: (a) Coherence; (a&b) (b) Adequacy of data
Example of components of appraisal instruments to assess Dimensions 2 & 3:
Both GRADE and CERQUAL: Quality of evidence for each specific finding

9. (9)
Justifiable evidence claims to
inform decisions
Evidence claim
Justification for claim
• Review Method
• Included Studies
• Evidence produced
Consideration of evidence and decisions
• Interpretation & integration with other information
Application of decisions
Outcomes of decisions

10. (10)
Systematic Review Methods +
The Science of Using Science
https://www.sagepub.com/en-us/nam/an-introduction-to-
systematic-reviews/book245742%20
http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3504