@EducEndowFoundn
OECD
Governing better through evidence-
informed policy making
Guidelines and standards for evidence
Stephen Fraser, EEF, 27 June 2017
@EducEndowFoundn
Who we are …
• The Education Endowment Foundation is an independent
grant-making charity dedicated to breaking the link between
family income and educational achievement.
• The EEF was founded in 2011 by lead charity the Sutton Trust,
in partnership with Impetus, with a £125m founding grant
from the UK Department for Education.
• In 2013, the EEF and the Sutton Trust joined the What Works
Network, as the government-designated What Works Centre
for Improving Education Outcomes for School-aged Children.
@EducEndowFoundn
What we do …
Evidence
Generation
Evidence
Mobilisation
Evidence
synthesis
• We support teachers, school leaders and
policy makers to use evidence to inform
their decision-making.
• We believe evidence can make the biggest
difference in the hands of practitioners.
• We believe that the evidence is best applied
in concert with professional judgement.
@EducEndowFoundn
Synthesis
Balancing:
• Accuracy
• Accessibility
• Applicability
• Acceptability
• Appropriateness
• Actionability
@EducEndowFoundn
The Teaching and Learning Toolkit – 2nd layer
• For each strand
• What is it?
• How effective is it?
• How secure is the evidence?
• What are the costs?
• What should I consider?
• Technical appendix
@EducEndowFoundn
Communicating confidence - toolkit
Rating Description
5
Very Extensive: Consistent high quality evidence from at least five robust and recent
meta-analyses where the majority of the included studies have good ecological validity
and where the outcome measures include curriculum measures or standardised tests in
school subject areas.
4
Extensive: Three or more meta-analyses from well-controlled experiments mainly
undertaken in schools using pupil attainment data with some exploration of causes of
any identified heterogeneity.
3
Moderate: Two or more rigorous meta-analyses of experimental studies of school age
students with cognitive or curriculum outcome measures.
2
Limited: At least one meta-analysis or systematic review with quantitative evidence of
impact on attainment or cognitive or curriculum outcome measures.
1
Very limited: Quantitative evidence of impact from single studies, but with effect size
data reported or calculable. No systematic reviews with quantitative data or meta-
analyses located.
@EducEndowFoundn
What we do …
Evidence
Generation
Evidence
Mobilisation
Evidence
synthesis
• We support teachers, school leaders and
policy makers to use evidence to inform
their decision-making.
• We believe evidence can make the biggest
difference in the hands of practitioners.
• We believe that the evidence is best applied
in concert with professional judgement.
@EducEndowFoundn
Communicating confidence - trials
Rating 1. Design 2. Power (MDES) 3. Attrition 4. Balance 5. Threats to validity
5
Fair and clear experimental design
(RCT)
< 0.2 < 10%
Well-balanced on
observables
No threats to validity
4
Fair and clear experimental design
(RCT, RDD)
< 0.3 < 20%
3
Well-matched comparison (quasi-
experiment)
< 0.4 < 30%
2
Matched comparison (quasi-
experiment)
< 0.5 < 40%
1
Comparison group with poor or no
matching
< 0.6 < 50%
0 No comparator > 0.6 > 50%
Imbalanced on
observables
Significant threats
@EducEndowFoundn
What we do …
Evidence
Generation
Evidence
Mobilisation
Evidence
synthesis
• We support teachers, school leaders and
policy makers to use evidence to inform
their decision-making.
• We believe evidence can make the biggest
difference in the hands of practitioners.
• We believe that the evidence is best applied
in concert with professional judgement.
@EducEndowFoundn
Affordable
Pick two
The scale-up challenge
@EducEndowFoundn
Guidance Reports
Combining the research evidence and our own learnings from trials, Guidance
Reports make clear, practical and evidence-based recommendations to help
teachers translate the evidence into practice.
@EducEndowFoundn
Stephen.Fraser@eefoundation.org.uk

Stephan Fraser - Evidence informed policy making - 27 June 2017

  • 1.
    @EducEndowFoundn OECD Governing better throughevidence- informed policy making Guidelines and standards for evidence Stephen Fraser, EEF, 27 June 2017
  • 2.
    @EducEndowFoundn Who we are… • The Education Endowment Foundation is an independent grant-making charity dedicated to breaking the link between family income and educational achievement. • The EEF was founded in 2011 by lead charity the Sutton Trust, in partnership with Impetus, with a £125m founding grant from the UK Department for Education. • In 2013, the EEF and the Sutton Trust joined the What Works Network, as the government-designated What Works Centre for Improving Education Outcomes for School-aged Children.
  • 3.
    @EducEndowFoundn What we do… Evidence Generation Evidence Mobilisation Evidence synthesis • We support teachers, school leaders and policy makers to use evidence to inform their decision-making. • We believe evidence can make the biggest difference in the hands of practitioners. • We believe that the evidence is best applied in concert with professional judgement.
  • 4.
    @EducEndowFoundn Synthesis Balancing: • Accuracy • Accessibility •Applicability • Acceptability • Appropriateness • Actionability
  • 5.
    @EducEndowFoundn The Teaching andLearning Toolkit – 2nd layer • For each strand • What is it? • How effective is it? • How secure is the evidence? • What are the costs? • What should I consider? • Technical appendix
  • 6.
    @EducEndowFoundn Communicating confidence -toolkit Rating Description 5 Very Extensive: Consistent high quality evidence from at least five robust and recent meta-analyses where the majority of the included studies have good ecological validity and where the outcome measures include curriculum measures or standardised tests in school subject areas. 4 Extensive: Three or more meta-analyses from well-controlled experiments mainly undertaken in schools using pupil attainment data with some exploration of causes of any identified heterogeneity. 3 Moderate: Two or more rigorous meta-analyses of experimental studies of school age students with cognitive or curriculum outcome measures. 2 Limited: At least one meta-analysis or systematic review with quantitative evidence of impact on attainment or cognitive or curriculum outcome measures. 1 Very limited: Quantitative evidence of impact from single studies, but with effect size data reported or calculable. No systematic reviews with quantitative data or meta- analyses located.
  • 7.
    @EducEndowFoundn What we do… Evidence Generation Evidence Mobilisation Evidence synthesis • We support teachers, school leaders and policy makers to use evidence to inform their decision-making. • We believe evidence can make the biggest difference in the hands of practitioners. • We believe that the evidence is best applied in concert with professional judgement.
  • 8.
    @EducEndowFoundn Communicating confidence -trials Rating 1. Design 2. Power (MDES) 3. Attrition 4. Balance 5. Threats to validity 5 Fair and clear experimental design (RCT) < 0.2 < 10% Well-balanced on observables No threats to validity 4 Fair and clear experimental design (RCT, RDD) < 0.3 < 20% 3 Well-matched comparison (quasi- experiment) < 0.4 < 30% 2 Matched comparison (quasi- experiment) < 0.5 < 40% 1 Comparison group with poor or no matching < 0.6 < 50% 0 No comparator > 0.6 > 50% Imbalanced on observables Significant threats
  • 9.
    @EducEndowFoundn What we do… Evidence Generation Evidence Mobilisation Evidence synthesis • We support teachers, school leaders and policy makers to use evidence to inform their decision-making. • We believe evidence can make the biggest difference in the hands of practitioners. • We believe that the evidence is best applied in concert with professional judgement.
  • 10.
  • 11.
    @EducEndowFoundn Guidance Reports Combining theresearch evidence and our own learnings from trials, Guidance Reports make clear, practical and evidence-based recommendations to help teachers translate the evidence into practice.
  • 12.