SlideShare a Scribd company logo
(1)
Justifiable Evidence Claims:
evidence standards to inform
decision making
David Gough
Governing better through evidence-informed
policy making OECD, Paris, 27th June 2017
Email: david.gough@ucl.ac.uk
Twitter: @ProfDavidGough
EPPI-Centre, Social Science Research Unit,
UCL Institute of Education, University College London
http://eppi.ioe.ac.uk
(2)
OECD and EC long term supporters
of evidence use in policy & practice
• 2005 OECD international comparisons of
use of research in educational policy
• 2007 OECD book on evidence and
education
• OECD Governance of Complex Education
Systems
• 2010-2013 EC DGEAC funding of the
Evidence in Education in Policy and
Practice in Europe (EIPPEE)
(3)
Evidence standard 1: Systematic
reviews
– not just any available study!
• Similar to primary research:
– Same expectations of rigor and transparency
– Same range of questions and assumptions
– Methods of review reflect methods and
assumptions of primary studies
– May include mixed methods
• A higher level of analysis:
– Data usually from pre-existing studies rather than
new primary data
– More justifiable evidence claims than from
individual studies alone
(4)
Evidence standard 2: Justifiable evidence
claims based on technique and relevance
Evidence claim
• Depends on research question. For e.g.: that an
intervention has a certain type and size of effect
Justification for claim: Research method
Sub-dimensions:
• Methods standards achieved
• Suitability of study design/method
• Relevance to study focus
- not only based on the technical execution of method
(5)
Evidence standard 3: Justifiable
evidence claims – 3 dimensions
Evidence claim:
• Depends on research question. For e.g.:
that an intervention has a certain type and
size of effect
Justification for claim: 3 Dimensions
• Review method: Standard+Suitability+Relevance
• Included studies: Standard+Suitability+Relevance
• Evidence produced: Nature + Extent
(6)
Dimension 1: Review method
Sub-dimensions:
a) Methods standards achieved: how well was review
undertaken methodologically?
b) Suitability of the review method: Is this an appropriate
method for answering this review question?
c) Relevance to review focus: Is the way that the review is
conducted appropriate for the specific focus of the
review question?
Example of components of appraisal instruments to assess Dimension 1:
AMSTAR: (a) Methodological quality
ROBIS: (a) Concerns with the review process; Risk of bias; (c) Relevance
(7)
Dimension 2: Included studies
Sub-dimensions:
a) Methods standards achieved: how well were the
‘included studies’ undertaken methodologically?
b) Suitability of the research methods of the ‘included
studies’: Were these appropriate methods for
answering this review question?
c) Relevance to review focus: Is the way that the
‘included studies’ were conducted appropriate for the
specific focus of the review question?
Example of components of appraisal instruments to assess Dimension 2:
GRADE: (a) Study design; (a&b) Study limitations; (c) Indirectness
CERQUAL: (a&b) Methodological limitations; (c) Relevance
(8)
Dimension 3: Evidence produced by
the review
Sub-dimensions:
a) Nature of the included studies: qualities of the
evidence when considered together (such as
heterogeneity, statistical independence).
b) Extent of evidence from the included studies:
further to issues of quality maybe issues of the
extent of evidence available to address the
review question.
Example of components of appraisal instruments to assess Dimension 3:
GRADE: (a&b) Inconsistency; Imprecision; Publication bias; Magnitude of
treatment effect; Impact of confounders; Dose /response
CERQUAL: (a) Coherence; (a&b) (b) Adequacy of data
Example of components of appraisal instruments to assess Dimensions 2 & 3:
Both GRADE and CERQUAL: Quality of evidence for each specific finding
(9)
Justifiable evidence claims to
inform decisions
Evidence claim
Justification for claim
• Review Method
• Included Studies
• Evidence produced
Consideration of evidence and decisions
• Interpretation & integration with other information
Application of decisions
Outcomes of decisions
(10)
Systematic Review Methods +
The Science of Using Science
https://www.sagepub.com/en-us/nam/an-introduction-to-
systematic-reviews/book245742%20
http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3504

More Related Content

What's hot

Introduction to EVBRES
Introduction to EVBRESIntroduction to EVBRES
Introduction to EVBRES
Caroline Blaine
 
Klara Brunnhuber: Research Waste when Justifying New Research
Klara Brunnhuber: Research Waste when Justifying New ResearchKlara Brunnhuber: Research Waste when Justifying New Research
Klara Brunnhuber: Research Waste when Justifying New Research
Caroline Blaine
 
Critical Appraisal
Critical Appraisal Critical Appraisal
Critical Appraisal
Balamurugan Tangiisuran
 
Evaluation-based advice with uncertain evidence
Evaluation-based advice with uncertain evidenceEvaluation-based advice with uncertain evidence
Evaluation-based advice with uncertain evidence
ACIL Allen Consulting
 
Assessing quality and bias in studies
Assessing quality and bias in studiesAssessing quality and bias in studies
Assessing quality and bias in studies
ILRI
 
Topic what specific health conditions increase the risk o
Topic         what specific health conditions increase the risk oTopic         what specific health conditions increase the risk o
Topic what specific health conditions increase the risk o
raju957290
 
How to choose publishable research topic
How to choose publishable research topicHow to choose publishable research topic
How to choose publishable research topic
Hasanain Ghazi
 
The Case for Competition: Learning About Evidence-Based Management Through Ca...
The Case for Competition: Learning About Evidence-Based Management Through Ca...The Case for Competition: Learning About Evidence-Based Management Through Ca...
The Case for Competition: Learning About Evidence-Based Management Through Ca...
Center for Evidence-Based Management
 
Operations Research: Methods, Challenges, Emerging Lessons, and Opportunities...
Operations Research: Methods, Challenges, Emerging Lessons, and Opportunities...Operations Research: Methods, Challenges, Emerging Lessons, and Opportunities...
Operations Research: Methods, Challenges, Emerging Lessons, and Opportunities...
CORE Group
 
Knowledge synthesis for policy and practice
Knowledge synthesis for policy and practiceKnowledge synthesis for policy and practice
Knowledge synthesis for policy and practice
Sax Institute
 
V3 canons park leading november 2016
V3 canons park leading november 2016V3 canons park leading november 2016
V3 canons park leading november 2016
Gary Jones
 
Causes of mode effect on survey management 2011
Causes of mode effect on survey management 2011Causes of mode effect on survey management 2011
Causes of mode effect on survey management 2011
natcenslides
 
Presentacion 4.4.5-4.7.2-rafael-castro
Presentacion 4.4.5-4.7.2-rafael-castroPresentacion 4.4.5-4.7.2-rafael-castro
Presentacion 4.4.5-4.7.2-rafael-castro
Hugo Díaz
 
DORA and the reinvention of research assessment
DORA and the reinvention of research assessmentDORA and the reinvention of research assessment
DORA and the reinvention of research assessment
Mark Patterson
 
systematic review : why & How
systematic review : why & Howsystematic review : why & How
systematic review : why & How
Hesham Al-Inany
 
Mona Nasser: Research waste when designing new research (role of funders)
Mona Nasser: Research waste when designing new research (role of funders)Mona Nasser: Research waste when designing new research (role of funders)
Mona Nasser: Research waste when designing new research (role of funders)
Caroline Blaine
 
EBR external presentation
EBR external presentationEBR external presentation
EBR external presentation
Caroline Blaine
 
Setting research objectives
Setting research objectivesSetting research objectives
Setting research objectives
Amna Khairy
 
Hans Lund: Background and Introduction to the COST Action EVBRES
Hans Lund: Background and Introduction to the COST Action EVBRESHans Lund: Background and Introduction to the COST Action EVBRES
Hans Lund: Background and Introduction to the COST Action EVBRES
Caroline Blaine
 
SURE: Helping get the most out of longitudinal data
SURE: Helping get the most out of longitudinal dataSURE: Helping get the most out of longitudinal data
SURE: Helping get the most out of longitudinal data
Sax Institute
 

What's hot (20)

Introduction to EVBRES
Introduction to EVBRESIntroduction to EVBRES
Introduction to EVBRES
 
Klara Brunnhuber: Research Waste when Justifying New Research
Klara Brunnhuber: Research Waste when Justifying New ResearchKlara Brunnhuber: Research Waste when Justifying New Research
Klara Brunnhuber: Research Waste when Justifying New Research
 
Critical Appraisal
Critical Appraisal Critical Appraisal
Critical Appraisal
 
Evaluation-based advice with uncertain evidence
Evaluation-based advice with uncertain evidenceEvaluation-based advice with uncertain evidence
Evaluation-based advice with uncertain evidence
 
Assessing quality and bias in studies
Assessing quality and bias in studiesAssessing quality and bias in studies
Assessing quality and bias in studies
 
Topic what specific health conditions increase the risk o
Topic         what specific health conditions increase the risk oTopic         what specific health conditions increase the risk o
Topic what specific health conditions increase the risk o
 
How to choose publishable research topic
How to choose publishable research topicHow to choose publishable research topic
How to choose publishable research topic
 
The Case for Competition: Learning About Evidence-Based Management Through Ca...
The Case for Competition: Learning About Evidence-Based Management Through Ca...The Case for Competition: Learning About Evidence-Based Management Through Ca...
The Case for Competition: Learning About Evidence-Based Management Through Ca...
 
Operations Research: Methods, Challenges, Emerging Lessons, and Opportunities...
Operations Research: Methods, Challenges, Emerging Lessons, and Opportunities...Operations Research: Methods, Challenges, Emerging Lessons, and Opportunities...
Operations Research: Methods, Challenges, Emerging Lessons, and Opportunities...
 
Knowledge synthesis for policy and practice
Knowledge synthesis for policy and practiceKnowledge synthesis for policy and practice
Knowledge synthesis for policy and practice
 
V3 canons park leading november 2016
V3 canons park leading november 2016V3 canons park leading november 2016
V3 canons park leading november 2016
 
Causes of mode effect on survey management 2011
Causes of mode effect on survey management 2011Causes of mode effect on survey management 2011
Causes of mode effect on survey management 2011
 
Presentacion 4.4.5-4.7.2-rafael-castro
Presentacion 4.4.5-4.7.2-rafael-castroPresentacion 4.4.5-4.7.2-rafael-castro
Presentacion 4.4.5-4.7.2-rafael-castro
 
DORA and the reinvention of research assessment
DORA and the reinvention of research assessmentDORA and the reinvention of research assessment
DORA and the reinvention of research assessment
 
systematic review : why & How
systematic review : why & Howsystematic review : why & How
systematic review : why & How
 
Mona Nasser: Research waste when designing new research (role of funders)
Mona Nasser: Research waste when designing new research (role of funders)Mona Nasser: Research waste when designing new research (role of funders)
Mona Nasser: Research waste when designing new research (role of funders)
 
EBR external presentation
EBR external presentationEBR external presentation
EBR external presentation
 
Setting research objectives
Setting research objectivesSetting research objectives
Setting research objectives
 
Hans Lund: Background and Introduction to the COST Action EVBRES
Hans Lund: Background and Introduction to the COST Action EVBRESHans Lund: Background and Introduction to the COST Action EVBRES
Hans Lund: Background and Introduction to the COST Action EVBRES
 
SURE: Helping get the most out of longitudinal data
SURE: Helping get the most out of longitudinal dataSURE: Helping get the most out of longitudinal data
SURE: Helping get the most out of longitudinal data
 

Similar to David Gough - Evidence informed policy making - 27 June 2017

Marketing Group assignment 1 Article review guideline.docx
Marketing Group assignment 1 Article review guideline.docxMarketing Group assignment 1 Article review guideline.docx
Marketing Group assignment 1 Article review guideline.docx
Derashmelaku
 
Multivariate Analysis.ppt
Multivariate Analysis.pptMultivariate Analysis.ppt
Multivariate Analysis.ppt
HarunorRashid74
 
ADOVH Research Methodology.pdf
ADOVH Research Methodology.pdfADOVH Research Methodology.pdf
ADOVH Research Methodology.pdf
ADOVH-University of South Africa
 
Business research methods
Business  research methodsBusiness  research methods
Business research methods
Hanumant singh
 
Over view on systematic review
Over view on systematic review Over view on systematic review
Over view on systematic review
David Peter
 
Bolouri qualitative method
Bolouri qualitative methodBolouri qualitative method
Bolouri qualitative method
Allame Tabatabaei
 
KNOWLEDGE SYNTHESIS IN NURSING REAL.pptx
KNOWLEDGE SYNTHESIS IN NURSING REAL.pptxKNOWLEDGE SYNTHESIS IN NURSING REAL.pptx
KNOWLEDGE SYNTHESIS IN NURSING REAL.pptx
YhenCorbeSongco
 
Part 1 Research workshop
Part 1 Research workshopPart 1 Research workshop
Part 1 Research workshop
Researchworkshop
 
Quality in scientific research final
Quality in scientific research finalQuality in scientific research final
Quality in scientific research final
مركز البحوث الأقسام العلمية
 
Literature Based Research Methodology
Literature Based Research MethodologyLiterature Based Research Methodology
Literature Based Research Methodology
huguette_comerasamy
 
Analysis Of Qualitative Methods Used In Computer And Educational Technologies...
Analysis Of Qualitative Methods Used In Computer And Educational Technologies...Analysis Of Qualitative Methods Used In Computer And Educational Technologies...
Analysis Of Qualitative Methods Used In Computer And Educational Technologies...
Kristen Carter
 
Systematic literature review technique.pptx
Systematic literature review technique.pptxSystematic literature review technique.pptx
Systematic literature review technique.pptx
TANMAY DAS GUPTA
 
1 HOW TO LAY OUT A RESEARCH PROPOSAL Title Page .docx
1 HOW TO LAY OUT A RESEARCH PROPOSAL  Title Page .docx1 HOW TO LAY OUT A RESEARCH PROPOSAL  Title Page .docx
1 HOW TO LAY OUT A RESEARCH PROPOSAL Title Page .docx
oswald1horne84988
 
Systematic Literature Review
Systematic Literature ReviewSystematic Literature Review
Systematic Literature Review
Tiago Silva da Silva
 
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth CobosMeta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
eMadrid network
 
Lecture-3 Research Proposal
Lecture-3 Research ProposalLecture-3 Research Proposal
Lecture-3 Research Proposal
Shankor Paul
 
Can i choose methodology to do my research and thesis
Can i choose methodology to do my research and thesisCan i choose methodology to do my research and thesis
Can i choose methodology to do my research and thesis
Dr. Oday QA
 
When is Qualitative Design Appropriate.pptx
When is Qualitative Design Appropriate.pptxWhen is Qualitative Design Appropriate.pptx
When is Qualitative Design Appropriate.pptx
KimberlyTatel
 
Yannis@patras seminar a_20150512a
Yannis@patras seminar a_20150512aYannis@patras seminar a_20150512a
Yannis@patras seminar a_20150512a
Yannis
 
ResearchDesignppt.pptx
ResearchDesignppt.pptxResearchDesignppt.pptx
ResearchDesignppt.pptx
Santhosh Krishna
 

Similar to David Gough - Evidence informed policy making - 27 June 2017 (20)

Marketing Group assignment 1 Article review guideline.docx
Marketing Group assignment 1 Article review guideline.docxMarketing Group assignment 1 Article review guideline.docx
Marketing Group assignment 1 Article review guideline.docx
 
Multivariate Analysis.ppt
Multivariate Analysis.pptMultivariate Analysis.ppt
Multivariate Analysis.ppt
 
ADOVH Research Methodology.pdf
ADOVH Research Methodology.pdfADOVH Research Methodology.pdf
ADOVH Research Methodology.pdf
 
Business research methods
Business  research methodsBusiness  research methods
Business research methods
 
Over view on systematic review
Over view on systematic review Over view on systematic review
Over view on systematic review
 
Bolouri qualitative method
Bolouri qualitative methodBolouri qualitative method
Bolouri qualitative method
 
KNOWLEDGE SYNTHESIS IN NURSING REAL.pptx
KNOWLEDGE SYNTHESIS IN NURSING REAL.pptxKNOWLEDGE SYNTHESIS IN NURSING REAL.pptx
KNOWLEDGE SYNTHESIS IN NURSING REAL.pptx
 
Part 1 Research workshop
Part 1 Research workshopPart 1 Research workshop
Part 1 Research workshop
 
Quality in scientific research final
Quality in scientific research finalQuality in scientific research final
Quality in scientific research final
 
Literature Based Research Methodology
Literature Based Research MethodologyLiterature Based Research Methodology
Literature Based Research Methodology
 
Analysis Of Qualitative Methods Used In Computer And Educational Technologies...
Analysis Of Qualitative Methods Used In Computer And Educational Technologies...Analysis Of Qualitative Methods Used In Computer And Educational Technologies...
Analysis Of Qualitative Methods Used In Computer And Educational Technologies...
 
Systematic literature review technique.pptx
Systematic literature review technique.pptxSystematic literature review technique.pptx
Systematic literature review technique.pptx
 
1 HOW TO LAY OUT A RESEARCH PROPOSAL Title Page .docx
1 HOW TO LAY OUT A RESEARCH PROPOSAL  Title Page .docx1 HOW TO LAY OUT A RESEARCH PROPOSAL  Title Page .docx
1 HOW TO LAY OUT A RESEARCH PROPOSAL Title Page .docx
 
Systematic Literature Review
Systematic Literature ReviewSystematic Literature Review
Systematic Literature Review
 
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth CobosMeta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
Meta-review of recognition of learning in LMS and MOOCs - Ruth Cobos
 
Lecture-3 Research Proposal
Lecture-3 Research ProposalLecture-3 Research Proposal
Lecture-3 Research Proposal
 
Can i choose methodology to do my research and thesis
Can i choose methodology to do my research and thesisCan i choose methodology to do my research and thesis
Can i choose methodology to do my research and thesis
 
When is Qualitative Design Appropriate.pptx
When is Qualitative Design Appropriate.pptxWhen is Qualitative Design Appropriate.pptx
When is Qualitative Design Appropriate.pptx
 
Yannis@patras seminar a_20150512a
Yannis@patras seminar a_20150512aYannis@patras seminar a_20150512a
Yannis@patras seminar a_20150512a
 
ResearchDesignppt.pptx
ResearchDesignppt.pptxResearchDesignppt.pptx
ResearchDesignppt.pptx
 

More from OECD Governance

OECD Survey on the Drivers of Trust in Public Institutions: 2024 Results
OECD Survey on the Drivers of Trust in Public Institutions: 2024 ResultsOECD Survey on the Drivers of Trust in Public Institutions: 2024 Results
OECD Survey on the Drivers of Trust in Public Institutions: 2024 Results
OECD Governance
 
Public Integrity Indicators Slides
Public Integrity Indicators SlidesPublic Integrity Indicators Slides
Public Integrity Indicators Slides
OECD Governance
 
Summary of the OECD expert meeting: Construction Risk Management in Infrastru...
Summary of the OECD expert meeting: Construction Risk Management in Infrastru...Summary of the OECD expert meeting: Construction Risk Management in Infrastru...
Summary of the OECD expert meeting: Construction Risk Management in Infrastru...
OECD Governance
 
Using AI led assurance to deliver projects on time and on budget - D. Amratia...
Using AI led assurance to deliver projects on time and on budget - D. Amratia...Using AI led assurance to deliver projects on time and on budget - D. Amratia...
Using AI led assurance to deliver projects on time and on budget - D. Amratia...
OECD Governance
 
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm (SE)
 ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm (SE) ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm (SE)
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm (SE)
OECD Governance
 
Building Client Capability to Deliver Megaprojects - J. Denicol, professor at...
Building Client Capability to Deliver Megaprojects - J. Denicol, professor at...Building Client Capability to Deliver Megaprojects - J. Denicol, professor at...
Building Client Capability to Deliver Megaprojects - J. Denicol, professor at...
OECD Governance
 
Procurement strategy in major infrastructure: The AS-IS and STEPS - D. Makovš...
Procurement strategy in major infrastructure: The AS-IS and STEPS - D. Makovš...Procurement strategy in major infrastructure: The AS-IS and STEPS - D. Makovš...
Procurement strategy in major infrastructure: The AS-IS and STEPS - D. Makovš...
OECD Governance
 
Procurement of major infrastructure projects 2017-22 - B. Hasselgren, Senior ...
Procurement of major infrastructure projects 2017-22 - B. Hasselgren, Senior ...Procurement of major infrastructure projects 2017-22 - B. Hasselgren, Senior ...
Procurement of major infrastructure projects 2017-22 - B. Hasselgren, Senior ...
OECD Governance
 
ECI Dutch Experience - A. Chao, Partner, Bird&Bird & J. de Koning, Head of Co...
ECI Dutch Experience - A. Chao, Partner, Bird&Bird & J. de Koning, Head of Co...ECI Dutch Experience - A. Chao, Partner, Bird&Bird & J. de Koning, Head of Co...
ECI Dutch Experience - A. Chao, Partner, Bird&Bird & J. de Koning, Head of Co...
OECD Governance
 
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, StockholmECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm
OECD Governance
 
EPEC's perception of market developments - E. Farquharson, Principal Adviser,...
EPEC's perception of market developments - E. Farquharson, Principal Adviser,...EPEC's perception of market developments - E. Farquharson, Principal Adviser,...
EPEC's perception of market developments - E. Farquharson, Principal Adviser,...
OECD Governance
 
Geographical scope of the lines in Design and Build - B.Dupuis, Executive Dir...
Geographical scope of the lines in Design and Build - B.Dupuis, Executive Dir...Geographical scope of the lines in Design and Build - B.Dupuis, Executive Dir...
Geographical scope of the lines in Design and Build - B.Dupuis, Executive Dir...
OECD Governance
 
Executive Agency of the Dutch Ministry of Infrastructure and Water Management...
Executive Agency of the Dutch Ministry of Infrastructure and Water Management...Executive Agency of the Dutch Ministry of Infrastructure and Water Management...
Executive Agency of the Dutch Ministry of Infrastructure and Water Management...
OECD Governance
 
Presentation of OECD Government at a Glance 2023
Presentation of OECD Government at a Glance 2023Presentation of OECD Government at a Glance 2023
Presentation of OECD Government at a Glance 2023
OECD Governance
 
The Protection and Promotion of Civic Space: Strengthening Alignment with Int...
The Protection and Promotion of Civic Space: Strengthening Alignment with Int...The Protection and Promotion of Civic Space: Strengthening Alignment with Int...
The Protection and Promotion of Civic Space: Strengthening Alignment with Int...
OECD Governance
 
Digital Government Review Türkiye - Assessment and Recommendations
Digital Government Review Türkiye - Assessment and RecommendationsDigital Government Review Türkiye - Assessment and Recommendations
Digital Government Review Türkiye - Assessment and Recommendations
OECD Governance
 
Buliding-Financial-Resilience-to climate-Impacts.pdf
Buliding-Financial-Resilience-to climate-Impacts.pdfBuliding-Financial-Resilience-to climate-Impacts.pdf
Buliding-Financial-Resilience-to climate-Impacts.pdf
OECD Governance
 
Session-7-OECD-Strengthening-climate.pptx
Session-7-OECD-Strengthening-climate.pptxSession-7-OECD-Strengthening-climate.pptx
Session-7-OECD-Strengthening-climate.pptx
OECD Governance
 
Session-6-Building-Financial-Resilience.pptx
Session-6-Building-Financial-Resilience.pptxSession-6-Building-Financial-Resilience.pptx
Session-6-Building-Financial-Resilience.pptx
OECD Governance
 
Session-5-Assurance-on-sustainability-Peter-Welch.pptx
Session-5-Assurance-on-sustainability-Peter-Welch.pptxSession-5-Assurance-on-sustainability-Peter-Welch.pptx
Session-5-Assurance-on-sustainability-Peter-Welch.pptx
OECD Governance
 

More from OECD Governance (20)

OECD Survey on the Drivers of Trust in Public Institutions: 2024 Results
OECD Survey on the Drivers of Trust in Public Institutions: 2024 ResultsOECD Survey on the Drivers of Trust in Public Institutions: 2024 Results
OECD Survey on the Drivers of Trust in Public Institutions: 2024 Results
 
Public Integrity Indicators Slides
Public Integrity Indicators SlidesPublic Integrity Indicators Slides
Public Integrity Indicators Slides
 
Summary of the OECD expert meeting: Construction Risk Management in Infrastru...
Summary of the OECD expert meeting: Construction Risk Management in Infrastru...Summary of the OECD expert meeting: Construction Risk Management in Infrastru...
Summary of the OECD expert meeting: Construction Risk Management in Infrastru...
 
Using AI led assurance to deliver projects on time and on budget - D. Amratia...
Using AI led assurance to deliver projects on time and on budget - D. Amratia...Using AI led assurance to deliver projects on time and on budget - D. Amratia...
Using AI led assurance to deliver projects on time and on budget - D. Amratia...
 
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm (SE)
 ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm (SE) ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm (SE)
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm (SE)
 
Building Client Capability to Deliver Megaprojects - J. Denicol, professor at...
Building Client Capability to Deliver Megaprojects - J. Denicol, professor at...Building Client Capability to Deliver Megaprojects - J. Denicol, professor at...
Building Client Capability to Deliver Megaprojects - J. Denicol, professor at...
 
Procurement strategy in major infrastructure: The AS-IS and STEPS - D. Makovš...
Procurement strategy in major infrastructure: The AS-IS and STEPS - D. Makovš...Procurement strategy in major infrastructure: The AS-IS and STEPS - D. Makovš...
Procurement strategy in major infrastructure: The AS-IS and STEPS - D. Makovš...
 
Procurement of major infrastructure projects 2017-22 - B. Hasselgren, Senior ...
Procurement of major infrastructure projects 2017-22 - B. Hasselgren, Senior ...Procurement of major infrastructure projects 2017-22 - B. Hasselgren, Senior ...
Procurement of major infrastructure projects 2017-22 - B. Hasselgren, Senior ...
 
ECI Dutch Experience - A. Chao, Partner, Bird&Bird & J. de Koning, Head of Co...
ECI Dutch Experience - A. Chao, Partner, Bird&Bird & J. de Koning, Head of Co...ECI Dutch Experience - A. Chao, Partner, Bird&Bird & J. de Koning, Head of Co...
ECI Dutch Experience - A. Chao, Partner, Bird&Bird & J. de Koning, Head of Co...
 
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, StockholmECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm
ECI in Sweden - A. Kadefors, KTH Royal Institute of Technology, Stockholm
 
EPEC's perception of market developments - E. Farquharson, Principal Adviser,...
EPEC's perception of market developments - E. Farquharson, Principal Adviser,...EPEC's perception of market developments - E. Farquharson, Principal Adviser,...
EPEC's perception of market developments - E. Farquharson, Principal Adviser,...
 
Geographical scope of the lines in Design and Build - B.Dupuis, Executive Dir...
Geographical scope of the lines in Design and Build - B.Dupuis, Executive Dir...Geographical scope of the lines in Design and Build - B.Dupuis, Executive Dir...
Geographical scope of the lines in Design and Build - B.Dupuis, Executive Dir...
 
Executive Agency of the Dutch Ministry of Infrastructure and Water Management...
Executive Agency of the Dutch Ministry of Infrastructure and Water Management...Executive Agency of the Dutch Ministry of Infrastructure and Water Management...
Executive Agency of the Dutch Ministry of Infrastructure and Water Management...
 
Presentation of OECD Government at a Glance 2023
Presentation of OECD Government at a Glance 2023Presentation of OECD Government at a Glance 2023
Presentation of OECD Government at a Glance 2023
 
The Protection and Promotion of Civic Space: Strengthening Alignment with Int...
The Protection and Promotion of Civic Space: Strengthening Alignment with Int...The Protection and Promotion of Civic Space: Strengthening Alignment with Int...
The Protection and Promotion of Civic Space: Strengthening Alignment with Int...
 
Digital Government Review Türkiye - Assessment and Recommendations
Digital Government Review Türkiye - Assessment and RecommendationsDigital Government Review Türkiye - Assessment and Recommendations
Digital Government Review Türkiye - Assessment and Recommendations
 
Buliding-Financial-Resilience-to climate-Impacts.pdf
Buliding-Financial-Resilience-to climate-Impacts.pdfBuliding-Financial-Resilience-to climate-Impacts.pdf
Buliding-Financial-Resilience-to climate-Impacts.pdf
 
Session-7-OECD-Strengthening-climate.pptx
Session-7-OECD-Strengthening-climate.pptxSession-7-OECD-Strengthening-climate.pptx
Session-7-OECD-Strengthening-climate.pptx
 
Session-6-Building-Financial-Resilience.pptx
Session-6-Building-Financial-Resilience.pptxSession-6-Building-Financial-Resilience.pptx
Session-6-Building-Financial-Resilience.pptx
 
Session-5-Assurance-on-sustainability-Peter-Welch.pptx
Session-5-Assurance-on-sustainability-Peter-Welch.pptxSession-5-Assurance-on-sustainability-Peter-Welch.pptx
Session-5-Assurance-on-sustainability-Peter-Welch.pptx
 

Recently uploaded

In BELA-BELA ^%[+27633867063*Abortion Pills For Sale In BELA-BELA Oakmoor
In BELA-BELA  ^%[+27633867063*Abortion Pills For Sale In BELA-BELA OakmoorIn BELA-BELA  ^%[+27633867063*Abortion Pills For Sale In BELA-BELA Oakmoor
In BELA-BELA ^%[+27633867063*Abortion Pills For Sale In BELA-BELA Oakmoor
periprospective
 
Online Girls Call Delhi 9711199171 Provide Best And Top Girl Service And No1 ...
Online Girls Call Delhi 9711199171 Provide Best And Top Girl Service And No1 ...Online Girls Call Delhi 9711199171 Provide Best And Top Girl Service And No1 ...
Online Girls Call Delhi 9711199171 Provide Best And Top Girl Service And No1 ...
AK47
 
Blog Version 2024 Population Estimates Summary
Blog Version 2024 Population Estimates SummaryBlog Version 2024 Population Estimates Summary
Blog Version 2024 Population Estimates Summary
ARCResearch
 
The role of skills and geography in job changes for the green transition- Wes...
The role of skills and geography in job changes for the green transition- Wes...The role of skills and geography in job changes for the green transition- Wes...
The role of skills and geography in job changes for the green transition- Wes...
OECD CFE
 
Item # 6 - 318 Tuxedo Ave. compat. review
Item # 6 - 318 Tuxedo Ave. compat. reviewItem # 6 - 318 Tuxedo Ave. compat. review
Item # 6 - 318 Tuxedo Ave. compat. review
ahcitycouncil
 
In Johannesburg ^%[+27633867063*Abortion Pills For Sale In Johannesburg Hazy...
In Johannesburg  ^%[+27633867063*Abortion Pills For Sale In Johannesburg Hazy...In Johannesburg  ^%[+27633867063*Abortion Pills For Sale In Johannesburg Hazy...
In Johannesburg ^%[+27633867063*Abortion Pills For Sale In Johannesburg Hazy...
ivory3872
 
Girls Call Surat 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Surat 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Surat 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Surat 000XX00000 Provide Best And Top Girl Service And No1 in City
gargruksana71
 
RBI Circular on LRS (Liberalized Remittance Scheme)
RBI Circular on LRS (Liberalized Remittance Scheme)RBI Circular on LRS (Liberalized Remittance Scheme)
RBI Circular on LRS (Liberalized Remittance Scheme)
Gaurav Kanudawala
 
Oakland_Public_Education_Fund_sum 2 (1).pptx
Oakland_Public_Education_Fund_sum 2 (1).pptxOakland_Public_Education_Fund_sum 2 (1).pptx
Oakland_Public_Education_Fund_sum 2 (1).pptx
itfernag
 
2024: The FAR - Federal Acquisition Regulations, Part 46
2024: The FAR - Federal Acquisition Regulations, Part 462024: The FAR - Federal Acquisition Regulations, Part 46
2024: The FAR - Federal Acquisition Regulations, Part 46
JSchaus & Associates
 
Global markets, value chains, livelihoods & cassava production systems: impli...
Global markets, value chains, livelihoods & cassava production systems: impli...Global markets, value chains, livelihoods & cassava production systems: impli...
Global markets, value chains, livelihoods & cassava production systems: impli...
Jonathan Newby
 
In MOFOLO ^%[+27633867063*Abortion Pills For Sale In MOFOLO ORANGE_FARM
In MOFOLO  ^%[+27633867063*Abortion Pills For Sale In MOFOLO ORANGE_FARMIn MOFOLO  ^%[+27633867063*Abortion Pills For Sale In MOFOLO ORANGE_FARM
In MOFOLO ^%[+27633867063*Abortion Pills For Sale In MOFOLO ORANGE_FARM
euginexenogeneic
 
Rapport 2024 sur la richesse mondiale (UBS)
Rapport 2024 sur la richesse mondiale (UBS)Rapport 2024 sur la richesse mondiale (UBS)
Rapport 2024 sur la richesse mondiale (UBS)
bernarddetre2022
 
Day care centers for poor childrens.pptx
Day care centers for poor childrens.pptxDay care centers for poor childrens.pptx
Day care centers for poor childrens.pptx
SERUDS INDIA
 
2024: The FAR - Federal Acquisition Regulations, Part 47
2024: The FAR - Federal Acquisition Regulations, Part 472024: The FAR - Federal Acquisition Regulations, Part 47
2024: The FAR - Federal Acquisition Regulations, Part 47
JSchaus & Associates
 
Lucknow Girls Call Kesar Bagh 08630512678 Girls Call Service
Lucknow Girls Call Kesar Bagh 08630512678  Girls Call ServiceLucknow Girls Call Kesar Bagh 08630512678  Girls Call Service
Lucknow Girls Call Kesar Bagh 08630512678 Girls Call Service
akarshkumar98776
 
Item # 8 - Advanced Water Well Technologies improv.
Item # 8 - Advanced Water Well Technologies improv.Item # 8 - Advanced Water Well Technologies improv.
Item # 8 - Advanced Water Well Technologies improv.
ahcitycouncil
 
Critical AI Literacy for Higher Ed Stakeholders
Critical AI Literacy for Higher Ed StakeholdersCritical AI Literacy for Higher Ed Stakeholders
Critical AI Literacy for Higher Ed Stakeholders
arlindno
 
Pune Girls Call Pune 0X0000000X Payment On Delevery Cash Hot Premium Genuine ...
Pune Girls Call Pune 0X0000000X Payment On Delevery Cash Hot Premium Genuine ...Pune Girls Call Pune 0X0000000X Payment On Delevery Cash Hot Premium Genuine ...
Pune Girls Call Pune 0X0000000X Payment On Delevery Cash Hot Premium Genuine ...
seenaoberoi
 
Women's Empowerment in Agrifood Governance (WEAGov) training course
Women's Empowerment in Agrifood Governance (WEAGov) training courseWomen's Empowerment in Agrifood Governance (WEAGov) training course
Women's Empowerment in Agrifood Governance (WEAGov) training course
jkyle5
 

Recently uploaded (20)

In BELA-BELA ^%[+27633867063*Abortion Pills For Sale In BELA-BELA Oakmoor
In BELA-BELA  ^%[+27633867063*Abortion Pills For Sale In BELA-BELA OakmoorIn BELA-BELA  ^%[+27633867063*Abortion Pills For Sale In BELA-BELA Oakmoor
In BELA-BELA ^%[+27633867063*Abortion Pills For Sale In BELA-BELA Oakmoor
 
Online Girls Call Delhi 9711199171 Provide Best And Top Girl Service And No1 ...
Online Girls Call Delhi 9711199171 Provide Best And Top Girl Service And No1 ...Online Girls Call Delhi 9711199171 Provide Best And Top Girl Service And No1 ...
Online Girls Call Delhi 9711199171 Provide Best And Top Girl Service And No1 ...
 
Blog Version 2024 Population Estimates Summary
Blog Version 2024 Population Estimates SummaryBlog Version 2024 Population Estimates Summary
Blog Version 2024 Population Estimates Summary
 
The role of skills and geography in job changes for the green transition- Wes...
The role of skills and geography in job changes for the green transition- Wes...The role of skills and geography in job changes for the green transition- Wes...
The role of skills and geography in job changes for the green transition- Wes...
 
Item # 6 - 318 Tuxedo Ave. compat. review
Item # 6 - 318 Tuxedo Ave. compat. reviewItem # 6 - 318 Tuxedo Ave. compat. review
Item # 6 - 318 Tuxedo Ave. compat. review
 
In Johannesburg ^%[+27633867063*Abortion Pills For Sale In Johannesburg Hazy...
In Johannesburg  ^%[+27633867063*Abortion Pills For Sale In Johannesburg Hazy...In Johannesburg  ^%[+27633867063*Abortion Pills For Sale In Johannesburg Hazy...
In Johannesburg ^%[+27633867063*Abortion Pills For Sale In Johannesburg Hazy...
 
Girls Call Surat 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Surat 000XX00000 Provide Best And Top Girl Service And No1 in CityGirls Call Surat 000XX00000 Provide Best And Top Girl Service And No1 in City
Girls Call Surat 000XX00000 Provide Best And Top Girl Service And No1 in City
 
RBI Circular on LRS (Liberalized Remittance Scheme)
RBI Circular on LRS (Liberalized Remittance Scheme)RBI Circular on LRS (Liberalized Remittance Scheme)
RBI Circular on LRS (Liberalized Remittance Scheme)
 
Oakland_Public_Education_Fund_sum 2 (1).pptx
Oakland_Public_Education_Fund_sum 2 (1).pptxOakland_Public_Education_Fund_sum 2 (1).pptx
Oakland_Public_Education_Fund_sum 2 (1).pptx
 
2024: The FAR - Federal Acquisition Regulations, Part 46
2024: The FAR - Federal Acquisition Regulations, Part 462024: The FAR - Federal Acquisition Regulations, Part 46
2024: The FAR - Federal Acquisition Regulations, Part 46
 
Global markets, value chains, livelihoods & cassava production systems: impli...
Global markets, value chains, livelihoods & cassava production systems: impli...Global markets, value chains, livelihoods & cassava production systems: impli...
Global markets, value chains, livelihoods & cassava production systems: impli...
 
In MOFOLO ^%[+27633867063*Abortion Pills For Sale In MOFOLO ORANGE_FARM
In MOFOLO  ^%[+27633867063*Abortion Pills For Sale In MOFOLO ORANGE_FARMIn MOFOLO  ^%[+27633867063*Abortion Pills For Sale In MOFOLO ORANGE_FARM
In MOFOLO ^%[+27633867063*Abortion Pills For Sale In MOFOLO ORANGE_FARM
 
Rapport 2024 sur la richesse mondiale (UBS)
Rapport 2024 sur la richesse mondiale (UBS)Rapport 2024 sur la richesse mondiale (UBS)
Rapport 2024 sur la richesse mondiale (UBS)
 
Day care centers for poor childrens.pptx
Day care centers for poor childrens.pptxDay care centers for poor childrens.pptx
Day care centers for poor childrens.pptx
 
2024: The FAR - Federal Acquisition Regulations, Part 47
2024: The FAR - Federal Acquisition Regulations, Part 472024: The FAR - Federal Acquisition Regulations, Part 47
2024: The FAR - Federal Acquisition Regulations, Part 47
 
Lucknow Girls Call Kesar Bagh 08630512678 Girls Call Service
Lucknow Girls Call Kesar Bagh 08630512678  Girls Call ServiceLucknow Girls Call Kesar Bagh 08630512678  Girls Call Service
Lucknow Girls Call Kesar Bagh 08630512678 Girls Call Service
 
Item # 8 - Advanced Water Well Technologies improv.
Item # 8 - Advanced Water Well Technologies improv.Item # 8 - Advanced Water Well Technologies improv.
Item # 8 - Advanced Water Well Technologies improv.
 
Critical AI Literacy for Higher Ed Stakeholders
Critical AI Literacy for Higher Ed StakeholdersCritical AI Literacy for Higher Ed Stakeholders
Critical AI Literacy for Higher Ed Stakeholders
 
Pune Girls Call Pune 0X0000000X Payment On Delevery Cash Hot Premium Genuine ...
Pune Girls Call Pune 0X0000000X Payment On Delevery Cash Hot Premium Genuine ...Pune Girls Call Pune 0X0000000X Payment On Delevery Cash Hot Premium Genuine ...
Pune Girls Call Pune 0X0000000X Payment On Delevery Cash Hot Premium Genuine ...
 
Women's Empowerment in Agrifood Governance (WEAGov) training course
Women's Empowerment in Agrifood Governance (WEAGov) training courseWomen's Empowerment in Agrifood Governance (WEAGov) training course
Women's Empowerment in Agrifood Governance (WEAGov) training course
 

David Gough - Evidence informed policy making - 27 June 2017

  • 1. (1) Justifiable Evidence Claims: evidence standards to inform decision making David Gough Governing better through evidence-informed policy making OECD, Paris, 27th June 2017 Email: david.gough@ucl.ac.uk Twitter: @ProfDavidGough EPPI-Centre, Social Science Research Unit, UCL Institute of Education, University College London http://eppi.ioe.ac.uk
  • 2. (2) OECD and EC long term supporters of evidence use in policy & practice • 2005 OECD international comparisons of use of research in educational policy • 2007 OECD book on evidence and education • OECD Governance of Complex Education Systems • 2010-2013 EC DGEAC funding of the Evidence in Education in Policy and Practice in Europe (EIPPEE)
  • 3. (3) Evidence standard 1: Systematic reviews – not just any available study! • Similar to primary research: – Same expectations of rigor and transparency – Same range of questions and assumptions – Methods of review reflect methods and assumptions of primary studies – May include mixed methods • A higher level of analysis: – Data usually from pre-existing studies rather than new primary data – More justifiable evidence claims than from individual studies alone
  • 4. (4) Evidence standard 2: Justifiable evidence claims based on technique and relevance Evidence claim • Depends on research question. For e.g.: that an intervention has a certain type and size of effect Justification for claim: Research method Sub-dimensions: • Methods standards achieved • Suitability of study design/method • Relevance to study focus - not only based on the technical execution of method
  • 5. (5) Evidence standard 3: Justifiable evidence claims – 3 dimensions Evidence claim: • Depends on research question. For e.g.: that an intervention has a certain type and size of effect Justification for claim: 3 Dimensions • Review method: Standard+Suitability+Relevance • Included studies: Standard+Suitability+Relevance • Evidence produced: Nature + Extent
  • 6. (6) Dimension 1: Review method Sub-dimensions: a) Methods standards achieved: how well was review undertaken methodologically? b) Suitability of the review method: Is this an appropriate method for answering this review question? c) Relevance to review focus: Is the way that the review is conducted appropriate for the specific focus of the review question? Example of components of appraisal instruments to assess Dimension 1: AMSTAR: (a) Methodological quality ROBIS: (a) Concerns with the review process; Risk of bias; (c) Relevance
  • 7. (7) Dimension 2: Included studies Sub-dimensions: a) Methods standards achieved: how well were the ‘included studies’ undertaken methodologically? b) Suitability of the research methods of the ‘included studies’: Were these appropriate methods for answering this review question? c) Relevance to review focus: Is the way that the ‘included studies’ were conducted appropriate for the specific focus of the review question? Example of components of appraisal instruments to assess Dimension 2: GRADE: (a) Study design; (a&b) Study limitations; (c) Indirectness CERQUAL: (a&b) Methodological limitations; (c) Relevance
  • 8. (8) Dimension 3: Evidence produced by the review Sub-dimensions: a) Nature of the included studies: qualities of the evidence when considered together (such as heterogeneity, statistical independence). b) Extent of evidence from the included studies: further to issues of quality maybe issues of the extent of evidence available to address the review question. Example of components of appraisal instruments to assess Dimension 3: GRADE: (a&b) Inconsistency; Imprecision; Publication bias; Magnitude of treatment effect; Impact of confounders; Dose /response CERQUAL: (a) Coherence; (a&b) (b) Adequacy of data Example of components of appraisal instruments to assess Dimensions 2 & 3: Both GRADE and CERQUAL: Quality of evidence for each specific finding
  • 9. (9) Justifiable evidence claims to inform decisions Evidence claim Justification for claim • Review Method • Included Studies • Evidence produced Consideration of evidence and decisions • Interpretation & integration with other information Application of decisions Outcomes of decisions
  • 10. (10) Systematic Review Methods + The Science of Using Science https://www.sagepub.com/en-us/nam/an-introduction-to- systematic-reviews/book245742%20 http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=3504