Making reform happen and evaluating reform in education
Upcoming SlideShare
Loading in...5

Making reform happen and evaluating reform in education



Presentation for the Norwegian Education Ministry - 22 June 2011

Presentation for the Norwegian Education Ministry - 22 June 2011



Total Views
Views on SlideShare
Embed Views



2 Embeds 2 1 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment
  • PartnershipsNetworksKnowledge brokeringInstitutional champions and thought leadersOwnershipUptake and implementationResistance to change/fatigueDuring the review visit, there were plenty of examples of such enabling structures having grown out of local social capital and a commonly perceived sense of urgency, or as a result of a particular funding arrangement for a defined period of time.
  • Complex governance structures without fluid communication channels:Vertically (national, regional)Horizontally (across government departments)
  • Emphasis on rational thinking but is this how decisions are made? How to capture after the fact explanation/revelation? How much input do drivers/funders have on process (as opposed to evidence base and research)?
  • 1) comprehensive evaluation of system and how to enhance capacity/ transparent information gaps. 2) Efforts to develop a systemic approach to innovation in VET are rare.. Only Switzerland and, to a lesser extent, Australia, can be said to have designed a systemic approach to innovation in VET. 3) Lack of a critical mass of codified, formal and research-based knowledge on VET, both at national and international levels. Knowledge brokerage institutions supporting the genesis and diffusion of innovations are scarce; knowledge based linkages between stakeholders weak. 4) Investing in VET innovations without carefully planning their evaluation should not be an option. Decisions about sustainability or scaling up of innovations cannot be taken on an informed, and eventually evidence-based, ground if there are not in place mechanisms intended to assess their effects. The innovation-related policies aimed at fostering innovations in VET cannot be assessed in the absence of feedback.
  • Explain irrationality politics

Making reform happen and evaluating reform in education Making reform happen and evaluating reform in education Presentation Transcript

  • Making reform happen and evaluating reform in education:A ‘knowledge management’ approach
    Dirk Van Damme
    Head of the Centre for Educational Research and Innovation - OECD
  • Outline
    Knowledge management in education
    Knowledge as part of systemic innovation
    Making reform happen in education
    The ‘GPS’ approach
    Governance and knowledge
    Conclusions and questions
  • Knowledge management in education
  • Educational R&D
    Striking findings that education, as a knowledge sector, has a very weak knowledge base itself…
    Low levels of educational R&D (but difficulties in finding a methodology for comparable data collection)
    Much lower than related public policy sectors such as health or social policy
    A weak empirical research capacity…
    Especially for quantitative research
    And a weak link between research and policy
  • Educational R&D
  • Educational R&D
  • Educational R&D
  • Evidence in Education
    The emergence of evidence in education…
    Methodologically sound solutions found for measurement issues in education
    Comparative education indicators
    Moving from inputs to outputs and outcomes
    The PISA shock
    The development of feedback systems, at student, school and system levels
    …dramatically changed the policy climate
  • Evidence in Education
    Important methodological issues and debates on what counts as (scientific research) evidence
    (quasi-)experimental design, randomised control trials, …
    Lack of large-scale longitudinal studies
    Scientific ideal versus pragmatically feasible
    Cost and capacity problems
    Ethical issues about educational experimenting, privacy issues related to data collection
  • Evidence in Education
    Research – Policy interaction
    Not a simple model of direct impact of evidence on policy, but mediation by all stakeholders and actors in a complex system
    The influence of knowledge on policy making may in fact be strongest not when it comes directly from the educational research community in direct advice to policy makers but when it is filtered through actors such as print or broadcast media, lobbyists, popularisers, etc. (EU, 2007, p.5)
    Mediation seems to be the weakest link in the knowledge chain
  • Evidence in Education
  • 12
  • Knowledge as part of systemic innovation
  • 14
  • Systemic Innovation in Education
    Investigate how systems go about change
    Processes and stakeholder relationships
    Knowledge Management perspective
    14 case studies: Australia, Denmark, Germany, Hungary, Mexico, and Switzerland
    Factors that facilitate/impede the use of evidence
    Lessons learned about the use of evidence
  • Enablers of innovation
    Consensus building
    Political vision
    Research evidence
    Brokerage: generation and dissemination of knowledge
    Legitimating rigour/quality
    Developing cooperation/trust
  • Barriers to innovation
    Change fatigue
    Competing policy agendas
    Accountability mechanisms and public policy agendas:
    Restricted risk management
    Short-term planning
  • Model of Change in Education
    Identification of needs
     What are the drivers of change?
    • Which stakeholders are involved?
    Evaluation & Monitoring
    = surveillance/ judgement of outcomes
    • How and when?
    • What criteria are used?
    • Summative or formative purpose?
    • What are the findings?
    Identification of needs
    Knowledge base
    What types of knowledge?
    • Tacit knowledge
    • Explicit knowledge
    What knowledge sources?
    Development of innovation
    Evaluation & Monitoring
    Knowledge base
    Implementation process
    • Without piloting: large-scale implementation
    • With piloting:
    Small-scale implementation
  • Systemic Innovation in Education
    Research evidence is interacting with various other forms of knowledge in innovation processes in education:
    Professional/Practitioner knowledge
    Teachers’ core pedagogical knowledge and beliefs as part of their professional identity
    Tacit knowledge
    Establishing a ‘culture of evaluation’ is critically important for the success of reform and planning of next reforms
  • Conclusions
    Policy Implications
    • Systemic innovation as useful analytical framework
    Targeted strategy to induce system-wide change
    • SI as guiding principle for innovation policy
    • Establish a formalised knowledge base
    • Monitoring and evaluation
    • Support link between systems research and innovation
    • Evidence-informed dialogue with stakeholders
    • Need for formalised knowledge base
    • Losing innovation opportunities
    • Not cost effective
  • Making reform happen in education
  • 22
  • Key lessons on education reform
    Making reform happen needs to:
    Actively engage stakeholders in formulating and implementing policy responses
    Make effective use of evidence to shape policies
    Explain clearly underlying principles and aims of reforms
    (from: OECD Education Policy Committee meeting at CEO level Seoul, Korea, 2008)
  • Key lessons on education reform
    More specific lessons from reform experience:
    Policy makers need to build consensus on the aims of education reform and actively engage stakeholders, especially teachers, in formulating and implementing policy responses
    Reforms can capitalise on external pressures as part of building a compelling case for change
    All political players and stakeholders need to develop more realistic expectations about the pace and nature of reforms to improve outcomes
    Reforms need to be backed by sustainable financing
    There is some shift away from reform initiatives per se towards building self-adjusting systems with rich feedback at all levels, incentives to react and tools to strengthen capacities to deliver better outcomes
  • Key lessons on education reform
    More specific lessons from reform experience:
    Investment is needed in change-management skills in the education system
    Teachers need reassurance that they will be given the tools to change and recognition of their professional motivation to improve outcomes for their students
    Evidence can be used more effectively to guide policy making, combining international benchmarks with national surveys and with inspectorates to provide a better diagnosis
    Evidence is most helpful when it is fed back to institutions along with information and tools about how they can use the information to improve outcomes
    “Whole-of-government” approaches can include education in more comprehensive reforms. These need effective co-ordination and overall leadership across all the relevant ministries
  • Making reform happen
    Case studies:
    Denmark: introducing a culture of evaluation
    Finland: introducing polytechnics into the tertiary education sector
    Portugal: tertiary education reform and teacher education reform
    Norway: improving lower secondary education
    Mexico: improving schools
    (Education and Training Policy division)
  • Making reform happen
    Mexico project video
  • Making reform happen
    Key elements relevant to Making Reform Happen in Mexico:
    greater understanding among stakeholders of each others’ roles and perspectives
    greater commitment to work together on lifting performance
    stronger capacity for undertaking education policy reform
    strong Mexican ownership of recommendations
    obstacles to reform identified and recommendations realistic
    long-term vision as well as immediate steps to take
    OECD in facilitating role, bringing international expertise to support Mexico’s reforms
  • The ‘GPS’ Approach
  • Leveraging the impact of knowledge
    The intent is to mobilise and integrate evidence from data, analyses and policy advice generated over the years by the Education Directorate and to make this knowledge accessible in a systematic way
    This service to member countries is an analogue of a ‘GPS navigation system’ which provides real-time evidence to guide the choices and trade-offs when they need it for policy decisions related to education
    Like the GPS system, the search by any user at any time should provide selected, focused and actionable information on specific issues for policy decision making by countries.
    (Indicators and Analysis Division)
  • Leveraging the impact of knowledge
    Overall objectives of this service to countries is:
    to provide a real time comprehensive access EDU’s rich evidence base on educational outcomes and policies experience so that they can used for policy decisions by governments.
    to offer a means by which countries can assess the contribution of educational reform to the growth and progress of their country, recognizing that the full impact of educational reform is cumulative over the trajectory of policy decisions
  • Objective 1: Real time comprehensive access to evidence means a shift from whole documents and other sources to extracted evidence. However the link to the document will be retained
    Evidence-link from Education Today, 2010
    Teachers are positive about the appraisal and feedback they receive, but in some countries a significant minority or even majority of teachers have not received any in recent years: Teachers across the different systems surveyed by TALIS tend to be positive about the appraisal and feedback
    they receive, reporting that on the whole it is fair and helpful for their work, and increases their job satisfaction. Approximately 13% of teachers surveyed by TALIS reported that they had received no feedback or evaluation in their current school in the previous five years; this average level rises to much higher levels in Ireland (26%), Italy (55%), Portugal (26%) and Spain (46%).
    High proportions of lower secondary teachers participate in professional development but many say that they would like more: Nearly 9 teachers in 10 surveyed by TALIS reported having taken part in a structured professional development activity in the preceding 18 months, though in Denmark, the Slovak Republic and Turkey around a quarter reported no participation during that period. Despite generally high levels of participation, more than half the teachers (55%) in the TALIS countries overall say that they would have liked more professional development, and lack of suitable opportunities is a significant factor in this. Approximately a third of the surveyed teachers reported a high level of need for training to help them teach students with special learning needs. Other professional development priorities include teaching with ICT and dealing with difficult student behaviour.
    Evidence-link: Up to Upper Secondary School, Analysis.
    Education Today, 2010, p20
  • Strand 1: Managing what we know
    • A digital tool to ensure OECD evidence is at your disposal at your desk when you need it and in a form that is suitable and adapted to the specific policy context in which you need it.
    • Three types of evidence – data, analyses and policy experience – will be available.
    • Work will continue to create the digital tool and to move evidence from the last three years into the data base. It is estimated that the complete data base will contain over 100,000 individual pieces of evidence which can be searched.
    • The tool will be available in 2013. Its utility is dependent on having a large enough data base for it to be useful to users. Further additions to the data base will continue adding to its value.
  • Strand 1: Managing what we know
    • The Evidence Navigator for Education will be available on the OECD Education web site and on OECD iLibrary in early 2013.
    • All evidence can be viewed at once, or evidence that is data, analysis or policy separately.
    • The data base will have evidence extracted from all products generated by the Education Directorate (CERI, ETP, INES, IA, CELE, IMHE) and all the surveys (PISA, PIAAC, TALIS, etc) which can be searched.
    • The extracted evidence includes both published and non-published information.
    • The search results show text, tables, figures, powerpoint slide or video.
    • If you want more information, the evidence will be linked to the original source at the extracted location. For instance, if you are interested in a figure, clicking on it will take you to the page in the book or report from which it is drawn.
    • Here is a short “trailer”.
  • Strand 2: Peer-learning from policy experiences
    Peer-learning from evaluated policies in other countries that have achieved successful outcomes
    Policy experience analysis will be incorporated into the GPS data base
    First volume of Strong Performers, Successful Reformers was published December 2010.
    OECD – Japan Seminar June 28-29 2011 will include a second volume.
  • Strand 3: Building tools for successful policy implementation
    Successful policy implementation is essential for reforms that achieve targets, within budget and on schedule. Implementation tools and delivery capacity are useful in policy planning.
    When available, such tools for managing implementation and assessing delivery capacity will be incorporated into the GPS data base for use by countries.
    A pilot of the tool developed by McKinsey and Michael Barber in the UK has been proposed to member countries in order to test its appropriateness and value.
  • Governance and knowledge
  • Governance and Knowledge
    Governing Complex Education Systems is a new CERI project dealing with the governance challenges of increasingly complex education systems
    Increasing number of actors and stakeholders
    Multilevel governance issues
    Decentralisation and re-centralisation
    What is the role of knowledge in governance?
    What is the role of governance in knowledge creation, dissemination and utilisation?
  • Governance and Knowledge
    General research questions:
    What models of governance are effective in complex education systems?
    What knowledge systems are necessary to support this?
  • Governance and Knowledge
    More specifically:
    How creating the capacity at central levels to handle complex flows of knowledge
    What knowledge options do policy makers have in making decisions and involving stakeholders
    How to provide the local levels in complex systems with sufficient knowledge to perform
    How to ensure that levels do share relevant knowledge
  • Steering
    Priority setting
    Policy Design
    Knowledge use
    Knowledge production
    Governance model
  • Governance and Knowledge
  • Conclusions and questions
  • Conclusions and Questions
    As most policy areas, educational policy has become more evidence-based in most countries
    What kind of evidence counts in educational policy and practice?
    What is the role of empirical scientific research?
    What is the role of comparative education indicators?
    Is evidence shared and discussed with all stakeholders before leading to policy decisions?
    Which channels do mediate between research, stakeholders and policy makers?
    How can we improve knowledge mediation?
  • Conclusions and Questions
    Innovations in education often lack a systemic approach, with a clear and knowledge-driven vision on implementation, scaling, monitoring and evaluation
    What role do various kinds of knowledge play in educational innovations?
    Is knowledge resulting from evaluating innovations used in designing new innovations?
    Is evidence from evaluations used to enrich the dialogue with practitioners and stakeholders?
  • Conclusions and Questions
    Educational systems have become more complex, partly as a result of decentralisation, partly as a result of multiplication of stakeholders. The governance of complex systems becomes a real challenge
    Can research evidence and other knowledge help in ‘binding’ the educational system? What are the functions knowledge can play in terms of governance?
    Under which conditions and in what forms should knowledge be developed, shared and discussed in order to have a productive impact on governance?
  • Thank you !