OECD Reviews of Evaluation and Assessment in Education: Czech Republic


Published on

The Review looks at the various components of assessment and evaluation frameworks. These include:
Student assessment; Teacher appraisal; School evaluation.
Education system evaluation;
Other types of evaluation (programme evaluation, evaluation of school leadership etc)

Published in: Education, Travel
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

OECD Reviews of Evaluation and Assessment in Education: Czech Republic

  1. 1. OECD Reviews ofEvaluation and Assessment in Education: Czech RepublicBy Paulo Santiago, Alison Gilmore, Deborah Nusche and Pamela Sammons www.oecd.org/edu/evaluationpolicy Conference, Prague The main conclusions and policy recommendations Presentation by Paulo Santiago Directorate for Education, OECD 15 March 2012
  2. 2. Outline of presentation Part 1 The OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes and the Review of Evaluation and Assessment in the Czech Republic Part 2 Conclusions and Recommendations • The Evaluation and Assessment Framework • Student assessment • Teacher appraisal • School evaluation • Education system evaluation
  3. 3. PART 1(1) The OECD Review on Evaluation and Assessment Frameworks for Improving School Outcomes and;(2) the Review of Evaluation and Assessment in the Czech Republic
  4. 4. OECD REVIEW - OBJECTIVES●Purpose: To explore how systems of evaluation and assessment can be used to improve the quality, equity and efficiency of school education.A Review of national approaches to evaluation and assessment in school education●The Review: Synthesises research-based evidence on the impact of evaluation and assessment strategies and disseminate this knowledge among countries. Identifies innovative and successful policy initiatives and practices. Facilitates exchanges of lessons and dialogue. Identifies policy options for governments to consider.
  5. 5. OECD REVIEW - SCOPE• The Review looks at the various components of assessment and evaluation frameworks. These include: • Student assessment; • Teacher appraisal; • School evaluation; • Education system evaluation; • Other types of evaluation (programme evaluation, evaluation of school leadership etc)• Comprehensive approach: Investigation of each component individually, as well as the coherence of the framework as a whole (including the links between the different components).
  6. 6. OECD REVIEW - KEY ISSUESKey policy issues for analysis– Governance: Designing a systemic framework for evaluation and assessment– Procedures: Ensuring the effectiveness of evaluation and assessment procedures– Competencies: Developing competencies for evaluation and for using feedback– Use of results: Making the best use of evaluation results– Implementation: Implementing evaluation and assessment policies
  7. 7. OECD REVIEW - BROAD FEATURES●Oversight by the Group of National Experts (GNE) on Evaluation and Assessment https://community.oecd.org/community/evaluationpolicyinschools●Countries working collaboratively with the Secretariat Countries do part of the work of the Review, a national co-ordinator is nominated, dialogue within countries is encouraged●Countries exchanging lessons and experiences, in particular through GNE meetings●Collection of views and perspectives from a range of stakeholders Through their participation in GNE meetings and their interaction with Country Review expert teams●A diverse set of external experts involved in the Country Reviews Involvement of top academics and policy makers to bring expertise into the Review●Ensuring a range of principles are respected Independence of analysis; inclusiveness in terms of the views and perspectives which are collected; extensive reviews of the literature; evidence-based policy advice; site visits to interact with school realities; contextualisation of policy advice.
  8. 8. OECD REVIEW - METHODOLOGY Analytical strand • Identifying the key questions for analysis and the background information needed from countries • Reviewing the literature and evidence on the impact of evaluation and assessment procedures • Gathering data on countries’ policies and practices (Country Background Reports) Country Review strand • Country Reviews provide specific advice to individual countries. • OECD-led Review Team with external experts • The scope and focus is determined by the country in consultation with the Secretariat Synthesis report • Comparative report to analyse policy options and highlight good practices across countries.  Collaboration with other OECD activities and international agencies • INES (in particular NESLI), PISA, TALIS, CERI’s ILE , Longitudinal Information Systems and Governing Complex Education Systems project, CELE’s on evaluating quality in educational facilities, GOV’s work on Public Sector Evaluation. • EC, Eurydice, the World Bank, Standing International Conference of Inspectorates(SICI), UNESCO, BIAC, TUAC.
  9. 9. OECD REVIEW - PARTICIPATION●Twenty four systems are preparing a Country Background Report: Australia, Austria, Belgium (Flemish Community), Belgium (French Community), Canada, Chile, Czech Republic, Denmark, Finland, France, Hungary, Iceland, Ireland, Korea, Luxembourg, Mexico, Netherlands, New Zealand, Norway, Poland, Portugal, Slovak Republic, Slovenia and Sweden.●Twelve countries also opted for a Country Review: Australia, Belgium (Flemish Community), Chile, Czech Republic, Denmark, Luxembourg, Mexico, New Zealand, Norway, Portugal, Slovak Republic and Sweden.●All OECD Member countries and Observers to the Education Policy Committee are members of the Group of National Experts on Evaluation and Assessment
  10. 10. REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC●Objectives: 1. To provide insights and suggestions for policy development in the Czech Republic; 2. To inform the wider international community about (a) the main features of the Czech Republic’s evaluation and assessment policies; and (b) innovative and effective approaches to evaluation and assessment in the Czech Republic; and 3. To inform the final synthesis report from the project.●Structure of visit (29 March – 5 April, 2011): • Review team with 4 members: 2 OECD Secretariat members; and researchers from New Zealand and the United Kingdom; • Interacted with about 200 individuals; had about 45 meetings amounting to about 50 hours of discussions; visited 6 schools in Prague, Ostrava and Liberec • Discussions were held with a wide range of national, regional and local authorities; education officials; the Schools Inspectorate; parents’ organisations; representatives of schools and school directors; teacher educators; and researchers with an interest in evaluation and assessment issues.
  11. 11. REVIEW of EVALUATION and ASSESSMENT in the CZECH REPUBLIC●Relevant features: 1. External perspective: looking from a distance; 2. Independent perspective: no review team member has a vested interest in the system; 3. Not an examination: qualitative analysis that seeks to provide an input of a specific nature to the internal debate; 4. Bring together research evidence, data and information available in a broad comprehensive comparative framework.●Country Review report released on 30 January, 2012 and available at: www.oecd.org/edu/evaluationpolicy
  12. 12. PART 2Conclusions and policy recommendations
  13. 13. Conclusions and policy recommendations The evaluation and assessment framework
  14. 14. The Evaluation and Assessment Framework Strengths  There are common references at the national level to provide the basis for evaluation and assessment 4-year Long-term policy objectives with indicators; Framework Education Programmes  There are good conditions for adapting learning to local needs School Education Programmes: curriculum innovation; collaborative work Regional 4-year Long-term policy objectives  Responsibilities across the evaluation and assessment framework are well articulated MEYS, CSI, Regions and municipalities, schools
  15. 15. The Evaluation and Assessment Framework Strengths (continued)  There is a range of initiatives to strengthen evaluation and assessment in the school system National standardised tests, common part of the school-leaving examination, external school evaluation, school self-evaluation mandatory, national indicators on education  There is an “open door” climate among teachers  There is a good principle of supporting policy work with specific expertise Institutes with specialised expertise, data collections, OECD projects  There are some reporting requirements: Education database, inspection reports
  16. 16. The Evaluation and Assessment Framework Challenges  The evaluation and assessment (E&A) framework needs to be completed and made coherent o There is no integrated E&A framework o The E&A framework is incomplete (formative assessment of students, moderation of marks, systematic teacher appraisal, school self-evaluation incipient, no framework for school leadership appraisal) o Some articulations within the E&A framework are not sufficiently developed  There is little attention to equity and inclusion in the evaluation and assessment framework  The Framework Education Programmes are not perceived as specific enough to guide teaching and assessment Little shared understanding about what constitutes adequate, good and excellent performance  It is unclear that the students are at the centre of the evaluation and assessment framework
  17. 17. The Evaluation and Assessment Framework Challenges (continued)  There is a narrow understanding of the purposes of evaluation and assessment Perceived as instrument to hold stakeholders accountable, to “control” and assess compliance with regulations  There is a need to strengthen competencies for evaluation and assessment across the system  The articulation between levels of government and the support from the centre are limited o Concerns about the lack of systematic application of national directions, inconsistency of practices and little capacity or commitment to developing quality frameworks. o Weak articulations between the different decision-making levels. o Limited provision from the centre of tools and guidelines to assist evaluation and assessment activities  There are challenges in the implementation of some evaluation and assessment initiatives
  18. 18. The Evaluation and Assessment Framework Policy recommendations Better articulate learning goals o Establish clear goals for education and make equity and inclusion more prominent o Clarify reference points and criteria for quality in evaluation and assessment Integrate the evaluation and assessment framework Develop a strategic plan or framework document that conceptualises a complete evaluation and assessment framework and articulates ways to achieve the coherence between its different components Strengthen some of the components of the evaluation and assessment framework Teacher appraisal, appraisal of school leaders, formative student assessment, school self-evaluation Further develop some articulations within the evaluation and assessment framework
  19. 19. The Evaluation and Assessment Framework Policy recommendations (continued)  Build on some key principles to effectively implement evaluation and assessment o Place the students at the centre of the evaluation and assessment framework o Ensure a good emphasis on the improvement function of evaluation and assessment o Communicate the rationale for evaluation and assessment o Recognise the importance of school leadership o Establish an implementation strategy  Develop evaluation and assessment capacity across the school system  Improve the articulation between levels of government and assure support from the centre
  20. 20. Conclusions and policy recommendations Student assessment
  21. 21. Student Assessment Strengths  Assessment is seen as part of the professional role of teachers o Teachers play the major role in assessing and reporting o Schools decide and publish assessment criteria o Variety of approaches  An external dimension to assessment was introduced o National component of school-leaving examination o Common assignments for the apprenticeship certificate o Addresses the need for checks and balances to ensure reliability in the application of standards  There is an increased focus on student outcomes o Including a move to identify expected minimum standards of achievement for students at key points in their education
  22. 22. Student Assessment Challenges  Approaches to learning and assessment remain markedly traditional o Traditional approach to the organisation of classrooms o Little emphasis in assessment practices on providing student feedback  Assessment for learning is not systematically used in Czech schools o Feedback often understood as ‘summative assessment done more often’  Summative assessment of students raises some concerns o Assessment is often norm-referenced o Achievement mixed with effort and motivation  There is limited consistency of student assessment across schools and classes  The national-level support for teacher-based student assessment is limited o Lack of guidance to assess against FEPs (e.g. in the form of exemplars), insufficient assessment tools for teachers, limited professional development.
  23. 23. Student Assessment Challenges (continued)  The national standardised tests entail a range of limitations and risks o Development of the standards being rushed o May be more appropriately regarded as specifications for the national tests, rather than standards o Potential negative consequences of high-stakes use: (i) ‘teaching to the test’; (ii) cross-curricular competencies ignored; (iii) classroom time spent on preparation of the test; (iv) schools selecting the students taking the test.  The assessment of students leads to little interaction among teachers o Low levels of teacher moderation of student assessments  Multiple purposes to school-leaving examinations raise some concerns o To provide a certificate of achievement to students o To compare performance of schools: need other sources of information to achieve this and value-added techniques
  24. 24. Student Assessment Policy recommendations  Develop educational standards covering the breadth of student learning objectives prior to developing national standardised tests o There is a need for clear external reference points in terms of expected levels of student performance at different levels of education o National tests not to be the vehicle to develop standards  Limit the undesired effects of national standardised tests Reflect further on the purposes of the national tests; should be trialled first; minimise negative impacts  Develop a broad strategy for student assessment and strengthen the role of formative assessment Assessment for learning; provide tools supporting the assessment of students; broad range of assessment approaches; formative ass.  Build teachers’ capability for student assessment
  25. 25. Student Assessment Policy recommendations (continued)  Develop a range of tools at the central level to support teacher-based student assessment e.g. marking rubrics; exemplars illustrating student performance, assessment tools  Put in place moderation processes to ensure the consistency of student summative assessment  Student assessment should be criterion-based rather than norm-referenced  Ensure student assessment is inclusive  e.g. students with special needs, avoid biases by socio-economic background and minority status (e.g. Roma students)  Build capacity of markers of external tests and examinations
  26. 26. Conclusions and policy recommendations Teacher appraisal
  27. 27. Teacher Appraisal Strengths  The principle that teachers should be evaluated is widely accepted o Teacher appraisal takes place in all schools and is an important and normal part of school activities  Teacher appraisal is focused on evaluating classroom teaching  The importance of teacher professional development is recognised in the legislation  Some structures for co-operation and exchange among teachers are in place o E.g. Subject commissions bringing together all teachers teaching a particular subject; lesson preparation; exchange of pedagogical approaches
  28. 28. Teacher Appraisal Strengths (continued)  There are plans to develop teaching standards and a new career system for teachers  The link between teacher appraisal and pay increments has potential to incentivise high performance o Teachers may be awarded additional pay increments and bonuses that are determined by the school leadership team o However, there are important questions regarding the transparency of how salary rewards are implemented
  29. 29. Teacher Appraisal Challenges  There is currently no shared understanding of what constitutes high quality teaching o Lack of a national framework defining standards for the teaching profession o There are no uniform performance criteria or reference frameworks against which teachers could be appraised  Teacher appraisal is not systematically implemented for all teachers The quality and extent of teacher appraisal approaches in individual schools depend on the capacity and leadership style of the school principals.  There is little tradition for educational leadership in schools Many school leaders have not been sufficiently prepared for their wide range of tasks, in particular leading teaching and learning processes in the school.
  30. 30. Teacher Appraisal Challenges (continued)  There are tensions between the accountability and improvement functions of teacher appraisal o Teacher appraisal traditionally been conceived as a summative and accountability-oriented process o There are risks involved in trying to achieve both the accountability function and the improvement function of teacher appraisal in one single process  The link between teacher appraisal and rewards is not transparent  Links between teacher appraisal and professional development could be enhanced o The provision of prof. development appears fragmented and not systematically linked to teacher appraisal o There is scope to better link teacher professional development to school development and improvement
  31. 31. Teacher Appraisal Policy recommendations  Develop a professional profile or standards for the teaching profession A clear and concise statement of what teachers are expected to know and be able to do is a key element in any teacher appraisal system as it provides a credible reference to make judgements about teacher competence  Strengthen teacher appraisal for improvement purposes (developmental appraisal) o Strengthening regular school-based formative appraisal with a professional development focus which is separate from the more summative appraisal processes. o To be validated externally by Schools Inspectorate  Further enhance the role of educational leadership
  32. 32. Teacher Appraisal Policy recommendations (continued)  Consider establishing a system of teacher certification to determine career progression  Access to each of the key stages of a career could be associated with formal processes of summative appraisal that complement the regular formative appraisal  The different career stages should match the different levels of expertise reflected in teaching standards  Need a stronger component external to the school to validate the process and ensure that practices are consistent across the country  Three core instruments: classroom observation, self appraisal and documentation of practices in a simplified portfolio.  Ensure links between developmental appraisal and appraisal for certification  Ensure appropriate articulation between teacher appraisal and school evaluation
  33. 33. Conclusions and policy recommendations School evaluation
  34. 34. School Evaluation Strengths  External school evaluation is established o Clear commitment to external accountability based around school evaluation with a regular cycle of external school evaluations o External school evaluation valued by stakeholders  The external evaluation model embodies a number of features of best practice o Well structured and systematic o Set of publicly-available criteria for external inspection is drawn up every year o Publication of inspection reports o A range of sources of information o Thematic reports  Schools facing greater challenges benefit from some follow-up The CSI undertakes a follow-up inspection to assess whether improvements were undertaken to address the challenges previously identified.
  35. 35. School Evaluation Strengths (continued)  Classroom observation is part of school evaluation processes Placing learning and teaching at the heart of the evaluation process sends clear signals about what matters.  There is a new emphasis on schools’ self-evaluation Has the potential to encourage schools and principals to place a greater emphasis on school improvement and development planning.  School leadership is promoted in school evaluation There is an explicit recognition that the processes of self-evaluation and external evaluation are hugely dependent on a principal’s capacity to stimulate engagement, to mobilise resources and to ensure appropriate training and support.
  36. 36. School Evaluation Challenges  External school evaluation seems to have limited emphasis on school improvement o The accountability function tends to emphasise compliance with legislation rather than the promotion of school improvement. o Advice is only given to “weaker” schools which are identified as those that do not meet the minimum standards as set by law. o There is not enough guidance about what will lead to school improvement and little attention is paid to identifying and disseminating best practice  There are a number of limitations in external school evaluation Difficult to take account of the socio-economic context of the school; not enough emphasis on pedagogical aspects, Criteria used in the CSI external evaluations are not stable enough.  There is little emphasis on student results/progress Limited ability to assess quality of learning and student progress.
  37. 37. School Evaluation Challenges (continued)  School self-evaluation needs to be strengthened Its penetration across the school system remains at an early stage of development  The use of data for school development is limited  The evaluation by organising bodies has a limited scope and impact  There is no full recognition of the role of school leaders and their appraisal is limited o While the school principal has a key role in the system and considerable responsibilities, this has not as yet been translated into a dedicated career structure o Limited preparation for the role of school principal o Little recognition and financial reward o The evaluation of school principals, conducted by organising bodies, is largely absent except in terms of the financial aspects of budget management.
  38. 38. School Evaluation Policy recommendations  Strengthen external school evaluation o Strengthen focus on school improvement and move away from the current “compliance” driven model. o Provide advice for improvement to all schools evaluated. o The school evaluation framework, the criteria and questions governing judgements and the methods employed should all focus much more directly on the quality of learning and teaching and their relationship to student outcomes.  Improve the alignment between external and self-evaluation and raise the profile of self-evaluation Criteria used in both processes sufficiently similar as to create a common language  Give stronger emphasis to the follow-up to external evaluation  Improve the capacity of schools to engage in school evaluation
  39. 39. School Evaluation Policy recommendations (continued)  Improve the instructional leadership skills of school principals  Plan to use data on student results effectively Any publication of results of school performance in students’ school-leaving examinations and/or national tests should be presented in ways that take account of intake differences including, for example, the socio-economic background of students.  Strengthen the evaluation of school principals  Improve the scope and role of organising bodies in school quality improvement o The regional and municipal authorities should strengthen their role in supporting school improvement. o Collaboration and networking amongst schools could be encouraged to help develop and spread good practice and enhance teachers’ professional skills.
  40. 40. Conclusions and policy recommendations Education system evaluation
  41. 41. Education System Evaluation Strengths  An Education Indicators Framework is established It involves well-established procedures for data collection in close articulation with schools.  There is a concern to assess the progress of the education system towards pre established objectives Principle of establishing educational objectives and the subsequent monitoring of the progress towards achieving them  The qualitative analysis of thematic reports provides valuable information for system monitoring Richness of contextualised qualitative analysis  The participation in international surveys is instrumental for system evaluation
  42. 42. Education System Evaluation Challenges  There is little emphasis on the evaluation of the education system o Limited policy attention thus far and there is no comprehensive strategic approach to it o The current narrow approach to system evaluation does not allow a broad enough assessment of the extent to which student learning objectives are being achieved  The absence of student performance data is a major gap in system monitoring Presently there is no mechanism for the Czech Republic to monitor at a national level the achievement of its students against learning objectives specified in the Framework Education Programmes  There are key information gaps at the system level There are no measures on students’ socio-economic background; There are gaps in the data collected from schools; Little emphasis on investigating inequities in the system; Limited information on the teaching and learning environment
  43. 43. Education System Evaluation Challenges (continued)  It is not possible to monitor student outcomes over time and across schools o Mostly due to the absence of national data on student performance o School-level results of national tests might be disclosed with no account for schools’ particular contexts. This can considerably distort considerations about the effectiveness of each school  Monitoring at the region and municipality levels is faced with considerable challenges o Regions have a limited intervention in quality assurance with their main tool being the evaluation of school principals o No overview of the different quality assurance systems in the regions, including strategies for school improvement o Limited articulation between Regions and municipalities  System-level information is not fully exploited Little analysis to inform educational planning; Limited use to inform school management; No comprehensive information system; Systematic sharing of data between schools is limited
  44. 44. Education System Evaluation Policy recommendations Raise the profile of system evaluation within the evaluation and assessment framework The challenge for system-level evaluation is to ensure that the measures of system performance are broad enough to capture the whole range of student learning objectives – policy making at the system level needs to be informed by high quality data and evidence, but not driven by the availability of such information. Develop national student performance data for system monitoring o Design national standardised tests for national monitoring and as a pedagogical tool [closely aligned with student learning objectives; can measure only a limited range of student learning objectives; publish test results at the school level is premature] o Develop strategies to monitor a wider range of curricular areas and broader outcomes [develop sample-based surveys] o Ensure that national monitoring covers broader outcomes [e.g. higher-order thinking skills and cross-curricular competencies]
  45. 45. Education System Evaluation Policy recommendations (continued) Prioritise efforts to meet information needs for national monitoring o Develop measures of the socio-economic background of students o The data collection from schools needs to be improved o Give more prominence to the analysis of inequities in the system o Improve the information on the teaching and learning environment Explore ways to more reliably track educational outcomes over time and across schools Make meaningful comparisons across schools if test results are published at the school level Strengthen the role of regions and municipalities in quality improvement Optimise the reporting and use of system-level data o Strengthen the analysis for educational planning and policy development o Improve feedback for local monitoring o Integrate available data and facilitate access by key agencies o Facilitate the sharing of student information across schools
  46. 46. Thank you (paulo.santiago@oecd.org)(www.oecd.org/edu/evaluationpolicy)