The document discusses indicators of education systems and frameworks for measuring educational quality and performance across countries. It addresses the need to (1) understand policy objectives and context, (2) consider multiple domains and levels from individual learners to national systems, and (3) recognize indicators that provide policy-relevant insights while also being technically feasible and comparable cross-culturally. International assessments like PISA aim to meet these criteria by measuring student performance, highlighting disparities, and informing policy, but require balancing breadth, depth, and country-specific needs.

1. Indicators of Education SystemsSeeing your education system in the mirror of other systems Madrid, 24 May 2010 Andreas SchleicherEducation Policy Advisor of the OECD Secretary-General

2. Indicators of education systems Or the Alchemists’ Stone? The Holy Grail? Know why you are looking You cannot improve what you cannot measure The yardstick for success is no longer just improvement by national standards but the best performing education systems globally Know what you are looking for Systemic framework Policy relevance Cross-country and cross-cultural validity Know howyou will recognise it when you find it What makes a good indicator system? Implications and lessons learned .

3. Know what you are looking for The Holy Grail was a well-described object, and there was only one true grail…

5. What countries care about, what they intend to do about it and how they define success

6. Do we understand the policy context and can we anticipate its future development ?

7. Do we understand the strategic requirements for change?

8. Are these

9. Technically feasible?

10. Politically and socially suitable?

11. Robust and cost-effective?

12. Do we understand the delivery challenge and delivery capacity?

13. Nature and size of the barriers that systems face to deliver reform goals

14. Can what works in one country by done in another by real people in real situations?

15. Avoiding big time and energy traps?

16. Do we understand past and present performance vis a vis the policy goals as well as the drivers of performance and their underlying system activities?

18. Domain 1 Dimensions of an indicators framework Domain 3 Domain 2 Antecedentscontextualise or constrain ed policy Policy Leversshape educational outcomes Outputs and Outcomesimpact of learning Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour LevelA Socio-economic background of learners Individual learner LevelB Student learning, teacher working conditions Quality of instructional delivery Teaching, learning practices and classroom climate Instructional settings The learning environment at school Community and school characteristics Output and performance of institutions LevelC Schools, other institutions National educ, social and economic context Social & economic outcomes of education Structures, resource alloc and policies Country or system LevelD

19. A world of change – highereducation Expenditure per student at tertiary level (USD) Cost per student Graduate supply Tertiary-type A graduation rate

20. A world of change – highereducation Expenditure per student at tertiary level (USD) United States Cost per student Finland Japan Spain Graduate supply Tertiary-type A graduation rate

21. A world of change – highereducation Expenditure per student at tertiary level (USD) Australia Finland United Kingdom Tertiary-type A graduation rate

22. A world of change – highereducation Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

23. A world of change – highereducation Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

24. A world of change – highereducation Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

25. A world of change – highereducation Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

26. A world of change – highereducation Expenditure per student at tertiary level (USD) Tertiary-type A graduation rate

27. A world of change – highereducation What about international students? Expenditure per student at tertiary level (USD) United States Australia A A United Kingdom Finland Spain A Tertiary-type A graduation rate

28. Domain 1 Dimensions of an indicators framework Domain 3 Domain 2 Antecedentscontextualise or constrain ed policy Policy Leversshape educational outcomes Outputs and Outcomesimpact of learning Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour LevelA Socio-economic background of learners Individual learner LevelB Student learning, teacher working conditions Quality of instructional delivery Teaching, learning practices and classroom climate Instructional settings The learning environment at school Community and school characteristics Output and performance of institutions LevelC Schools, other institutions National educ, social and economic context Social & economic outcomes of education Structures, resource alloc and policies Country or system LevelD

29. Domain 1 Dimensions of an indicators framework Domain 3 Domain 2 Antecedentscontextualise or constrain ed policy Policy Leversshape educational outcomes Outputs and Outcomesimpact of learning Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour LevelA Socio-economic background of learners Individual learner LevelB Student learning, teacher working conditions Quality of instructional delivery Teaching, learning practices and classroom climate Instructional settings The learning environment at school Community and school characteristics Output and performance of institutions LevelC Schools, other institutions National educ, social and economic context Social & economic outcomes of education Structures, resource alloc and policies Country or system LevelD

30. Components of the private net present value for a male with higher education 27K$ 56K$ 170K$ 105K$ 35K$ 26K$ 367K$ Net present value in USD equivalent

31. Public cost and benefits for a male obtaining post-secondary education Public costs Public benefits Net present value, USD equivalent (numbers in orange shownegative values) USD equivalent

32. Domain 1 Dimensions of an indicators framework Domain 3 Domain 2 Antecedentscontextualise or constrain ed policy Policy Leversshape educational outcomes Outputs and Outcomesimpact of learning Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour LevelA Socio-economic background of learners Individual learner LevelB Student learning, teacher working conditions Quality of instructional delivery Teaching, learning practices and classroom climate Instructional settings The learning environment at school Community and school characteristics Output and performance of institutions LevelC Schools, other institutions National educ, social and economic context Social & economic outcomes of education Structures, resource alloc and policies Country or system LevelD

33. Spending choices on secondary schoolsContribution of various factors to upper secondary teacher compensation costsper student as a percentage of GDP per capita (2004) Percentage points

34. Domain 1 Dimensions of an indicators framework Domain 3 Domain 2 Antecedentscontextualise or constrain ed policy Policy Leversshape educational outcomes Outputs and Outcomesimpact of learning Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour LevelA Socio-economic background of learners Individual learner LevelB Student learning, teacher working conditions Quality of instructional delivery Teaching, learning practices and classroom climate Instructional settings The learning environment at school Community and school characteristics Output and performance of institutions LevelC Schools, other institutions National educ, social and economic context Social & economic outcomes of education Structures, resource alloc and policies Country or system LevelD

35. Domain 1 Dimensions of an indicators framework Domain 3 Domain 2 Antecedentscontextualise or constrain ed policy Policy Leversshape educational outcomes Outputs and Outcomesimpact of learning Quality and distribution of knowledge & skills Individ attitudes, engagement and behaviour LevelA Socio-economic background of learners Individual learner LevelB Student learning, teacher working conditions Quality of instructional delivery Teaching, learning practices and classroom climate Instructional settings The learning environment at school Community and school characteristics Output and performance of institutions LevelC Schools, other institutions National educ, social and economic context Social & economic outcomes of education Structures, resource alloc and policies Country or system LevelD

36. OECD’s PISA assessment of the knowledge and skills of 15-year-olds Coverage of world economy 83% 77% 81% 85% 86% 87%

37. High science performance Average performanceof 15-year-olds in science – extrapolate and apply Poland 2000 … 18 countries perform below this line Low science performance

38. High science performance Average performanceof 15-year-olds in science – extrapolate and apply High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low science performance

39. High science performance Durchschnittliche Schülerleistungen im Bereich Mathematik High average performance Large socio-economic disparities High average performance High social equity Strong socio-economic impact on student performance Socially equitable distribution of learning opportunities Low average performance Large socio-economic disparities Low average performance High social equity Low science performance

40. Consistency in quality standardsVariation in the performance of 15-year-olds in mathematics 20

41. Consistency in quality standardsVariation in the performance of 15-year-olds in mathematics Variation of performance within schools Variation of performance between schools OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.

42. Pooled international dataset, effects of selected school/system factors on science performance after accounting for all other factors in the model School principal’s positive evaluation of quality of educational materials(gross only) Schools with more competing schools(gross only) Schools with greater autonomy (resources)(gross and net) School activities to promote science learning(gross and net) One additional hour of self-study or homework (gross and net) One additional hour of science learning at school (gross and net) School results posted publicly (gross and net) Academically selective schools (gross and net) but no system-wide effect Schools practicing ability grouping (gross and net) One additional hour of out-of-school lessons (gross and net) 20 Each additional 10% of public funding(gross only) School principal’s perception that lack of qualified teachers hinders instruction(gross only) Effect after accounting for the socio-economic background of students, schools and countries Measured effect OECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a

43. Know how you will recognise it when you find it The Alchemists’ stone was to be recognisedby transforming ordinary metal into gold…

44. Cross-cultural invariance of PISA

45. Increased likelihood of postsec. particip. at age 19/21 associated with PISA reading proficiency at age 15 (Canada)after accounting for school engagement, gender, mother tongue, place of residence, parental, education and family income (reference group PISA Level 1) Odds ratioCollege entry School marks at age 15 PISA performance at age 15

46. Latin America then… Hanushek 2009

47. Latin America then and now… Hanushek 2009

48. Latin America then and now… Why quality is the key Hanushek 2009

49. High policy relevance Quick wins Must haves Outcomes of education Financing of education Participation and progression Processes and structures Low technical quality/costly High technical quality Money pits Low-hanging fruits … Low policy relevance

50. Implications and lessons learned The medieval Alchemists’ followed the dictates of a well-established science but that was built on wrong foundations The search for the Holy Grail was overburdened by false clues and cryptic symbols

51. Some things are always difficult Focus on added value of international comparisons Not every policy question canbebestanswered through international comparisons Trading off breadth and depths Not everything that isimportantneedstobe dealt withwithexcruciatingdetail Seek outputs that are as comparable as possible… … but as country-specific as necessary Focus coverage as much as feasible… … but keep as large as necessary to be useful for policy formation .

52. High policy value A real-time assessment environment that bridges the gap between formative and summative assessment . Quick wins Must haves Examine individual, institutional and systemic factors associated with performance Extending the range of indicators through which educational quality is assessed Monitor educational progress Measuring growth in learning Low feasibility High feasibility Establish the relative standing of students and schools Assuming that every new indicator is orthogonal to all others Money pits Low-hanging fruits Low policy value

53. www.oecd.org; www.pisa.oecd.org All national and international publications The complete micro-level database email: pisa@oecd.org Andreas.Schleicher@OECD.org … and remember: Without data, you are just another person with an opinion Thank you !