Indicators of Education SystemsSeeing your education system in the mirror of other systems<br />Madrid, 24 May 2010<br />A...
Indicators of education systems<br />Or the Alchemists’ Stone?<br />The Holy Grail?<br />Know why you are looking<br />You...
Know what you are looking for<br />The Holy Grail was a well-described object, and there was only one true grail…<br />
More than compilations of statistics<br /><ul><li>Do we understand the policy objectives
What countries care about, what they intend to do about it and how they define success
Do we understand the policy context and can we anticipate its future development ?
Do we understand the strategic requirements for change?
Are these
Technically feasible?
Politically and socially suitable?
Robust and cost-effective?
Do we understand the delivery challenge and delivery capacity?
Nature and size of the barriers that systems face to deliver reform goals
Can what works in one country by done in another by real people in real situations?
Avoiding big time and energy traps?
Do we understand past and present performance vis a vis the policy goals as well as the drivers of performance and their u...
What is added value of international comparisons?</li></li></ul><li>Domain 1<br />Individual learner<br />LevelA<br />Leve...
Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain...
A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Cost per student<br />Gradua...
A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />United States<br />Cost per ...
A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Australia<br />Finland<br />...
A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation r...
A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation r...
A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation r...
A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation r...
A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation r...
A world of change – highereducation<br />What about international students?<br />Expenditure per student at tertiary level...
Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain...
Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain...
Components of the private net present value for a male with higher education<br />27K$<br />56K$<br />170K$<br />105K$<br ...
Public cost and benefits for a male obtaining <br />post-secondary education<br />Public costs<br />Public benefits<br />N...
Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain...
Spending choices on secondary schoolsContribution of various factors to upper secondary teacher compensation costsper stud...
Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain...
Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain...
OECD’s PISA assessment of the knowledge and skills of 15-year-olds<br />Coverage of world economy<br />83%<br />77%<br />8...
High science performance<br />Average performanceof 15-year-olds in science – extrapolate and apply<br />Poland 2000<br />...
High science performance<br />Average performanceof 15-year-olds in science – extrapolate and apply<br />High average perf...
High science performance<br />Durchschnittliche Schülerleistungen im Bereich Mathematik<br />High average performance<br /...
Consistency in quality standardsVariation in the performance of 15-year-olds in mathematics<br />20<br />
Consistency in quality standardsVariation in the performance of 15-year-olds in mathematics<br />Variation of performance ...
Upcoming SlideShare
Loading in …5
×

2010 e-madrid (spanish presidency) - rev 1.0

811 views

Published on

Spanish EU Presidency discusses how indicators can help improve education

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
811
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Such an OECD knowledge base needs to reflect three dimension, (1) an understanding of what countries are trying to achieve, (2) the evidence that we have, and (3) what we know about policy delivery in different contexts. As concerning the policy context: Do we understand what countries care about, what they intend to do about it and how they define success? Do we understand the policy context, and that is not just about the context as it stands, but how we anticipate its future development, because in an area like education where the delivery chain of reforms is so long, we need to think far into the future. Do we understand the strategic requirements for change, and do we judge various approaches to change technically feasible? And not everything that is feasible is politically suitable. And not everything that can be done is robust and cost-effective. With regard to the evidence: Are our indicators and analyses adequately capturing past and present performance of the education system vis a vis the reform goals, and have we been able to pin-point the drivers of performance and their underlying system activities? How does what we produce at the OECD, whether that is data, country reviews or thematic review relate to and add value to what other players provide?Finally, to what extent is our analysis useful and actionable in national contexts. Do we understand the challenges for reform delivery in countries and their delivery capacity? Is what we propose doable by real people in real situations and avoiding big time and energy traps?
  • The pace of change is most clearly visible in higher education, and I want to bring two more dimensions into the picture here. Each dot on this chart represents one country. The horizontal axis shows you the college graduation rate, the proportion of an age group that comes out of the system with a college degree. The vertical axis shows you how much it costs to educate a graduate per year.
  • *Lets now add where the money comes from into the picture, the larger the dot, the larger the share of private spending on college education, such as tuition.The chart shows the US as the country with the highest college graduation rate, and the highest level of spending per student. The US is also among the countries with the largest share of resources generated through the private sector. That allows the US to spend roughly twice as much per student as Europe. US, FinlandThe only thing I have not highlighted so far is that this was the situation in 1995. And now watch this closely as you see how this changed between 1995 and 2005.
  • You see that in 2000, five years, later, the picture looked very different. While in 1995 the US was well ahead of any other country – you see that marked by the dotted circle, in 2000 several other countries had reached out to this frontier. Look at Australia, in pink.
  • Thatwasallveryquick, letusgothroughthisdevelopmentonceagain
  • The best way to find out whether what students have learned at school matters for their life is to actuallywatch what happens to them after they leave school. This is exactly what we have done that with around 30,000 students in Canada. We tested them in the year 2000 when they were 15 years old in reading, math and science, and since then we are following up with them each year on what choices they make and how successful they are in their transition from school to higher education and work.The horizontal axis shows you the PISA level which 15-year-old Canadians had scored in 2000. Level 2 is the baseline level on the PISA reading test and Level 5 the top level in reading.The red bar shows you how many times more successful someone who scored Level 2 at age 15 was at age 19 to have made a successful transition to university, as compared to someone who did not make it to the baseline PISA level 1. And to ensure that what you see here is not simply a reflection of social background, gender, immigration or school engagement, we have already statistically accounted for all of these factors. The orange bar. …How would you expect the picture to be like at age 21? We are talking about test scores here, but for a moment, lets go back to the judgements schools make on young people, for example through school marks. You can do the same thing here, you can see how well school marks at age 15 predict the subsequent success of youths. You see that there is some relationship as well, but that it is much less pronounced than when we use the direct measure of skills.
  • Let us go back to the 1960s. The chart shows you the wealth of world regions and the average years of schooling in these regions, which is the most traditional measure of human capital. Have a look at Latin America, it ranked third in wealth and third in years of schooling, so in the 1960s the world seemed pretty much in order.
  • But when you look at economic growth between 1960 and 2000, you see that something went wrong. Despite the fact that Latin America did well in terms of years of schooling, only Sub-Saharan Africa did worse in terms of economic growth. So in 2000, Latin America had fallen back considerably in terms of GDP per capita.You can draw two conclusions from this: Either education is not as important for economic growth as we thought, or we have for a long time been measuring the wrong thing.
  • Now let me add one additional element, and that is a measure of the quality of education, in the form of the score of the different world regions on international tests like PISA or TIMSS. And you see now that the world looks in order again, there seems a close relationship between test scores and economic growth. You can see that even more clearly when you put this into graphical form. This is one of the charts produced by Professor Hanushek. And, as Professor Hanushek will explain, the relationship holds even when you account for other factors, it even holds when you compare growth in economies with growth in learning outcomes, which is the closest we can come to examining causality.So what this tells you is that it is not simply years of schooling or the number of graduates we produce, but indeed the quality of learning outcomes that counts.
  • 2010 e-madrid (spanish presidency) - rev 1.0

    1. 1. Indicators of Education SystemsSeeing your education system in the mirror of other systems<br />Madrid, 24 May 2010<br />Andreas SchleicherEducation Policy Advisor of the OECD Secretary-General<br />
    2. 2. Indicators of education systems<br />Or the Alchemists’ Stone?<br />The Holy Grail?<br />Know why you are looking<br />You cannot improve what you cannot measure<br />The yardstick for success is no longer just improvement by national standards but the best performing education systems globally<br />Know what you are looking for<br />Systemic framework<br />Policy relevance<br />Cross-country and cross-cultural validity<br />Know howyou will recognise it when you find it<br />What makes a good indicator system?<br />Implications and lessons learned .<br />
    3. 3. Know what you are looking for<br />The Holy Grail was a well-described object, and there was only one true grail…<br />
    4. 4. More than compilations of statistics<br /><ul><li>Do we understand the policy objectives
    5. 5. What countries care about, what they intend to do about it and how they define success
    6. 6. Do we understand the policy context and can we anticipate its future development ?
    7. 7. Do we understand the strategic requirements for change?
    8. 8. Are these
    9. 9. Technically feasible?
    10. 10. Politically and socially suitable?
    11. 11. Robust and cost-effective?
    12. 12. Do we understand the delivery challenge and delivery capacity?
    13. 13. Nature and size of the barriers that systems face to deliver reform goals
    14. 14. Can what works in one country by done in another by real people in real situations?
    15. 15. Avoiding big time and energy traps?
    16. 16. Do we understand past and present performance vis a vis the policy goals as well as the drivers of performance and their underlying system activities?
    17. 17. What is added value of international comparisons?</li></li></ul><li>Domain 1<br />Individual learner<br />LevelA<br />LevelB<br />Instructional settings<br />LevelC<br />Schools, other institutions<br />Country or system<br />LevelD<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain ed policy<br />Policy Leversshape educational outcomes<br />Outputs and Outcomesimpact of learning<br />Quality and distribution of knowledge & skills<br />Individ attitudes, engagement and behaviour<br />Socio-economic background of learners<br />Student learning, teacher working conditions<br />Quality of instructional delivery<br />Teaching, learning practices and classroom climate<br />The learning environment at school<br />Community and school characteristics<br />Output and performance of institutions<br />National educ, social and economic context<br />Social & economic outcomes of education<br />Structures, resource alloc and policies<br />
    18. 18. Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain ed policy<br />Policy Leversshape educational outcomes<br />Outputs and Outcomesimpact of learning<br />Quality and distribution of knowledge & skills<br />Individ attitudes, engagement and behaviour<br />LevelA<br />Socio-economic background of learners<br />Individual learner<br />LevelB<br />Student learning, teacher working conditions<br />Quality of instructional delivery<br />Teaching, learning practices and classroom climate<br />Instructional settings<br />The learning environment at school<br />Community and school characteristics<br />Output and performance of institutions<br />LevelC<br />Schools, other institutions<br />National educ, social and economic context<br />Social & economic outcomes of education<br />Structures, resource alloc and policies<br />Country or system<br />LevelD<br />
    19. 19. A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Cost per student<br />Graduate supply<br />Tertiary-type A graduation rate <br />
    20. 20. A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />United States<br />Cost per student<br />Finland<br />Japan<br />Spain<br />Graduate supply<br />Tertiary-type A graduation rate <br />
    21. 21. A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Australia<br />Finland<br />United Kingdom<br />Tertiary-type A graduation rate <br />
    22. 22. A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation rate <br />
    23. 23. A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation rate <br />
    24. 24. A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation rate <br />
    25. 25. A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation rate <br />
    26. 26. A world of change – highereducation<br />Expenditure per student at tertiary level (USD)<br />Tertiary-type A graduation rate <br />
    27. 27. A world of change – highereducation<br />What about international students?<br />Expenditure per student at tertiary level (USD)<br />United States<br />Australia<br />A<br />A<br />United Kingdom<br />Finland<br />Spain<br />A<br />Tertiary-type A graduation rate <br />
    28. 28. Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain ed policy<br />Policy Leversshape educational outcomes<br />Outputs and Outcomesimpact of learning<br />Quality and distribution of knowledge & skills<br />Individ attitudes, engagement and behaviour<br />LevelA<br />Socio-economic background of learners<br />Individual learner<br />LevelB<br />Student learning, teacher working conditions<br />Quality of instructional delivery<br />Teaching, learning practices and classroom climate<br />Instructional settings<br />The learning environment at school<br />Community and school characteristics<br />Output and performance of institutions<br />LevelC<br />Schools, other institutions<br />National educ, social and economic context<br />Social & economic outcomes of education<br />Structures, resource alloc and policies<br />Country or system<br />LevelD<br />
    29. 29. Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain ed policy<br />Policy Leversshape educational outcomes<br />Outputs and Outcomesimpact of learning<br />Quality and distribution of knowledge & skills<br />Individ attitudes, engagement and behaviour<br />LevelA<br />Socio-economic background of learners<br />Individual learner<br />LevelB<br />Student learning, teacher working conditions<br />Quality of instructional delivery<br />Teaching, learning practices and classroom climate<br />Instructional settings<br />The learning environment at school<br />Community and school characteristics<br />Output and performance of institutions<br />LevelC<br />Schools, other institutions<br />National educ, social and economic context<br />Social & economic outcomes of education<br />Structures, resource alloc and policies<br />Country or system<br />LevelD<br />
    30. 30. Components of the private net present value for a male with higher education<br />27K$<br />56K$<br />170K$<br />105K$<br />35K$<br />26K$<br />367K$<br />Net present value in USD equivalent<br />
    31. 31. Public cost and benefits for a male obtaining <br />post-secondary education<br />Public costs<br />Public benefits<br />Net present value, USD equivalent<br />(numbers in orange shownegative values)<br />USD equivalent<br />
    32. 32. Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain ed policy<br />Policy Leversshape educational outcomes<br />Outputs and Outcomesimpact of learning<br />Quality and distribution of knowledge & skills<br />Individ attitudes, engagement and behaviour<br />LevelA<br />Socio-economic background of learners<br />Individual learner<br />LevelB<br />Student learning, teacher working conditions<br />Quality of instructional delivery<br />Teaching, learning practices and classroom climate<br />Instructional settings<br />The learning environment at school<br />Community and school characteristics<br />Output and performance of institutions<br />LevelC<br />Schools, other institutions<br />National educ, social and economic context<br />Social & economic outcomes of education<br />Structures, resource alloc and policies<br />Country or system<br />LevelD<br />
    33. 33. Spending choices on secondary schoolsContribution of various factors to upper secondary teacher compensation costsper student as a percentage of GDP per capita (2004)<br />Percentage points<br />
    34. 34. Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain ed policy<br />Policy Leversshape educational outcomes<br />Outputs and Outcomesimpact of learning<br />Quality and distribution of knowledge & skills<br />Individ attitudes, engagement and behaviour<br />LevelA<br />Socio-economic background of learners<br />Individual learner<br />LevelB<br />Student learning, teacher working conditions<br />Quality of instructional delivery<br />Teaching, learning practices and classroom climate<br />Instructional settings<br />The learning environment at school<br />Community and school characteristics<br />Output and performance of institutions<br />LevelC<br />Schools, other institutions<br />National educ, social and economic context<br />Social & economic outcomes of education<br />Structures, resource alloc and policies<br />Country or system<br />LevelD<br />
    35. 35. Domain 1<br />Dimensions of an indicators framework<br />Domain 3<br />Domain 2<br />Antecedentscontextualise or constrain ed policy<br />Policy Leversshape educational outcomes<br />Outputs and Outcomesimpact of learning<br />Quality and distribution of knowledge & skills<br />Individ attitudes, engagement and behaviour<br />LevelA<br />Socio-economic background of learners<br />Individual learner<br />LevelB<br />Student learning, teacher working conditions<br />Quality of instructional delivery<br />Teaching, learning practices and classroom climate<br />Instructional settings<br />The learning environment at school<br />Community and school characteristics<br />Output and performance of institutions<br />LevelC<br />Schools, other institutions<br />National educ, social and economic context<br />Social & economic outcomes of education<br />Structures, resource alloc and policies<br />Country or system<br />LevelD<br />
    36. 36. OECD’s PISA assessment of the knowledge and skills of 15-year-olds<br />Coverage of world economy<br />83%<br />77%<br />81%<br />85%<br />86%<br />87%<br />
    37. 37. High science performance<br />Average performanceof 15-year-olds in science – extrapolate and apply<br />Poland 2000<br />… 18 countries perform below this line<br />Low science performance<br />
    38. 38. High science performance<br />Average performanceof 15-year-olds in science – extrapolate and apply<br />High average performance<br />Large socio-economic disparities<br />High average performance<br />High social equity<br />Strong socio-economic impact on student performance<br />Socially equitable distribution of learning opportunities<br />Low average performance<br />Large socio-economic disparities<br />Low average performance<br />High social equity<br />Low science performance<br />
    39. 39. High science performance<br />Durchschnittliche Schülerleistungen im Bereich Mathematik<br />High average performance<br />Large socio-economic disparities<br />High average performance<br />High social equity<br />Strong socio-economic impact on student performance<br />Socially equitable distribution of learning opportunities<br />Low average performance<br />Large socio-economic disparities<br />Low average performance<br />High social equity<br />Low science performance<br />
    40. 40. Consistency in quality standardsVariation in the performance of 15-year-olds in mathematics<br />20<br />
    41. 41. Consistency in quality standardsVariation in the performance of 15-year-olds in mathematics<br />Variation of performance within schools<br />Variation of performance between schools<br />OECD (2004), Learning for tomorrow’s world: First results from PISA 2003, Table 4.1a, p.383.<br />
    42. 42. Pooled international dataset, effects of selected school/system factors on science performance after accounting for all other factors in the model<br />School principal’s positive evaluation of quality of educational materials(gross only)<br />Schools with more competing schools(gross only)<br />Schools with greater autonomy (resources)(gross and net)<br />School activities to promote science learning(gross and net)<br />One additional hour of self-study or homework (gross and net)<br />One additional hour of science learning at school (gross and net)<br />School results posted publicly (gross and net)<br />Academically selective schools (gross and net) but no system-wide effect<br />Schools practicing ability grouping (gross and net)<br />One additional hour of out-of-school lessons (gross and net)<br />20<br />Each additional 10% of public funding(gross only)<br />School principal’s perception that lack of qualified teachers hinders instruction(gross only)<br />Effect after accounting for the socio-economic background of students, schools and countries<br />Measured effect<br />OECD (2007), PISA 2006 – Science Competencies from Tomorrow’s World, Table 6.1a <br />
    43. 43. Know how you will recognise it when you find it<br />The Alchemists’ stone was to be recognisedby transforming ordinary metal into gold…<br />
    44. 44. Cross-cultural invariance of PISA<br />
    45. 45. Increased likelihood of postsec. particip. at age 19/21 associated with PISA reading proficiency at age 15 (Canada)after accounting for school engagement, gender, mother tongue, place of residence, parental, education and family income (reference group PISA Level 1)<br />Odds ratioCollege entry<br />School marks at age 15<br />PISA performance at age 15<br />
    46. 46. Latin America then…<br />Hanushek 2009<br />
    47. 47. Latin America then and now…<br />Hanushek 2009<br />
    48. 48. Latin America then and now…<br />Why quality is the key<br />Hanushek 2009<br />
    49. 49. High policy relevance<br />Quick wins<br />Must haves<br />Outcomes of education<br />Financing of education<br />Participation and progression<br />Processes and structures<br />Low technical quality/costly<br />High technical quality<br />Money pits<br />Low-hanging fruits<br />…<br />Low policy relevance<br />
    50. 50. Implications and lessons learned<br />The medieval Alchemists’ followed the dictates of a well-established science but that was built on wrong foundations<br />The search for the Holy Grail was overburdened by false clues and cryptic symbols<br />
    51. 51. Some things are always difficult<br />Focus on added value of international comparisons<br />Not every policy question canbebestanswered through international comparisons <br />Trading off breadth and depths<br />Not everything that isimportantneedstobe dealt withwithexcruciatingdetail<br />Seek outputs that are as comparable as possible…<br />… but as country-specific as necessary<br />Focus coverage as much as feasible…<br />… but keep as large as necessary to be useful for policy formation .<br />
    52. 52. High policy value<br />A real-time assessment environment that bridges the gap between formative and summative assessment .<br />Quick wins<br />Must haves<br />Examine individual, institutional and systemic factors associated with performance<br />Extending the range of indicators through which educational quality is assessed<br />Monitor educational progress<br />Measuring growth in learning<br />Low feasibility<br />High feasibility<br />Establish the relative standing of students and schools<br />Assuming that every new indicator is orthogonal to all others<br />Money pits<br />Low-hanging fruits<br />Low policy value<br />
    53. 53. www.oecd.org; www.pisa.oecd.org<br />All national and international publications<br />The complete micro-level database<br />email: pisa@oecd.org<br />Andreas.Schleicher@OECD.org<br />… and remember:<br /> Without data, you are just another person with an opinion<br />Thank you !<br />

    ×