Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (18)
Similar to Opening the ‘black box’ was melbourne 2 oct 2015
Similar to Opening the ‘black box’ was melbourne 2 oct 2015 (20)
More from dvndamme
More from dvndamme (15)
Recently uploaded
Recently uploaded (20)
Opening the ‘black box’ was melbourne 2 oct 2015
- 1. OPENING THE ‘BLACK BOX’: ASSESSING TEACHING AND LEARNING EXCELLENCE AS A GAME-CHANGE Dirk Van Damme OECD/EDU/IMEP
- 4. Financial inputs in higher education increasing • Total (public & private) financial investment grew – Between 2005 and 2011 on average across OECD increase of 10% in per student expenditure and 27% in total expenditure – With huge differences between countries, increases higher in countries with below-average expenditure, catching up – Yearly per student expenditure is now 14 KUS$ – Total expenditure increased from 1.3% GDP in 2000 to 1.6% GDP in 2011 Inputs 4
- 5. United States Switzerland Denmark Sweden Norway Finland Netherlands Germany Japan Ireland Belgium FranceAustria Spain Israel United Kingdom Brazil Italy KoreaPolandPortugal Czech Republic HungaryChile Slovak RepublicMexico Estonia Iceland Russian Federation New Zealand Slovenia R² = 0.079 6 000 10 000 14 000 18 000 22 000 26 000 -30 -20 -10 0 10 20 30 40 50 60 70 Annual expenditure per student (2011, USD) Change in expenditure per student between 2005 and 2011 (%) OECD average OECD average Financial inputs in higher education increasing Inputs 5
- 6. • Private expenditure has increased a lot – 31% of total expenditure (0.5% GDP) comes from private sources, mainly tuition fees – Increase from 25% in 2000 – Total private expenditure increased with 32% since 2005 – >50% in Israel, US, Australia, Japan, UK, Korea and Chile Financial inputs in higher education increasing Inputs 6
- 7. • Increase in total per student expenditure slows down since crisis – Negative growth in almost half of countries between 2008 and 2011 – Expenditure cannot catch up with increasing student numbers • Increasing concerns about levels of private expenditure, student debt – 5 million international students investing even much higher amounts of money But strong signs of stagnating funding Inputs 7
- 8. 80 90 100 110 120 130 140 150 Estonia SlovakRepublic Chile Hungary Korea CzechRepublic Finland Slovenia Denmark RussianFederation Israel Japan UnitedKingdom Italy Poland OECDaverage EU21average Netherlands Switzerland France Sweden Germany Australia Brazil Spain Norway Mexico Belgium Portugal Austria UnitedStates Ireland Iceland Index of change (2008=100) Change in expenditure Change in the number of students (in full-time equivalents) Change in expenditure per student But strong signs of stagnating funding Inputs 8
- 9. But strong signs of stagnating funding Inputs • Efficiency and value-for-money become very important policy considerations – Both for governments and students/families – Cost of higher education becoming political issue in many countries • What are students actually ‘buying’? – Very weak relationship between cost and actual ‘product’, benefits and outcomes – Value-for-money depends enormously on institution and field of study 9
- 12. • Share of tertiary educated in the population increased, but still huge disparities across countries – 34% of adult population in OECD now has a tertiary qualification, up from 22% in 2000 Outputs Increases in graduate output 12
- 13. Outputs Increases in graduate output 0 10 20 30 40 50 60 China SouthAfrica Indonesia Brazil Turkey Italy Chile Mexico Portugal SlovakRepublic CzechRepublic Colombia Austria Hungary Poland Slovenia Greece Germany Latvia EU21average France Spain OECDaverage Netherlands Denmark Iceland Belgium Sweden Switzerland Estonia Norway Luxembourg Finland Ireland NewZealand UnitedKingdom Australia Korea UnitedStates Israel Japan Canada Russian… 2000 2012% 13
- 14. • Employment benefits remain high • On average across OECD academic degree gives a wage premium of 75% compared to upper secondary educated workers • Total net present value over lifetime is around 160 000 US$ for man, 100 000 US$ for woman – But with huge differences across fields of study • Or internal rate of return of 16% • High social benefits in health, interpersonal trust, volunteering, political participation Outputs Employment, earnings and social benefits remain high 14
- 15. Outputs Employment, earnings and social benefits remain high 15 0 50 000 100 000 150 000 200 000 250 000 300 000 350 000 400 000 450 000 500 000 Turkey Denmark Spain Estonia Sweden NewZealand Greece Korea Japan Canada SlovakRepublic Poland Norway Israel CzechRepublic France Australia Finland OECDaverage Portugal EU21average Austria UnitedKingdom Netherlands Italy Belgium Slovenia Germany UnitedStates Hungary Ireland EquivalentUSD Private net returns Public net returns
- 16. • Is graduate output higher than the economy’s need for high-skilled labour? – Graduate unemployment – Filtering-down effect? – Over-qualification and over-skilling – Huge field-of-study mismatches • Is polarization in labour markets, with high employment/high earnings because of skill- biased technological change going to last? Outputs But increasing concerns about quality of output 16
- 17. Outputs But increasing concerns about quality of output 17
- 18. Outputs But increasing concerns about quality of output • Enormous differences in the skills equivalent of tertiary qualifications – Foundation skills such as literacy and numeracy – Can societies accept that higher education graduates have low literacy and numeracy skills? 18
- 19. 19 Outputs But increasing concerns about quality of output
- 20. 20 0 5 10 15 20 25 30 35 40 Japan Finland Netherlands Sweden Australia Norway Flanders(Belgium) England(UK) England/N.Ireland (UK) UnitedStates CzechRepublic OECDaverage Poland Canada NorthernIreland(UK) Austria Germany Ireland France Denmark Estonia SlovakRepublic Korea RussianFederation Spain Italy Below upper secondary education Upper secondary or post-secondary non-tertiary education Tertiary education Proportion of 25-64 year-olds scoring at PIAAC numeracy level 4 and 5, by educational attainment of the population (2012) Outputs But increasing concerns about quality of output
- 21. 21 Outputs But increasing concerns about quality of output • Concerns about the quality and added-value of a university experience – Academically Adrift: limited improvement in academic skills – What is the relative contribution of selection versus teaching and learning in the production of high-quality graduates; what is the actual ‘learning gain’ – Doubts on the quality of the teaching and learning experience at elite universities
- 22. 22 Outputs But increasing concerns about quality of output
- 24. Inputs Increasing investments by taxpayers and families, but concerns about value- for-money Black box Outputs High numbers of graduates, still high returns, but concerns about quality 24
- 25. Inputs Increasing investments by taxpayers and families, but concerns about value- for-money Black box Outputs High numbers of graduates, still high returns, but concerns about quality ? 25
- 26. • What do we know about how quality of teaching and learning results in high-quality output, and socially interesting outcomes? • Information asymmetry: both public and private financers of higher education have very little understanding of what they actually are spending money for • Increase of investment has not been accompanied by an empowerment of the input side to make smart choices through better information Black box Black box problem is problem of transparency 26
- 27. • How to improve transparency? – Performance management systems: essentially bureaucratic control systems, very rarely comparative and tuned to better decision-making – Quality assurance arrangements: increasingly relativistic (‘fitness-for-purpose’), bureaucratic and inward oriented (internal QA), unfit for external transparency, nor empowering students – Student satisfaction surveys: empowering, but perception-based, largely unrelated to teaching and learning quality Black box Black box problem is problem of transparency 27
- 28. • How to improve transparency? – Input measures as proxies for teaching and learning quality: above a threshold level input has a weak relationship to quality (cfr PISA) – Research measurement and bibliometric indicators as proxies for teaching and learning quality: most inaccurate, unfair to wide range of institutions and institutional diversity – Reputations: act as (too?) powerful tools, also in rankings, but inaccurate, often outdated, inimical to newcomers and innovators Black box Black box problem is problem of transparency 28
- 29. • Sound metrics of teaching and learning are very much needed – To reassure governments and families about the value-for-money of investments – To reward institutions who invest in improving teaching and learning and are not compensated through other measures – To value institutional diversification – To reward and foster quality improvement through mutual learning – To compensate for the over-reliance of rankings on research and reputation metrics Black box Black box problem is problem of transparency 29
- 30. • What are the risks of not improving transparency? – Erosion of the symbolic power of degrees, the only monopoly of the higher education sector • Employers turning to alternative modes of selection • Emergence of alternative modes of qualification (employer credentials, badges, recognition of prior learning, etc.) Black box Black box problem is problem of transparency 30
- 31. Black box Black box problem is problem of transparency 31
- 32. • What are the risks of not improving transparency? – Blaming the student becomes the main excuse – Decreasing trust of governments, employers, families and wider society in the value of higher education • Degree inflation • Concerns about declining standards in some countries – Gradual erosion of the financial health of higher education institutions if value-for-money concerns are left unanswered • Financial bubbles of student debt – Markets no longer accept non-transparency • Cfr Volkswagen Black box Black box problem is problem of transparency 32
- 33. Inputs Increasing investments by taxpayers and families, but concerns about value- for-money Black box Outputs High numbers of graduates, still high returns, but concerns about quality ? 33
- 34. Inputs Increasing investments by taxpayers and families, but concerns about value- for-money Black box Outputs High numbers of graduates, still high returns, but concerns about quality Opening the black box 34
- 35. • We need to better understand what happens in the teaching and learning environments in universities – With the investment inputs provided – Explaining the benefits and returns and understanding why they sometimes fail – In order to incentivize improvements in teaching and learning – Empower students and families – Restore trust 35 Black box Opening the black box
- 36. • Comparative assessment of learning outcomes of graduates is the most promising approach to measure teaching and learning excellence – OECD’s AHELO project – National research projects in Germany, UK, Italy – CLA and various other initiatives in US – OECD-CEA partnership to implement CLA+ in countries – European Commission supported CALOHEE project in Tuning framework 36 Black box Opening the black box
- 37. • Strong resistance by parts of the academic community, but do they have a strong case? – No consensus on academic skills that matter – Risk of standardization – Institutional diversity too large to use limited number of metrics – Methodological concerns – Cost and burden • …exactly the same arguments used 20 years ago when the PISA programme was born • …and very similar to arguments used 10 years ago against measuring research excellence 37 Black box Opening the black box
- 38. • Developing reliable metrics of teaching and learning excellence in universities is the next big systemic challenge in the development of higher education worldwide • In the short term universities might think it’s not in their interest and that non-transparency is the better option • But in the longer term that might be a very risky approach, in which the costs largely exceed the short-term profits 38 Black box Opening the black box
- 39. • The future progress of universities will not come from – Continued massification and ever higher numbers of students – Continued increases in public and private funding • But will be the outcome of better and more visible quality improvements in teaching and learning • The impact of teaching and learning excellence will be similar to the way research excellence has changed the system 39 Finally
- 41. Thank you ! dirk.vandamme@oecd.org www.oecd.org/edu/ceri twitter @VanDammeEDU 41