Published on

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Last November, Times Higher Education did an extraordinary thing. Some say a foolish things. Others thought we were principled and brave. After six years of publishing a well known and well established global rankings table, we decided to start all over again – a new data provider, an entirely new database of global institutions, a new mission, a new methodology. It was a New Start, and of course, it produced some new – and controversial -- findings. Today, I am going to briefly explain WHY we started again. HOW we went about developing an entirely new way of benchmarking universities, and WHY we think we are close to establishing the most sophisticated and relevant global ranking system in the world. Then I’ll reflect on some of the findings of the first of a brand new exercise.
  • First, let me introduce my magazine. This is Times Higher Education. We were founded in 1971 as a tabloid newspaper – previously known as the Times Higher Education Supplement. We re-launched in 2008 as a weekly magazine, and become simply Times Higher Education – THE for short. We come out every Thursday.
  • But we are also a daily website. All news, opinions, book reviews and features from the weekly print magazine are published on-line every Thursday. But we also have daily higher education news updates and on-line opinions section. 100,000 unique visitors every week from all around the world. We received around one million visits in 24 hours on 16 September – rankings day.
  • And this is our parent company – TSL Education Ltd, based in Bloomsbury, London. You’ll see from the slide that all of our businesses are in education. Through our publications and websites, our mission is to provide authorative, robust and trusted information for all those in education. TES centenary this year – 100 years in the business. Our world rankings must meet those very high standards and that 100 year pedigree. As experts in education, we know that universities are too complex, and do too many amazing, intangible things, to be reduced to a relatively crude set of numbers. So why do we rank at all?
  • Well first of all, higher education is rapidly globalising. 3.3 million is OECD figure for 2008.
  • The world of higher education is changing. We believe strongly that rankings, despite their limitations, help us understand this dramatic process.
  • And rankings do serve a valid purpose. This slide shows a series of quotes from the US Institute of Higher Education policy on the influence of rankings. QUOTE Love them or hate them, rankings are here to stay.
  • And that same IHEP report also highlighted the influence and power of rankings.
  • Simon Marginson – a strong critic of rankings, but someone who sees their increasing power.
  • Here’s Ben Wildavsky. I think that the key point from this is “Refined and improved”.
  • So with rankings filling a very important global information void, and with their reach becoming ever greater, we felt we had a very serious responsibility. This is the position we were in, at the end of last year. This was the mission as laid out by my esteemed editor, Ann Mroz.
  • So to deliver on this mission, we brought in Thomson Reuters, the world’s leading research data specialist. A new brand was born. For us, it is a perfect partnership. Thomson Reuters brings exceptional expertise with data, especially with citations data. We bring 40 years of getting under the skin of the higher education world.
  • And by partnering with Thomson Reuters, we also got the expertise of Jonathan Adams. He is director of research evaluation for Thomson Reuters. He truly is a world leading expert – Australia, REF, Europe. Evidence Ltd. Leeds University. Government advisor. Framework 7, Australia, REF etc etc. Couldn’t ask for a more informed partner.
  • This is the foundation stone of the new rankings. Thomson Reuters has invested vast resources into creating its Global Institutional Profiles Project, with full-time staff stationed around the world to build highly So sophisticated data-driven profiles of more than 1,000 universities around the world. That project is not a ranking project, but the database it builds will be the foundation stone of our rankings.
  • One of the first things Thomson Reuters did is carry out a survey to find out what people thought of rankings, what they wanted from rankings. Interesting findings. People clearly valued and wanted rankings, but they were seriously concerned about the validity and rigour of the existing ranking systems, including the old THE-QS world university ranking. The survey also provided detailed feedback on indicators and proxies that were valued…
  • Some specifics: READ And we’ve delivered on this.
  • Times Higher Education also convened a special editorial board meeting – with some of the sector’s leading figures, to delve into the issue. Three very big concerns about the rankings that we had been publishing between 2004 and 2009. READ: After establishing what was wrong with our old rankings, we then we established a platform group, to scrutinise our proposals in very great deal. 50 people, in 15 countries, across every continent. What did we come up with?...
  • The first major improvement we made was a new reputation survey. We brought in professional polling company, Ipsos Mori. Really pleased with the results: Large numbers of responses. But size is not the only thing that counts. Good mix of experience – teaching and research Good geographical spread. Particularly pleased to see such a high response rate from Asia and the Middle East Good balance of subject areas. But that was only one of many, many major improvements.
  • Citations Data. TR owns the Web of Science. Excellent resource. But… It is not just the size and depth of the citations database, it’s the expertise and understanding of the data that comes with working directly with the data owners, Thomson Reuters. Not just buying in a lump of data! Also have the expertise to understand it. It will be normalised
  • And here are the data points we are collecting directly from the universities themselves. So the rankings from 2010 will be made up of three key data sets: reputation survey results; bibliometric data already held by Thomson Reuters, and data collected by Thomson Reuters from individual institutions. And now over to the hard part – how do we pull this all together into a final methodology?
  • Here it is. The final methodology for 2010-11. 13 indicators, across five broad areas of a university’s activity, covering the three core missions of every university: Teaching, research and knowledge transfer. New teaching section – proxies, but help us get a sense of the learning environment. 30 per cent. No real measures of teaching quality! AHELO perhaps in future. Reputation reduced to 35 per cent. Down from 50 per cent in old system. New income measures – no more than 10 per cent. Research 60 per cent. We are very proud of it, but we also know it is a work in progress.
  • We can not rely solely on research indicators. What about US and European institutions with excellent reputations but not as much research culture as Harvard and Oxford? We want a rounder picture. Also – quantity is nothing without quality. Resources: a uni that has $20,000 for every students will be in a position to deliver a richer environment than one with $10,000. Illogical to argue that it does not matter. A legitimate area of interest. Yes, hard to compare different systems, govt funds, private funds, competitive research grants vs block grants. But broad brush of total funds, and balance within that which applies to research. Normalisation by country! Controversy The results caused a shock, but with so many different innovations, We expect the results to be different from other systems.
  • Reduced role for reputation. Many more indicators
  • Early reactions have been very positive. But we know we have work to do. Citations indicator caused controversy – statistical outliers.
  • Mass media coverage all around the world. More than ever before. Hundreds and hundreds of news items.
  • And here is a brief summary the results. The top ten. US take all of the top 5 and 7 of top 10. UK does not do as well as under the old system.
  • Key themes: US dominance the overwhelming story – 72 institutions in the top 200. UK firmly holding on as the world’s second strongest higher education nation, but new methodology paints a less rosy picture than in the past. Asia rising. Strong showing for smaller research-intensives – quality not quantity. Strong showing for science and technology institutions. Income. More normalisation needed?
  • A visualisation of the top 100. Source: many eyes , 16 September 2010
  • Here’s a run down of country strength, compared against OECD data on the national spend on tertiary education. US 3.1 UK only 1.3 If there’s one quick conclusion from that, it is that money talks!
  • You can find out much more, and dig much deeper into our data, on our fully interactive website
  • And you can get even deeper into the results with our IPhone App. The app allows the user to filter the results against cost of living and tuition fees and other criteria, and it includes data on 400 institutions. But for me, the crucial feature is that you can manipulate our weightings. I think it is a crucial part of the rankers role to be as transparent and accountable as possible, so we want to give the user as much power as possible.
  • But to make sure we’re as accountable as it is possible to be, we need constant criticism and input. Only with the engagement of the higher education sector will we achieve a tool that is as rigorous and as transparent and as useful as the sector needs and deserves. So please use the sites and tools above to make sure you have your say and tell us what you think.

    1. 1. THE WORLD UNIVERSITY RANKINGS URAP ‘10, Middle East Technical University , 3 December, 2010 Phil Baty Editor Times Higher Education World University Rankings
    2. 2. About Times Higher Education The weekly magazine for all higher education professionals
    3. 3. About www.timeshighereducation.co.uk The dedicated website for higher education news, jobs and resources
    4. 4. About TSL Education Times Educational Supplement (TES) Times Higher Education (THE) TES Prime TSL Events TES Hirewire THE HireWire
    5. 5. Why Rank? Rapid globalisation of higher education <ul><li>There are 3.3 million students enrolled in higher education outside their country of origin, a 53 per cent increase since 1999 </li></ul><ul><li>Universities now have 162 satellite campuses outside their home countries, an increase of 43 per cent in just the past three years </li></ul><ul><li>Around 20 per cent of all academics working in the UK are appointed from overseas </li></ul><ul><li>Sir Drummond Bone said: “World class research is inherently international” </li></ul>
    6. 6. Why Rank? Rapid globalisation of higher education “ We are living through one of those tipping points where in five years, (commentators will say) that this was the period when the landscape changed for ever, when the speed of reputational growth and decline suddenly accelerated. “ We all accept that higher education is borderless - ideas know no boundaries, do not accord any significance to geography and maps - and that is equally true of reputations and university rankings.” Peter Upton Director, British Council, Hong Kong
    7. 7. Why Rank? Rankings have a useful function <ul><li>“ Rankings often serve in place of formal accreditation systems in countries where such accountability measures do not exist.” </li></ul><ul><li>“ Prompt change in areas that directly improve student learning experiences” </li></ul><ul><li>“ Encourage institutions to move beyond their internal conversations to participate in broader national and international discussions.” </li></ul><ul><li>“ Foster collaboration, such as research partnerships, student and faculty exchange programmes.” </li></ul><ul><li>US Institute for Higher Education Policy, May 2009 </li></ul>
    8. 8. Why Rank? “ The term world class universities has begun to appear in higher education discussions, in institutional mission statements, and government education policy worldwide” “ Many staffing and organisational decisions at institutions worldwide have been affected by ranking-related goals and outcomes.” “ Rankings play an important role in persuading the Government and universities to rethink core national values” US Institute for Higher Education Policy
    9. 9. Rankings: Increasing influence “ University rankings are powerful. They compel public attention and shape the behaviour of universities and policy makers.. The rankings date only from 2003 and 2004 but already they are everywhere in the sector and beyond. They set university reputations”. Simon Marginson, centre for the study of higher education, University of Melbourne .
    10. 10. Rankings: increasing influence “ Rankings are an unmistakable reflection of global academic competition… they seem destined to be a fixture on the global education scene for years to come… As they are refined and improved they can and should play an important role in helping universities get better.” Ben Wildavsky , The Great Brain Race (Princeton University Press, May 2010)
    11. 11. Times Higher Education’s responsibility “ The responsibility weighs heavily on our shoulders. We are very much aware that national policies and multimillion-pound decisions are influenced by the rankings…. We feel we have a duty to improve how we compile them. “ We believe universities deserve a rigorous, robust and transparent set of rankings – a serious tool for the sector, not just an annual curiosity.” Ann Mroz , Editor, Times Higher Education, November 2009
    12. 12. The development of a new world ranking system In November 2009 we signed a deal with Thomson Reuters , to work with us to develop and fuel a new and improved global ranking for the future.
    13. 13. Then what? “ In addition to unmatched data quality, Thomson Reuters provides a proven history of bibliometric expertise and analysis. We are proud that our data continues to be chosen by leading organisations around the world and we’re happy to provide insight and consultation on such a widely respected indicator,” Jonathan Adams , director of research evaluation, Thomson Reuters
    14. 14. Then what?
    15. 15. Thomson Reuters’ stakeholder survey. Key findings: * About 40 per cent globally said rankings were ‘extremely/very useful’ and a further 45 per cent said they were ‘somewhat useful’ * But methodologies were perceived unfavorably by many and there was widespread concern about data quality * 74 percent of respondents believe that institutions manipulate their data to move up in rankings.
    16. 16. Thomson Reuters’ stakeholder survey. Key findings: <ul><li>92 per cent said that faculty output (publications) was a must have/nice to have </li></ul><ul><li>91 per cent said that faculty impact (citations) was a must have/nice to have </li></ul><ul><li>86 per cent said they wanted faculty/student ratios </li></ul><ul><li>84 per cent wanted income from research grants </li></ul><ul><li>79 per cent wanted peer “reputation” measure </li></ul>
    17. 17. The development of a new world ranking system Consultative meetings. Key points: * Previous exercise (2004-2009) relied too heavily on subjective opinion (50 per cent of weighting) with weak sample * Previous exercise’s use of citations data biased against fields with lower average citations * SSR too crude as proxy for teaching quality
    18. 18. Dramatic improvements: reputation survey 13,388 responses breakdown
    19. 19. Dramatic improvements: citations Thomson Reuters’ Web of Science platform provides academics and university administrators with access to the world’s leading citation databases, covering: * 12,000 of the highest-impact academic journals * More than 110,000 conference proceedings
    20. 20. Dramatic improvements: more data variables Number of academic staff, including proportion who are of international origin Number of research-only staff Number of undergraduates admitted, including the proportion of international students Number of bachelor’s degrees awarded Number of doctoral students admitted, including the proportion funded by competitive research scholarships Number of doctorates awarded Total institutional income Research grant income from public sources and charities, and research contract income from industry and commerce
    21. 21. The 2010-11 methodology
    22. 22. The 2010-11 methodology: key innovations Research excellence is an ultimate arbiter of an internationally outstanding academic environment (but must be balanced with other indicators) Resources Scale Accounting for subject factors Accounting for international factors
    23. 23. Methodologies compared 100%   100%   TOTAL     10% Employer survey Reputation 2.50% Industry income     Knowledge transfer 2% Ratio of international to domestic students 5% Ratio of international to domestic students   3% Ratio of international to domestic staff 5% Ratio of international to domestic staff International mix 0.75% Public research income/total research income       4.50% Papers per academic and research staff       5.25% Research income (scaled)       19.50% Reputational survey research 40% Academic peer review Research 32.50% Citation impact – normalised 20% Total citations/numbers of researchers Citations 2.25% PhD awards/bachelor’s awards       2.25% Income per academic       4.50% Undergraduates admitted per academic 20% Ratio of staff to students   6% PhD awards per academic       15% Reputational teaching survey     Teaching Weighting 2010 and beyond Weighting 2004-2009 Area
    24. 24. Early reactions to the new system “ The new methodology employed by Times Higher Education is less heavily weighted towards subjective assessments of reputation and uses more robust citation measures. This bolstered confidence in the evaluation method.” Steve Smith , president, Universities UK “ I congratulate THE for reviewing the methodology to produce this new picture of the best in higher education worldwide.” David Willetts , UK minister for higher education and science “ Anything that looks at quality rather than simply quantity is something that plays to us. It’s great that Times Higher Education has recognised what we’ve been trying to do.” David Turpin , president, University of Victoria “ This year Times Higher Education consulted widely to pinpoint weaknesses in other ranking systems and in their previous approach…. These are welcome developments.” David Naylor , president, University of Toronto “ The data is likely to be more reliable, and the methodology more rigorously framed and implemented.” Kris Olds, Professor, University of Wisconsin Madison, and editor of the Global Higher Education blog
    25. 25. More media interest than ever before UK: BBC Radio 4 Today Programme, BBC Breakfast News, Daily Telegraph, Daily Mail and The Guardian US: Huffington Post, Bloomberg, Business Week and Christian Science Monitor, Forbes Canada: The Montreal Gazette, The Vancouver Sun, The Globe & Mail and Toronto Sun Europe: La Tribune (front page), Le Figaro, Liberation (France) Irish Independent, Irish Times, Der Standard (Austria), Der Spiegel, Bilt (Germany), El Pais (Spain), De Standaard (Belgium) Far-east: Mainichi Daily News, Wall Street Journal (Japan) Malaysia Star, China News, China Times, Focus Taiwan and Korea Daily Australia: Sydney Morning Herald, The Age
    26. 26. THE World University Rankings 2010-11: Results 1. Harvard University 2. California Institute of Technology 3. Massachusetts Institute of Technology 4. Stanford University 5. Princeton University 6. University of Cambridge/University of Oxford 8. University of California, Berkeley 9. Imperial College London 10. Yale University
    27. 27. THE World University Rankings 2010-11: Results 112 Bilkent University 183 Middle East Technical University
    28. 28. THE World University Rankings 2010-11: Results
    29. 29. THE World University Rankings 2010-11: Results
    30. 30. THE World University Rankings 2010-11: Results
    31. 31. Rankings website: http://bit.ly/thewur
    32. 32. THE World University Rankings: IPhone App
    33. 33. Over to you • Visit the Global Institutional Profiles Project website: http ://science.thomsonreuters.com/globalprofilesproject • See the results in full, with our interactive tables: http://bit.ly/thewur • Join our rankings Facebook group. www.facebook.com/THEWorldUniRank • Keep up to date with all the rankings news on Twitter: @THEWorldUniRank
    34. 34. Thank you. Stay in touch. Phil Baty Times Higher Education T. 020 3194 3298 E. phil.baty@tsleducation.com