Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
AGL FORUM - LEAGUE TABLES, AND
OTHER THINGS WITH NUMBERS
@mikehamlyn
blogs.staffs.ac.uk/mikehamlyn/
HOW DO WE SEE OURSELVES?
A reflection
But distorted through
commercial algorithms
Not easy to reverse engineer
There are n...
COVERAGE
• Why is this important?
• What does the research say?
• Some league table results and
methodologies
• What can w...
IMPORTANCE
• Strategic plan
• Impact on recruitment?
• Feelgood factor
• Measures of what we should want to
improve
PROBLEMS WITH LEAGUE
TABLES
• from VC of Greenwich
– Counting what can be measured rather than
measuring what counts
– Off...
IMPACT OF NSS, LEAGUE TABLES?
• The study shows that NSS scores do matter for university applications. However,
contrary t...
…BUT WE STILL WANT TO BE
SUCCESSFUL
• Want to have a great student experience
• Students who achieve and can gain
employme...
3 TABLES,
1 SET OF DATA,
3 RESULTS
Year Complete
University
Guide
Guardian
Guide
Good
University
Guide (was
Sunday
Times)
...
WHAT IS BEING MEASURED –
AND HOW?
What?
• Spend (HESA return)
• Staffing (HESA return)
• Good degrees (HESA)
• Retention (...
CUG2016 (103RD IN 2016 UP FROM 105TH )
2016 table 2015 table movement
Entry Standards 118/126 116/123 
Student Satisfacti...
2010 2011 2012 2013 2014 2015 2016
position 80 80 99 108 113 105 103
0
20
40
60
80
100
120
CUG league table position
2010 ...
GUARDIAN 2016, 83RD FROM 90TH
year position Average
Teaching
Score
NSS
Teaching
(%)
NSS
Overall
(%)
Expendit
ure per
stude...
2010 2011 2012 2013 2014 2015 2016
NSS Teaching (%) 82 80 82 83 86 85.9 86.8
NSS Overall (%) 78 82 80 85 83.6 84.6
NSS Fee...
As well as
Coventry’s meteoric
rise, we can see a
number of our
comparators
performing less well
this year,
particularly
S...
TIMES GOOD UNI GUIDE
Performance Measure Score Rank
Teaching Quality 81.80% 49=
Student Experience 82.80% 80=
Research Qua...
HOW TO DO WELL IN A LEAGUE
TABLE?
• Data returns
– Appropriate returns for staffing, students and
finance to HESA
– Approp...
LESSONS?
• Review document to see subject level
scores
• Note – these will not necessarily relate to
AM data or NSS scores...
METRICS MATTER INTERNALLY
• Portfolio Performance Review
• QBR and Business Insight
• Raising attainment roadmap
– Data on...
GOOD DEGREES AND ETHNICITY
(it doesn’t matter what your previous background was,
there is a systemic problem…)
FULFILLING OUR POTENTIAL. TEACHING
EXCELLENCE, SOCIAL MOBILITY AND
STUDENT CHOICE
• Productivity challenge
• Transparency ...
READING THE GREEN PAPER
• Introduction of TEF
– Assessment of TEF
– Criteria and metrics
– Social mobility and WP
• The HE...
TEACHING EXCELLENCE
FRAMEWORK
• The teaching excellence framework (TEF) is set to allow most English
universities to incre...
http://wonkhe.com/blogs/the-incredible-machine-our-visual-guide-to-the-tef/
METRICS WILL MATTER EXTERNALLY….
NEW PROVIDERS
• Removing barriers to entry – eg need for 4 years track
record prior to DAP
• University title not limited ...
THE FUTURE HE LANDSCAPE
AN END OF HEFCE?
• New Office for Students
• Joining together HEFCE and OFFA
• What about other sector organisations?
• De...
OUR RESPONSE?
• Read the Green paper!
• Engage in the workshops to decide the university
response
• Read WonkHE – excellen...
CONCLUSIONS
• League tables are here to stay
• Externally, metrics will become more important
• Internally we can learn to...
Agl forum november 2015 league tables
Upcoming SlideShare
Loading in …5
×

Agl forum november 2015 league tables

545 views

Published on

Presentation to the StaffsUni Academic Group Leader Forum on league tables and metrics

Published in: Education
  • Be the first to comment

  • Be the first to like this

Agl forum november 2015 league tables

  1. 1. AGL FORUM - LEAGUE TABLES, AND OTHER THINGS WITH NUMBERS @mikehamlyn blogs.staffs.ac.uk/mikehamlyn/
  2. 2. HOW DO WE SEE OURSELVES? A reflection But distorted through commercial algorithms Not easy to reverse engineer There are no facts, only interpretations
  3. 3. COVERAGE • Why is this important? • What does the research say? • Some league table results and methodologies • What can we learn? • How to “win” • Some other things with numbers – (the spending review is being announced right now……)
  4. 4. IMPORTANCE • Strategic plan • Impact on recruitment? • Feelgood factor • Measures of what we should want to improve
  5. 5. PROBLEMS WITH LEAGUE TABLES • from VC of Greenwich – Counting what can be measured rather than measuring what counts – Offering one view of the world – Heavily skewed by research performance – Produced by media organisations to sell digital and print media • (also mainly about FTUG……..) http://blogs.gre.ac.uk/vc/2014/12/22/university-league-tables-part-1-everyone-loathes-unless-theyre-top/
  6. 6. IMPACT OF NSS, LEAGUE TABLES? • The study shows that NSS scores do matter for university applications. However, contrary to what many university managers might think, the effect of changes in NSS scores on demand for places is quite small. Moving from the bottom of the scale (around 65% satisfaction) to the top of the scale (about 95% satisfaction) will only result in a degree course gaining about seven more applicants for every 100 it already receives. And it isn’t the separately published NSS data themselves which matter. The timing of the effects of the NSS scores, relative to their publication, reveals that the NSS impact on student demand works indirectly by affecting university rankings in published league tables, such as The Times Good University Guide. • A likely explanation for this result is that leagues tables, which rank departments on a single scale, are more readily available and easily understood by applicants than more complex NSS data. In other words, the format in which information is presented to prospective students matters. But here too the effects are small: a 10 place move up a table of 100 universities only increases applications by around 2-3%. An improvement in position encourages a slightly more able pool of applicants (based on A- level tariff points). The position effect of changes in league table is also slightly bigger in subjects and places where there are more providers from which to choose, and amongst universities that are in the middle in terms of their entry qualifications http://blogs.lse.ac.uk/impactofsocialsciences/2015/11/10/student-satisfaction-league-tables-and-university-applications/
  7. 7. …BUT WE STILL WANT TO BE SUCCESSFUL • Want to have a great student experience • Students who achieve and can gain employment • Impact on recruitment from overseas • Impact on partnership activity
  8. 8. 3 TABLES, 1 SET OF DATA, 3 RESULTS Year Complete University Guide Guardian Guide Good University Guide (was Sunday Times) 2012 99th 77th 105th 2013 108th 96th 107th 2014 113th 92nd 108th 2015 105th 90th 101st 2016 103rd 83rd 95th
  9. 9. WHAT IS BEING MEASURED – AND HOW? What? • Spend (HESA return) • Staffing (HESA return) • Good degrees (HESA) • Retention (HESA) • Experience (NSS) • Research (REF) • Employability (DLHE) Who? •  •  •  •  •  •  • 
  10. 10. CUG2016 (103RD IN 2016 UP FROM 105TH ) 2016 table 2015 table movement Entry Standards 118/126 116/123  Student Satisfaction 75 72  Research Assessment/Quality 104 111  Research - impact 64 n/a n/a Graduate Prospects 109 102  Student staff ratio 64 62  Academic Services Spend 55 82  Facilities Spend 100 66  Good Honours 97 103  Degree Completion 115 102  Overall 103 105 
  11. 11. 2010 2011 2012 2013 2014 2015 2016 position 80 80 99 108 113 105 103 0 20 40 60 80 100 120 CUG league table position 2010 2011 2012 2013 2014 2015 2016 Good Honours 53 50.8 52.4 50.6 56.3 58.5 63.2 0 10 20 30 40 50 60 70 Good Honours 2010 2011 2012 2013 2014 2015 2016 Student Satisfaction 3.8 3.8 3.9 3.9 4 4.02 4.04 3.5 3.7 3.9 4.1 4.3 4.5 4.7 4.9 Student Satisfaction 2010 2011 2012 2013 2014 2015 2016 Graduate Prospects 69 67.4 60.7 58.9 48.9 53.6 54.7 0 10 20 30 40 50 60 70 80 Graduate Prospects
  12. 12. GUARDIAN 2016, 83RD FROM 90TH year position Average Teaching Score NSS Teaching (%) NSS Overall (%) Expendit ure per student / 10 Student:s taff ratio Career prospect s (%) Value added score/10 Entry Tariff NSS Feedback (%) 2010 55 59 82 7 19.1 65 5.2 249 63 2011 69 54.6 80 78 6.3 19.0 63 5.2 236 65 2012 77 51.9 82 82 5.7 21.7 56 4.7 239 66 2013 96 45.1 83 80 6.0 21.6 55 4.0 246 68 2014 92 48.6 86 85 5.52 20.3 48 3.8 254 72 2015 90 48.8 85.9 83.6 5.1 17.7 55.3 4.2 273 71.1 2016 83 56.0 86.8 84.6 6.2 17.0 55.7 4.9 274.1 72.9
  13. 13. 2010 2011 2012 2013 2014 2015 2016 NSS Teaching (%) 82 80 82 83 86 85.9 86.8 NSS Overall (%) 78 82 80 85 83.6 84.6 NSS Feedback (%) 63 65 66 68 72 71.1 72.9 60 65 70 75 80 85 90 95 100 %Result NSS Performance 2010 2011 2012 2013 2014 2015 2016 Career prospects (%) 65 63 56 55 48 55.3 55.7 0 10 20 30 40 50 60 70 80 90 100 %graduateprospects Career prospects (%)
  14. 14. As well as Coventry’s meteoric rise, we can see a number of our comparators performing less well this year, particularly Sunderland, Plymouth, Derby, UCLAN and Glyndwr.
  15. 15. TIMES GOOD UNI GUIDE Performance Measure Score Rank Teaching Quality 81.80% 49= Student Experience 82.80% 80= Research Quality 16.50% 55 UCAS Entry Points 274 118 Graduate Prospects 58.40% 111 Firsts and 2(i)s 63.20% 98= Completion Rate 78.40% 115 Student-Staff Ratio 16.8:1 59= Services/Facilties Spend £1,620 81 Note difference from CUG! Note uses latest data
  16. 16. HOW TO DO WELL IN A LEAGUE TABLE? • Data returns – Appropriate returns for staffing, students and finance to HESA – Appropriate recording of graduate destinations in DLHE • In Faculty – Focus on student experience and outcomes • University – Support for employability, experience, attainment
  17. 17. LESSONS? • Review document to see subject level scores • Note – these will not necessarily relate to AM data or NSS scores as we know them • Issues: – Entry tariff – Good degrees – Graduate employment <
  18. 18. METRICS MATTER INTERNALLY • Portfolio Performance Review • QBR and Business Insight • Raising attainment roadmap – Data on low performing (and overly high performing) modules – Observations of teaching – Improved data on evaluation of modules – Appraisals • More useful annual monitoring • Data on student cohorts – eg BME, WP etc
  19. 19. GOOD DEGREES AND ETHNICITY (it doesn’t matter what your previous background was, there is a systemic problem…)
  20. 20. FULFILLING OUR POTENTIAL. TEACHING EXCELLENCE, SOCIAL MOBILITY AND STUDENT CHOICE • Productivity challenge • Transparency challenge • Driving up teaching standards • Boosting social mobility • Fairer deal for new and existing providers • Reforming higher education architecture • Reducing complexity and bureaucracy in research funding Johnson - purpose of the TEF is ‘to accelerate market shift’ in HE provision and to encourage ‘market exit’ in areas where some providers are uncompetitive. Has described some teaching as “lamentable”
  21. 21. READING THE GREEN PAPER • Introduction of TEF – Assessment of TEF – Criteria and metrics – Social mobility and WP • The HE Sector – Opening up to new providers – Provider exit and student protection • Simplifying architecture – Office for students – Further deregulation • Reducing complexity in research funding
  22. 22. TEACHING EXCELLENCE FRAMEWORK • The teaching excellence framework (TEF) is set to allow most English universities to increase their tuition fees in line with inflation from 2017-18, with institutions being invited to apply the following year for higher awards that pave the way for variable fees. • Initial gateway is successful QA review • Initial metrics proposed for the higher levels of the TEF include the National Student Survey (NSS) on teaching quality and the learning environment, graduate employment figures from sources such as the Destinations of Leavers from Higher Education (DLHE) surveys and information on student retention published by the Higher Education Statistics Agency (Hesa). • (this looks a bit like a league table) • Success at higher levels of the TEF would also be linked to meeting targets on helping students from disadvantaged backgrounds. • Is there really a problem with HE teaching? And do these metrics help? https://www.timeshighereducation.com/news/higher-education-green-paper-glance
  23. 23. http://wonkhe.com/blogs/the-incredible-machine-our-visual-guide-to-the-tef/ METRICS WILL MATTER EXTERNALLY….
  24. 24. NEW PROVIDERS • Removing barriers to entry – eg need for 4 years track record prior to DAP • University title not limited by restrictions on size or location of student body (currently 1000) • I am opening the University of Shelton • Above the chip shop • Personal view – this is the narrative of a deregulatory government who believe that markets operate in all spheres • Will also be more protection for providers who exit ( as would be needed in increasingly marketised HE environment)
  25. 25. THE FUTURE HE LANDSCAPE
  26. 26. AN END OF HEFCE? • New Office for Students • Joining together HEFCE and OFFA • What about other sector organisations? • Dependent partly on spending review • Greater flexibility for HECs (eg us)
  27. 27. OUR RESPONSE? • Read the Green paper! • Engage in the workshops to decide the university response • Read WonkHE – excellent set of resources and commentary, fast becoming a significant HE voice
  28. 28. CONCLUSIONS • League tables are here to stay • Externally, metrics will become more important • Internally we can learn to use metrics to help shape our future • But the future should be more complex than FTUG and NSS/DLHE @mikehamlyn blogs.staffs.ac.uk/mikehamlyn/

×