2. Teaching Excellence Framework: Lessons Learned
• Process fair and transparent
• Clear and robust findings on basis of metrics,
submissions and guidance
• Balance of evidence between metrics and
submissions broadly right
• No significant biases
• No changes to overall structure or methodology
• Some areas for improvement identified to
strengthen accountability and fully recognise
excellence.
3. TEF – the Chair’s 10 conclusions
• TEF focuses on things surveys say matter to students: routes to work, retention,
assessment, quality of teaching.
• TEF is part of an increased focus on outcomes for students, shifting from the focus on
inputs, processes and outputs.
• Metrics matter, but need to be benchmarked. Metrics generate hypotheses and
anchor initial judgements
• At the core of TEF is a strategic clarity: the relationship between institutional policies,
institutional practices and student outcomes.
• The best accounts did not describe initiatives but systematically demonstrated the
difference they had made.
• Initiatives carried conviction when part of a coherent strategy for improvement.
• Technology tools matter less than how tools are used.
• Genuine student involvement stood out: an embedded culture of engagement at
every level.
• Too many institutions used context as excuse rather than analysing and responding to
the challenges it offered.
• Excellence is distributed across the sector and matches its diversity.
4. Principal refinements for TEF Year Three
Core:
• NSS weighting; part-time provision;
high & low absolute values
Supplementary:
• LEO & grade inflation
Link with other processes:
• Director of Fair Access;
Quality Assessment
There will be no changes to the overall structure or methodology of the TEF
There will be the same method and process of assessment, with peer review
central and the same criteria, ratings, core metrics and the same use of the
provider submission.
5. TEF Year Three: Key dates
Date Activity
9 October 2017 DfE publish TEF specification and Lessons Learned report
19 October 2017 HE providers have access to their metrics on TEF extranet
HEFCE publishes TEF Year Three procedural guidance
TEF extranet opens for provider applications to TEF Year Three
November 2017 Provider webinars and briefing events
18 January 2018 Deadline for provider applications
Early June 2018 Outcomes published
6. Subject Pilot specification
Model A
Top-down, ‘by exception’ model gives provider-level rating and subjects
the same rating as provider where metrics the same, with fuller
assessment (and potentially different ratings) where metrics differ.
Model B
‘Bottom-up’ model gives subject-level ratings, feeding into the
provider-level assessment and rating. Subjects are grouped for
submissions, but ratings are awarded at subject-level.
Teaching intensity - piloting two measures in a selection of
subjects in both models
7. Subject pilot provider recruitment
83 providers have volunteered:
• 77 England
• 2 Scotland
• 1 Northern Ireland
• 3 Wales
Mix of provider types
• 46 HEIs (including 5 small and specialist)
• 22 FECs (both franchised and own provision)
• 15 Alternative providers
A good spread of providers showed interest
in participating in either one or both models
8. Panel member recruitment
• 3 groups of applicants
• 3 types of panel
• 15 types of role
• 100 vacancies
No of applications
Experts (employer/PSRB/WP) 85
Academic panel members / assessors 595
Student panel members / assessors 169
9. Future of TEF
• Ongoing Evaluative work
• 1 January 2018 HERA clause 25 commences 1 January 2018
• January to April 2018 HEFCE implements TEF on behalf of OfS
• April 2018 OfS responsible for TEF
• AY 2018/19 Independent Review of TEF
• AY 2019/20 TEF subject level specified (5 year cycle)
10. Copyright
The copyright in this presentation is held either by the
Higher Education Funding Council for England (HEFCE) or
by the originating authors.
Please contact customerservices@hefce.ac.uk for further
information and re-use requests.
11. Thank you for listening
c.millward@hefce.ac.uk
Follow us on Twitter at
#hefcecon2017
Editor's Notes
The operation of Year Two of the TEF has demonstrated that the process has operated in a fair and transparent way.
Assessors were able to come to clear and robust findings on the basis of the metrics, provider submissions and guidance; the balance of evidence between metrics and submissions was broadly right; the moderation process worked well; and the results are generally perceived as credible and reflecting teaching excellence across the sector. Statistical analysis of the results has confirmed that there were no significant biases by region, gender, ethnicity, social disadvantage, subject breakdown and research income.
We are therefore making no changes to the overall structure or methodology of the TEF, and will be retaining the method and process of assessment, the centrality of peer review, the criteria, the ratings, the core metrics and the use of the provider submission.
The lessons learned exercise did, however, also highlight a number of areas for improvement: either refinements to the existing assessment process to ensure excellence is fully recognised, or steps to strengthen the way that TEF holds providers to account for delivering excellent teaching.
TESOF – may need to change this logo…..
The NSS remains a key component of TEF but, in order to achieve a more balanced assessment, the weight of each NSS metric will be halved for the purposes of determining the initial hypothesis.
Part-time provision
In order to recognise excellence in part-time provision appropriately, we will offer providers with a majority of part-time provision (≤35%) the opportunity to submit an additional page of quantitative information alongside their metrics and refine the assessment procedure for providers with similar numbers of full-time and part-time students.
Very high and low absolute values
Whilst benchmarking will remain at the heart of TEF assessment, the flagging and benchmarking system has limitations at the extreme ends of the metrics. We will therefore explicitly mark the top and bottom 10% of absolute values for each metric. These indicators will be taken into account in the calculation of the initial hypothesis, where a provider is not already flagged.
Longitudinal Education Outcomes (LEO) data
We will include supplementary metrics derived from LEO data in order to exploit the rich data about graduate employment now available. As supplementary metrics, these will not affect the initial hypothesis but will be considered alongside the provider submission.
Grade inflation metric - DfE very keen on implementing this strictly…. Expect it to be mandatory that you declare it if you have it. (but don’t worry if you don’t have it) and it will have to be only for providers with DAPs and presumably only for those students that the provider with DAPs teaches? Check spec carefully on this aspect when it is published.
A new supplementary metric on grade inflation will be linked to the existing criterion on ‘Rigour and Stretch’ to aid assessors in making judgement in this area and allow providers that are taking genuine steps to tackle grade inflation to be recognised for doing so.
Allow the Director for Fair Access, or successor, to comment on whether ‘gaming’ has taken place
Power of referral
Where the assessment process suggests concerns about the underlying quality of a provider, the TEF Chair will have the right to refer the provider to the Office for Students with a recommendation that the OfS should consider an investigation to confirm whether the provider continues to meet baseline quality requirements.
Plus
01 November 2017 Webinar: TEF Year Three Metrics
02 November 2017 Webinar: Introduction to TEF Year Three
7, 10, 14, 15 & 17 November TEF Year Three Briefings - Invite only events (1 member of staff & 1 student from each eligible provider will be invited to attend)
and
February - May 2018 Assessment period
and
June - July 2018 Appeals process
August 2018 Publication of any changes to TEF outcomes as a result of appeals
WP and Employment experts and Employer/PSRB representatives
3 broad categories of applicants
15 different role types across TEF 3 and Pilot Main and Subject Panels
110 vacancies
Transition to the Office for Students and Independent Review 12. The Government intends to commence Clause 25 of the Higher Education and Research Act 2017 (HERA) on 1 January 2018, in line with the creation of the Office for Students (OfS). Between January and April 2018, HEFCE will implement the TEF on behalf of the OfS and, from April, the OfS will be solely responsible. This will not cause disruption for providers involved in the process or for students. 13. We therefore anticipate the Independent Review will take place in academic year 2018/19, in line with the timetable set out in HERA. We intend for the Independent Review to report in time to determine the assessment specification for 2019/20, which will also be the first year of assessments at subject-level. 14. The assessments taking place in academic year 2019/20 will therefore constitute the completion of the TEF development process. At this point, subject to the findings of the Independent Review, we anticipate that TEF will move to a five-year cycle: TEF ratings will last for five years, with providers able to reapply for assessment either three, four or five years after their last application. This will deliver value for money for the sector and be a proportionate approach to adopt once the TEF has been independently reviewed.