Download to read offline
Uploaded bySEDA
Louise Naylor
This document discusses measuring teaching excellence and learning gain in higher education. It defines learning gain as the improvement in students' skills, knowledge, and development between two points in time. Teaching excellence is defined broadly as high-quality teaching, learning environments, and student outcomes. The document examines potential metrics to measure these concepts and challenges around benchmarking standards across institutions. It also summarizes early findings from a national study measuring learning gain at multiple universities.
More Related Content
![Teacher Formation - Hong Kong Workshop [Peter Knight and Jo Tait]](https://cdn.slidesharecdn.com/ss_thumbnails/teacher-formation-hong-kong-workshop-peter-knight-and-jo-tait-12317-thumbnail.jpg?width=640&height=640&fit=bounds)
![‘[T]here were actually some individuals speaking within those groups […] who ...](https://cdn.slidesharecdn.com/ss_thumbnails/careylove-day2v2-250530135103-f9d244cf-thumbnail.jpg?width=640&height=640&fit=bounds)
Louise Naylor
- 1. The UK’s Europeanuniversity The Holy Grail for HE: achieving transformational learning (gain) through excellent teaching? Dr Louise Naylor 12 May, 2017 l.h.naylor@kent.ac.uk
- 2. In search ofthe Holy Grail….. The question of how to measure the student learning gained in higher education has been much debated. The debate is healthy — any attempt to measure a process which is as multidimensional as learning can encourage a focus on what is effective — but it also risks distorting activity. There is an evident need to consider different approaches to the measurement of gains in learning and consider their impact on practice and on outcomes. Debate on the measurement of the learning students gain in higher education has been prominent internationally, but the concept has not been studied extensively in the English higher education context. McGrath et al (2015) Learning Gain in HE (RR-996-HEFCE) The Holy Grail for HE - teaching excellence and learning gain
- 3. Definitions Learning Gain The conceptof ‘learning gain’ is defined as the ‘distance travelled’, or the difference between the skills, competencies, content knowledge and personal development demonstrated by students at two points in time. This allows for a comparison of academic abilities and how participation in higher education has contributed to such intellectual development (‘value-added’). McGrath et al., RAND report (2015) The Holy Grail for HE - teaching excellence and learning gain
- 4. Definitions Teaching Excellence We takea broad view of teaching excellence, including the teaching itself, the learning environments in which it takes place, and the outcomes it delivers. We expect higher education to deliver well designed courses, robust standards, support for students, career readiness and an environment that develops the ‘soft skills’ that employers consistently say they need. These include capacity for critical thinking, analysis and teamwork, along with the vital development of a student’s ability to learn. (p43) The Holy Grail for HE - teaching excellence and learning gain
- 5. ‘Great teaching increasesthe likelihood of good outcomes’ Learning Gain (ie student outcomes) Improvement in: knowledge skills work-readiness (employability) personal development Teaching Excellence (ie staff performance) • ‘Innovative, inspiring, engaging, stimulating, challenging’….. • Good or ‘best’ practice/exemplars • Contextualised (discipline, level, mode) Gunn & Fisk (2013) The Holy Grail for HE - teaching excellence and learning gain
- 6. Choosing the Metrics– Match up Learning Gain Grades (GPA) - entry to output PDP portfolio Progress tracking (ALIS) Surveys – NSS, UKES Careers registration • Other? Teaching Excellence NSS DLHE HESA (non-continuation) • Teaching qualifications? • Student engagement? • Contact time? • HMRC? (Salary) • Other? The Holy Grail for HE - teaching excellence and learning gain
- 7. TEF Year 2– student outcomes/learning gain Criteria • Employment and Further Study • Employability and Transferable Skills • Positive Outcomes for All Examples of evidence Learning gain and distance-travelled by all students Career enhancement and progression/impact of initiatives aimed at graduate employability Impact of initiatives aimed at preparing students for further study and research Longer-term employment outcomes and progression of graduates including into highly-skilled employment Extent of student involvement in enterprise and entrepreneurship Number, impact and success of graduate start-ups Use and effectiveness of initiatives used to help measure and record student progress, such as Grade Point Average (GPA) Impact of initiatives aimed at closing gaps for students from different backgrounds The Holy Grail for HE - teaching excellence and learning gain
- 8. Critical thinking &problem- solving Attitudes towards study experience Engagement 12 test questions exploring respondents ability to: • summarise the main conclusion (CT) • assess the impact of additional evidence (CT) • apply principles (CT) • detect reasoning errors (CT) • identify similarity (PS) • find procedures (PS) • make relevant selections (PS). Three sets of survey questions exploring: • Positive attitude toward Literacy (6 questions) • Openness to diversity (7 questions) • Academic motivation (8 questions) Three sets of survey questions exploring: • course emphasis on critical thinking • interaction with staff • time spent between various academic and non-academic activities source: Cambridge Assessment source: Pascarella/Wabash source: HEA/UKES questions 1, 5 and 18 2016-17 (1st term) 2016-17 (3rd term) 2017-18 (3rd term) 2018-19 (3rd term) 2021-22? 1. Critical thinking & problem-solving 2. Attitudes towards study experience 1. Critical thinking & problem-solving 2. Attitudes towards study experience 3. Engagement Link results to graduate outcomes? NMMLGP - 10 HEIs TEST REPEATED
- 9. NMMLGP – earlyfindings Achievements Institution Emails sent Response flag Total Response rate (full response)Full response Partial response University of Kent 3,211 273 115 388 9% Challenges • Timescale for launch • Student engagement (7%) • Incomplete surveys (~30%) • Anonymity vs support • Longitudinal attrition (3yrs) • Benchmarking options: • None; All responses; Institution only; All responses + institution • Initial results The Holy Grail for HE - teaching excellence and learning gain In addition, the table below gives an overview of the response rate for your institution:
- 10. Choosing the Metrics– Mix Up 0 1 2 3 4 5 6 7 8 9 METRICS LG Metric TE Metric LG + TEF • Place in rank order your most preferred (3) to least preferred metric (1) for learning gain and teaching excellence • Choose ideal combined rank eg • no overlap -1 • full synergy – 9 The Holy Grail for HE - teaching excellence and learning gain
- 11. Next steps forTEF* ….. • ‘Universal condemnation of the TEF’ • ‘Proposed metrics are factually flawed’ • ‘Written submissions as opening the way to game-playing that characterised the excesses of the REF’ • ‘Only 4% academics believe it will accurately assess teaching quality’ (so not excellence then?) *THE Poppletonian - 9 March 2017 UUK Survey (83) Submission: SMT (55%); Senate (22%) Steering Group - 88% Average staff - 27 TEF judgement -76% (core) Impact: Accelerated plans (47%) ‘Proxy’ metrics; raised profile of teaching Future: Viability of subject level? Areas for development: • student use (79%) • new metrics & assessment (53%) • reduce burden (47%) The Holy Grail for HE - teaching excellence and learning gain
- 12. and for LearningGain….. Supporting excellence and improving student outcomes The Holy Grail for HE - teaching excellence and learning gain #learninggain
- 13. The ‘elusive’ HolyGrail in HE National Benchmarked Standards: • How/can these be defined? Definitions • How/should they be measured? Metrics • Is it possible to benchmark? Comparability • Will these influence the student experience in a positive way? Impact • Will this differentiate the sector as planned? Market • Will this provide information students want or understand? Communication The Holy Grail for HE - teaching excellence and learning gain
- 14.
Editor's Notes
- #2 With the introduction of the Teaching Excellence Framework as part of the HE&R Bill and the ‘quest’ for comparable national metrics to measure Learning Gain, the sector is being challenged to reconcile the relationships between these two concepts using quantitative and qualitative assessments. Although the HE Bill has now gone through Parliament (now Act), the outcomes of TEF2 are delayed to mid June (post election) and our LG project has only just started, so this workshop will explore through our own experience to date, whether these exercises are likely to provide appropriate ways of conceptualizing and differentiating the sector and more importantly, whether the links are as simple as suggested and will achieve the desired outcomes in terms of enhancing the student learning experience. This session will explore: 1. Definitions & 2. Metrics - for TE and LG 3. Consider how these might be combined to benefit the Student Learning Experience
- #6 P43- The report goes onto assert that ‘great teaching increased the likelihood of good outcomes’ but if we look more closely at the two concepts, one is about student outcomes (example measures) and the other about staff performance in teaching (example measures) – are these two inextricably linked and if so, in what ways? SEDA list (March 2017)– quite a debate about when the term T&L first appeared and relationship between the two; which comes first ‘chicken or egg’; are these mutually dependent or independent? It is possible that we can have: ‘Learning without teaching and teaching without learning’ and even the DfE publication hedges bets on this – notes ‘increases the likelihood’ but not necessarily causative correlation between teaching and learning outcomes Gunn and Fisk (2013) - Highly contextualised and individual dependent – suggest that no ‘one size fits all’ and warn that chasing national benchmark standards will be difficult
- #7 Learning Gain RAND p64-68 Mix of both qualitative and quantitative methodologies (these are some current indicators, but you will have others): Grades – problem with comparability across sector/ relating tariff entry to outputs not always direct (not a good predictor) PDP – students reflect on development but highly individualised and not comparable even within institutions Progress tracking- diagnostics skills audit – measures only skills aspect of LG and often done at discipline not institutional level (except Med Schools) – how can this be standardised? Mixed methods eg ALIS – Advanced Level Information System combines grade and test results as predictor of students’ performance – provides score of likely achievement, rather than Learning Gain NSS administered nationally (UK-wide) but it is about ‘satisfaction’ with teaching & learning environment/ does not necessarily equate to quality or excellence – not a measure of LG (taken only at end of study) UKES not originally a measure of LG, questionable association btw students scores and levels of engagement (attitudes , intrinsic motivation?); low response rates and sector engagement (~38 institutions) – 3 questions added to NMMLGP pilot Careers registration – ask student to measure readiness for career; potential cultural, demographic and geographical impacts – would required extensive validation Take 10 mins to talk about this at your table – compare the ways in which learning gain and teaching excellence are measured within your own institutions (adding Others as appropriate) and then identify areas of similarity or difference. Invite feedback on similarities or differences (eg other) in each category – feedback from each group
- #8 Relationship btw TE and LG? Employment and Further Study •(SO1) Students achieve their educational and professional goals, in particular progression to further study or highly skilled employment Employability and Transferrable Skills •(SO2) Students acquire knowledge, skills and attributes that are valued by employers and that enhance their personal and/or professional lives Positive Outcomes for All •(SO3) Positive outcomes are achieved by its students from all backgrounds, in particular those from disadvantaged backgrounds or those who are at greater risk of not achieving positive outcomes
- #9 Three strands; Administered by IFF (externally contracted) Targeted at first year, first time u/gs Longitudinal survey - up to 4 years to include Year out – placement or abroad
- #10 Benchmarking for returning survey data to students – complex and varying responses Initial results – more variation within than btw institutions – median 5 – 3 to 8 responses across all institutions show a Gaussian distribution of results, suggesting CT can discriminate across participants highest levels of correlation re questions related to diversity ie ‘Meeting people form different cultures is an important part of my University education’ ‘Contact with individuals whose background are different from my own is an essential part of my University education’
- #11 Post-it sorting exercise: Which metric provide the most valid measure of Learning Gain and Teaching Excellence? Rank - most preferred as highest number; least preferred as lowest number How could/or could these be included within the TEF /LG to positively benefit the student experience? eg Synergy; value-add; integration of multiple metrics (do we not do this anyway? Institutional vs national level? Mutually dependent or independent) Feedback from groups
- #12 (un) Popular view UUK Survey Submission to be core part to TEF judgement; 11-13% secondary information or explanatory/defence against metrics 53% no impact on plans (yet) Concerns re metrics and ability to measure T&L (‘proxy’) but thought the exercise had raised the profile of T&L Future directions Trial year; lessons learnt, delay in subject level TEF 3, balance btw metrics and provider submission (tbc)
- #13 A cautionary tale: controversy over focussing on standard testing as a measure of the quality of education: Whatever ‘path’/metrics are chosen, we must ensure that evaluation activities ‘support excellence and improve learning outcomes’ by making more effective professionals/teachers. Evaluation activities associated with outcomes that do not make teaching more effective, are a waste of time. Perhaps timely to remember that ‘you don’t fatten a pig by weighing it’! (ie the evaluation is weighing, but the teaching is the fattening activity in this analogy).
- #14 Is a nuanced approach more appropriate/ more desirable? (Gunn & Fisk) – any metrics need to be contextualised within institutions, normalised for cohorts and applied appropriately within disciplines – they also need to work locally for staff and students!