1. The UK’s European university
The Holy Grail for HE:
achieving transformational learning
(gain) through excellent teaching?
Dr Louise Naylor
12 May, 2017 l.h.naylor@kent.ac.uk
2. In search of the Holy Grail…..
The question of how to measure the student learning
gained in higher education has been much debated. The
debate is healthy — any attempt to measure a process
which is as multidimensional as learning can encourage a
focus on what is effective — but it also risks distorting
activity. There is an evident need to consider different
approaches to the measurement of gains in learning and
consider their impact on practice and on outcomes. Debate
on the measurement of the learning students gain in higher
education has been prominent internationally, but the
concept has not been studied extensively in the English
higher education context.
McGrath et al (2015) Learning Gain in HE (RR-996-HEFCE)
The Holy Grail for HE - teaching excellence and learning gain
3. Definitions
Learning Gain
The concept of ‘learning gain’ is
defined as the ‘distance travelled’, or
the difference between the skills,
competencies, content knowledge
and personal development
demonstrated by students at two
points in time. This allows for a
comparison of academic abilities and
how participation in higher education
has contributed to such intellectual
development (‘value-added’).
McGrath et al., RAND report (2015)
The Holy Grail for HE - teaching excellence and learning gain
4. Definitions
Teaching Excellence
We take a broad view of teaching
excellence, including the teaching itself,
the learning environments in which it
takes place, and the outcomes it delivers.
We expect higher education to deliver
well designed courses, robust standards,
support for students, career readiness
and an environment that develops the
‘soft skills’ that employers consistently
say they need. These include capacity
for critical thinking, analysis and
teamwork, along with the vital
development of a student’s ability to
learn. (p43)
The Holy Grail for HE - teaching excellence and learning gain
5. ‘Great teaching increases the likelihood of good outcomes’
Learning Gain
(ie student outcomes)
Improvement in:
knowledge
skills
work-readiness
(employability)
personal development
Teaching Excellence
(ie staff performance)
• ‘Innovative, inspiring,
engaging, stimulating,
challenging’…..
• Good or ‘best’
practice/exemplars
• Contextualised
(discipline, level, mode)
Gunn & Fisk (2013)
The Holy Grail for HE - teaching excellence and learning gain
6. Choosing the Metrics – Match up
Learning Gain
Grades (GPA) - entry to
output
PDP portfolio
Progress tracking (ALIS)
Surveys – NSS, UKES
Careers registration
• Other?
Teaching Excellence
NSS
DLHE
HESA (non-continuation)
• Teaching qualifications?
• Student engagement?
• Contact time?
• HMRC? (Salary)
• Other?
The Holy Grail for HE - teaching excellence and learning gain
7. TEF Year 2 – student outcomes/learning gain
Criteria
• Employment and
Further Study
• Employability and
Transferable Skills
• Positive Outcomes
for All
Examples of evidence
Learning gain and distance-travelled by all students
Career enhancement and progression/impact of
initiatives aimed at graduate employability
Impact of initiatives aimed at preparing students for
further study and research
Longer-term employment outcomes and progression of
graduates including into highly-skilled employment
Extent of student involvement in enterprise and
entrepreneurship
Number, impact and success of graduate start-ups
Use and effectiveness of initiatives used to help measure
and record student progress, such as Grade Point
Average (GPA)
Impact of initiatives aimed at closing gaps for students
from different backgrounds
The Holy Grail for HE - teaching excellence and learning gain
8. Critical thinking & problem-
solving
Attitudes towards study
experience
Engagement
12 test questions exploring
respondents ability to:
• summarise the main conclusion
(CT)
• assess the impact of additional
evidence (CT)
• apply principles (CT)
• detect reasoning errors (CT)
• identify similarity (PS)
• find procedures (PS)
• make relevant selections (PS).
Three sets of survey questions
exploring:
• Positive attitude toward Literacy
(6 questions)
• Openness to diversity (7
questions)
• Academic motivation (8
questions)
Three sets of survey questions
exploring:
• course emphasis on critical
thinking
• interaction with staff
• time spent between various
academic and non-academic
activities
source: Cambridge Assessment source: Pascarella/Wabash source: HEA/UKES questions 1, 5 and 18
2016-17 (1st term) 2016-17 (3rd term) 2017-18 (3rd term) 2018-19 (3rd term) 2021-22?
1. Critical thinking &
problem-solving
2. Attitudes towards
study experience
1. Critical thinking &
problem-solving
2. Attitudes towards
study experience
3. Engagement
Link results to graduate
outcomes?
NMMLGP - 10 HEIs
TEST REPEATED
9. NMMLGP – early findings
Achievements
Institution
Emails
sent
Response flag
Total
Response
rate (full
response)Full
response
Partial
response
University of
Kent
3,211 273 115 388 9%
Challenges
• Timescale for launch
• Student engagement (7%)
• Incomplete surveys (~30%)
• Anonymity vs support
• Longitudinal attrition (3yrs)
• Benchmarking options:
• None; All responses;
Institution only; All
responses + institution
• Initial results
The Holy Grail for HE - teaching excellence and learning gain
In addition, the table below gives an overview of the response rate for your institution:
10. Choosing the Metrics – Mix Up
0
1
2
3
4
5
6
7
8
9
METRICS
LG Metric TE Metric LG + TEF
• Place in rank order
your most preferred
(3) to least preferred
metric (1) for
learning gain and
teaching excellence
• Choose ideal
combined rank eg
• no overlap -1
• full synergy – 9
The Holy Grail for HE - teaching excellence and learning gain
11. Next steps for TEF* …..
• ‘Universal condemnation of the
TEF’
• ‘Proposed metrics are factually
flawed’
• ‘Written submissions as opening
the way to game-playing that
characterised the excesses of
the REF’
• ‘Only 4% academics believe it
will accurately assess teaching
quality’ (so not excellence then?)
*THE Poppletonian - 9 March 2017
UUK Survey (83)
Submission:
SMT (55%); Senate (22%)
Steering Group - 88%
Average staff - 27
TEF judgement -76% (core)
Impact:
Accelerated plans (47%)
‘Proxy’ metrics; raised
profile of teaching
Future:
Viability of subject level?
Areas for development:
• student use (79%)
• new metrics & assessment
(53%)
• reduce burden (47%)
The Holy Grail for HE - teaching excellence and learning gain
12. and for Learning Gain…..
Supporting excellence and improving student outcomes
The Holy Grail for HE - teaching excellence and learning gain
#learninggain
13. The ‘elusive’ Holy Grail in HE
National Benchmarked Standards:
• How/can these be defined?
Definitions
• How/should they be measured?
Metrics
• Is it possible to benchmark?
Comparability
• Will these influence the student
experience in a positive way? Impact
• Will this differentiate the sector as
planned? Market
• Will this provide information students
want or understand? Communication
The Holy Grail for HE - teaching excellence and learning gain
With the introduction of the Teaching Excellence Framework as part of the HE&R Bill and the ‘quest’ for comparable national metrics to measure Learning Gain, the sector is being challenged to reconcile the relationships between these two concepts using quantitative and qualitative assessments.
Although the HE Bill has now gone through Parliament (now Act), the outcomes of TEF2 are delayed to mid June (post election) and our LG project has only just started, so this workshop will explore through our own experience to date, whether these exercises are likely to provide appropriate ways of conceptualizing and differentiating the sector
and more importantly, whether the links are as simple as suggested and will achieve the desired outcomes in terms of enhancing the student learning experience.
This session will explore:
1. Definitions &
2. Metrics - for TE and LG
3. Consider how these might be combined to benefit the Student Learning Experience
P43- The report goes onto assert that ‘great teaching increased the likelihood of good outcomes’ but
if we look more closely at the two concepts, one is about student outcomes (example measures) and the other about staff performance in teaching (example measures) – are these two inextricably linked and if so, in what ways?
SEDA list (March 2017)– quite a debate about when the term T&L first appeared and relationship between the two; which comes first ‘chicken or egg’; are these mutually dependent or independent?
It is possible that we can have:
‘Learning without teaching and teaching without learning’ and even the DfE publication hedges bets on this – notes ‘increases the likelihood’ but not necessarily causative correlation between teaching and learning outcomes
Gunn and Fisk (2013) - Highly contextualised and individual dependent – suggest that no ‘one size fits all’ and warn that chasing national benchmark standards will be difficult
Learning Gain
RAND p64-68
Mix of both qualitative and quantitative methodologies (these are some current indicators, but you will have others):
Grades – problem with comparability across sector/ relating tariff entry to outputs not always direct (not a good predictor)
PDP – students reflect on development but highly individualised and not comparable even within institutions
Progress tracking- diagnostics skills audit – measures only skills aspect of LG and often done at discipline not institutional level (except Med Schools) – how can this be standardised? Mixed methods eg ALIS – Advanced Level Information System combines grade and test results as predictor of students’ performance – provides score of likely achievement, rather than Learning Gain
NSS administered nationally (UK-wide) but it is about ‘satisfaction’ with teaching & learning environment/ does not necessarily equate to quality or excellence – not a measure of LG (taken only at end of study)
UKES not originally a measure of LG, questionable association btw students scores and levels of engagement (attitudes , intrinsic motivation?); low response rates and sector engagement (~38 institutions) – 3 questions added to NMMLGP pilot
Careers registration – ask student to measure readiness for career; potential cultural, demographic and geographical impacts – would required extensive validation
Take 10 mins to talk about this at your table – compare the ways in which learning gain and teaching excellence are measured within your own institutions (adding Others as appropriate) and then identify areas of similarity or difference.
Invite feedback on similarities or differences (eg other) in each category – feedback from each group
Relationship btw TE and LG?
Employment and Further Study
•(SO1) Students achieve their educational and professional goals, in particular progression to further study or highly skilled employment
Employability and Transferrable Skills
•(SO2) Students acquire knowledge, skills and attributes that are valued by employers and that enhance their personal and/or professional lives
Positive Outcomes for All
•(SO3) Positive outcomes are achieved by its students from all backgrounds, in particular those from disadvantaged backgrounds or those who are at greater risk of not achieving positive outcomes
Three strands;
Administered by IFF (externally contracted)
Targeted at first year, first time u/gs
Longitudinal survey - up to 4 years to include Year out – placement or abroad
Benchmarking for returning survey data to students – complex and varying responses
Initial results –
more variation within than btw institutions – median 5 – 3 to 8
responses across all institutions show a Gaussian distribution of results, suggesting CT can discriminate across participants
highest levels of correlation re questions related to diversity ie
‘Meeting people form different cultures is an important part of my University education’
‘Contact with individuals whose background are different from my own is an essential part of my University education’
Post-it sorting exercise:
Which metric provide the most valid measure of Learning Gain and Teaching Excellence? Rank - most preferred as highest number; least preferred as lowest number
How could/or could these be included within the TEF /LG to positively benefit the student experience? eg Synergy; value-add; integration of multiple metrics (do we not do this anyway? Institutional vs national level? Mutually dependent or independent)
Feedback from groups
(un) Popular view
UUK Survey
Submission to be core part to TEF judgement; 11-13% secondary information or explanatory/defence against metrics
53% no impact on plans (yet)
Concerns re metrics and ability to measure T&L (‘proxy’) but thought the exercise had raised the profile of T&L
Future directions
Trial year; lessons learnt, delay in subject level TEF 3, balance btw metrics and provider submission (tbc)
A cautionary tale: controversy over focussing on standard testing as a measure of the quality of education:
Whatever ‘path’/metrics are chosen, we must ensure that evaluation activities ‘support excellence and improve learning outcomes’ by making more effective professionals/teachers.
Evaluation activities associated with outcomes that do not make teaching more effective, are a waste of time.
Perhaps timely to remember that ‘you don’t fatten a pig by weighing it’! (ie the evaluation is weighing, but the teaching is the fattening activity in this analogy).
Is a nuanced approach more appropriate/ more desirable? (Gunn & Fisk) – any metrics need to be contextualised within institutions, normalised for cohorts and applied appropriately within disciplines – they also need to work locally for staff and students!