The document discusses learning gain and why measuring it is important. It outlines the session which will clarify what learning gain means, consider drivers for interest in measuring it like the Teaching Excellence Framework, introduce types of learning gain measures, and discuss how learning technologies could provide data. Challenges of developing robust learning gain measures are also examined. Examples of UK universities measuring skills, engagement, and attributes are provided.
Data sharing and analytics in research and learningJisc
Learning analytics: progress and solutions - Niall Sclater and Michael Webb, both Jisc
Reading analytics - Clifford Lynch, CNI
Sharing data safely and it's re-use for analytics – David Fergusson, Francis Crick
Jisc and CNI conference, 6 July 2016
Jisc Change Agents' Network webinar 30 June 2015Ellen Lessner
Dr. Eleanor Quince, University of Southampton and Charlotte Medland, a student on the project, presented an overview of the Mission Employable; a student-led employability activity.
'Reflect and review' the webinar series led by Sarah Knight.
Making a difference with technology enhanced learning - Esther Barrett, Andre...Jisc
Led by Esther Barrett, subject specialist in teaching, learning and assessment, Jisc.
With contributions from Andrew Jaffrey, head of the office for digital learning and Richard Beggs, instructional design consultant - both from Ulster University.
There will be a focus how technology can support learning and teaching for a better student experience. Local providers will be sharing how their technology-based approaches have made a difference for learners and teachers.
Jisc Connect more in Northern Ireland, 23 June 2016.
Data sharing and analytics in research and learningJisc
Learning analytics: progress and solutions - Niall Sclater and Michael Webb, both Jisc
Reading analytics - Clifford Lynch, CNI
Sharing data safely and it's re-use for analytics – David Fergusson, Francis Crick
Jisc and CNI conference, 6 July 2016
Jisc Change Agents' Network webinar 30 June 2015Ellen Lessner
Dr. Eleanor Quince, University of Southampton and Charlotte Medland, a student on the project, presented an overview of the Mission Employable; a student-led employability activity.
'Reflect and review' the webinar series led by Sarah Knight.
Making a difference with technology enhanced learning - Esther Barrett, Andre...Jisc
Led by Esther Barrett, subject specialist in teaching, learning and assessment, Jisc.
With contributions from Andrew Jaffrey, head of the office for digital learning and Richard Beggs, instructional design consultant - both from Ulster University.
There will be a focus how technology can support learning and teaching for a better student experience. Local providers will be sharing how their technology-based approaches have made a difference for learners and teachers.
Jisc Connect more in Northern Ireland, 23 June 2016.
From Jisc's student experience experts group meeting in Birmingham on 21 April 2016.
https://www.jisc.ac.uk/events/student-experience-experts-group-meeting-20-apr-2016
Link into your professional network - Jisc Digifest 2016Jisc
This session will explore how helping teachers to build confidence in their own technical and professional networking skills, showing teachers how to use and become proficient with LinkedIn and how to transfer those skills to students can lead to employment for students.
The session will show case the Learning Futures/Education and Training Foundation funded resources for the FE and skills sector that its is anticipated may be embedded into a future Jisc service that is currently in the R&D phase.
Implementing analytics part 2 - Moriamo OduyemiJisc
With contribution from Moriamo Oduyemi, head of corporate information systems, University of Abertay.
Jisc Connect more in Northern Ireland, 23 June 2016
The changing face of assessment and feedback: how technology can make a diffe...Jisc
Over the past two years, the Jisc Assessment and Feedback programme has worked with over 30 institutions in the UK further and higher education sector to pilot new approaches that address a range of challenges to better meet the needs of learners, employers and staff. This workshop will share some of the experiences, approaches and lessons learned from these projects around key themes including:
Influencing change in assessment and feedback practices through a principle-led approach
Assessment and employability: the role of technology in supporting the development of skills and competences to enhance employment prospects
Feedback and feed forward: the role of technology in supporting learner engagement with feedback and improving progression
Electronic assessment management and how technology can support assessment lifecycle processes to make more effective use of resources
A range of resources will also be shared that can help to inform organisational good practice in enhancing assessment and feedback through technology.
The session will involve a mixture of presentation and discussion, giving participants opportunities to ask questions, discuss the themes emerging and how they relate to their contexts, and contribute to discussions around future priorities related to technology-enhanced assessment and feedback
Equipping the researcher - patterns in the UK and USJisc
UK and US academic practices – Christine Wolff, Ithaka S+R and David Prosser, RLUK
Digital scholarship centres – Harriet Hemmassi, Brown University and Joan Lippincott, CNI
Software carpentry and software skills and practice – Neil Chue Hong, Software Sustainability Institute
Jisc and CNI conference, 6 July 2016
FE digital student findings and recommendationsJisc
Findings and recommendations from the FE digital student project. Presented by Sarah Knight and Paul Bailey at the Learning and teaching practice experts group on 22 April 2015
Jisc toolkit: supporting the digital experience of new studentsJisc
This resource is based on the findings from Jisc’s student digital experience insights survey 2019. It's designed to be fully customised for your new students as part of their induction and onboarding processes.
A presentation by Clare Killen, content insight manager, Jisc
From Jisc's student experience experts group meeting in Birmingham on 21 April 2016.
https://www.jisc.ac.uk/events/student-experience-experts-group-meeting-20-apr-2016
Learning and teaching reimagined - how are student needs changing?Jisc
Presentation slides from our first learning and teaching reimagined series, which focused on how student needs are changing.
The rapid move to online learning brought about by COVID-19 has caused profound changes to higher education and the student experience.
But how much do we really know about the needs of our students? On what evidence are we basing these views? Even if we are confident that we do have a full and accurate picture of these needs, what difference is it making to our planning and decision making?
As part of our learning and teaching reimagined programme with UUK, Advance HE and Emerge Education, this webinar provided the opportunity to share your own understanding of your students’ needs and to hear those of others – not least from students themselves.
It explores the value of different types of evidence and, crucially, how to then build on this insight to ensure that the student voice permeates through, and plays an active role in, influencing your strategic planning.
The benefits and challenges of open access: lessons from practice - Helen Bla...Jisc
Led by Helen Blanchett, subject specialist, scholarly communications, Jisc.
With contribution from Andrew Simpson, associate university librarian (procurement and metadata and systems), Portsmouth University.
In this session you’ll hear in this session you’ll hear about the benefits and challenges of open access.
Connect more in London, 28 June 2016
Engaging students by closing the feedback loopJisc
Anish Bagga presented the findings, best practices,and potential consequences of an ineffective feedback system and how Unitu has discovered a great way to close the feedback loop. Delivered at the Learning and teaching practice experts group on 22 April 2015
Wellbeing and responsibility: a new ethics for digital educatorsHelen Beetham
Slides for Jisc Learning and Teaching Experts' group June 2015 summarising work of Jisc Digital Student project and 'Framing digital capabilities' project. Summarises findings and draws out implications for 'digital wellbeing' as an emerging concern for staff and students.
Growing a whole institution culture of commitment to student engagementJisc
As the student engagement agenda has gained momentum in UK higher and further education, there are numerous interesting and complex issues that arise and seemingly prevent a whole institutional commitment to working with students as partners.
Issues such as departmental autonomy, traditional hierarchies and power dynamics, and lack of time invested in innovative student engagement all contribute to a landscape where engaging students remains a project rather than a culture, and something done in a few departments rather than across a whole institution.
Find out more at https://www.jisc.ac.uk/events/growing-a-whole-institution-culture-of-commitment-to-student-engagement-20-jan-2016
Assessing OER impact across varied organisations and learners: experiences fr...Beck Pitt
This presentation was co-authored by Tim Coughlan (Nottingham), Beck Pitt (OU), Patrick McAndrew (OU) and Nassim Ebrahimi (Anne Arundel).
It was presented at OER13, Nottingham, UK which took place 26-27 March 2013.
From Jisc's student experience experts group meeting in Birmingham on 21 April 2016.
https://www.jisc.ac.uk/events/student-experience-experts-group-meeting-20-apr-2016
Link into your professional network - Jisc Digifest 2016Jisc
This session will explore how helping teachers to build confidence in their own technical and professional networking skills, showing teachers how to use and become proficient with LinkedIn and how to transfer those skills to students can lead to employment for students.
The session will show case the Learning Futures/Education and Training Foundation funded resources for the FE and skills sector that its is anticipated may be embedded into a future Jisc service that is currently in the R&D phase.
Implementing analytics part 2 - Moriamo OduyemiJisc
With contribution from Moriamo Oduyemi, head of corporate information systems, University of Abertay.
Jisc Connect more in Northern Ireland, 23 June 2016
The changing face of assessment and feedback: how technology can make a diffe...Jisc
Over the past two years, the Jisc Assessment and Feedback programme has worked with over 30 institutions in the UK further and higher education sector to pilot new approaches that address a range of challenges to better meet the needs of learners, employers and staff. This workshop will share some of the experiences, approaches and lessons learned from these projects around key themes including:
Influencing change in assessment and feedback practices through a principle-led approach
Assessment and employability: the role of technology in supporting the development of skills and competences to enhance employment prospects
Feedback and feed forward: the role of technology in supporting learner engagement with feedback and improving progression
Electronic assessment management and how technology can support assessment lifecycle processes to make more effective use of resources
A range of resources will also be shared that can help to inform organisational good practice in enhancing assessment and feedback through technology.
The session will involve a mixture of presentation and discussion, giving participants opportunities to ask questions, discuss the themes emerging and how they relate to their contexts, and contribute to discussions around future priorities related to technology-enhanced assessment and feedback
Equipping the researcher - patterns in the UK and USJisc
UK and US academic practices – Christine Wolff, Ithaka S+R and David Prosser, RLUK
Digital scholarship centres – Harriet Hemmassi, Brown University and Joan Lippincott, CNI
Software carpentry and software skills and practice – Neil Chue Hong, Software Sustainability Institute
Jisc and CNI conference, 6 July 2016
FE digital student findings and recommendationsJisc
Findings and recommendations from the FE digital student project. Presented by Sarah Knight and Paul Bailey at the Learning and teaching practice experts group on 22 April 2015
Jisc toolkit: supporting the digital experience of new studentsJisc
This resource is based on the findings from Jisc’s student digital experience insights survey 2019. It's designed to be fully customised for your new students as part of their induction and onboarding processes.
A presentation by Clare Killen, content insight manager, Jisc
From Jisc's student experience experts group meeting in Birmingham on 21 April 2016.
https://www.jisc.ac.uk/events/student-experience-experts-group-meeting-20-apr-2016
Learning and teaching reimagined - how are student needs changing?Jisc
Presentation slides from our first learning and teaching reimagined series, which focused on how student needs are changing.
The rapid move to online learning brought about by COVID-19 has caused profound changes to higher education and the student experience.
But how much do we really know about the needs of our students? On what evidence are we basing these views? Even if we are confident that we do have a full and accurate picture of these needs, what difference is it making to our planning and decision making?
As part of our learning and teaching reimagined programme with UUK, Advance HE and Emerge Education, this webinar provided the opportunity to share your own understanding of your students’ needs and to hear those of others – not least from students themselves.
It explores the value of different types of evidence and, crucially, how to then build on this insight to ensure that the student voice permeates through, and plays an active role in, influencing your strategic planning.
The benefits and challenges of open access: lessons from practice - Helen Bla...Jisc
Led by Helen Blanchett, subject specialist, scholarly communications, Jisc.
With contribution from Andrew Simpson, associate university librarian (procurement and metadata and systems), Portsmouth University.
In this session you’ll hear in this session you’ll hear about the benefits and challenges of open access.
Connect more in London, 28 June 2016
Engaging students by closing the feedback loopJisc
Anish Bagga presented the findings, best practices,and potential consequences of an ineffective feedback system and how Unitu has discovered a great way to close the feedback loop. Delivered at the Learning and teaching practice experts group on 22 April 2015
Wellbeing and responsibility: a new ethics for digital educatorsHelen Beetham
Slides for Jisc Learning and Teaching Experts' group June 2015 summarising work of Jisc Digital Student project and 'Framing digital capabilities' project. Summarises findings and draws out implications for 'digital wellbeing' as an emerging concern for staff and students.
Growing a whole institution culture of commitment to student engagementJisc
As the student engagement agenda has gained momentum in UK higher and further education, there are numerous interesting and complex issues that arise and seemingly prevent a whole institutional commitment to working with students as partners.
Issues such as departmental autonomy, traditional hierarchies and power dynamics, and lack of time invested in innovative student engagement all contribute to a landscape where engaging students remains a project rather than a culture, and something done in a few departments rather than across a whole institution.
Find out more at https://www.jisc.ac.uk/events/growing-a-whole-institution-culture-of-commitment-to-student-engagement-20-jan-2016
Assessing OER impact across varied organisations and learners: experiences fr...Beck Pitt
This presentation was co-authored by Tim Coughlan (Nottingham), Beck Pitt (OU), Patrick McAndrew (OU) and Nassim Ebrahimi (Anne Arundel).
It was presented at OER13, Nottingham, UK which took place 26-27 March 2013.
Assessing OER impact across varied organisations and learners: experiences fr...OER Hub
This presentation was co-authored by Tim Coughlan (Nottingham), Beck Pitt (OU), Patrick McAndrew (OU) and Nassim Ebrahimi (Anne Arundel).
It was presented at OER13, Nottingham, UK which took place 26-27 March 2013.
Strategic Visions & Values: Inclusive Curricula and Leadership in Learning an...Richard Hall
Presentation for the Leadership in Learning and Teaching event at Durham University on 1 May 2019.
Project resources:
Universal Design for Learning: Evaluation Interim Report: https://www.dora.dmu.ac.uk/handle/2086/17106
A Literature Review of Universal Design for Learning: https://www.dora.dmu.ac.uk/handle/2086/17059
Freedom to Achieve: Project Evaluation Report: https://www.dora.dmu.ac.uk/handle/2086/16793
"The Student Technology Committee: Students as Stakeholders in Medical Educat...Amol Utrankar
Jared Shenson, Ryan Adams, Amol Utrankar, S. Toufeeq Ahmed, and Anderson Spickard, III
Vanderbilt University School of Medicine
Presented at AAMC IT in Academic Medicine 2015, San Diego, CA.
Evaluating higher education learning outcomes presentation to cesCesToronto
The Higher Education Quality Council of Ontario (HEQCO) has a number of projects focused on defining and measuring learning outcomes working with Ontario's colleges and universities in partnership with international organisations. This presentation provides an overview of how learning outcomes are increasing being viewed as a means to evaluate higher education quality, and presents the challenges and successes of developing, demonstrating and assessing higher education learning outcomes in Ontario.
Sri Lanka Technological Campus (SLTC) had a problem. They had a strong association with the national telecom, Sri Lanka Telecom (SLT). It didn't help that SLT was one of the shareholders of SLTC, and was used by previous communications as a brand USP. This led to people thinking that SLTC is a state-owned university that offered free tuition. The brand strategy proposed included a full rebrand, renaming SLTC to Innovation & Technology Campus, Sri Lanka.
Similar to Understanding learning gain and why this might matter to you (20)
Palestine last event orientationfvgnh .pptxRaedMohamed3
An EFL lesson about the current events in Palestine. It is intended to be for intermediate students who wish to increase their listening skills through a short lesson in power point.
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Read| The latest issue of The Challenger is here! We are thrilled to announce that our school paper has qualified for the NATIONAL SCHOOLS PRESS CONFERENCE (NSPC) 2024. Thank you for your unwavering support and trust. Dive into the stories that made us stand out!
Unit 8 - Information and Communication Technology (Paper I).pdfThiyagu K
This slides describes the basic concepts of ICT, basics of Email, Emerging Technology and Digital Initiatives in Education. This presentations aligns with the UGC Paper I syllabus.
Embracing GenAI - A Strategic ImperativePeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Instructions for Submissions thorugh G- Classroom.pptxJheel Barad
This presentation provides a briefing on how to upload submissions and documents in Google Classroom. It was prepared as part of an orientation for new Sainik School in-service teacher trainees. As a training officer, my goal is to ensure that you are comfortable and proficient with this essential tool for managing assignments and fostering student engagement.
Biological screening of herbal drugs: Introduction and Need for
Phyto-Pharmacological Screening, New Strategies for evaluating
Natural Products, In vitro evaluation techniques for Antioxidants, Antimicrobial and Anticancer drugs. In vivo evaluation techniques
for Anti-inflammatory, Antiulcer, Anticancer, Wound healing, Antidiabetic, Hepatoprotective, Cardio protective, Diuretics and
Antifertility, Toxicity studies as per OECD guidelines
Acetabularia Information For Class 9 .docxvaibhavrinwa19
Acetabularia acetabulum is a single-celled green alga that in its vegetative state is morphologically differentiated into a basal rhizoid and an axially elongated stalk, which bears whorls of branching hairs. The single diploid nucleus resides in the rhizoid.
Introduction to AI for Nonprofits with Tapp NetworkTechSoup
Dive into the world of AI! Experts Jon Hill and Tareq Monaur will guide you through AI's role in enhancing nonprofit websites and basic marketing strategies, making it easy to understand and apply.
Honest Reviews of Tim Han LMA Course Program.pptxtimhan337
Personal development courses are widely available today, with each one promising life-changing outcomes. Tim Han’s Life Mastery Achievers (LMA) Course has drawn a lot of interest. In addition to offering my frank assessment of Success Insider’s LMA Course, this piece examines the course’s effects via a variety of Tim Han LMA course reviews and Success Insider comments.
2. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Session outline and objectives
1. To clarify what learning gain means in the context of UK HE
2. To consider some drivers for the current interest in LG, including TEF
3. To introduce the types of measure used to gauge LG, and the
challenges associated with developing these measures
4. To consider how learning technologies could be deployed to provide
robust data for learning gain measures.
3. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Definitions of learning gain
“an attempt to measure the improvement in knowledge, skills,
work-readiness and personal development made by students during
their time spent in higher education” HEFCE Learning Gain website
“the ‘distance travelled’, or the difference between the skills,
competencies, content knowledge and personal development
demonstrated by students at two points in time”
RAND report: McGrath et al, 2015, p. ix)
4. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Attempts to measure learning gain are not new
But, as noted in the RAND report (2015, p. ix)
“Debate on the measurement of the learning students
gain in higher education has been prominent
internationally, but the concept has not been studied
extensively in the English higher education context.”
5. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Drivers for measuring learning gain
“There is growing interest around the world in measuring how
much students in higher education learn …
Amongst other factors, such measures can be seen as
important to the debates about the quality and impact of
higher education, how we evidence the value of investment in
it, and how we evidence students’ skills acquisition for
employers.” RAND report: McGrath et al 2015, p. xi
6. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
The need to measure impact
Rich, 2015. pp 3 - 4
“… if universities are to command public investment, then a public
good has to be served and observed.”
“… what has been lacking is any measure of the actual value added
by universities to students and to the economy.”
“Without equations to demonstrate impact, it is hard to measure
the public good and, in an austere world driven by econometrics,
what is hard to measure is hard to fund.”
7. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Transparency for students
The desire to improve career prospects is the primary reason students cite
for going to university (Kandiko Howson, 2012)
‘Many higher education applicants … imagine that the relationship
between a course of study and a career is mechanistic, that higher
education is about direct preparation or training for a job’ Rich, 2015 p. 18
“Higher education cannot reasonably offer [employment] guarantees to
students. We need to shift the way we frame the student experience, to
be more transparent about what they will really gain from it.” Rich, 2015 p. 18
8. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Learning Gain and the Teaching Excellence
Framework
• Good teaching seen as a means to increasing graduates’ earning
potential, and also their overall contribution to the economy and to
society (the public good) (BIS 2015)
• Proposed TEF2 will provide ‘better understanding of the range of skills
and knowledge [students] bring from their course’ (BIS 2015 p. 21)
• BIS envisage incorporating evidence of learning gain as part of TEF2
(BIS 2016 pp. 11, 12, 16 and 30)
Department for Business, Innovation and Skills, May 2016
9. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
RAND report 2015: 5 main groups of measure
• Standardised tests: measure the skills students acquire; generic or specialised
• Grades: measure the progress in students’ achievement by comparing the difference between
grades at selected points in time
• Self-reporting surveys: ask students to report the extent to which they think they have gained
knowledge and developed skills; run at a number of points throughout a degree programme
• Other qualitative methods: e.g. PDPs; encourage students to reflect on their learning, acquired
skills and skills gaps; stimulate productive discussions between students and their tutors
• Mixed methods: draw on a range of tools and indicators to track improvement in performance,
(e.g. through a combination of grades, student learning data and student surveys); can be used to
predict students’ performance. See Wabash study for a comprehensive example
https://www.youtube.com/watch?v=N4OKKsW-YdQ
10. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
RAND report 2015: types of LG measure
• Proxies, e.g. engagement, experience ratings, employment (including
HMRC data) ‘could prove efficient since they exploit existing metrics’
but validity is questionable
• Direct measures include ‘before’ and ‘after’ grades; standardised tests
(generic & discipline specific); student self-report surveys (again,
validity concerns)
• e.g. UKES pilot included 12 items on students’ perceived skills development;
NSS contains one question on personal development
11. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Unintended consequences and caveats
• Gaming by institutions
• Discouraging learning itself
• Quality is relative and must take into account the diversity of missions and goals in HE
• Quality of learning is more than just learning gain, and quality of HE is more than just
quality learning
• Skills and competencies may not be comparable across disciplines
• Contextual and demographic characteristics are a factor, but difficult to determine which
• Learning gain should not be the sole determinant of the quality of HE (cf TEF proposals).
RAND report: McGrath et al 2015 pp. 11, 12
12. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Challenges of developing LG measures which…
• Have a clearly stated purpose
• Identify the dimensions of analysis (personal development, generic skills etc)
• Consider complexity of LG and diversity of students, HEIs, state mission etc
• Adopt methods which have comparable inputs and outputs
• Are practical, can be implemented effectively and are cost-effective
• Remain aware of unintended consequences, for example, need to prevent ‘gaming’
• Achieve buy-in from HEIs and other stakeholders
• Acknowledge the limits of the measures
• Validate methods, for example by controlling for factors which could impact on
validity, such as sociodemographic or neighbourhood effects
13. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
UEL/Roehampton/Brunel
Focus on academic skills and engagement with learning
Same measures being trialled across all 3 HEIs
Academic Behavioural
Confidence (ABC) scale to test
students; also to predict grades
and where students grades will
lie in relation to cohort
“A psychometric means of assessing
the confidence that undergraduate
university students have in their own
anticipated study behaviours in
relation to their degree programme”
(Sander and Sanders 2009:19)
Where are digital capabilities?
16. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Russell Group consortium example 1
Career planning / readiness
• Using existing methodology
developed by Bob Gilworth (Leeds)
• Formative tool embedded in annual
registration process
• Students pick one of 10 statements
which they feel most closely reflects
their current career thinking (3 x
‘Decide’, 3 x ‘Plan’, 3 x ‘Compete’
and 1 ‘sorted’)
http://careerweb.leeds.ac.uk/info/13/career_planning
17. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Russell Group consortium example 2
Graduate attributes
• ‘Realise2’ questionnaire
• Blue chip organisations (eg Ernst
& Young) use this as part of their
graduate recruitment process to
test for a set range of graduate
attributes (strengths)
“Through 180 questions, 60 strengths are accurately assessed
according to the three dimensions of Energy,
Performance and Use – with each user receiving their unique,
personalised Profile, revealing their realised strengths,
learned behaviours, weaknesses and unrealised strengths.”
http://www.cappeu.com/R2StrengthsProfiler
18. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
Discussion
1) Is your HEI using or developing measures of learning gain or learner analytics? If
so, briefly tell others in your group what is being measured and how
2) How are, or could, learning (or other) technologies be used to support the robust
measurement of learning gain? Consider:
a) Student engagement data sources: already recorded / available but not
recorded
b) Administration of tests: existing and new
3) Which other professional and academic departments will you need to collaborate
with in order to do this?
4) Are you ready for it?
20. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
References
• Britton, J. et al (2016) How English domiciled graduate earnings vary with gender, institution attended, subject and socio-economic
background, IFS Working Paper W16/06. Nuffield Foundation.
• Cacioppo, J. T., & Petty, R. E. (1984). The need for cognition: Relationships to attitudinal processes. In R. P. McGlynn, J. E. Maddux, C.
Stoltenberg, & J. H. Harvey (Eds.), Social perception in clinical and counseling psychology. Lubbock, Texas Tech University Press. [see
also http://www.liberalarts.wabash.edu/ncs/ for overview article]
• Collini, S. (2012) What are universities for? Penguin, London
• Fulfilling our Potential: Teaching Excellence, Social Mobility and Student Choice. Department for Business, Innovation & Skills, 2015
https://www.gov.uk/government/consultations/higher-education-teaching-excellence-social-mobility-and-student-choice
• Greatbatch, D & Holland, J. (2016) Teaching Quality in Higher Education: Literature Review and Qualitative Research May 2016
Department for Business, Innovation & Skills https://www.gov.uk/government/publications/teaching-quality-in-higher-education-
literature-review-and-qualitative-research
• Teaching Excellence Framework Technical Consultation for Year Two. Department for Business, Innovation & Skills, 2015
https://www.gov.uk/government/consultations/teaching-excellence-framework-year-2-technical-consultation
• Kandiko Howson, C. (2012) Student expectations and perceptions of higher education. Kings College London and Quality Assurance
Agency
21. CONSULTING
www.coronyedwards.co.uk
Corony
Edwards
References contd.
• Learning Gain: HEFCE Policy Guide http://www.hefce.ac.uk/lt/lg/
• McGrath, C. H. et al. (2015) Learning gain in higher education. RAND Corporation, Santa Monica, CA
http://www.rand.org/pubs/research_reports/RR996.html (The HEFCE ‘RAND report’)
• Meltzer, D. (2002) ‘The relationship between mathematics preparation and conceptual learning gains in physics: A possible “hidden
variable” in diagnostic pretest scores’. American Journal of Physics. 70:12, 1259-68
• Morrison, B, (2005) Evaluating learning gain in a self-access language learning centre Language Teaching Research. 9:3 267-293
• Rankin, E. & Dale, L. (1969) ‘Cloze Residual Gain - A Technique for Measuring Learning Through Reading’ in Schick, G. & May, M. Eds.
The Psychology of Reading Behavior; Eighteenth Yearbook of the National Reading Conference. National Reading Conference,
Milwaukee
• Rich, J. (2015) Employability: degrees of value. HEPI Occasional Paper 12. Higher Education Policy Institute, Oxford
http://www.hepi.ac.uk/2015/12/10/employability-degrees-value/
• Sander, P & Sanders, L. (2009) Measuring academic behavioural confidence: the ABC scale revisited. Studies in Higher Education,
34:1, 19-35
• Thorndike, E. L.; Bregman, E. O.; Tilton, J.; Woodyard, E. (1928) Adult Learning. Macmillan, Oxford (Abstract at
http://psycnet.apa.org/index.cfm?fa=search.displayRecord&uid=1929-00108-000)
Editor's Notes
NB employability/ employment features as a theme from the start in HEFCE’s discourse.
Skills are generally defined as the abilities or proficiencies to apply cognitive or practical knowledge and use know-how to complete tasks and solve problems in certain areas.
A competence, in addition to including a skill, covers abilities, behaviours and knowledge that are fundamental to the use of such skill. A competence is a proven ability to use knowledge, skills and personal, social and/or methodological abilities, in work or study situations and in professional and personal development.
See RAND report top of p7 for some of the skills and competencies that Ss might develop during their HE.
learning gain enumerates actual change in performance between two points in time
learning outcome describes the output level.
Cf value added, which is typically measured by the difference between a student’s actual test score at a second point in time and his or her predicted test score at a first point in time [NB that value added models are used in performance tables and systems for school improvement in the UK, which is an international leader in the development and use of such models
Earliest example I found was Thorndike et al 1928
Recent International debates on how to measure the quality of education have heightened interest in this originally US-based concept in England and elsewhere.
Interest has grown because learning gain has the potential to say more about the extent to which higher education fulfils its key mission of contributing to students’ knowledge, skills and understanding than traditional output indicators, such as the number of graduates. (RAND p1)
Note that employability / employment is a key theme from the outset.
The change in funding arrangements in UK HE (fees etc) over the last decade = pressure to justify to students, employers and governments the investment in higher education.
Commission on the Future of Higher Education in the United States (the ‘Spellings Commission’) 2006 - questioned the learning outcomes from US higher education, and recommended that post-secondary institutions should measure and report ‘meaningful student learning outcomes’ (ie learning gain).
(OECD) feasibility study to compare generic and discipline-specific skills in economics and engineering across HEIs: Assessment of Higher Education Learning Outcomes (AHELO).
The Bologna Process, a European intergovernmental series of agreements, also covered the debate during discussions on learning outcomes.
Interestingly ambiguous choice of wording, where ‘value’ often seems to equate to ‘employment prospects’ or ‘earnings’. Is this fundamentally about money, and is this now all that really matters as an outcome of a university education, or am I just being cynical?
RAND says ‘Value is measured not only in terms of the qualification acquired, but also in terms of what is learned during higher education - including the growth in what students know/understand/can do throughout their degree programme’ – but to what end?
Note that the HEAR does to some extent address the need to specify what has been learned rather than simply on the results of students’ formal assessment. It does not, however, demonstrate progress made during the course of studies.
What students really gain is (or should be) the transferrable skills that employers say they want.
Note findings of very recent experimental report into graduate earnings (Institute for Fiscal Studies) using HMRC PAYE and Student Loans Company data for graduates, with real data on earnings ten years’ post-graduation by subject.
Green paper and TEF consultation is primarily concerned with teaching excellence (among other themes), but explicitly identifies learning gain as an aspect of excellent teaching that should be measured.
So it seems that measurement of learning gain is coming, whatever our views.
Wabash presentation on HEFCE LG website. 49 colleges, 17,000 students, very wide range of measures to identify not just LG, but also the types of intervention, activity or good practice that will help students to improve in a liberal arts context
UK is behind the US and Europe in the use of LG measures
• There are two main types of standardised test:
o Discipline-specific test, such as the Progress Testing administered in some schools of
medicine in England.
o Generic skills test, which are not discipline specific, such as the Collegiate Learning
Assessment (CLA), which relies on open-ended questions and the Measure for Academic
Proficiency and Progress (MAPP), which is a more traditional test relying on multiple choice
questions. Both these examples originated and are
‘It is important to note that unintended consequences observed in the context of value added measurements in secondary education, such as ‘gaming’, could also apply to the context of higher education if the objective is to use the measure for accountability purposes’ [such as TEF]
‘Learning gain should not be considered as a sole determinant of the quality of higher education. Quality is relative to the many missions and goals that higher education has - there is both a broader conception of what is meant by quality of learning than solely learning gain, and there are more aspects to what is understood as quality of higher education in institutions, than just quality learning’
These are the principles or criteria which acceptable measures of LG must meet
See also p. 17 of RAND report for a framework of questions to use when evaluating methods to measure learning gain. These look at:
Validity and comparability (subjective or objective tests, and how validated)
Longitudinal or cross-sectional data, LG or value added measure
Representativeness (eg sample size)
Survey 1 at entry, surveys 2 & 3 in lectures to maximise response rates; have got ethics clearance, ready to start (Brunel already started data collection)
Brunel – will investigate impact of 2 interventions on student performance in ABC – peer assisted learning and ‘Headstart’ (a week long transition programme for WP students covering study skills, support sources etc)
Roehampton is also using the ABC questionnaire in a handful of volunteer faculties, and may also follow up with focus groups to get feedback on the ABC tool and questions and to verify answers
UEL will administer ABC on a larger scale, currently across their Faculties of Education and Psychology
NB Also Durham skills audit – pre entry self-report Qs, linked to Durham graduate attributes. See vignette in RAND report P.28
RG consortium includes 17 RG HEIs. Led by Warwick.
workstream leads are Birmingham, Nottingham, Cambridge, Warwick.
Some RG HEIs (eg Manchester, Uni of London) also leading separate projects.
University College London, University of Birmingham, University of Bristol, University of Cambridge, University of Durham, University of Exeter, University of Leeds, University of Liverpool, University of Manchester, University of Nottingham, University of Southampton, University of Sheffield, University of York, Imperial College London, King’s College London, Newcastle University, Queen Mary University of London
Embedding transferrable or employability skills into the curriculum is not a new idea, but are you doing enough of this – and assessing it – to make sure that Sunderland will perform well in any new measure of LG?