Learning gains has increasingly become apparent within the HE literature, gained traction in government policies in the UK, and are at the heart of Teaching Excellence Framework (TFL). As such, this raises a question to what extent teaching and learning environment can actually predict students’ learning gains using principles of learning analytics. In this presentation, which is joined work with University of Surrey and Oxford Brookes, I will focus on some preliminary findings based upon developing and testing an Affective-Behaviour-Cognition learning gains model using longitudinal approach. The main aim of the research is to examine whether learning gains occur on all three levels of Affective-Behaviour-Cognition model and whether any particular student or course characteristics can predict learning gains or lack of learning and dropout. For more info, see https://abclearninggains.com/

1. The power of learning
analytics to measure learning
gains: an OU, Surrey and
Oxford Brookes story
https://twitter.com/LearningGains
https://abclearninggains.com/
The results of the ABC project were made possible due to Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Jekaterina Rogaten, Bart Rienties, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise
Whitelock. Please contract Jekaterina.rogaten@open.ac.uk if you want to know more about ABC learning gains

2. Why measure Learning Gains
Talk to your neighbour:
1. What do you think Learning Gains are?
2. Why should universities worry about Learning Gains?
3. Why should I/we need to think about Learning Gains?

3. Why measure Learning Gains
• Green paper UK (November 2015) states – learning gain (and student outcomes) is a
key aspect to consider when recognizing teaching excellence and quality.
• Learning Gains are key elements of TEF, but how to measure?
• 13 Pilot projects have been awarded a total of 4m to advance our understand of
learning gains in HE and develop valid and reliable assessment of learning gains.
• Our project is ABC mixed-method learning gains research and for quantitative part of
the study we are looking at what affective, behavioural and cognitive learning gains
students show.

4. Agenda for today
1. What are learning gains?
2. How are we working within ABC project to measure learning gains?
3. How are learning gains measured: a meta-analysis
4. Do students make learning gains, and what is the power of LA to predict learning
gains?
5. Policy implications

5. 1 What are learning gains?
In line with Cronbach & Furby, 1970 and Lord, 1956, we
define Learning Gains as:
Growth or change in knowledge, skills, and abilities over
time that can be linked to the desired learning outcomes or
learning goals of the course
Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective.
Educational Research Review. Impact factor: 3.860

6. 2. How are we working within ABC project to measure learning gains?
Affect
Behaviour
Cognition
Academic
performance
VLE
Satisfaction

7. Quantitative
Qualitative
Academic
performance
VLE
Satisfaction
Interviews
Diary
Module
design
Socio-
Demographic
Structure of the
ABC project

8. Open University Team
• Strength
• Learning analytics
• Quantitative data analysis
• Assessment
• Online learning and learning design
• Cross-sector thinking and working
• Responsibilities
• Overall leading the project
• Leading the Phase 1 of the project
(secondary data analysis)
• Phase 1 ethics applications
• Developing models of learning gains
Dr Bart Rienties
Dr Jekaterina Rogaten
Prof Denise Whitelock
Dr Simon Cross
Prof Allison Littlejohn

9. Open University
1. Learning Analytics
Largest VLE datasets of affect, behaviour and
cognition of 200K students, with 2 large scale
implementations of predictive modelling
2. Learning Design
Extensive modelling of learning designs (OULDI)
3. Student Experience on a Module
Extensive datasets of students’ experiences with
blended and online learning

10. Oxford Brooks University Team
Prof Rhona Sharpe
Dr Ian Scott
Strength
Expertise in learners’ experience
Mixed-methods approach
Measures of learning and engagement
Learning analytics
Responsibilities
Co-leading Phase 2 of the project
In depth-interviews and diary data collection
Testing validity of self report measures of learning gains
Triangulation

11. Oxford Brookes University
1. Academic Performance Tracking Tool
Presents staff with data from student information
system, NSS and module evaluations in form of
dashboards used for module and programme
reviews.
2. Grade Point Average
Provides students with information about their
progress. Provides staff with a more nuanced
indicator of attainment.
3. Survey of Student Engagement
Developed own scales for monitoring student
engagement with Assessment Compact, Academic
Advising and Graduate Attributes.

12. University of Surrey Team
Responsibilities
Co-leading Phase 2 of the project
Testing the validity of the self-reported measures of learning
gains (+OB)
Testing the validity of learning analytics modelling (+OU)
Dissemination: website and Twitter
Prof Ian Kinchin Prof Steven Warburton Dr Simon Lygo-Baker
Strength
University pedagogy and concept mapping
Educational technologies
Expertise in innovative teaching approaches development
Ceri Hitchings

13. University of Surrey
• Business Intelligence - Progression Analysis project has built / is building:
• Live data on students who are at risk, to act as an early warning indicator so that active support
measures can be put in place;
• Visualisation of data via dashboards for academics to put in place appropriate actions;
• Extension into WP & Outreach domain with specific data feeds, modelling, and a WP progression
dashboard.
• Institutional success:
• Ability to makes links between what gains our students make in correlation with an institution which
has high levels of satisfaction and has enhanced learning and teaching across all Faculty.

14. 3. How are learning gains measured: a meta-analysis
• The concept of learning gain is
primarily used to examine the
effect of any particular
educational ‘intervention’
• There is a gradual increase in
studies examining learning
gains all across the world
• All learning gains can be
classified into ABC 53%
16% 21%
10% Behaviour-Cognitive
Learning Gains
Affective-Cognitive
Learning Gains
Affective-Behaviour-Cognitive
Learning Gains
Cognitive Learning Gains
Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective.
Educational Research Review. Impact factor: 3.860

15. A wide divergence in methods and reported
effects
Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective.
Educational Research Review. Impact factor: 3.860
Results will be shared when published

16. 4. Do students make learning gains, and what is
the power of LA to predict learning gains?
• Learning gains are usually measured using pre-post standardised testing
• It is resource intensive and becomes even more so if one wants to estimate
learning gains across various disciplines and number of universities
• Using assessment results for estimating learning gains has number of
advantages:
• Assessment data readily available
• Widely recognized as appropriate measure of learning
• Relatively free from self-reported biases
• Allows a direct comparison of research finding with the results of other studies
Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia

17. Three-level Growth Curve Model
Level 1
Level 2
Level 3
TMA1
Student1
TMA3 TMA1 TMA2 TMA3 TMA1 TMA2 TMA3TMA2
Student2 Student3
Module1 Module2
TMA1 TMA2 TMA3
Student4
TMA1 TMA2 TMA3
Student5
Module3
Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia
VLE VLEVLE

18. Study 1
• Participants
• 11,909 Social Science students of whom
72% were females and 28% were males
with average age of M = 30.6, SD = 9.9
• 5,791 Science students of whom 58.2%
were females and 41.8% were males with
average age of M = 29.8, SD = 9.6.
• Measures
• Tutor Marked Assessments (TMA)
• Socio demographics (gender, ethnicity,
prior educational qualification)
• Across 111 modules
Study 1: Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia
Study 2: Rogaten, J., Rienties, B., Whitelock, D. (Submitted: 17-10-2016). Multi-level growth modelling of learning gains: the impact of VLE engagement on learning Arts and Business. Paper submitted to LAK2017.
Study 2
• Participants
• 2,875 Arts students of whom 64% were
females and 36% were males with
average age of M = 34, SD = 13.3
• 5,082 students of whom 54% were
females and 46% were males with
average age of M = 29.6, SD = 9.3
• Measures
• Tutor Marked Assessments (TMA)
• Socio demographics (gender, ethnicity,
prior educational qualification, work part-
time or full-time, and disability)
• VLE engagement
• Across 36 modules

19. Three level growth model
Study 1: Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia
Study 2: Rogaten, J., Rienties, B., Whitelock, D. (Submitted: 17-10-2016). Multi-level growth modelling of learning gains: the impact of VLE engagement on learning Arts and Business. Paper submitted to LAK2017.
Results will be shared when published

20. The power of learning
analytics to measure learning
gains: an OU, Surrey and
Oxford Brookes story
https://twitter.com/LearningGains
https://abclearninggains.com/
The results of the ABC project were made possible due to Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Jekaterina Rogaten, Bart Rienties, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise
Whitelock. Please contract Jekaterina.rogaten@open.ac.uk if you want to know more about ABC learning gains