Jekaterina Rogaten1 , Bart Rienties1 , Denise Whitelock1 , Simon Cross1 , Allison Littlejohn1 , Rhona Sharpe2 , Simon Lygo-Baker3 , Ian Scott2 , Steven Warburton3 , Ian Kinchin3 1The Open University UK, UK, 2Oxford Brooks University, UK, 3University of Surrey, UK Research Domain: Learning, teaching and assessment (LTA) In the UK, the introduction of the Teaching Excellence Framework (TEF) has increased interest in appropriate and valid measurement approaches of learning gains in Higher Education. Usually learning gains are measured using pre-post testing, but this study examines whether academic performance can be effectively used as proxy to estimate students’ learning progress. Academic performance of 21,192 online learners from two major faculties was retrieved from university database. A three-level growth-curve model was estimated and results showed that 16% to 46% of variance in students’ initial academic performance, and 51% to 77% of variance in their subsequent learning gains was due to them studying at a particular module. In addition, the results illustrate that students who studied in modules with initial high student achievements exhibited lower learning gains than students learning in modules with low initial student achievements. The importance of assessment and learning design for learning gains are outlined. www.abclearninggains.com @learninggains

1. Multilevel Modelling of Learning
Gains: The Impact of Module
Particulars on Students’ Learning
in Higher Education.
Jekaterina Rogaten
https://twitter.com/LearningGains
https://abclearninggains.com/
The results of the ABC project were made possible due to Jekaterina Rogaten, Bart Rienties, Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise
Whitelock. Please contract Jekaterina.rogaten@open.ac.uk if you want to know more about ABC learning gains

2. Research Team
Dr Bart Rienties
Dr Jekaterina Rogaten
Prof Denise Whitelock
Dr Simon Cross
Prof Allison Littlejohn Dr Ian ScottProf Rhona Sharpe
Dr Simon Lygo-Baker
Prof Steven Warburton
Prof Ian Kinchin

3. Learning gains
We define Learning Gains as:
Growth or change in knowledge, skills, and abilities over
time that can be linked to the desired learning outcomes or
learning goals of the course
Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective.
Educational Research Review. Impact factor: 3.860

4. ABC project to measure learning gains?
Affect
Behaviour
Cognition
Academic
performance
VLE
Satisfaction

5. How are learning gains measured: a meta-analysis
• The concept of learning gain is
primarily used to examine the
effect of any particular
educational ‘intervention’
• There is a gradual increase in
studies examining learning
gains all across the world
• All learning gains can be
classified into ABC 53%
16% 21%
10% Behaviour-Cognitive
Learning Gains
Affective-Cognitive
Learning Gains
Affective-Behaviour-Cognitive
Learning Gains
Cognitive Learning Gains
Rogaten, J., Rienties, B, Cross, S., Whitelock, D., Sharpe, R., Lygo-Baker, S., Littlejohn, A. (Submitted: 07-07-2016). Reviewing the concept of learning gains in higher education: an affective, behaviour and cognitive perspective.
Educational Research Review. Impact factor: 3.860

6. Do students make learning gains, and what
is the power of LA to predict learning gains?
• Learning gains are usually measured using pre-post standardised testing
• It is resource intensive and becomes even more so if one wants to estimate
learning gains across various disciplines and number of universities
• Using assessment results for estimating learning gains has number of
advantages:
• Assessment data readily available
• Widely recognized as appropriate measure of learning
• Relatively free from self-reported biases
• Allows a direct comparison of research finding with the results of other studies
Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia

7. Modelling learning gains
• Pre-tests and post-tests are a standard method
• Raw gain
• True gain
• The normalised gain
• Normalised change
• ANCOVA on pre-test and post-test scores
• ANOVA on raw gain scores.
• ANOVA on residuals
• Repeated measures ANOVA

8. Three-level Growth Curve Model
Level 1
Level 2
Level 3
TMA1
Student1
TMA3 TMA1 TMA2 TMA3 TMA1 TMA2 TMA3TMA2
Student2 Student3
Module1 Module2
TMA1 TMA2 TMA3
Student4
TMA1 TMA2 TMA3
Student5
Module3
Rogaten, J., Rienties, B, Whitelock, D. (2016). Assessing learning gains, TEA Conference, Tallinn, Estonia

9. Current Study
• Participants
• STEM - 10,038students
• 74.4% males and 25.6% females
• Business and Law - 11,154 students
• 45.3% males and 54.7% females
• Academic performance for each Tutor Marked Assessment (TMA)
• TMA for 200 modules was retrieved from our university database for 2012/13, 2013/14
and 2014/15 academic years.

10. Math and Computing
(M=71.6; SD = 22.2)

11. Math and Computing

12. Business and Law
(M=64; SD = 15.2)

13. Business and Law

14. Basic Model
STEM Business and Law
Variance in students’ initial
achievements
46% 16%
variance in students’ learning
gains
77% 51%
Module-level intercept-slope
correlation
-.69** -0.43**
Student-level intercept-slope
correlation
-.15* -.1
VPC module 28.3% 9.6%
VPC student 33.8% 50.4%
VPC TMAs 37.9% 39.9%
Students in courses
with the low initial
achievements show
higher learning gains
than students in
courses with the high
initial achievements
Students with low
initial achievement
show higher learning
gains.

15. Key messages
• The module that any one student student is enrolled in accounts for a
substantial portion of variance, not just in the student’s initial academic
achievements, but also in the learning gains within the module.
• Multilevel modelling is a more accurate method compared with simple linear
models to estimate students’ learning gains because simple models are not
able to detect differences between modules when looking at the faculty level
performance
• Low learning gains - or negative learning gains of high achieving students -
does not imply that students are losing knowledge, however it highlights the
complexity of factors that have to be taken into account when using students’
academic performance as a proxy for learning gains.

16. Multilevel Modelling of Learning
Gains: The Impact of Module
Particulars on Students’ Learning
in Higher Education.
Jekaterina Rogaten
https://twitter.com/LearningGains
https://abclearninggains.com/
The results of the ABC project were made possible due to Jekaterina Rogaten, Bart Rienties, Simon Cross, Ceri Hitching, Ian Kinchin, Simon Lygo-Baker, Allison Littlejohn, Ian Scott, Rhona Sharpe, Steve Warburton, and Denise
Whitelock. Please contract Jekaterina.rogaten@open.ac.uk if you want to know more about ABC learning gains