The Elo rating system has been recognised as an effective method for modelling students and items within adaptive educational systems. The existing Elo-based models have the limiting assumption that items are only tagged with a single concept and are mainly studied in the context of adaptive testing systems. In this paper, we introduce a multivariate Elo-based learner model that is suitable for the domains where learning items can be tagged with multiple concepts, and investigate its fit in the context of adaptive learning. To evaluate the model, we first compare the predictive performance of the proposed model against the standard Elo-based model using synthetic and public data sets. Our results from this study indicate that our proposed model has superior predictive performance compared to the standard Elo-based model, but the difference is rather small. We then investigate the fit of the proposed multivariate Elo-based model by integrating it into an adaptive learning system which incorporates the principles of open learner models (OLMs). The results from this study suggest that the availability of additional parameters derived from multivariate Elo-based models have two further advantages: guiding adaptive behaviour for the system and providing additional insight for students and instructors.

1. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 1
A Multivariate Elo-based Learner Model for
Adaptive Educational Systems
Authors: Solmaz Abdi, Hassan Khosravi, Shazia Sadiq, Dragan Gasevic
Presenter: Dr Hassan Khosravi
Senior Lecturer at The University of Queensland
h.khosravi@uq.edu.au
@haskhosravi
hassan-khosravi.net
The 12th International Conference on Educational Data Mining

2. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 2
Introduction
The M-ELO Approach
Evaluation: Predictive Performance
Conclusion and Future Work
Background
Evaluation: Fit for Adaptive Learning

3. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 3
Introduction
• Educators continue to face significant challenges in
providing high quality instruction in large diverse online or
on-campus classes.
• Providing tailored
learning resources.
• Monitoring
students’
achievements.
• Providing helpful
feedback and
guidance.
https://myams.org/wp-content/uploads/2015/01/lecture-22903.jpg

4. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 4
Adaptive Educational Systems
Adaptive educational systems make use of data about
students, learning process, and learning products to adapt
the level or type of instruction for each student.
Domain Model Learner Model
Content Repository
Recommendation Engine

5. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 5
Modelling Learners in Adaptive
Educational Systems
• Conventional approaches for learner modelling:
– Bayesian knowledge Tracing (BKT) and its extensions.
– Item Response Theory (IRT) and its extensions
• Neither of these approaches are well-suited for adaptive educational
systems as they generally require pre-calibration on big samples
(Planek, 2016).
• The Elo rating system has been shown to be an effective alternative
for modelling students in adaptive educational system (Wauters et
al., 2011).

6. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 6
The Elo Rating
• The Elo rating system was originally used to rate chess players.
This rating system is self-correcting, meaning that the ratings, in the
long run, should correctly reflect the skill level of the player
Jack 920 Jane 1600
Outcome Jack’s rating Jane’s rating
Jane wins 910 1610
Jack wins 1000 1520
The sum of the updates to the ratings of
the players is always zero (zero sum games)

7. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 7
ELO Rating in Education
• A similar comparison can be conducted between a student
and a question being attempted by the student.
James - 920
Outcome Jame’s
rating
Q38 ‘s
difficulty
Answered
incorrectly
910 1610
Answered
correctly
1000 1520
This rating system is self-correcting, meaning that, in the long run, it should
correctly reflect the mastery of the student and difficulty level of the question.
The sum of the updates to the ratings of
the players is always zero (zero sum games)
Question 38 - 1600

8. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 8
Existing Studies and Our Contribution
Existing studies
• Use repositories that
contain items that are
pure
• Are studied in the
context of adaptive
testing systems.
Our Contribution
• Introduce a new variant
that models students and
items using repositories
that contain non-pure
items.
• Investigate its fit in the
context of adaptive
learning systems.

9. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 9
The M-ELO Approach
Evaluation: Predictive Performance
Conclusion and Future Work
Introduction
Evaluation: Fit for Adaptive Learning
Background

10. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 10
Adaptive Educational Systems
Adaptive Testing
• Conducting an exam using a
sequence of questions that are
successively administered.
• Aiming to maximise the precision
of the score within a reasonable
timeframe.
– Exam terminates upon estimating
a student’s ability with a
confidence level that exceeds a
user-specific threshold.
• Students competencies are not
expected to change while using
the system to do an exam.
Adaptive Learning
• Assisting students in their
learning by recommending
learning resources.
• Aiming to provide rich feedback
to students on their learning.
• Students can spend theoretically,
an infinite amount of time on a
learning item or on the system.
• Students competencies are
expected to improve via receiving
feedback or decline as a result of
forgetting over time.

11. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 11
Open Learner Models
Open learner models (OLMs) are learner models that are
externalised and made accessible to students or other
stakeholders often through visualisation, as an important
means of supporting learning.
Visualisations showing comparison of learning outcome achievements (Law et al., 2015)

12. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 12
RiPPLE: Adaptive Learning Meets Crowdsourcing
RiPPLE (Recommendation in Personalised Peer Learning Environments)
recommends personalised learning activities to students based on
their knowledge state from a pool of crowdsourced learning activities.

13. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 14
Background
Evaluation: Predictive Performance
Conclusion and Future Work
Introduction
Evaluation: Fit for Adaptive Learning
The M-ELO Approach

14. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 15
Notation
Learner model
!" A set of # students enrolled in the course, where %& is an arbitrary student
∆( A set of knowledge components contributed to the course, where )* is an arbitrary knowledge
component (domain model)
+, A set of - items contributed to the course, ./ is an arbitrary item (content model)
W,×( Matrix, where 1/* is 2
3⁄ if item ./ is tagged with 5 knowledge components including knowledge
component )*, and 0 otherwise.
6"×, Matrix, where 7&/ is 1 if student %& answers item ./ correctly and 0 if answered incorrectly
Modelling learners and Items
8 " Vector, The Elo-based learner model, where 8& indicates %&’s proficiency level on the entire
domain
L"×, Matrix, The Elo based learner model based on multi-concept M-Elo (where l&* represents student
%&’s Elo rating on knowledge component )* , approximating the proficiency level of the student on
that certain knowledge component
9/ Vector, The Elo rating of questions in the question repository, where :/ indicates the
approximated difficulty of question ./

15. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 16
Standard Elo-based Learner Model
7
James - 920
Outcome Jame’s
rating
Q38 ‘s
difficulty
Answered
incorrectly
910 1610
Answered
correctly
1000 1520
The sum of the updates to the ratings of
the players is always zero (zero sum games)
Question 38 - 1600

16. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 17
Standard Elo-based Learner Model
7
;(7&/|8&, :/) = @(8& − :/)
:/ ≔ :/ + D(; 7&/|8&, :/ − 7&/)
Probability of %& answering ./ correctly
Logistic function
Updating the estimate of :/
8& ≔ 8& + D(7&/ − ;(7&/|8&, :/))
Updating %&’s Elo rating
K = Sensitivity of the estimations to the
student’s last attempt. Can be replaced with:
! E =
G
1 + I ∗ E
2
Number of prior updates
3
1
Follows the principles
of a zero-sum game

17. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 18
Single-Concept Multivariate Elo-based Learner model
(Doebler et al., 2015; Pelanek et al., 2017)
7
James
Outcome Jame’s rating Q38 ‘s
difficulty
Answered
incorrectly
Answered
correctly
DBMS ER … Map-ER …
1100 1040 … 910 …
DBMS ER … Map-ER …
1170 1040 … 1000 …
1610
1520
DBMS ER … Map-ER
1100 1040 … 920
Question 40 - 1600
Question 38 – 1600
(Map-ER)

18. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 19
Multi-Concept Multivariate Elo-based Learner model
7
James
Outcome Jame’s rating Q40 ‘s
difficulty
Answered
incorrectly
Answered
correctly
DBMS ER … Map-ER …
1080 1040 … 910 …
DBMS ER … Map-ER …
1170 1130 … 970 …
1630
1510
DBMS ER … Map-ER
1100 1040 … 920
Question 40 - 1600
(DBMS, Map-ER)

19. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 20
Multi-Concept Multivariate Elo-based
learner model (M-Elo)
%&’s average competency on the knowledge
components associated with item ./
1
lK&/ = ∑ l&*×1/*
(
*M2
;(7&/|lK&/, :/) = @(lK& − :/)
:/ ≔ :/ + D(; 7&/|lK&/, :/ − 7&/)
Probability of %& answering ./ correctly
Logistic function
Updating the estimate of :/
l&* ≔ l&* + a N D(7&/ − ;(7&/|l&*, :/))
Updating %&’s Elo rating on each knowledge
component )* the question is tagged with
Sensitivity of the estimations to the student’s last
attempt. Can be replaced with:
! E =
G
1 + I ∗ E
a =
|; 7&/ lK&/, :/ − 7&/|
∑ (|7&/ − ;(7&/|l&*, :/)×1/*|)(
&M2
2
3 4
Number of prior updates
Follows the principles
of a zero-sum game

20. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 21
Background
The M-ELO Approach
Conclusion and Future Work
Introduction
Evaluation: Fit for Adaptive Learning
Evaluation: Predictive Performance

21. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 22
Evaluation: Predictive Performance
• Comparing the predictive performance of M-Elo
against Elo using
– A suite of simulated data sets
– Publicly available data sets
• Three Metrics:
– Area under the curve (AUC)
– Root Mean Squared Error (RMSE)
– Accuracy (ACC)

22. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 23
Synthetic Data Set
Dirichlet distribution
(topic level gaps)
σ
Sparsity
U
Users
L
# topics Discrete uniform
distribution
(topics)
Q
Questions
Latent Trait
Models
A
T
Normal distribution
(difficulty,
discrimination)
Normal
distribution
D
Tags
Difficulties
Answers
• For all experiments, 100 students, 1000 learning items and 70000 answers were
sampled. 70% of the created data set was used for training, and the remaining 30%
reserved for testing.
• Each experiment was repeated for 5 times and the reported values are the average
results across the five runs (Desmarais et al., 2010; Khosravi et al., 2017).

23. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 24
Synthetic Data Sets - Results
• For L=10, as σ is increased and students with more diversity in their abilities
across different concepts are generated, M-Elo outperforms Elo.
• For L=100, the same trend is observable; however, the intersection point for
where M-Elo outperforms Elo occurs for larger value of σ.

24. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 25
Public Data Sets
Data Set Alg2005 Alg2006 BAlg2006
Students 575 1840 1146
KC 112 714 493
Items 147,914 319,151 19,954
Multi-KC 51,171 21,415 1,650
Interactions 609,979 1,825,030 1,822,697
Data sets from PSLC datashop
• AlgebraI 2005-2006 (Alg2005)
• AlgebraI 2006-2007 (Alg2006)
• Bridge to Algebra 2006-2007
(BAlg2006)
Data cleaning
• Using the train/test split provided by KDD Cup 2010
• Discarding interactions with no assigned concepts
• Using Step Name column as the learning item
• Each learning item is associated with one or more concepts (KC)
covered in the course

25. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 26
Public Data Sets - Results
Data Set AUC
Elo M-Elo
Alg2005 0.726 0.750
Alg2006 0.687 0.695
BAlg2006 0.676 0.712
RMSE
Elo M-Elo
0.392 0.385
0.394 0.390
0.368 0.361
ACC
Elo M-Elo
0.787 0.790
0.784 0.797
0.827 0.828
• M-Elo outperforms Elo on all three data sets.
• It may be possible to hypothesise that students often
have different competency levels on different
concepts, but these differences are often not too
significant.

26. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 27
Background
The M-ELO Approach
Conclusion and Future Work
Introduction
Evaluation: Predictive Performance
Evaluation: Fit for Adaptive Learning

27. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 28
Case study
• Location: The University of Queensland (UQ)
• Course: Introduction to relational databases
• General Topics: 17 concepts such mainly on
ER diagrams, relational models, functional
dependencies and SQL.
• Number of students: 521
• Number of items: 1,632
• Number of attempts: 91,340
The learner model is shared with students based on the principles of OLMs through
a visualisation widget

28. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 29
Guiding Adaptivity
• Estimating the knowledge state of students on each
topic as well as estimating the difficulty level of
each item.
• Recommending learning items based on knowledge
gaps at the right level of difficulty.
The adaptivity at the concept level is possible because of the
availability of the additional parameters learned by M-Elo,
which is not achievable using standard Elo rating

29. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 30
Insights from Student Survey (N=53)
• Motivation: The visualisation used by RiPPLE for showing my
knowledge state increase my motivation to study or further use the
system
• Rationality: Having the ability to visually see my knowledge state,
help to understand the rationale behind suggestions made by the
system
• Trust: Having the ability to visually see my knowledge state,
increases my trust in recommended items

30. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 31
Insights from the Case Study
• Challenge: Maintaining a balance between the students' proficiency
level and the learning items' difficulty level in our pilot.
• Outcome: Students lost motivation in answering questions as the
large loss of rating in answering the question incorrectly outweighed
the small rating gain received in answering the question correctly.
The zero-sum game principles of the Elo rating might not be ideal for
adaptive learning systems.
600
800
1000
1200
1400
Rating
Time
Avg student and question rating over time
Student rating question rating
Potential explanation: Students have
full access to the internet, textbooks
and colleagues as well as an infinite
amount of time for answering a
question.

31. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 32
Insights from the Case Study
• Challenge: reducing the sensitivity of the estimations over time as
the number of attempts is increased means that students are not able
to make significant changes to their knowledge state despite
answering questions correctly.
• Outcome: students tend to get discouraged from using the system
once they have used it for a while.
Reducing the sensitivity of the estimations over time might not be
ideal for adaptive learning systems.
600
800
1000
1200
1400
Rating
Time
Avg student and question rating over time
Student rating question rating
This seems better suited for adaptive
testing where students are not expected
to learn in one sitting and the uncertainty
function helps stabilise the ratings.

32. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 33
Background
The M-ELO Approach
Evaluation: Fit for Adaptive Learning
Introduction
Evaluation: Predictive Performance
Conclusion and Future Work

33. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 34
Conclusion and Future Work
• We introduced a new multi-variate Elo-based learner
model called M-Elo where learning items can be tagged
with one or more concepts.
• M-Elo outperforms the standard Elo-based model in real-
world environments, is interpretable by students and
provides better concept-level adaptivity.
• Future work focuses on tailoring M-Elo to make a better fit
for adaptive learning than adaptive testing.

34. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 35
Thank you!
Presenter: Dr Hassan Khosravi
Senior Lecturer at The University of Queensland
h.khosravi@uq.edu.au
@haskhosravi
hassan-khosravi.net

35. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 36
The Elo Rating
• The Elo rating system was originally used to rate chess players.
• Principle: each player is assigned a rating, which is updated after
each match.
– The rating of the winner is increased and the rating of the loser is
decreased.
– If a strong player beats a weak player, the result is not surprising
and the update is small, whereas if the opposite happens, the
update is large.
– The sum of the updates to the ratings of the players is always
zero (zero sum games)
This rating system is self-correcting, meaning
that the ratings, in the long run, should
correctly reflect the skill level of the player

36. A Multivariate Elo-based Learner Model for Adaptive Educational Systems Page 37
ELO Rating in Education
• A similar comparison can be conducted between a student
and a question being attempted by the student.
• Principle: students and questions are assigned a rating, which is
updated after each attempt.
• If the question is answered correctly, the student's rating increases
and the rating of the question decreases.
• If the question is answered incorrectly, the student's rating decreases
and the rating of the question increases.
• The update to the ratings is proportional to the difference between
the ratings of the student and the question.
• The sum of the updates to the ratings of the players is always zero
(zero sum games)