The recent advances in computer-assisted learning systems and the availability of open educational resources today promise a pathway to providing cost-efficient high-quality education to large masses of learners. One of the most ambitious use cases of computer-assisted learning is to build a lifelong learning recommendation system. Unlike short-term courses, lifelong learning presents unique challenges, requiring sophisticated recommendation models that account for a wide range of factors such as background knowledge of learners or novelty of the material while effectively maintaining knowledge states of masses of learners for significantly longer periods of time (ideally, a lifetime). This work presents the foundations towards building a dynamic, scalable and transparent recommendation system for education, modelling learner's knowledge from implicit data in the form of engagement with open educational resources. We i) use a text ontology based on Wikipedia to automatically extract knowledge components of educational resources and, ii) propose a set of online Bayesian strategies inspired by the well-known areas of item response theory and knowledge tracing. Our proposal, TrueLearn, focuses on recommendations for which the learner has enough background knowledge (so they are able to understand and learn from the material), and the material has enough novelty that would help the learner improve their knowledge about the subject and keep them engaged. We further construct a large open educational video lectures dataset and test the performance of the proposed algorithms, which show clear promise towards building an effective educational recommendation system.

1. TrueLearn
A Family of Bayesian Algorithms to Match
Lifelong Learners to Open Educational Resources
Department of Computer Science
Sahan Bulathwela

2. 2
Overview
• Background
• Related Work
• TrueLearn
• Conclusion
Department of Computer Science

3. 3
•X Modal, X Cultural, X Lingual, X
Domain, and X Site Global OER
Network (https://x5gon.org/)
•Convergence of OER media and
provide equal accessibility to all
•Leveraging AI to index, understand
and match content with appropriate
lifelong learners
•Through content understanding,
quality assurance and user
modelling
Department of Computer Science

4. 4
Open Educational Resources (OERs) are
teaching, learning and research materials in any
medium, digital or otherwise, that reside in the
public domain or have been released under an
open license that permits no-cost access, use,
adaptation and redistribution by others with no or
limited restrictions.
Department of Computer Science
- Hewlett Foundation -

5. 5
Related Work

6. 6
What drives learner engagement?
• Rapid creation of educational resources [Pawlowski et. al (2007), Ehlers et. al (2018)]
need scalable quality assurance [Camilleri (2014)].
• Detecting quality of educational resources is essential
•In contrast to traditional content recommenders, personalised Learning Systems
demand for features that account for novelty and learning trajectories [Drachsler et.
al (2017), Bauman and Tuzhilin (2018)].
•User Interests and goals also drive finding relevant educational resources to learners
Department of Computer Science

7. 7
Drivers of Learner Engagement
Department of Computer Science
[Bulathwela, Perez-Ortiz, Yilmaz
and Shawe-Taylor (AAAI ’20a)]

8. 8
Q1: A Education Recommender
•Problem Source:
• In contrast to a traditional content recommender [Hu et. al (ICDM ’08) ,Covington et. al (RecSys ’16), Graepel
et. al (ICML ’10)], personalised Learning Systems demand for additional features [Drachsler et. al (2017)]
•Current Approaches in Education:
• Collaborative Approaches [Milicevic et al. (2010), Bobadilla et al. (2010)]: Content features not considered, no
trajectory, good for cross modality
• Content based [Lops et al. (2011), Hammouda and Kamel (SDM ‘06)]: Only content similarity is considered, no
trajectory, over-specialization
• Hybrid and Other [Bauman and Tuzhilin (2018), Romero et. al (2009), Salehi et. al (2014), Drachsler et. al (IJLT
‘08)] complex, computationally expensive, complex to interpret, expert labelling etc.
Department of Computer Science

9. 9
Desired features of an educational
recommender
•Cross Modal, Cross Lingual support allows better personalisation leading to
recommending materials that are more likely to help learners achieve their desired
learning outcomes [Lane (2008)].
•Transparency builds trust and enables meta-cognition processes [Bull and Kay
(2016)].
•Data Efficiency as explicit feedback is expensive to acquire [Hu et. al (ICDM ’08),
Covington et. al (RecSys ’16), Graepel et. al (ICML ’10)]
•Incorporate novelty, interests of learners [Drachsler et. al (2017)]
•Scalability is essential to provide quality education to masses of lifelong learners
[José González-Brenes (EDM ‘14)]
Department of Computer Science

10. 10
Personalised Learning Systems
Department of Computer Science
• One-on-one tutoring → 2σ improvement of
performance [Bloom (1984)].
• ITS usually consists of a domain model, pedagogy
model and a learning model [Holmes, Bialik and Fadel
(2019)] .
• Domain model: Subject materials [Hung (2014)]
• Pedagogy Model: Effective teaching [Chaiklin (2003)]
• Learner Model: Learner state [Self (1974), Corbett and
Anderson (1994)]

11. 11
Content Analytics
•Problem Source:
• Very little work on Context Agnostic Engagement [Guo et al (L@S ‘14), Wu et al (ICWSM ‘18), Dalip et al
(JEIQ’ 11)]
• Extracting Knowledge Components (KCs) [Corbett and Anderson (1994), Yudelson et. al (AIED ‘13)]
•Current Approaches:
• Handcrafted by domain experts [Lindsey et. al (NIPS ’14), Pirolli and Kairam (2013), Dalip et al (JEIQ’ 11)] :
Expensive, domain specific and not scalable
• Automation [Chaplot et. al (AIED ‘18), Piech et. al (NIPS ‘15), Pirolli and Kairam (2013)]: Challenges in
interpretability, stability of solutions
• Wikipedia based ontology [Brank et. al (SiKDD ‘17), Mohamed Amir Yosef et. al (PVLDB ‘11)]: Scalable,
human-interpretable, domain-agnostic, may not align with pedagogy.
Department of Computer Science

12. 12
Learning Analytics
•Problem Source:
Scarcity of easy to interpret learner models that incorporate critical factors such as
learner knowledge, content novelty and learner interests.
•Current Approaches:
• Item Response Theory:
Rasch (1960) → Elo (1978), Pela´nek (2017) → Herbrich et. al (2007) → …
• Knowledge Tracing:
Corbett and Anderson (1994) → Yudelson et. al (AIED ‘13) → Piech et. al (NIPS ‘15)
• Novelty :[Abuhamdeh and Csikszentmihalyi (2012), Lomas et. al (CHI ‘17)]
• Interest/ goals : [Bauman and Tuzhilin (2018), Jiang et al (LAK ‘19)]
Department of Computer Science

13. 13
Learning Analytics (IRT)
•IRT: focuses on designing, analysing and scoring ability tests.
•Done by modelling the learner skill and item difficulty
simultaneously [Rasch (1960)].
•IRT is geared towards learning static skill level of user (not
dynamic)
•This idea has been extended to game skill learning in
adaptations such as Elo algorithm [Elo (1978), Pela´nek (2017)]
•TrueSkill® algorithm improves Elo algorithm by adding team
game modelling capabilities and adding a dynamic factor
[Herbrich et. al (2007)]
Department of Computer Science

14. 14
Learning Analytics (KT)
•Mostly used in Intelligent Tutoring Systems (ITS)
•Tries to infer learner mastery of a skill [Corbett and
Anderson (1994)]
•Use mastery of skill to predict the outcome for next question
•Improvements such as individualization has been proposed
[Yudelson et. Al (AIED ‘13)]
•Recently, Deep Knowledge Tracing [Piech et. al (NIPS ‘15)]
•DKT is data hungry and has interpretability challenge
Department of Computer Science

15. 15
TrueLearn

16. 16
TrueLearn with Novelty
•A Family of Bayesian Algorithms inspired by
TrueSkill algorithm [Herbrich et. al ( NIPS’07)]
•Retains a humanly-intuitive leaner
representation
•Infers learner knowledge state
•Incorporates novelty of educational resources
•Predicts engagement [Guo et. al (L@S’14)]
• Recognised Innovation at UNESCO Mobile
Learning Week 2020. [UNESCO (2020)]
Department of Computer Science

17. 17
Hypotheses
Department of Computer Science

18. 18
Models
•Naïve baselines:
• Persistence
• Majority
•Alternate baselines from IRT and KT:
• Online Knowledge Tracing model
• Vanilla TrueSkill
•Contributions:
• Dynamic Depth TrueLearn Model (Background Knowledge)
• Fixed Depth TrueLearn Model (Background)
• TrueLearn Novelty Model (Background + Novelty)
Department of Computer Science

19. 19
Factor Graphs of Models
Department of Computer Science

20. 20
Results
Department of Computer Science
Weighted average test performance for accuracy, precision, recall and F1. Models
labelled with (∆) are trained with positive and negative engagement. Models labelled
with (∗) learn multiple skill parameters, one per Wikipedia page
[Bulathwela, Perez-Ortiz, Yilmaz and
Shawe-Taylor (AAAI ’20b)]

21. 21
Results
Department of Computer Science
[Bulathwela, Perez-Ortiz, Yilmaz and
Shawe-Taylor (AAAI ’20b)]

22. 22
Conclusion
• Educational recommenders demand additional
features that go beyond a conventional
recommender
• Entity Linking provides a scalable and domain-
agnostic content representation
• TrueLearn algorithms are capable of
maintaining a humanly intuitive representation
while preserving its predictive performance.
• TrueLearn performs significantly better with
unseen topics compared to Knowledge Tracing
counterparts.
Department of Computer Science

23. 23
Future Directions
• Semantic TrueLearn
• Quality + Knowledge + Novelty + Interests
• Integrate TrueLearn to X5Learn Learning
Platform
Department of Computer Science

24. 24
• AI powered learning platform to discover
and learn from OERs.
http://x5learn.org/
• State-of-the-art User Interface
components to improve findability of
information
• Video:
https://youtu.be/7h_6KUIVO8s
• Create playlists using OERs that can be
shared with students
• Convergence of OER media and provide
equal accessibility to all
Department of Computer Science
[Bulathwela, Kreitmayer, Perez-Ortiz, Yilmaz
and Shawe-Taylor (IUI ’20)]

25. 25
References
• [Bulathwela, Perez-Ortiz, Yilmaz and Shawe-Taylor (AAAI ’20a)] Sahan Bulathwela, Maria Perez-Ortiz, Emine
Yilmaz and John Shawe-Taylor. Towards an integrative educational recommender for lifelong learners. In
Proceedings of the 2020 AAAI Conference on Artificial Intelligence. 2020
• [Bulathwela, Perez-Ortiz, Yilmaz and Shawe-Taylor (AAAI ’20b)] Sahan Bulathwela, Maria Perez-Ortiz, Emine
Yilmaz and John Shawe-Taylor. TrueLearn: A Family of Bayesian Algorithms to Match Lifelong Learners with Open
Educational Resources. In Proceedings of the 2020 AAAI Conference on Artificial Intelligence. 2020
• [Bulathwela, Kreitmayer, Perez-Ortiz, Yilmaz and Shawe-Taylor (IUI ’20)] Sahan Bulathwela, Stefan Kreitmayer, and
María Pérez-Ortiz. 2020. What’s in it for me? Augmenting Recommended Learning Resources with Navigable
Annotations. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion (IUI
’20). Association for Computing Machinery, New York, NY, USA, 114–115.
DOI:https://doi.org/10.1145/3379336.3381457
• [Camilleri (2014)] Anthony F. Camilleri, Ulf Daniel Ehlers, and Jan Pawlowski. State of the art review of quality
issues related to open educational resources (OER), volume 52 S. - JRC Scientific and Policy Reports of
Publications Office of the European Union 2014. 2014
Department of Computer Science

26. 26
References
• [Convington et. al (RecSys’16)] Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube
recommendations. In Proc. of ACM Conf. on Recommender Systems, 2016.
• [Drachsler et. al (2017)] Hendrik Drachsler, Hans Hummel & Rob Koper (2008). Personal recommender systems for learners
in lifelong learning: requirements, techniques and model. International Journal of Learning Technology, 3(4), 404–423.
• [Ehlers et. al (2018)] Max Ehlers, Robert Schuwer, and Ben Janssen. Oer in tvet: Open educational resources for skills
development, 2018
• [José González-Brenes (EDM ‘14)] José González-Brenes, Yun Huang, and Peter Brusilovsky. "General features in
knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge." The 7th International
Conference on Educational Data Mining. University of Pittsburgh, 2014
• [Greapel et. al (ICML’10)] Thore Graepel, Joaquin Quionero Candela, Thomas Borchert, and Ralf Herbrich. Web-scale
bayesian click-through rate prediction for sponsored search advertising in microsoft’s bing search engine. In Proc. of Int. Conf.
on Machine Learning, 2010
• [Guo et. al (L@S’14)] Philip J. Guo, Juho Kim, and Rob Rubin. How video production affects student engagement: An
empirical study of mooc videos. In Proc. of the First ACM Conf. on Learning @ Scale, 2014.
Department of Computer Science

27. 27
References
• [Herbrich et. al ( NIPS’07)] Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill(tm): A bayesian skill rating
system. In Advances in Neural Information Processing Systems 20, pages 569–576. MIT Press, January 2007.
• [Hu et. al ICDM’08] Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback
datasets. In Proc. of Int. Conf. on Data Mining, volume 8, pages 263–272. Citeseer, 2008.
• [Lane (2010)] Lane, Andrew (2008). Who puts the education into open educational content? In: Katz, Richard
N. ed. The Tower and the Cloud: Higher Education in the Age of Cloud Computing. Boulder, Colorado: Educause,
pp. 158–168.
• [Pawlowski et. al (2007)] Jan M. Pawlowski, Volker Zimmermann, and Imc Ag. Open content: A concept for the
future of e-learning and knowledge management?, 2007
• [UNESCO (2020)] UNESCO, Artificial Intelligence and Inclusion Programme, extracted from:
https://en.unesco.org/sites/default/files/mlw2020-programme-en.pdf#page=5, Retrieved on 27/05/2020
Department of Computer Science

28. TrueLearn
A Family of Bayesian Algorithms to Match
Lifelong Learners to Open Educational Resources
Department of Computer Science
Sahan Bulathwela