Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

TrueLearn: A Family of Bayesian Algorithms to Match Lifelong Learners to Open Educational Resources (AAAI '20)


Published on

The recent advances in computer-assisted learning systems and the availability of open educational resources today promise a pathway to providing cost-efficient high-quality education to large masses of learners. One of the most ambitious use cases of computer-assisted learning is to build a lifelong learning recommendation system. Unlike short-term courses, lifelong learning presents unique challenges, requiring sophisticated recommendation models that account for a wide range of factors such as background knowledge of learners or novelty of the material while effectively maintaining knowledge states of masses of learners for significantly longer periods of time (ideally, a lifetime). This work presents the foundations towards building a dynamic, scalable and transparent recommendation system for education, modelling learner's knowledge from implicit data in the form of engagement with open educational resources. We i) use a text ontology based on Wikipedia to automatically extract knowledge components of educational resources and, ii) propose a set of online Bayesian strategies inspired by the well-known areas of item response theory and knowledge tracing. Our proposal, TrueLearn, focuses on recommendations for which the learner has enough background knowledge (so they are able to understand and learn from the material), and the material has enough novelty that would help the learner improve their knowledge about the subject and keep them engaged. We further construct a large open educational video lectures dataset and test the performance of the proposed algorithms, which show clear promise towards building an effective educational recommendation system.

Published in: Education
  • Be the first to comment

  • Be the first to like this

TrueLearn: A Family of Bayesian Algorithms to Match Lifelong Learners to Open Educational Resources (AAAI '20)

  1. 1. TrueLearn A Family of Bayesian Algorithms to Match Lifelong Learners to Open Educational Resources Department of Computer Science Sahan Bulathwela
  2. 2. 2 Overview • Background • Related Work • TrueLearn • Conclusion Department of Computer Science
  3. 3. 3 •X Modal, X Cultural, X Lingual, X Domain, and X Site Global OER Network ( •Convergence of OER media and provide equal accessibility to all •Leveraging AI to index, understand and match content with appropriate lifelong learners •Through content understanding, quality assurance and user modelling Department of Computer Science
  4. 4. 4 Open Educational Resources (OERs) are teaching, learning and research materials in any medium, digital or otherwise, that reside in the public domain or have been released under an open license that permits no-cost access, use, adaptation and redistribution by others with no or limited restrictions. Department of Computer Science - Hewlett Foundation -
  5. 5. 5 Related Work
  6. 6. 6 What drives learner engagement? • Rapid creation of educational resources [Pawlowski et. al (2007), Ehlers et. al (2018)] need scalable quality assurance [Camilleri (2014)]. • Detecting quality of educational resources is essential •In contrast to traditional content recommenders, personalised Learning Systems demand for features that account for novelty and learning trajectories [Drachsler et. al (2017), Bauman and Tuzhilin (2018)]. •User Interests and goals also drive finding relevant educational resources to learners Department of Computer Science
  7. 7. 7 Drivers of Learner Engagement Department of Computer Science [Bulathwela, Perez-Ortiz, Yilmaz and Shawe-Taylor (AAAI ’20a)]
  8. 8. 8 Q1: A Education Recommender •Problem Source: • In contrast to a traditional content recommender [Hu et. al (ICDM ’08) ,Covington et. al (RecSys ’16), Graepel et. al (ICML ’10)], personalised Learning Systems demand for additional features [Drachsler et. al (2017)] •Current Approaches in Education: • Collaborative Approaches [Milicevic et al. (2010), Bobadilla et al. (2010)]: Content features not considered, no trajectory, good for cross modality • Content based [Lops et al. (2011), Hammouda and Kamel (SDM ‘06)]: Only content similarity is considered, no trajectory, over-specialization • Hybrid and Other [Bauman and Tuzhilin (2018), Romero et. al (2009), Salehi et. al (2014), Drachsler et. al (IJLT ‘08)] complex, computationally expensive, complex to interpret, expert labelling etc. Department of Computer Science
  9. 9. 9 Desired features of an educational recommender •Cross Modal, Cross Lingual support allows better personalisation leading to recommending materials that are more likely to help learners achieve their desired learning outcomes [Lane (2008)]. •Transparency builds trust and enables meta-cognition processes [Bull and Kay (2016)]. •Data Efficiency as explicit feedback is expensive to acquire [Hu et. al (ICDM ’08), Covington et. al (RecSys ’16), Graepel et. al (ICML ’10)] •Incorporate novelty, interests of learners [Drachsler et. al (2017)] •Scalability is essential to provide quality education to masses of lifelong learners [José González-Brenes (EDM ‘14)] Department of Computer Science
  10. 10. 10 Personalised Learning Systems Department of Computer Science • One-on-one tutoring → 2σ improvement of performance [Bloom (1984)]. • ITS usually consists of a domain model, pedagogy model and a learning model [Holmes, Bialik and Fadel (2019)] . • Domain model: Subject materials [Hung (2014)] • Pedagogy Model: Effective teaching [Chaiklin (2003)] • Learner Model: Learner state [Self (1974), Corbett and Anderson (1994)]
  11. 11. 11 Content Analytics •Problem Source: • Very little work on Context Agnostic Engagement [Guo et al (L@S ‘14), Wu et al (ICWSM ‘18), Dalip et al (JEIQ’ 11)] • Extracting Knowledge Components (KCs) [Corbett and Anderson (1994), Yudelson et. al (AIED ‘13)] •Current Approaches: • Handcrafted by domain experts [Lindsey et. al (NIPS ’14), Pirolli and Kairam (2013), Dalip et al (JEIQ’ 11)] : Expensive, domain specific and not scalable • Automation [Chaplot et. al (AIED ‘18), Piech et. al (NIPS ‘15), Pirolli and Kairam (2013)]: Challenges in interpretability, stability of solutions • Wikipedia based ontology [Brank et. al (SiKDD ‘17), Mohamed Amir Yosef et. al (PVLDB ‘11)]: Scalable, human-interpretable, domain-agnostic, may not align with pedagogy. Department of Computer Science
  12. 12. 12 Learning Analytics •Problem Source: Scarcity of easy to interpret learner models that incorporate critical factors such as learner knowledge, content novelty and learner interests. •Current Approaches: • Item Response Theory: Rasch (1960) → Elo (1978), Pela´nek (2017) → Herbrich et. al (2007) → … • Knowledge Tracing: Corbett and Anderson (1994) → Yudelson et. al (AIED ‘13) → Piech et. al (NIPS ‘15) • Novelty :[Abuhamdeh and Csikszentmihalyi (2012), Lomas et. al (CHI ‘17)] • Interest/ goals : [Bauman and Tuzhilin (2018), Jiang et al (LAK ‘19)] Department of Computer Science
  13. 13. 13 Learning Analytics (IRT) •IRT: focuses on designing, analysing and scoring ability tests. •Done by modelling the learner skill and item difficulty simultaneously [Rasch (1960)]. •IRT is geared towards learning static skill level of user (not dynamic) •This idea has been extended to game skill learning in adaptations such as Elo algorithm [Elo (1978), Pela´nek (2017)] •TrueSkill® algorithm improves Elo algorithm by adding team game modelling capabilities and adding a dynamic factor [Herbrich et. al (2007)] Department of Computer Science
  14. 14. 14 Learning Analytics (KT) •Mostly used in Intelligent Tutoring Systems (ITS) •Tries to infer learner mastery of a skill [Corbett and Anderson (1994)] •Use mastery of skill to predict the outcome for next question •Improvements such as individualization has been proposed [Yudelson et. Al (AIED ‘13)] •Recently, Deep Knowledge Tracing [Piech et. al (NIPS ‘15)] •DKT is data hungry and has interpretability challenge Department of Computer Science
  15. 15. 15 TrueLearn
  16. 16. 16 TrueLearn with Novelty •A Family of Bayesian Algorithms inspired by TrueSkill algorithm [Herbrich et. al ( NIPS’07)] •Retains a humanly-intuitive leaner representation •Infers learner knowledge state •Incorporates novelty of educational resources •Predicts engagement [Guo et. al (L@S’14)] • Recognised Innovation at UNESCO Mobile Learning Week 2020. [UNESCO (2020)] Department of Computer Science
  17. 17. 17 Hypotheses Department of Computer Science
  18. 18. 18 Models •Naïve baselines: • Persistence • Majority •Alternate baselines from IRT and KT: • Online Knowledge Tracing model • Vanilla TrueSkill •Contributions: • Dynamic Depth TrueLearn Model (Background Knowledge) • Fixed Depth TrueLearn Model (Background) • TrueLearn Novelty Model (Background + Novelty) Department of Computer Science
  19. 19. 19 Factor Graphs of Models Department of Computer Science
  20. 20. 20 Results Department of Computer Science Weighted average test performance for accuracy, precision, recall and F1. Models labelled with (∆) are trained with positive and negative engagement. Models labelled with (∗) learn multiple skill parameters, one per Wikipedia page [Bulathwela, Perez-Ortiz, Yilmaz and Shawe-Taylor (AAAI ’20b)]
  21. 21. 21 Results Department of Computer Science [Bulathwela, Perez-Ortiz, Yilmaz and Shawe-Taylor (AAAI ’20b)]
  22. 22. 22 Conclusion • Educational recommenders demand additional features that go beyond a conventional recommender • Entity Linking provides a scalable and domain- agnostic content representation • TrueLearn algorithms are capable of maintaining a humanly intuitive representation while preserving its predictive performance. • TrueLearn performs significantly better with unseen topics compared to Knowledge Tracing counterparts. Department of Computer Science
  23. 23. 23 Future Directions • Semantic TrueLearn • Quality + Knowledge + Novelty + Interests • Integrate TrueLearn to X5Learn Learning Platform Department of Computer Science
  24. 24. 24 • AI powered learning platform to discover and learn from OERs. • State-of-the-art User Interface components to improve findability of information • Video: • Create playlists using OERs that can be shared with students • Convergence of OER media and provide equal accessibility to all Department of Computer Science [Bulathwela, Kreitmayer, Perez-Ortiz, Yilmaz and Shawe-Taylor (IUI ’20)]
  25. 25. 25 References • [Bulathwela, Perez-Ortiz, Yilmaz and Shawe-Taylor (AAAI ’20a)] Sahan Bulathwela, Maria Perez-Ortiz, Emine Yilmaz and John Shawe-Taylor. Towards an integrative educational recommender for lifelong learners. In Proceedings of the 2020 AAAI Conference on Artificial Intelligence. 2020 • [Bulathwela, Perez-Ortiz, Yilmaz and Shawe-Taylor (AAAI ’20b)] Sahan Bulathwela, Maria Perez-Ortiz, Emine Yilmaz and John Shawe-Taylor. TrueLearn: A Family of Bayesian Algorithms to Match Lifelong Learners with Open Educational Resources. In Proceedings of the 2020 AAAI Conference on Artificial Intelligence. 2020 • [Bulathwela, Kreitmayer, Perez-Ortiz, Yilmaz and Shawe-Taylor (IUI ’20)] Sahan Bulathwela, Stefan Kreitmayer, and María Pérez-Ortiz. 2020. What’s in it for me? Augmenting Recommended Learning Resources with Navigable Annotations. In Proceedings of the 25th International Conference on Intelligent User Interfaces Companion (IUI ’20). Association for Computing Machinery, New York, NY, USA, 114–115. DOI: • [Camilleri (2014)] Anthony F. Camilleri, Ulf Daniel Ehlers, and Jan Pawlowski. State of the art review of quality issues related to open educational resources (OER), volume 52 S. - JRC Scientific and Policy Reports of Publications Office of the European Union 2014. 2014 Department of Computer Science
  26. 26. 26 References • [Convington et. al (RecSys’16)] Paul Covington, Jay Adams, and Emre Sargin. Deep neural networks for youtube recommendations. In Proc. of ACM Conf. on Recommender Systems, 2016. • [Drachsler et. al (2017)] Hendrik Drachsler, Hans Hummel & Rob Koper (2008). Personal recommender systems for learners in lifelong learning: requirements, techniques and model. International Journal of Learning Technology, 3(4), 404–423. • [Ehlers et. al (2018)] Max Ehlers, Robert Schuwer, and Ben Janssen. Oer in tvet: Open educational resources for skills development, 2018 • [José González-Brenes (EDM ‘14)] José González-Brenes, Yun Huang, and Peter Brusilovsky. "General features in knowledge tracing to model multiple subskills, temporal item response theory, and expert knowledge." The 7th International Conference on Educational Data Mining. University of Pittsburgh, 2014 • [Greapel et. al (ICML’10)] Thore Graepel, Joaquin Quionero Candela, Thomas Borchert, and Ralf Herbrich. Web-scale bayesian click-through rate prediction for sponsored search advertising in microsoft’s bing search engine. In Proc. of Int. Conf. on Machine Learning, 2010 • [Guo et. al (L@S’14)] Philip J. Guo, Juho Kim, and Rob Rubin. How video production affects student engagement: An empirical study of mooc videos. In Proc. of the First ACM Conf. on Learning @ Scale, 2014. Department of Computer Science
  27. 27. 27 References • [Herbrich et. al ( NIPS’07)] Ralf Herbrich, Tom Minka, and Thore Graepel. Trueskill(tm): A bayesian skill rating system. In Advances in Neural Information Processing Systems 20, pages 569–576. MIT Press, January 2007. • [Hu et. al ICDM’08] Yifan Hu, Yehuda Koren, and Chris Volinsky. Collaborative filtering for implicit feedback datasets. In Proc. of Int. Conf. on Data Mining, volume 8, pages 263–272. Citeseer, 2008. • [Lane (2010)] Lane, Andrew (2008). Who puts the education into open educational content? In: Katz, Richard N. ed. The Tower and the Cloud: Higher Education in the Age of Cloud Computing. Boulder, Colorado: Educause, pp. 158–168. • [Pawlowski et. al (2007)] Jan M. Pawlowski, Volker Zimmermann, and Imc Ag. Open content: A concept for the future of e-learning and knowledge management?, 2007 • [UNESCO (2020)] UNESCO, Artificial Intelligence and Inclusion Programme, extracted from:, Retrieved on 27/05/2020 Department of Computer Science
  28. 28. TrueLearn A Family of Bayesian Algorithms to Match Lifelong Learners to Open Educational Resources Department of Computer Science Sahan Bulathwela