Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Recommendations for Open Online Education: An Algorithmic Study


Published on

Recommending courses to students in online platforms is studied widely. Almost all studies target closed platforms, that belong to a University or some other educational provider. This makes the course recommenders situation specific. Over the last years, a demand has developed for recommender system that suit open online platforms. Those platforms have some common characteristics, such as the lack of rich user profiles with content metadata. Instead they log user interactions within the platform that can be used for analysis and personalization. In this paper, we investigate how user interactions and activities tracked within open online learning platforms can be used to provide recommendations. We present a study in which we investigate the application of several state-of-the-art recommender algorithms, including a graph-based recommender approach. We use data from the OpenU open online learning platform that is in use by the Open University of the Netherlands. The results show that user-based and memory-based methods perform better than model-based and factorization methods. Particularly, the graph-based recommender system proves to outperform the classical approaches on prediction accuracy of recommendations in terms of recall. We conclude that, if the algorithms are chosen wisely, recommenders can contribute to a better experience of learners in open online courses.
Soude Fazeli, Enayat Rajabi, Leonardo Lezcano, Hendrik Drachsler, Peter Sloep

Published in: Education
  • Be the first to comment

Recommendations for Open Online Education: An Algorithmic Study

  1. 1. Recommendations for Open Online Education: An Algorithmic Study Soude Fazeli1, Enayat Rajabi2, Leonardo Lezcano3, Hendrik Drachsler1, Peter Sloep1 1 Open University Netherlands, 2 Dalhousie University, 3 eBay Inc. 27.07.2016, ICALT 2016, Austin, Texas, USA
  2. 2. 3 • Hendrik Drachsler Associate Professor Learning Technologies • Research topics: Personalization, Recommender Systems, Learning Analytics, Mobile devices • Application domains: Schools, HEI, Medical education WhoAmI 2006 - 2009 @HDrachsler
  3. 3. 27/07/16Hendrik Drachsler 3
  4. 4. Context of the study • Goal: Personalization of Learning (based on prior knowledge) • Problem: Selection from a huge variety of possibilities (Information overflow) • Solution: Recommender systems that points a target user to content of interest based on her user profile Recommendations for Open Education: An Algorithimic Study Pagina 4
  5. 5. Problem definition Recommendations for Open Education: An Algorithimic Study Pagina 5 Institutional Course RecSys Open Education RecSys VS. Rich learner and course metadata Sparse learner and course metadata
  6. 6. Pagina 6 RQ: How to recommend courses to learners in open education platforms? Recommendations for Open Education: An Algorithimic Study Research Question
  7. 7. Pagina 7 1. Content-based 2. Collaborative filtering ✓ Recommendations for Open Education: An Algorithimic Study Recommender system algorithms Our Input Data are mainly user indirect ratings, thus collaborative filtering are more relevant for us
  8. 8. 8 Drachsler, H., Verbert, K., Santos, O., and Manouselis, N. (2015). Recommender Systems for Learning. 2nd Handbook on Recommender Systems. Berlin:Springer Recommender system algorithms
  9. 9. Pagina 9 • Memory-based • Use statistical approaches to infer similarity between users based on the users’ data stored in memory • k-Nearest Neighbour method (kNN, with neighbourhood size k) • Similarity metrics: Pearson correlation, Cosine similarity, and the Jaccard coefficient. • Model-based • Use probabilistic approaches to create a model of users’ feedback • Matrix factorization, and Bayesian networks • are faster than memory-based algorithms • more costly (required resources and maintenance) In this study, we use both memory-based (both user-based and item-based) and model-based algorithms to test which one performs best on the Open U platform. Recommendations for Open Education: An Algorithimic Study Collaborative Filtering (CF) algorithms
  10. 10. Pagina 10 H1: Item-based outperforms user-based approaches H2: Model-based outperforms memory-based approaches Recommendations for Open Education: An Algorithimic Study Hypothesis
  11. 11. Experiment Pagina 11 1. Dataset • From Open Education Platform: OpenU A broad national online learning platform for lifelong learning • Data collected: from March 2009 until September 2013 • Users: OpenU Users are professionals from various domains Dataset Users Learning objects Transactions Sparsity (%) OpenU 3462 105 92,689 98.14 Recommendations for Open Education: An Algorithimic Study
  12. 12. Pagina 12 • Figure 1: Course completion in related to the students’ activity • Each blue X: the Percentage of Online Interactions (POI) for a given student and a given course, relative to the highest online interactions of a student in that course. • Online interactions = student’s contributions to chat sessions and forum messages. The course completion rate for OpenU students goes up dramatically with increases in students’ interactions (course-mates and the academic staff) Recommendations for Open Education: An Algorithimic Study Experiment 1. Data set
  13. 13. Pagina 13 Experiment 2. Algorithms 2.1. Memory-based • Most CF algorithms are based on kNN methods: • Find like-minded users and introduces them as the target user’s nearest neighbours • The appropriate similarity measure depends on whether the input data is: • Explicit (e.g. 5-star ratings) or • Implicit user feedback (e.g. views, downloads, clicks, etc.) • Open U = Implicit user feedback (activities) -> Jaccard coefficient and Cosine are appropriate Recommendations for Open Education: An Algorithimic Study
  14. 14. Pagina 14 Experiment 2. Algorithms 2.2. Model-based • Bayesian Personalized Ranking (BPR) method proposed by Rendle et al. • They applied their BPR to the state-of-the-art matrix factorization models to improve the learning process in the Bayesian model used (BPRMF). • MostPopular approach • Makes recommendations based on general popularity of items • Items are weighted based on how often they have been seen in the past S. Rendle, C. Freudenthaler, Z. Gantner, and L. Schmidt-Thieme, “BPR: Bayesian Personalized Ranking from Implicit Feedback,” in UAI ’09 Proceedings of the Twenty- Fifth Conference on Uncertainty in Artificial Intelligence, 2009, pp. 452–461
  15. 15. Pagina 15 Experiment 2. Algorithms 2.3. Graph-based • Implicit networks: a graph – Nodes: users; Edges: similarity relationships; Weights: similarity values • Improve the process of finding nearest neighbors – By invoking graph search algorithms – Memory-based and user-based – For more information, see our ECTEL2014 paper: S. Fazeli, B. Loni, H. Drachsler, and P. Sloep, “Which Recommender system Can Best Fit Social Learning Platforms?,” in 9th European Conference on Technology Enhanced Learning, EC-TEL 2014, 2014, pp. 84–97.
  16. 16. Pagina 16 Experiment 3. Settings • Metrics • Precision (ratio number of relevant items recommended to the total number of recommended items) • Recall shows the probability that a relevant item is recommended • Both Precision and recall range from 0 to 1. • The number of courses in this experiment is 105 thus • The number of top-n items to be recommended is 5 (approx. 5% of the courses) and 10 (approx.10% of the courses). • For each memory-based CF algorithm, we evaluated six neighbourhood sizes (k={5,10,20,30,50,100}). Recommendations for Open Education: An Algorithimic Study
  17. 17. Pagina 17 1. Memory-based • User-based with Jaccard (UB1) • User-based with Cosine (UB2) • Item-based with Jaccard (IB1) • Item-based with Cosine (IB2) 2. Model-based • MostPopular (MB1) • Bayesian Personalized Ranking with Matrix Factorization (MB2) 3. Graph-based • User-based with T-index (UB3) Experiment Which algorithm and parameters are best suited for the users of the Open U learning platform?
  18. 18. Experimental study 3. Results Pagina 18 Values for the highest- scoring neighbourhood size are in bold, the highest values among all are underlined
  19. 19. Discussions H1: Item-based outperformed user-based methods. User-based CFs exceeded all expectations - contrary to what the recommender systems literature suggests. • Item-based results were expected to trump the user-based since the number of items (courses) is much smaller than the number of users for our dataset. • User-based algorithms performed better on the Open U data than those that make use of similarities between items (courses). • Therefore: we reject H1. Pagina 19 Recommendations for Open Education: An Algorithimic Study
  20. 20. Discussions H2: Matrix factorization methods outperform memory-based methods. • The user-based recommenders (UB1, UB2, UB3), which are memory- based, widely outperform the model-based ones (MB1, MB2). • We expected the matrix factorization (model-based CFs) to perform better since they often prove to outperform prediction accuracy of recommendations particularly when explicit user feedback is available (e.g. 5-star ratings). • So we reject also H2. Pagina 20 Recommendations for Open Education: An Algorithimic Study
  21. 21. Pagina 21 • This study sought to find out how best to generate personalized recommendations from user activities within an open online course platform. • The results show that user-based and memory- based methods perform better than item and model-based factorization methods. • The UB1 algorithms seem to be most suited to provide accurate recommendations to the users of our Open U platform. Recommendations for Open Education: An Algorithimic Study Conclusion
  22. 22. Pagina 22 Ongoing and Further work 1. Integrating the selected recommender algorithms in the OpenU platform to provide online recommendations. 2. Studying how the graph-based approach can help to improve the process of finding like-minded neighbours in terms of social network analysis (SNA) 3. User study – To measure novelty and serendipity of the recommendations made for OpenU users. Recommendations for Open Education: An Algorithimic Study
  23. 23. Pagina 23 soude.fazeli[at]ou[dot]nl @SoudeFazeli Recommendations for Open Education: An Algorithimic Study