Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EC-TEL 2016: Which Algorithms Suit Which Learning Environments?

301 views

Published on

A comparative study of recommender systems in TEL

Published in: Internet
  • Be the first to comment

  • Be the first to like this

EC-TEL 2016: Which Algorithms Suit Which Learning Environments?

  1. 1. W I S S E N T E C H N I K L E I D E N S C H A F T http://kti.tugraz.at Which Algorithms Suit Which Learning Environments? A Comparative Study of Recommender Systems in TEL S. Kopeinik, D. Kowald, E. Lex, Graz University of Technology, Austria Knowledge Technologies Institute, Cognitive Science Section October 24, 2016
  2. 2. 2 Outline A study comparing a variety of recommendation strategies on 6 empirical TEL datasets Considering 2 application cases Findings: The performance of algorithms strongly depends on the characteristics of the datasets The number of users per resource is a crucial factor A hybrid combination of a cognitive-inspired and a popularity based approach works best for tag recommendations S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  3. 3. 3 Introduction Recommender Systems (RS) ... are software components that suggest items of interest or of relevance to a user’s needs [Kon, 2004, Ricci et al., 2011]. Recommendations are related to decision making processes: Ease information overload Sales assistance Popular examples: Amazon.com, YouTube, Netflix, Tripadvisor, Last.fm, ... S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  4. 4. 4 Introduction RS in Technology Enhanced Learning . . . are adaptational tasks to fit the learner’s needs [H¨am¨al¨ainen and Vinni, 2010]. Typical recommendation services include: Peer recommendations Activity recommendations Learning resource recommendations Tag recommendations S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  5. 5. 5 Introduction Motivation So far there are no generally suggested or commonly applied recommender system in TEL [Khribi et al., 2015] Learning data is sparse, especially in informal learning environments [Manouselis et al., 2011] Available data varies greatly, but available implicit usage data typically includes learner ids, information about learning resources, timestamps [Verbert et al., 2012] Research Question 1 How accurate do state-of-the-art resource recommendation algorithms, using only implicit usage data, perform on different TEL datasets? S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  6. 6. 6 Introduction Motivation Lack of learning object meta-data Shifting the task to the crowd [Bateman et al., 2007] Tagging is a mechanism to collectively annotate learning objects [Xu et al., 2006] fosters reflection and deep learning [Kuhn et al., 2012] needs to be done regularly and thoroughly Research Question 2 Which computationally inexpensive state-of-the-art tag recommendation algorithm performs best on TEL datasets? S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  7. 7. 7 Evaluation Study Setup Evaluation of two recommender application cases a) Recommendation of learning resources b) Recommendation of tags For each dataset 1. Sort user activities in chronological order (timestamp) 2. Split data into training and test set Application Training Set Test Set Resources 80% 20% Tags n-1 1 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  8. 8. 8 Evaluation Algorithms Well-established, computationally inexpensive tag and resource recommendation strategies Most Popular (MP) [J¨aschke et al., 2007] . . . counts frequency of occurrence Collaborative Filtering (CF) [Schafer et al., 2007] . . . calculates neighbourhood of users or items Content-based Filtering (CB) [Basilico and Hofmann, 2004] . . . calculates similarity of user profiles and item content S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  9. 9. 9 Evaluation Algorithms Approaches that have been suggested in the context of TEL Usage Context-based Similarity (UCbSim) [Niemann and Wolpers, 2013] . . . calculates item similarities based on co-occurrences in user sessions Base Level Learning Equation (BLLAC) [Kowald et al., 2015] . . . mimics human semantic memory retrieval as a function of recency and frequency of tag use Sustain [Seitlinger et al., 2015] . . . simulates category learning as a dynamic clustering approach S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  10. 10. 10 Evaluation Metrics [Marinho et al., 2012, Sakai, 2007] Recall (R) The proportion of correctly recommended items to all items relevant to the user. Precision (P) The proportion of correctly recommended items to all recommended items. F-measure (F) The harmonic mean of R and P. Discounted Cumulative Gain (nDCG) A ranking quality metric that calculates usefulness scores of items based on relevance and position. S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  11. 11. 11 Evaluation Datasets Six datasets from different application domains: BibSonomy, CiteULike (Social Bookmarking) KDD15 (MOOCs) Mace, TravelWell (Open Social Learning) Aposdle (Workplace Learning) |P| |U| |R| |T| |Tp| |ATr | |ATpr | |ARu| |AUr | SPt SPtp BibSonomy 82539 2437 28000 30889 0 4.1 0 33.8 3 0 100 CiteULike 105333 7182 42320 46060 0 3.5 0 14.7 2.5 0 100 KDD15 262330 15236 5315 0 3160 0 1.8 17.2 49.4 100 1.1 TravelWell 2572 97 1890 4156 153 3.5 1.7 26.5 1.4 3.2 28.7 MACE 23017 627 12360 15249 0 2.4 0 36.7 1.9 31.2 100 Aposdle 449 6 430 0 98 0 1.1 74.8 1 100 0 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  12. 12. 11 Evaluation Datasets Six datasets from different application domains: BibSonomy, CiteULike (Social Bookmarking) KDD15 (MOOCs) Mace, TravelWell (Open Social Learning) Aposdle (Workplace Learning) |P| |U| |R| |T| |Tp| |ATr | |ATpr | |ARu| |AUr | SPt SPtp BibSonomy 82539 2437 28000 30889 0 4.1 0 33.8 3 0 100 CiteULike 105333 7182 42320 46060 0 3.5 0 14.7 2.5 0 100 KDD15 262330 15236 5315 0 3160 0 1.8 17.2 49.4 100 1.1 TravelWell 2572 97 1890 4156 153 3.5 1.7 26.5 1.4 3.2 28.7 MACE 23017 627 12360 15249 0 2.4 0 36.7 1.9 31.2 100 Aposdle 449 6 430 0 98 0 1.1 74.8 1 100 0 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  13. 13. 12 Discussion Results: Resource Recommender Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU BibSonomy P@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467 F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496 nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541 CiteULike P@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553 F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650 nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717 KDD15 P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377 F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059 nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608 TravelWell P@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382 F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204 nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220 MACE P@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190 F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205 nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215 Aposdle P@5 .0 .0 .0 .0333 .0 .0 .0 F@5 .0 .0 .0 .0049 .0 .0 .0 nDCG@5 .0 .0 .0 .0042 .0 .0 .0 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  14. 14. 12 Discussion Results: Resource Recommender Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU BibSonomy P@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467 F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496 nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541 CiteULike P@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553 F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650 nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717 KDD15 P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377 F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059 nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608 TravelWell P@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382 F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204 nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220 MACE P@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190 F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205 nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215 Aposdle P@5 .0 .0 .0 .0333 .0 .0 .0 F@5 .0 .0 .0 .0049 .0 .0 .0 nDCG@5 .0 .0 .0 .0042 .0 .0 .0 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  15. 15. 12 Discussion Results: Resource Recommender Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU BibSonomy P@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467 F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496 nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541 CiteULike P@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553 F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650 nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717 KDD15 P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377 F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059 nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608 TravelWell P@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382 F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204 nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220 MACE P@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190 F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205 nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215 Aposdle P@5 .0 .0 .0 .0333 .0 .0 .0 F@5 .0 .0 .0 .0049 .0 .0 .0 nDCG@5 .0 .0 .0 .0042 .0 .0 .0 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  16. 16. 12 Discussion Results: Resource Recommender Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU BibSonomy P@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467 F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496 nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541 CiteULike P@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553 F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650 nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717 KDD15 P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377 F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059 nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608 TravelWell P@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382 F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204 nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220 MACE P@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190 F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205 nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215 Aposdle P@5 .0 .0 .0 .0333 .0 .0 .0 F@5 .0 .0 .0 .0049 .0 .0 .0 nDCG@5 .0 .0 .0 .0042 .0 .0 .0 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  17. 17. 13 Discussion Datasets |ATr | |ATpr | |ARu| |AUr | BibSonomy 4.1 0 33.8 3 CiteULike 3.5 0 14.7 2.5 KDD15 0 1.8 17.2 49.4 TravelWell 3.5 1.7 26.5 1.4 MACE 2.4 0 36.7 1.9 Aposdle 0 1.1 74.8 1 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  18. 18. 14 Discussion Results: Resource Recommender Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU BibSonomy P@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467 F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496 nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541 CiteULike P@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553 F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650 nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717 KDD15 P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377 F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059 nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608 TravelWell P@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382 F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204 nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220 MACE P@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190 F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205 nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215 Aposdle P@5 .0 .0 .0 .0333 .0 .0 .0 F@5 .0 .0 .0 .0049 .0 .0 .0 nDCG@5 .0 .0 .0 .0042 .0 .0 .0 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  19. 19. 15 Discussion Findings 1. The performance of most algorithms (F@5) strongly correlates (.958) with the average number of users per resource 2. Good performance values can only be reached for the MOOCs dataset 3. Algorithms based on implicit usage data don’t satisfy the requirements of small-scale environments like Aposdle 4. The performance of algorithms strongly depends on the characteristics of the datasets S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  20. 20. 16 Discussion Results: Tag Recommender Dataset Metric MPU MPR MPU,R CFU BLLAC BLLAC+MPR BibSonomy P@5 .1991 .0572 .2221 .2066 .2207 .2359 F@5 .2535 .0688 .2814 .2606 .2795 .2987 nDCG@5 .3449 .0841 .3741 .3492 .3851 .4022 CiteULike P@5 .1687 .0323 .1829 .1698 .1897 .2003 F@5 .2310 .0427 .2497 .2315 .2597 .2738 nDCG@5 .3414 .0600 .3632 .3457 .4016 .4140 TravelWell P@5 .1000 .0366 .1333 .0800 .1300 .1400 F@5 .1376 .0484 .1724 .1096 .1708 .1872 nDCG@5 .2110 .0717 .2253 .1622 .2525 .2615 MACE P@5 .0576 .0173 .0618 .0631 .0812 .0812 F@5 .0799 .0259 .0869 .0893 .1114 .1138 nDCG@5 .1146 .0463 .1296 .1502 .1670 .1734 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  21. 21. 16 Discussion Results: Tag Recommender Dataset Metric MPU MPR MPU,R CFU BLLAC BLLAC+MPR BibSonomy P@5 .1991 .0572 .2221 .2066 .2207 .2359 F@5 .2535 .0688 .2814 .2606 .2795 .2987 nDCG@5 .3449 .0841 .3741 .3492 .3851 .4022 CiteULike P@5 .1687 .0323 .1829 .1698 .1897 .2003 F@5 .2310 .0427 .2497 .2315 .2597 .2738 nDCG@5 .3414 .0600 .3632 .3457 .4016 .4140 TravelWell P@5 .1000 .0366 .1333 .0800 .1300 .1400 F@5 .1376 .0484 .1724 .1096 .1708 .1872 nDCG@5 .2110 .0717 .2253 .1622 .2525 .2615 MACE P@5 .0576 .0173 .0618 .0631 .0812 .0812 F@5 .0799 .0259 .0869 .0893 .1114 .1138 nDCG@5 .1146 .0463 .1296 .1502 .1670 .1734 S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  22. 22. 17 Discussion Conclusion Learning Resource Recommendation: A dense user resource matrix is crucial The performance of most algorithms (F@5) strongly correlates (.958) with the average number of users per resource For small-scale learning environments, a thorough description of user and learning resources is necessary Algorithms based on implicit usage data don’t satisfy the requirements of small-scale environments like Aposdle MOOCs are not representative for other, typically sparse TEL environments Good performance values can only be reached for the MOOCs dataset Tag Recommendation: BLLAC+ MPR clearly outperforms the remaining algorithms MPU,R, an alternative for runtime-sensitive environments S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  23. 23. 18 Discussion Acknowledgement Special thanks are dedicated to Katja Niemann who provided us with the datasets MACE and TravelWell. For the KDD15 data, we would like to gratefully acknowledge the organizers of KDD Cup 2015 as well as XuetangX for making the datasets available. This work is funded by the Know-Center and the EU-IP Learning Layers (Grant Agreement: 318209). The Know-Center is funded within the Austrian COMET Program under the auspices of the Austrian Ministry of Transport, Innovation and Technology, the Austrian Ministry of Economics and Labor and by the State of Styria. S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  24. 24. 19 Discussion References I [Kon, 2004] (2004). Introduction to recommender systems: Algorithms and evaluation. ACM Trans. Inf. Syst., 22(1):1–4. [Basilico and Hofmann, 2004] Basilico, J. and Hofmann, T. (2004). Unifying collaborative and content-based filtering. In Proc. of ICML’04, page 9. ACM. [Bateman et al., 2007] Bateman, S., Brooks, C., Mccalla, G., and Brusilovsky, P. (2007). Applying collaborative tagging to e-learning. In Proceedings of the 16th international world wide web conference (WWW2007). [H¨am¨al¨ainen and Vinni, 2010] H¨am¨al¨ainen, W. and Vinni, M. (2010). Classifiers for educational data mining. Handbook of Educational Data Mining, Chapman & Hall/CRC Data Mining and Knowledge Discovery Series, pages 57–71. [J¨aschke et al., 2007] J¨aschke, R., Marinho, L., Hotho, A., Schmidt-Thieme, L., and Stumme, G. (2007). Tag recommendations in folksonomies. In Proc. of PKDD’07, pages 506–514. Springer. S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  25. 25. 20 Discussion References II [Khribi et al., 2015] Khribi, M. K., Jemni, M., and Nasraoui, O. (2015). Recommendation systems for personalized technology-enhanced learning. In Ubiquitous Learning Environments and Technologies, pages 159–180. Springer. [Kowald et al., 2015] Kowald, D., Kopeinik, S., Seitlinger, P., Ley, T., Albert, D., and Trattner, C. (2015). Refining frequency-based tag reuse predictions by means of time and semantic context. In Mining, Modeling, and Recommending’Things’ in Social Media, pages 55–74. Springer. [Kuhn et al., 2012] Kuhn, A., McNally, B., Schmoll, S., Cahill, C., Lo, W.-T., Quintana, C., and Delen, I. (2012). How students find, evaluate and utilize peer-collected annotated multimedia data in science inquiry with zydeco. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3061–3070. ACM. [Manouselis et al., 2011] Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., and Koper, R. (2011). Recommender systems in technology enhanced learning. In Recommender systems handbook, pages 387–415. Springer. S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  26. 26. 21 Discussion References III [Marinho et al., 2012] Marinho, L. B., Hotho, A., J¨aschke, R., Nanopoulos, A., Rendle, S., Schmidt-Thieme, L., Stumme, G., and Symeonidis, P. (2012). Recommender systems for social tagging systems. Springer Science & Business Media. [Niemann and Wolpers, 2013] Niemann, K. and Wolpers, M. (2013). Usage context-boosted filtering for recommender systems in tel. In Scaling up Learning for Sustained Impact, pages 246–259. Springer. [Ricci et al., 2011] Ricci, F., Rokach, L., and Shapira, B. (2011). Introduction to recommender systems handbook. Springer. [Sakai, 2007] Sakai, T. (2007). On the reliability of information retrieval metrics based on graded relevance. Information processing & management, 43(2):531–548. S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016
  27. 27. 22 Discussion References IV [Schafer et al., 2007] Schafer, J. B., Frankowski, D., Herlocker, J., and Sen, S. (2007). Collaborative filtering recommender systems. In The adaptive web, pages 291–324. Springer. [Seitlinger et al., 2015] Seitlinger, P., Kowald, D., Kopeinik, S., Hasani-Mavriqi, I., Ley, T., and Lex, E. (2015). Attention please! a hybrid resource recommender mimicking attention-interpretation dynamics. arXiv preprint arXiv:1501.07716. [Verbert et al., 2012] Verbert, K., Manouselis, N., Drachsler, H., and Duval, E. (2012). Dataset-driven research to support learning and knowledge analytics. Educational Technology & Society, 15(3):133–148. [Xu et al., 2006] Xu, Z., Fu, Y., Mao, J., and Su, D. (2006). Towards the semantic web: Collaborative tag suggestions. In Collaborative web tagging workshop at WWW2006, Edinburgh, Scotland. S. Kopeinik, D. Kowald, E. Lex, KTI-CSS October 24, 2016

×