EC-TEL 2016: Which Algorithms Suit Which Learning Environments?

W I S S E N T E C H N I K L E I D E N S C H A F T
http://kti.tugraz.at
Which Algorithms Suit Which Learning
Environments? A Comparative Study of
Recommender Systems in TEL
S. Kopeinik, D. Kowald, E. Lex,
Graz University of Technology, Austria
Knowledge Technologies Institute, Cognitive Science Section
October 24, 2016

2
Outline
A study comparing a variety of recommendation strategies
on 6 empirical TEL datasets
Considering 2 application cases
Findings:
The performance of algorithms strongly depends on the
characteristics of the datasets
The number of users per resource is a crucial factor
A hybrid combination of a cognitive-inspired and a
popularity based approach works best for tag
recommendations
S. Kopeinik, D. Kowald, E. Lex, KTI-CSS
October 24, 2016

3
Introduction
Recommender Systems (RS)
... are software components that suggest items of interest or of
relevance to a user’s needs [Kon, 2004, Ricci et al., 2011].
Recommendations are related to decision making processes:
Ease information overload
Sales assistance
Popular examples: Amazon.com, YouTube, Netﬂix, Tripadvisor,
Last.fm, ...
October 24, 2016

4
Introduction
RS in Technology Enhanced Learning
. . . are adaptational tasks to fit the learner’s
needs [Hämäläinen and Vinni, 2010].
Typical recommendation services include:
Peer recommendations
Activity recommendations
Learning resource recommendations
Tag recommendations
October 24, 2016

5
Introduction
Motivation
So far there are no generally suggested or commonly
applied recommender system in TEL [Khribi et al., 2015]
Learning data is sparse, especially in informal learning
environments [Manouselis et al., 2011]
Available data varies greatly, but available implicit usage
data typically includes learner ids, information about
learning resources, timestamps [Verbert et al., 2012]
Research Question 1
How accurate do state-of-the-art resource recommendation algorithms,
using only implicit usage data, perform on different TEL datasets?
October 24, 2016

6
Introduction
Motivation
Lack of learning object meta-data
Shifting the task to the crowd [Bateman et al., 2007]
Tagging is a mechanism to collectively annotate
learning objects [Xu et al., 2006]
fosters reﬂection and deep learning
[Kuhn et al., 2012]
needs to be done regularly and thoroughly
Research Question 2
Which computationally inexpensive state-of-the-art tag recommendation
algorithm performs best on TEL datasets?
October 24, 2016

7
Evaluation
Study Setup
Evaluation of two recommender application cases
a) Recommendation of learning resources
b) Recommendation of tags
For each dataset
1. Sort user activities in chronological order
(timestamp)
2. Split data into training and test set
Application Training Set Test Set
Resources 80% 20%
Tags n-1 1
October 24, 2016

8
Evaluation
Algorithms
Well-established, computationally inexpensive tag and
resource recommendation strategies
Most Popular (MP) [J¨aschke et al., 2007]
. . . counts frequency of occurrence
Collaborative Filtering (CF) [Schafer et al., 2007]
. . . calculates neighbourhood of users or items
Content-based Filtering (CB) [Basilico and Hofmann, 2004]
. . . calculates similarity of user proﬁles and item
content
October 24, 2016

9
Evaluation
Algorithms
Approaches that have been suggested in the context of TEL
Usage Context-based Similarity (UCbSim)
[Niemann and Wolpers, 2013]
. . . calculates item similarities based on
co-occurrences in user sessions
Base Level Learning Equation (BLLAC) [Kowald et al., 2015]
. . . mimics human semantic memory retrieval as a
function of recency and frequency of tag use
Sustain [Seitlinger et al., 2015]
. . . simulates category learning as a dynamic
clustering approach
October 24, 2016

10
Evaluation
Metrics [Marinho et al., 2012, Sakai, 2007]
Recall (R)
The proportion of correctly recommended items to
all items relevant to the user.
Precision (P)
The proportion of correctly recommended items to
all recommended items.
F-measure (F)
The harmonic mean of R and P.
Discounted Cumulative Gain (nDCG)
A ranking quality metric that calculates usefulness
scores of items based on relevance and position.
October 24, 2016

11
Evaluation
Datasets
Six datasets from different application domains:
BibSonomy, CiteULike (Social Bookmarking)
KDD15 (MOOCs)
Mace, TravelWell (Open Social Learning)
Aposdle (Workplace Learning)
|P| |U| |R| |T| |Tp| |ATr | |ATpr | |ARu| |AUr | SPt SPtp
BibSonomy 82539 2437 28000 30889 0 4.1 0 33.8 3 0 100
CiteULike 105333 7182 42320 46060 0 3.5 0 14.7 2.5 0 100
KDD15 262330 15236 5315 0 3160 0 1.8 17.2 49.4 100 1.1
TravelWell 2572 97 1890 4156 153 3.5 1.7 26.5 1.4 3.2 28.7
MACE 23017 627 12360 15249 0 2.4 0 36.7 1.9 31.2 100
Aposdle 449 6 430 0 98 0 1.1 74.8 1 100 0
October 24, 2016

12
Discussion
Results: Resource Recommender
Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU
BibSonomy
P@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467
F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496
nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541
CiteULike
P@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553
F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650
nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717
KDD15
P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377
F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059
nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608
TravelWell
P@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382
F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204
nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220
MACE
P@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190
F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205
nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215
Aposdle
P@5 .0 .0 .0 .0333 .0 .0 .0
F@5 .0 .0 .0 .0049 .0 .0 .0
nDCG@5 .0 .0 .0 .0042 .0 .0 .0
October 24, 2016

13
Discussion
Datasets
|ATr | |ATpr | |ARu| |AUr |
BibSonomy 4.1 0 33.8 3
CiteULike 3.5 0 14.7 2.5
KDD15 0 1.8 17.2 49.4
TravelWell 3.5 1.7 26.5 1.4
MACE 2.4 0 36.7 1.9
Aposdle 0 1.1 74.8 1
October 24, 2016

14
Discussion
Results: Resource Recommender
Dataset Metric MP CFR CBT CFU UCbSim SUSTAIN SUSTAIN+CFU
BibSonomy
P@5 .0154 .0336 .0197 .0410 .0336 .0336 .0467
F@5 .0099 .0383 .0238 .0426 .0367 .0363 .0496
nDCG@5 .0088 .0416 .0270 .0440 .0371 .0392 .0541
CiteULike
P@5 .0048 .0592 .0353 .0412 .0558 .0503 .0553
F@5 .0050 .0694 .0404 .0477 .0627 .0597 .0650
nDCG@5 .0048 .0792 .0427 .0511 .0686 .0704 .0717
KDD15
P@5 .0018 .2488 .1409 .2355 .2570 .2436 .2377
F@5 .0029 .3074 .1612 .3050 .3314 .3025 .3059
nDCG@5 .0053 .3897 .1927 .3618 .3529 .3227 .3608
TravelWell
P@5 .0127 .0212 .0382 .0425 .0297 .0382 .0382
F@5 .0056 .0232 .0240 .0414 .0365 .0427 .0204
nDCG@5 .0072 .0220 .0275 .0305 .0491 .0446 .0220
MACE
P@5 .0167 .0079 .0023 .0251 .0213 .0065 .0190
F@5 .0201 .0079 .0019 .0266 .0177 .0076 .0205
nDCG@5 .0248 .0082 .0014 .0264 .0165 .0079 .0215
Aposdle
P@5 .0 .0 .0 .0333 .0 .0 .0
F@5 .0 .0 .0 .0049 .0 .0 .0
nDCG@5 .0 .0 .0 .0042 .0 .0 .0
October 24, 2016

15
Discussion
Findings
1. The performance of most algorithms (F@5) strongly
correlates (.958) with the average number of users per
resource
2. Good performance values can only be reached for the
MOOCs dataset
3. Algorithms based on implicit usage data don’t satisfy the
requirements of small-scale environments like Aposdle
4. The performance of algorithms strongly depends on the
characteristics of the datasets
October 24, 2016

16
Discussion
Results: Tag Recommender
Dataset Metric MPU MPR MPU,R CFU BLLAC BLLAC+MPR
BibSonomy
P@5 .1991 .0572 .2221 .2066 .2207 .2359
F@5 .2535 .0688 .2814 .2606 .2795 .2987
nDCG@5 .3449 .0841 .3741 .3492 .3851 .4022
CiteULike
P@5 .1687 .0323 .1829 .1698 .1897 .2003
F@5 .2310 .0427 .2497 .2315 .2597 .2738
nDCG@5 .3414 .0600 .3632 .3457 .4016 .4140
TravelWell
P@5 .1000 .0366 .1333 .0800 .1300 .1400
F@5 .1376 .0484 .1724 .1096 .1708 .1872
nDCG@5 .2110 .0717 .2253 .1622 .2525 .2615
MACE
P@5 .0576 .0173 .0618 .0631 .0812 .0812
F@5 .0799 .0259 .0869 .0893 .1114 .1138
nDCG@5 .1146 .0463 .1296 .1502 .1670 .1734
October 24, 2016

17
Discussion
Conclusion
Learning Resource Recommendation:
A dense user resource matrix is crucial
The performance of most algorithms (F@5) strongly correlates (.958) with the average
number of users per resource
For small-scale learning environments, a thorough
description of user and learning resources is necessary
Algorithms based on implicit usage data don’t satisfy the requirements of small-scale
environments like Aposdle
MOOCs are not representative for other, typically sparse
TEL environments
Good performance values can only be reached for the MOOCs dataset
Tag Recommendation:
BLLAC+ MPR clearly outperforms the remaining algorithms
MPU,R, an alternative for runtime-sensitive environments
October 24, 2016

18
Discussion
Acknowledgement
Special thanks are dedicated to Katja Niemann who provided
us with the datasets MACE and TravelWell. For the KDD15
data, we would like to gratefully acknowledge the organizers of
KDD Cup 2015 as well as XuetangX for making the datasets
available. This work is funded by the Know-Center and the
EU-IP Learning Layers (Grant Agreement: 318209). The
Know-Center is funded within the Austrian COMET Program
under the auspices of the Austrian Ministry of Transport,
Innovation and Technology, the Austrian Ministry of Economics
and Labor and by the State of Styria.
October 24, 2016

19
Discussion
References I
[Kon, 2004] (2004).
Introduction to recommender systems: Algorithms and evaluation.
ACM Trans. Inf. Syst., 22(1):1–4.
[Basilico and Hofmann, 2004] Basilico, J. and Hofmann, T. (2004).
Unifying collaborative and content-based filtering.
In Proc. of ICML’04, page 9. ACM.
[Bateman et al., 2007] Bateman, S., Brooks, C., Mccalla, G., and Brusilovsky, P. (2007).
Applying collaborative tagging to e-learning.
In Proceedings of the 16th international world wide web conference (WWW2007).
[Hämäläinen and Vinni, 2010] Hämäläinen, W. and Vinni, M. (2010).
Classifiers for educational data mining.
Handbook of Educational Data Mining, Chapman & Hall/CRC Data Mining and Knowledge Discovery Series, pages 57–71.
[Jäschke et al., 2007] Jäschke, R., Marinho, L., Hotho, A., Schmidt-Thieme, L., and Stumme, G. (2007).
Tag recommendations in folksonomies.
In Proc. of PKDD’07, pages 506–514. Springer.
October 24, 2016

20
Discussion
References II
[Khribi et al., 2015] Khribi, M. K., Jemni, M., and Nasraoui, O. (2015).
Recommendation systems for personalized technology-enhanced learning.
In Ubiquitous Learning Environments and Technologies, pages 159–180. Springer.
[Kowald et al., 2015] Kowald, D., Kopeinik, S., Seitlinger, P., Ley, T., Albert, D., and Trattner, C. (2015).
Reﬁning frequency-based tag reuse predictions by means of time and semantic context.
In Mining, Modeling, and Recommending’Things’ in Social Media, pages 55–74. Springer.
[Kuhn et al., 2012] Kuhn, A., McNally, B., Schmoll, S., Cahill, C., Lo, W.-T., Quintana, C., and Delen, I. (2012).
How students ﬁnd, evaluate and utilize peer-collected annotated multimedia data in science inquiry with zydeco.
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 3061–3070. ACM.
[Manouselis et al., 2011] Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H., and Koper, R. (2011).
Recommender systems in technology enhanced learning.
In Recommender systems handbook, pages 387–415. Springer.
October 24, 2016

21
Discussion
References III
[Marinho et al., 2012] Marinho, L. B., Hotho, A., J¨aschke, R., Nanopoulos, A., Rendle, S., Schmidt-Thieme, L., Stumme, G.,
and Symeonidis, P. (2012).
Recommender systems for social tagging systems.
Springer Science & Business Media.
[Niemann and Wolpers, 2013] Niemann, K. and Wolpers, M. (2013).
Usage context-boosted ﬁltering for recommender systems in tel.
In Scaling up Learning for Sustained Impact, pages 246–259. Springer.
[Ricci et al., 2011] Ricci, F., Rokach, L., and Shapira, B. (2011).
Introduction to recommender systems handbook.
Springer.
[Sakai, 2007] Sakai, T. (2007).
On the reliability of information retrieval metrics based on graded relevance.
Information processing & management, 43(2):531–548.
October 24, 2016

22
Discussion
References IV
[Schafer et al., 2007] Schafer, J. B., Frankowski, D., Herlocker, J., and Sen, S. (2007).
Collaborative ﬁltering recommender systems.
In The adaptive web, pages 291–324. Springer.
[Seitlinger et al., 2015] Seitlinger, P., Kowald, D., Kopeinik, S., Hasani-Mavriqi, I., Ley, T., and Lex, E. (2015).
Attention please! a hybrid resource recommender mimicking attention-interpretation dynamics.
arXiv preprint arXiv:1501.07716.
[Verbert et al., 2012] Verbert, K., Manouselis, N., Drachsler, H., and Duval, E. (2012).
Dataset-driven research to support learning and knowledge analytics.
Educational Technology & Society, 15(3):133–148.
[Xu et al., 2006] Xu, Z., Fu, Y., Mao, J., and Su, D. (2006).
Towards the semantic web: Collaborative tag suggestions.
In Collaborative web tagging workshop at WWW2006, Edinburgh, Scotland.
October 24, 2016

EC-TEL 2016: Which Algorithms Suit Which Learning Environments?

More Related Content

Viewers also liked

Similar to EC-TEL 2016: Which Algorithms Suit Which Learning Environments?

EC-TEL 2016: Which Algorithms Suit Which Learning Environments?