Your SlideShare is downloading. ×
Recommender Systems            and Learning Analytics in TELHendrik DrachslerCentre for Learning Sciences and TechnologyOp...
Goals of the lecture1. Crash course Recommender Systems (RecSys)2. Overview of RecSys in TEL3. Open research issues for Re...
Introduction intoRecommender Systems     Introduction       Objectives                                 Technologies       ...
Introduction::Application areas Application areas • E-commerce websites (Amazon) • Video, Music websites (Netflix, last.fm)...
Introduction::DefinitionUsing the opinions of a community of users tohelp individuals in that community to identify moreeff...
Introduction::DefinitionUsing the opinions of a community of users tohelp individuals in that community to identify moreeff...
Introduction::Example           6
Introduction::Example           6
Introduction::Example           6
Introduction::Example           6
Introduction::Example           6
Introduction::Example           6
Introduction::Example           6
Introduction::Example           6
Introduction::ExampleWhat did we learn from the small exercise?  • There are different kinds of recommendations  a. People...
Introduction:: The Long TailAnderson, C., (2004). The Long Tail. Wired Magazine.                                       7
Introduction:: The Long Tail“We are leaving the age of information andentering the age of recommendation”.                ...
Introduction:: Age of RecSys?      ...10 minutes on Google.                   8
Introduction:: Age of RecSys?      ...10 minutes on Google.                   8
Introduction:: Age of RecSys?... another 10 minutes, research on RecSys is  becoming main stream.Some examples:– ACM RecSy...
Objectivesof RecSys       probabilistic combination of                – Item-based method                – User-based meth...
Objectives::Aims• Converting Browsers into    Buyers• Increasing Cross-sales• Building Loyalty                            ...
Objectives::RecSys TasksFind good itemspresenting a ranked list ofrecommendendations.                                     ...
Objectives::RecSys TasksFind good items                             Receive sequence of itemspresenting a ranked list of  ...
Objectives::RecSys TasksFind good items                             Receive sequence of itemspresenting a ranked list of  ...
RecSys Technologies1. Predict how much a user  may like a certain product2. Create a list of Top-N  best items3. Adjust it...
RecSys Technologies1. Predict how much a user  may like a certain product2. Create a list of Top-N  best items3. Adjust it...
Technologies::Collaborative filtering  User-based filtering  (Grouplens, 1994)Take about 20-50 people who sharesimilar taste...
Technologies::Collaborative filtering  User-based filtering                             Item-based filtering  (Grouplens, 199...
Technologies::Content-based filtering  Information needs of user and characteristics of items are    represented in keyword...
Technologies::Hybrid RecSysCombination of techniques to overcomedisadvantages and advantages of single techniques. Advanta...
Technologies::Hybrid RecSysCombination of techniques to overcomedisadvantages and advantages of single techniques. Advanta...
Evaluationof RecSys                probabilistic combination of                – Item-based method                – User-b...
Evaluation::General idea    Most of the time based on performance measures      (“How good are your recommendations?”)For ...
Evaluation::Reference datasets         ... and various commercial datasets.                19
Evaluation::Approaches                       1. Simulation•User preference•Prediction accuracy•Coverage•Confidence•Trust•No...
Evaluation::Metrics Precision – The portion of recommendations that were successful. (Selected by the algorithm and by the...
Evaluation::Metrics Precision – The portion of recommendations that were successful. (Selected by the algorithm and by the...
Evaluation::Metrics Precision – The portion of recommendations that were successful. (Selected by the algorithm and by the...
Evaluation::Metrics Precision – The portion of recommendations that were successful. (Selected by the algorithm and by the...
Evaluation::Metrics                                       5 Conclusion:                                       4 Pearson is...
Evaluation::Metrics                                             5 Conclusion:                                             ...
RecSys::TimeToThinkWhat do you expect that a RecSys in aMUP/PLE should do with respect to ...• Aims• Tasks• Technology    ...
Goals of the lecture1. Crash course Recommender Systems (RecSys)2. Overview of RecSys in TEL3. Open research issues for Re...
Recommender Systemsfor TEL    Introduction        Objectives                                 Technologies                 ...
TEL RecSys::Definition     Using the experiences of a community of     learners to help individual learners in that     com...
FindTEL RecSys::Learning spectrum                     M                                                   fo              ...
The Long TailGraphic: Wilkins, D., (2009).   28
The Long Tail of LearningGraphic: Wilkins, D., (2009).   28
The Long Tail of Learning          Formal                                 InformalGraphic: Wilkins, D., (2009).   28
TEL RecSys::Technologies           29
TEL RecSys:: Technologies            30
TEL RecSys:: Technologies            31
TEL RecSys:: Technologies          RecSys Task:          Find good items          Hybrid RecSys:          •Content-based o...
TEL RecSys::Tasks  Find good items  e.g. relevant items for a learning     task or a learning goal                        ...
TEL RecSys::Tasks  Find good items  e.g. relevant items for a learning     task or a learning goal Receive sequence of ite...
TEL RecSys::Tasks  Find good items  e.g. relevant items for a learning     task or a learning goal Receive sequence of ite...
Evaluation of TEL RecSys         probabilistic combination of                – Item-based method                – User-bas...
TEL RecSys::Review study             34
TEL RecSys::Review studyManouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011).Recommender Sys...
TEL RecSys::Review study      Conclusions:      Half of the systems (11/20) still at design or prototyping stage       onl...
Thus...“The performance resultsof different researchefforts in recommendersystems are hardlycomparable.”(Manouselis et al....
Thus...TEL recommender “The performance resultsexperiments lack of different research efforts in recommendertransparency. ...
TEL RecSys::Evaluation/datasets              36
TEL RecSys::Evaluation/datasetsDrachsler, H., Bogers, T., Vuorikari, R., Verbert, K., Duval, E., Manouselis, N., Beham, G....
Evaluation::Metrics          MAE – Mean Absolute Error:                                                   Deviation of rec...
Evaluation::Metrics          MAE – Mean Absolute Error:                                                   Deviation of rec...
Evaluation::Metrics         MAE – Mean Absolute Error:                                                   Deviation of reco...
TEL RecSys::EvaluationCombined approach by          Kirkpatrick model by Drachsler et al. 2008        Manouselis et al. 20...
TEL RecSys::Evaluation 1. Accuracy 2. Coverage 3. PrecisionCombined approach by          Kirkpatrick model by Drachsler et...
TEL RecSys::Evaluation 1. Accuracy 2. Coverage 3. Precision 1. Effectiveness of learning 2. Efficiency of learning 3. Drop ...
TEL RecSys::Evaluation 1. Accuracy                           1. Reaction of learner 2. Coverage                           ...
Goals of the lecture1. Crash course Recommender Systems (RecSys)2. Overview of RecSys in TEL3. Open research issues for Re...
TEL RecSys::Open issues1. Evaluation of TEL RecSys2. Publicly available datasets3. Comparable experiments4. Body of knowle...
Goals of the lecture1. Crash course Recommender Systems (RecSys)2. Overview of RecSys in TEL3. Open research issues for Re...
Greller, W., & Drachsler, H., 2011.                          42
Greller, W., & Drachsler, H., 2011.                          42
Greller, W., & Drachsler, H., 2011.                          42
Greller, W., & Drachsler, H., 2011.                          42
Greller, W., & Drachsler, H., 2011.                          42
Greller, W., & Drachsler, H., 2011.                          42
Greller, W., & Drachsler, H., 2011.                          42
Learning Analytics::TimeToThink •   Consider the Learning Analytics     framework and imagine some great TEL     RecSys th...
Thank you for attending this lecture! This silde is available at: http://www.slideshare.com/Drachsler Email:       hendrik...
Upcoming SlideShare
Loading in...5
×

Guest lecture Recommender Systems in TEL at RWTH Aachen, Germany

2,918

Published on

Published in: Technology, Education

Transcript of "Guest lecture Recommender Systems in TEL at RWTH Aachen, Germany"

  1. 1. Recommender Systems and Learning Analytics in TELHendrik DrachslerCentre for Learning Sciences and TechnologyOpen University of the Netherlands 1
  2. 2. Goals of the lecture1. Crash course Recommender Systems (RecSys)2. Overview of RecSys in TEL3. Open research issues for RecSys in TEL4.TEL RecSys and Learning Analytics 2
  3. 3. Introduction intoRecommender Systems Introduction Objectives Technologies Evaluation 3
  4. 4. Introduction::Application areas Application areas • E-commerce websites (Amazon) • Video, Music websites (Netflix, last.fm) • Content websites (CNN, Google News) • Information Support SystemsMajor claims • Highly application-oriented research area, every domain and task needs a specific RecSys • Always build around content or products they never exist as on their own 4
  5. 5. Introduction::DefinitionUsing the opinions of a community of users tohelp individuals in that community to identify moreeffectively content of interest from a potentiallyoverwhelming set of choices.Resnick & Varian (1997). Recommender Systems, Communications of the ACM, 40(3). 5
  6. 6. Introduction::DefinitionUsing the opinions of a community of users tohelp individuals in that community to identify moreeffectively content of interest from a potentiallyoverwhelming set of choices.Resnick & Varian (1997). Recommender Systems, Communications of the ACM, 40(3).Any system that produces personalizedrecommendations as output or has the effect ofguiding the user in a personalized way to interestingor useful objects in a large space of possible options.Burke R. (2002). Hybrid Recommender Systems: Survey and Experiments,User Modeling & User Adapted Interaction, 12, pp. 331-370. 5
  7. 7. Introduction::Example 6
  8. 8. Introduction::Example 6
  9. 9. Introduction::Example 6
  10. 10. Introduction::Example 6
  11. 11. Introduction::Example 6
  12. 12. Introduction::Example 6
  13. 13. Introduction::Example 6
  14. 14. Introduction::Example 6
  15. 15. Introduction::ExampleWhat did we learn from the small exercise? • There are different kinds of recommendations a. People who bought X also bought Y b. there are more advanced personalized recommendations • When registering, we have to tell the RecSys what we like (and what not). Thus, it requires information to offer suitable recommendations and it learns our preferences. 6
  16. 16. Introduction:: The Long TailAnderson, C., (2004). The Long Tail. Wired Magazine. 7
  17. 17. Introduction:: The Long Tail“We are leaving the age of information andentering the age of recommendation”. Anderson, C. (2004)Anderson, C., (2004). The Long Tail. Wired Magazine. 7
  18. 18. Introduction:: Age of RecSys? ...10 minutes on Google. 8
  19. 19. Introduction:: Age of RecSys? ...10 minutes on Google. 8
  20. 20. Introduction:: Age of RecSys?... another 10 minutes, research on RecSys is becoming main stream.Some examples:– ACM RecSys conference– ICWSM: Weblog and Social Media– WebKDD: Web Knowledge Discovery and Data Mining– WWW: The original WWW conference– SIGIR: Information Retrieval– ACM KDD: Knowledge Discovery and Data Mining– LAK: Learning Analytics and Knowledge– Educational data mining conference– ICML: Machine Learning– ...... and various workshops, books, and journals. 9
  21. 21. Objectivesof RecSys probabilistic combination of – Item-based method – User-based method – Matrix Factorization – (May be) content-based method The idea is to pick from my previous list 20-50 movies that share similar audience with “Taken”, then how much I will like depend on how much I liked those early movies – In short: I tend to watch this movie because I have watched those movies … or 10 – People who have watched those movies also liked this movie (Amazon style)
  22. 22. Objectives::Aims• Converting Browsers into Buyers• Increasing Cross-sales• Building Loyalty Foto by markhillarySchafer, Konstan & Riedel, (1999). RecSys in e-commerce. Proc. of the 1st ACM onelectronic commerce, Denver, Colorado, pp. 158-169. 11
  23. 23. Objectives::RecSys TasksFind good itemspresenting a ranked list ofrecommendendations. probabilistic combination of – Item-based method – User-based method – Matrix Factorization – (May be) content-based methodFind all good itemsuser wants to identify all The idea is to pick from myitems that might be previous list 20-50 movies that share similar audience withinteresting, e.g. medical “Taken”, then how much I will like depend on how much I liked thoseor legal cases early movies – In short: I tend to watch this movie because I have watched thoseHerlocker, Konstan, Borchers, & Riedl (2004). Evaluating Collaborative Filtering movies … orRecommender Systems. ACM Transactions on–Informationhave watched those pp. 5-53. 12 People who Systems, 22(1), movies also liked this movie (Amazon style)
  24. 24. Objectives::RecSys TasksFind good items Receive sequence of itemspresenting a ranked list of sequence of related items isrecommendendations. recommended to the user, e.g. music recommender probabilistic combination of – Item-based method – User-based method – Matrix FactorizationFind all good items Annotation in context – (May be) content-based methoduser wants to identify all predicted usefulness of anitems that might be item that pick from mythatis currently The idea is to the user previous list 20-50 moviesinteresting, e.g. medical viewing, e.g. linkslike share similar audience with within a “Taken”, then how much I willor legal cases websitehow much I liked those depend on early movies – In short: I tend to watch this movie because I have watched thoseHerlocker, Konstan, Borchers, & Riedl (2004). Evaluating Collaborative Filtering movies … orRecommender Systems. ACM Transactions on–Informationhave watched those pp. 5-53. 12 People who Systems, 22(1), movies also liked this movie (Amazon style)
  25. 25. Objectives::RecSys TasksFind good items Receive sequence of itemspresenting a ranked list of sequence of related items isrecommendendations. recommended to the user, e.g. music recommender There are more tasks available... of probabilistic combination – Item-based method – User-based method – Matrix FactorizationFind all good items Annotation in context – (May be) content-based methoduser wants to identify all predicted usefulness of anitems that might be item that pick from mythatis currently The idea is to the user previous list 20-50 moviesinteresting, e.g. medical viewing, e.g. linkslike share similar audience with within a “Taken”, then how much I willor legal cases websitehow much I liked those depend on early movies – In short: I tend to watch this movie because I have watched thoseHerlocker, Konstan, Borchers, & Riedl (2004). Evaluating Collaborative Filtering movies … orRecommender Systems. ACM Transactions on–Informationhave watched those pp. 5-53. 12 People who Systems, 22(1), movies also liked this movie (Amazon style)
  26. 26. RecSys Technologies1. Predict how much a user may like a certain product2. Create a list of Top-N best items3. Adjust its prediction based on feedback of the target user and like- minded usersHanani et al., (2001). Information Filtering: Overview of Issues, Research and Systems", User Modeling and User-Adapted Interaction, 11. 13
  27. 27. RecSys Technologies1. Predict how much a user may like a certain product2. Create a list of Top-N best items3. Adjust its prediction based on feedback of the Just some examples target user and like- there are more minded users technologies available.Hanani et al., (2001). Information Filtering: Overview of Issues, Research and Systems", User Modeling and User-Adapted Interaction, 11. 13
  28. 28. Technologies::Collaborative filtering User-based filtering (Grouplens, 1994)Take about 20-50 people who sharesimilar taste with you, afterwardspredict how much you might like anitem depended on how much the othersliked it.You may like it because your“friends” liked it. 14
  29. 29. Technologies::Collaborative filtering User-based filtering Item-based filtering (Grouplens, 1994) (Amazon, 2001)Take about 20-50 people who share Pick from your previous list 20-50 itemssimilar taste with you, afterwards that share similar people with “thepredict how much you might like an target item”, how much you will like theitem depended on how much the others target item depends on how much theliked it. others liked those earlier items.You may like it because your You tend to like that item because“friends” liked it. you have liked those items. 14
  30. 30. Technologies::Content-based filtering Information needs of user and characteristics of items are represented in keywords, attributes, tags that describe past selections, e.g., TF-IDF. 15
  31. 31. Technologies::Hybrid RecSysCombination of techniques to overcomedisadvantages and advantages of single techniques. Advantages Disadvantages probabilistic combination of – Item-based method• No content analysis • Cold-start problem – User-based method – Matrix Factorization• Quality improves • Over-fitting – (May be) content-based method• No cold-start problem • New user / item problem The idea is to pick from my• No new user / item • Sparsity previous list 20-50 movies that share similar audience with problem “Taken”, then how much I will like depend on how much I liked those early movies – In short: I tend to watch this movie because I have watched those movies … or 16 – People who have watched those movies also liked this movie (Amazon style)
  32. 32. Technologies::Hybrid RecSysCombination of techniques to overcomedisadvantages and advantages of single techniques. Advantages Disadvantages probabilistic combination of – Item-based method• No content analysis • Cold-start problem – User-based method – Matrix Factorization• Quality improves • Over-fitting – (May be) content-based method• No cold-start problem • New user / item problem The idea is to pick from my• No new user / item • Sparsity previous list 20-50 movies that share similar audience with problem “Taken”, then how much I will like Just some examples there depend on how much I liked those early movies –are more watch this movie In short: I tend to (dis)advantages because I have watched those 16 movies … or available. – People who have watched those movies also liked this movie (Amazon style)
  33. 33. Evaluationof RecSys probabilistic combination of – Item-based method – User-based method – Matrix Factorization – (May be) content-based method The idea is to pick from my previous list 20-50 movies that share similar audience with “Taken”, then how much I will like depend on how much I liked those early movies – In short: I tend to watch this movie because I have watched those movies … or 17 – People who have watched those movies also liked this movie (Amazon style)
  34. 34. Evaluation::General idea Most of the time based on performance measures (“How good are your recommendations?”)For example:•Predict what rating will a user give an item?•Will the user select an item?•What is the order of usefulness of items to a user?Herlocker, Konstan, Riedl (2004). Evaluating Collaborative Filtering RecommenderSystems. ACM Transactions on Information Systems, 22(1), 5-53. 18
  35. 35. Evaluation::Reference datasets ... and various commercial datasets. 19
  36. 36. Evaluation::Approaches 1. Simulation•User preference•Prediction accuracy•Coverage•Confidence•Trust•Novelty 2. User study•Serendipity•Diversity•Utility•Risk•Robustness +•Privacy•Adaptivity•Scalability 20
  37. 37. Evaluation::Metrics Precision – The portion of recommendations that were successful. (Selected by the algorithm and by the user) Recall – The portion of relevant items selected by algorithm compared to a total number of relevant items available. F1 - Measure balances Precision and Recall into a single measurement.Gunawardana, A., Shani, G., (2009). A Survey of Accuracy Evaluation Metrics ofRecommendation Tasks, Journal of Machine Learning Research, 10(Dec):2935−2962,2009. 21
  38. 38. Evaluation::Metrics Precision – The portion of recommendations that were successful. (Selected by the algorithm and by the user) Recall – The portion of relevant items selected by algorithm compared to a total number of relevant items available. F1 - Measure balances Precision and Recall into a single measurement.Gunawardana, A., Shani, G., (2009). A Survey of Accuracy Evaluation Metrics ofRecommendation Tasks, Journal of Machine Learning Research, 10(Dec):2935−2962,2009. 21
  39. 39. Evaluation::Metrics Precision – The portion of recommendations that were successful. (Selected by the algorithm and by the user) Recall – The portion of relevant items selected by algorithm compared to a total number of relevant items available. F1 - Measure balances Precision and Recall into a single measurement.Gunawardana, A., Shani, G., (2009). A Survey of Accuracy Evaluation Metrics ofRecommendation Tasks, Journal of Machine Learning Research, 10(Dec):2935−2962,2009. 21
  40. 40. Evaluation::Metrics Precision – The portion of recommendations that were successful. (Selected by the algorithm and by the user) Recall – The portion of relevant items selected by algorithm compared to a total number of relevant items available. F1 - Measure balances Precision Just some examples there and Recall into a single are more metrics available measurement. like MAE, RSME.Gunawardana, A., Shani, G., (2009). A Survey of Accuracy Evaluation Metrics ofRecommendation Tasks, Journal of Machine Learning Research, 10(Dec):2935−2962,2009. 21
  41. 41. Evaluation::Metrics 5 Conclusion: 4 Pearson is better RMSE than Cosine, 3 Pearson because less 2 errors in predicting Cosine 1 TOP-N items. 0 Netflix BookCrossingGunawardana, A., Shani, G., (2009). A Survey of Accuracy Evaluation Metrics ofRecommendation Tasks, Journal of Machine Learning Research, 10(Dec):2935−2962,2009. 22
  42. 42. Evaluation::Metrics 5 Conclusion: 4 Pearson is better RMSE than Cosine, 3 Pearson because less 2 errors in predicting Cosine 1 TOP-N items. 0 Netflix BookCrossing News Story Clicks Conclusion: 80% Cosine better than Precision 60% Pearson, because 40% of higher precision 20% and recall value on TOP-N items. 0% 5% 10% 15% 20% 25% 30% 35% 40% RecallGunawardana, A., Shani, G., (2009). A Survey of Accuracy Evaluation Metrics ofRecommendation Tasks, Journal of Machine Learning Research, 10(Dec):2935−2962,2009. 22
  43. 43. RecSys::TimeToThinkWhat do you expect that a RecSys in aMUP/PLE should do with respect to ...• Aims• Tasks• Technology Blackmore’s custom-built LSD Drive• Evaluation http://www.flickr.com/photos/ rootoftwo/ 23
  44. 44. Goals of the lecture1. Crash course Recommender Systems (RecSys)2. Overview of RecSys in TEL3. Open research issues for RecSys in TEL4.TEL RecSys and Learning Analytics 24
  45. 45. Recommender Systemsfor TEL Introduction Objectives Technologies Evaluation 25
  46. 46. TEL RecSys::Definition Using the experiences of a community of learners to help individual learners in that community to identify more effectively learning content from a potentially overwhelming set of choices.Extended definition by Resnick & Varian (1997). Recommender Systems, Communications of the ACM, 40(3). 26
  47. 47. FindTEL RecSys::Learning spectrum M fo co Maj sy fa We apCross, J., Informal learning. Pfeifer. (2006). 27
  48. 48. The Long TailGraphic: Wilkins, D., (2009). 28
  49. 49. The Long Tail of LearningGraphic: Wilkins, D., (2009). 28
  50. 50. The Long Tail of Learning Formal InformalGraphic: Wilkins, D., (2009). 28
  51. 51. TEL RecSys::Technologies 29
  52. 52. TEL RecSys:: Technologies 30
  53. 53. TEL RecSys:: Technologies 31
  54. 54. TEL RecSys:: Technologies RecSys Task: Find good items Hybrid RecSys: •Content-based on interests •Collaborative filtering 31
  55. 55. TEL RecSys::Tasks Find good items e.g. relevant items for a learning task or a learning goal The idea is to pick from my previous list 20-50 movies that share similar audience with “Taken”, then how much I will like depend on how much I liked those early movies – In short: I tend to watch this movie because I have watched thoseDrachsler, H., Hummel, H., Koper, R., (2009). Identifying the goal, user model and conditions of movies … or recommender systems for formal and informal–learning. Journal watched those 32 People who have of Digital Information. 10(2). movies also liked this movie (Amazon style)
  56. 56. TEL RecSys::Tasks Find good items e.g. relevant items for a learning task or a learning goal Receive sequence of items e.g. recommend a learning path to achieve a certain competence The idea is to pick from my previous list 20-50 movies that share similar audience with “Taken”, then how much I will like depend on how much I liked those early movies – In short: I tend to watch this movie because I have watched thoseDrachsler, H., Hummel, H., Koper, R., (2009). Identifying the goal, user model and conditions of movies … or recommender systems for formal and informal–learning. Journal watched those 32 People who have of Digital Information. 10(2). movies also liked this movie (Amazon style)
  57. 57. TEL RecSys::Tasks Find good items e.g. relevant items for a learning task or a learning goal Receive sequence of items e.g. recommend a learning path to achieve a certain competenceAnnotation in context The idea is to pick from mye.g. take into account location, previous list 20-50 movies that share similar audience with time, noise level, prior “Taken”, then how much I will like knowledge, peers around depend on how much I liked those early movies – In short: I tend to watch this movie because I have watched thoseDrachsler, H., Hummel, H., Koper, R., (2009). Identifying the goal, user model and conditions of movies … or recommender systems for formal and informal–learning. Journal watched those 32 People who have of Digital Information. 10(2). movies also liked this movie (Amazon style)
  58. 58. Evaluation of TEL RecSys probabilistic combination of – Item-based method – User-based method – Matrix Factorization – (May be) content-based method The idea is to pick from my previous list 20-50 movies that share similar audience with “Taken”, then how much I will like depend on how much I liked those early movies – In short: I tend to watch this movie because I have watched those movies … or 33 – People who have watched those movies also liked this movie (Amazon style)
  59. 59. TEL RecSys::Review study 34
  60. 60. TEL RecSys::Review studyManouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011).Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F. Ricci,L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp. 387-415).Berlin: Springer. 34
  61. 61. TEL RecSys::Review study Conclusions: Half of the systems (11/20) still at design or prototyping stage only 8 systems evaluated through trials with human users.Manouselis, N., Drachsler, H., Vuorikari, R., Hummel, H. G. K., & Koper, R. (2011).Recommender Systems in Technology Enhanced Learning. In P. B. Kantor, F. Ricci,L. Rokach, & B. Shapira (Eds.), Recommender Systems Handbook (pp. 387-415).Berlin: Springer. 34
  62. 62. Thus...“The performance resultsof different researchefforts in recommendersystems are hardlycomparable.”(Manouselis et al., 2010) Kaptain Kobold http://www.flickr.com/photos/ kaptainkobold/3203311346/ 35
  63. 63. Thus...TEL recommender “The performance resultsexperiments lack of different research efforts in recommendertransparency. They need systems are hardlyto be repeatable to test: comparable.”• Validity•(Manouselis et al., 2010) Verification• Compare results Kaptain Kobold http://www.flickr.com/photos/ kaptainkobold/3203311346/ 35
  64. 64. TEL RecSys::Evaluation/datasets 36
  65. 65. TEL RecSys::Evaluation/datasetsDrachsler, H., Bogers, T., Vuorikari, R., Verbert, K., Duval, E., Manouselis, N., Beham, G.,Lindstaedt, S., Stern, H., Friedrich, M., & Wolpers, M. (2010). Issues and Considerationsregarding Sharable Data Sets for Recommender Systems in Technology Enhanced Learning.Presentation at the 1st Workshop Recommnder Systems in Technology Enhanced Learning(RecSysTEL) in conjunction with 5th European Conference on Technology EnhancedLearning (EC-TEL 2010): Sustaining TEL: From Innovation to Learning and Practice.September, 28, 2010, Barcelona, Spain. 36
  66. 66. Evaluation::Metrics MAE – Mean Absolute Error: Deviation of recommendations from the user-specified ratings. The lower the MAE, the more accurately the RecSys predicts user ratings.Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Beham, G., Duval, E.,(2011). Dataset-driven Research for Improving Recommender Systems for Learning. LearningAnalytics & Knowledge: February 27-March 1,37 2011, Banff, Alberta, Canada
  67. 67. Evaluation::Metrics MAE – Mean Absolute Error: Deviation of recommendations from the user-specified ratings. The lower the MAE, the more accurately the RecSys predicts user ratings. Outcomes: Tanimoto similarity + item-based CF was the most accurate.Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Beham, G., Duval, E.,(2011). Dataset-driven Research for Improving Recommender Systems for Learning. LearningAnalytics & Knowledge: February 27-March 1,37 2011, Banff, Alberta, Canada
  68. 68. Evaluation::Metrics MAE – Mean Absolute Error: Deviation of recommendations from the user-specified ratings. The lower the MAE, the more accurately the RecSys predicts user ratings.Outcomes:•User-based CF Algorithm thatpredicts the top 10 most relevant Outcomes:items for a user has a F1 score Tanimoto similarity +of almost 30%. item-based CF was•the most accurate. Implicit ratings like download rates, bookmarks can successfully used in TEL.Verbert, K., Drachsler, H., Manouselis, N., Wolpers, M., Vuorikari, R., Beham, G., Duval, E.,(2011). Dataset-driven Research for Improving Recommender Systems for Learning. LearningAnalytics & Knowledge: February 27-March 1,37 2011, Banff, Alberta, Canada
  69. 69. TEL RecSys::EvaluationCombined approach by Kirkpatrick model by Drachsler et al. 2008 Manouselis et al. 2010 38
  70. 70. TEL RecSys::Evaluation 1. Accuracy 2. Coverage 3. PrecisionCombined approach by Kirkpatrick model by Drachsler et al. 2008 Manouselis et al. 2010 38
  71. 71. TEL RecSys::Evaluation 1. Accuracy 2. Coverage 3. Precision 1. Effectiveness of learning 2. Efficiency of learning 3. Drop out rate 4. SatisfactionCombined approach by Kirkpatrick model by Drachsler et al. 2008 Manouselis et al. 2010 38
  72. 72. TEL RecSys::Evaluation 1. Accuracy 1. Reaction of learner 2. Coverage 2. Learning improved 3. Precision 3. Behaviour 4. Results 1. Effectiveness of learning 2. Efficiency of learning 3. Drop out rate 4. SatisfactionCombined approach by Kirkpatrick model by Drachsler et al. 2008 Manouselis et al. 2010 38
  73. 73. Goals of the lecture1. Crash course Recommender Systems (RecSys)2. Overview of RecSys in TEL3. Open research issues for RecSys in TEL4.TEL RecSys and Learning Analytics 39
  74. 74. TEL RecSys::Open issues1. Evaluation of TEL RecSys2. Publicly available datasets3. Comparable experiments4. Body of knowledge5. Privacy and data protection6. Design learning driven RecSys 40
  75. 75. Goals of the lecture1. Crash course Recommender Systems (RecSys)2. Overview of RecSys in TEL3. Open research issues for RecSys in TEL4.TEL RecSys and Learning Analytics 41
  76. 76. Greller, W., & Drachsler, H., 2011. 42
  77. 77. Greller, W., & Drachsler, H., 2011. 42
  78. 78. Greller, W., & Drachsler, H., 2011. 42
  79. 79. Greller, W., & Drachsler, H., 2011. 42
  80. 80. Greller, W., & Drachsler, H., 2011. 42
  81. 81. Greller, W., & Drachsler, H., 2011. 42
  82. 82. Greller, W., & Drachsler, H., 2011. 42
  83. 83. Learning Analytics::TimeToThink • Consider the Learning Analytics framework and imagine some great TEL RecSys that could support you in your stakeholder role alternatively • Name a learning task where a TEL RecSys would be useful for. 43
  84. 84. Thank you for attending this lecture! This silde is available at: http://www.slideshare.com/Drachsler Email: hendrik.drachsler@ou.nl Skype: celstec-hendrik.drachsler Blogging at: http://www.drachsler.de Twittering at: http://twitter.com/HDrachsler 44

×