Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
An architecture for evaluatingrecommender systems in real worldscenariosMaster Thesis Manuel Blechschmidt 2011  Supervisor...
2                       Christmas 2009 ...    Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 1...
Agenda3      ■ Motivation and Current Research      ■ Solution           □ Use Cases & Requirements           □ Wireframes...
4    Motivation and Current Research    Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
5                                    Experiment    Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschm...
Choice6    Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Motivation7      ■ The choice overload problem is well known in psychology           □ It is necessary to do a preselectio...
Current Algorithms and Developments8      ■ Matrix Factorization (best RMSE 0.855 for NetFlix Dataset)           □ SVD    ...
Empirical Studies9      ■ Current empirical studies (RecSys 2010)         □ Understanding Choice Overload in Recommender S...
Current Problems10       ■ Not a lot of big empirical studies how recommender quality         influence consumer behavior ...
Evaluating in real world11       ■ Most of the academia persons do not know enough persons which         are willing to te...
12     Solution     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Master Thesis13       ■ Building and maintaining an evaluation platform for recommender         systems in real world scen...
Solution: Use Cases14     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Roles15       ■ 5 Roles with different point of views and different interests and         goals       ■ The roles are desc...
Use Cases and Requirements16       ■ Use Cases and Requirements are described based on IEEE 830       ■ A use case is defi...
Use Case Example C1 Design User     Interaction17       ■ Id: C1 Name: Design User Interaction       ■ Summary:           ...
C1 Design User Interaction18     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
C1 Design User Interaction19     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
C1 Design User Interaction20     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
C1 Design User Interaction21     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Implemented Architecture22     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Logical Modularization23     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Survey Module Entities24     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Survey Module Services25     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
26                                            Demo     Evaluate Recommender Systems in Real World Scenarios | Manuel Blech...
Implemented User Interaction     chocStore27     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmid...
28     Related Work     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Related Work: Competition29       ■ NetFlix Grand Prize 2006 – 2009            □ 1.000.000 $ to make CineMatch 10% better ...
Related Work: Platforms30       ■ GroupLens Research of University of Minnesota            □ MovieLens 1997 http://moviele...
Further Research31       ■ Implement more user interactions            □ Item-to-Item recommender       ■ Prove that the p...
32     Conclusion     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Conclusion33       ■ An enterprise ready platform was defined and implemented       ■ Companies already applied for using ...
Questions34                                       Questions?     Evaluate Recommender Systems in Real World Scenarios | Ma...
Backup: What is a recommender?35     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
36     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
37     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
38     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
39     Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
Upcoming SlideShare
Loading in …5
×

An architecture for evaluating recommender systems in real world scenarios

2,185 views

Published on

The choice overload problem is well known in psychology
□ It is necessary to do a preselection for the customer
■ Recommender systems are already very successful to decrease the choice overload problem in some domains
□ Product-to-Product Recommendation → Amazon.com
□ Movie Recommendation → NetFlix
■ Algorithms already produce great results
Evaluation still very difficult for Research. kaggle.com and tunedit.org are hosting competitions.

Published in: Technology, Education

An architecture for evaluating recommender systems in real world scenarios

  1. 1. An architecture for evaluatingrecommender systems in real worldscenariosMaster Thesis Manuel Blechschmidt 2011 Supervisor Prof. Dr. Christoph Meinel M.Sc. Rehab Alnemr
  2. 2. 2 Christmas 2009 ... Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  3. 3. Agenda3 ■ Motivation and Current Research ■ Solution □ Use Cases & Requirements □ Wireframes □ Implementation ■ Related Work ■ Conclusion Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  4. 4. 4 Motivation and Current Research Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  5. 5. 5 Experiment Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  6. 6. Choice6 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  7. 7. Motivation7 ■ The choice overload problem is well known in psychology □ It is necessary to do a preselection for the customer ■ Recommender systems are already very successful to decrease the choice overload problem in some domains □ Product-to-Product Recommendation → Amazon.com □ Movie Recommendation → NetFlix ■ Algorithms already produce great results ■ Already research in soft factores like: Diversity, Serendepity, Trust, Explanations → not a lot of emprical studies how these influences customers → no cross domain data sets → not a lot of business intereset integration Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  8. 8. Current Algorithms and Developments8 ■ Matrix Factorization (best RMSE 0.855 for NetFlix Dataset) □ SVD □ SVD++ R.M.Bell, Y. Koren, and C. Volinsky □ TimeSVD++ R.M.Bell, Y. Koren, and C. Volinsky ■ Collaborative Filtering □ Item based □ User based ■ Performance gains □ ALS1 István Pilászy, Dávid Zibriczky, Domonkos Tikk ■ Some of the algorithms already implemented in a distributed manner Mahout, MyMedia Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  9. 9. Empirical Studies9 ■ Current empirical studies (RecSys 2010) □ Understanding Choice Overload in Recommender Systems 174 participants □ Eye-Tracking Product Recommendersʼ Usage 18 participants □ Recommender Algorithms in Activity Motivating Games 180 participants □ Group-Based Recipe Recommendations: Analysis of Data Aggregation Strategies 170 participants □ A User-Centric Evaluation Framework of Recommender Systems 807 participants □ Information Overload and Usage of Recommendations 466 participants □ ... Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  10. 10. Current Problems10 ■ Not a lot of big empirical studies how recommender quality influence consumer behavior especially □ Acurarcy □ Familiarity □ Serendipity □ Attractiveness □ Enjoyability □ Novelty □ Diversity □ Context Compatibility ■ Taken from A User-Centric Evaluation Framework of Recommender Systems Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  11. 11. Evaluating in real world11 ■ Most of the academia persons do not know enough persons which are willing to test the algorithms. Therefore the following things are difficult: □ Evaluating User Interfaces □ Evaluating Maintenance □ Evaluating Scalibility □ Evaluating Performance Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  12. 12. 12 Solution Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  13. 13. Master Thesis13 ■ Building and maintaining an evaluation platform for recommender systems in real world scenarios ■ Maintenance challenges in running a recommender system ■ Empirical study about user behavior □ Brand loyalty □ Pricing □ Timing Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  14. 14. Solution: Use Cases14 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  15. 15. Roles15 ■ 5 Roles with different point of views and different interests and goals ■ The roles are describeded with description and goals ■ Example: □ Provider □ A provider is a legal personality which has as primary goal to optimize a particular objective. In an economic context this is most of the time a business goal like raise profit or optimize conversion rates. … □ Goals: – optimizing an objective – get forecasts – ensure privacy of his data Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  16. 16. Use Cases and Requirements16 ■ Use Cases and Requirements are described based on IEEE 830 ■ A use case is defined by: □ Id □ Name □ Summary □ Roles □ Preconditions □ Postconditions □ Wireframes □ More optional attributes Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  17. 17. Use Case Example C1 Design User Interaction17 ■ Id: C1 Name: Design User Interaction ■ Summary: When a user interaction should be run like a newsletter or an item-to-item recommendation the consultant has to do the following steps: … ■ Roles: Consultant ■ Preconditions □ User is logged in □ User has the Consultant role □ At least one user interaction is implemented □ At least one provider is associated with the consultant □ The provider has the necessary data which is needed for the user interaction ■ Postconditions □ Provider received an email for approving the user interaction □ User interaction is created in the system Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  18. 18. C1 Design User Interaction18 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  19. 19. C1 Design User Interaction19 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  20. 20. C1 Design User Interaction20 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  21. 21. C1 Design User Interaction21 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  22. 22. Implemented Architecture22 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  23. 23. Logical Modularization23 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  24. 24. Survey Module Entities24 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  25. 25. Survey Module Services25 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  26. 26. 26 Demo Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  27. 27. Implemented User Interaction chocStore27 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  28. 28. 28 Related Work Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  29. 29. Related Work: Competition29 ■ NetFlix Grand Prize 2006 – 2009 □ 1.000.000 $ to make CineMatch 10% better □ Lots research of papers ■ KDD Cup 2011 Recommending Music Items based on the Yahoo! Music Dataset ■ ECML/PKDD’2007 DISCOVERY CHALLENGE □ User 1 User’s behaviour prediction Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  30. 30. Related Work: Platforms30 ■ GroupLens Research of University of Minnesota □ MovieLens 1997 http://movielens.umn.edu/ ■ RichRelevance RecLab 2011 □ RecLab: A System For eCommerce Recommender Research with Real Data, Context and Feedback ■ Knowledge and Data Engineering Group of Uni Kassel □ 2006 BibSonomy is a system for sharing bookmarks and lists of literature. Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  31. 31. Further Research31 ■ Implement more user interactions □ Item-to-Item recommender ■ Prove that the platform is scalable ■ Run the platform for a long time and evaluate usage ■ Integrate more companies ■ Promote plattform in science and economics ■ Take part at research projects together with companies Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  32. 32. 32 Conclusion Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  33. 33. Conclusion33 ■ An enterprise ready platform was defined and implemented ■ Companies already applied for using ■ One example user interaction was implemented □ chocStore ■ Statistical test can be applied to the data to give scientific results Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  34. 34. Questions34 Questions? Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  35. 35. Backup: What is a recommender?35 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  36. 36. 36 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  37. 37. 37 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  38. 38. 38 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11
  39. 39. 39 Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

×