The choice overload problem is well known in psychology □ It is necessary to do a preselection for the customer ■ Recommender systems are already very successful to decrease the choice overload problem in some domains □ Product-to-Product Recommendation → Amazon.com □ Movie Recommendation → NetFlix ■ Algorithms already produce great results Evaluation still very difficult for Research. kaggle.com and tunedit.org are hosting competitions.

1. An architecture for evaluating
recommender systems in real world
scenarios
Master Thesis Manuel Blechschmidt 2011
Supervisor
Prof. Dr. Christoph Meinel
M.Sc. Rehab Alnemr

2. 2
Christmas 2009 ...
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

3. Agenda
3
■ Motivation and Current Research
■ Solution
□ Use Cases & Requirements
□ Wireframes
□ Implementation
■ Related Work
■ Conclusion
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

4. 4
Motivation and Current Research
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

5. 5
Experiment
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

6. Choice
6
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

7. Motivation
7
■ The choice overload problem is well known in psychology
□ It is necessary to do a preselection for the customer
■ Recommender systems are already very successful to decrease
the choice overload problem in some domains
□ Product-to-Product Recommendation → Amazon.com
□ Movie Recommendation → NetFlix
■ Algorithms already produce great results
■ Already research in soft factores like: Diversity, Serendepity, Trust,
Explanations
→ not a lot of emprical studies how these influences customers
→ no cross domain data sets
→ not a lot of business intereset integration
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

8. Current Algorithms and Developments
8
■ Matrix Factorization (best RMSE 0.855 for NetFlix Dataset)
□ SVD
□ SVD++ R.M.Bell, Y. Koren, and C. Volinsky
□ TimeSVD++ R.M.Bell, Y. Koren, and C. Volinsky
■ Collaborative Filtering
□ Item based
□ User based
■ Performance gains
□ ALS1 István Pilászy, Dávid Zibriczky, Domonkos Tikk
■ Some of the algorithms already implemented in a distributed
manner Mahout, MyMedia
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

9. Empirical Studies
9
■ Current empirical studies (RecSys 2010)
□ Understanding Choice Overload in Recommender Systems
174 participants
□ Eye-Tracking Product Recommendersʼ Usage
18 participants
□ Recommender Algorithms in Activity Motivating Games
180 participants
□ Group-Based Recipe Recommendations: Analysis of Data Aggregation
Strategies
170 participants
□ A User-Centric Evaluation Framework of Recommender Systems
807 participants
□ Information Overload and Usage of Recommendations
466 participants
□ ...
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

10. Current Problems
10
■ Not a lot of big empirical studies how recommender quality
influence consumer behavior especially
□ Acurarcy
□ Familiarity
□ Serendipity
□ Attractiveness
□ Enjoyability
□ Novelty
□ Diversity
□ Context Compatibility
■ Taken from A User-Centric Evaluation Framework of Recommender
Systems
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

11. Evaluating in real world
11
■ Most of the academia persons do not know enough persons which
are willing to test the algorithms. Therefore the following things
are difficult:
□ Evaluating User Interfaces
□ Evaluating Maintenance
□ Evaluating Scalibility
□ Evaluating Performance
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

12. 12
Solution
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

13. Master Thesis
13
■ Building and maintaining an evaluation platform for recommender
systems in real world scenarios
■ Maintenance challenges in running a recommender system
■ Empirical study about user behavior
□ Brand loyalty
□ Pricing
□ Timing
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

14. Solution: Use Cases
14
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

15. Roles
15
■ 5 Roles with different point of views and different interests and
goals
■ The roles are describeded with description and goals
■ Example:
□ Provider
□ A provider is a legal personality which has as primary goal to
optimize a particular objective. In an economic context this is
most of the time a business goal like raise profit or optimize
conversion rates. …
□ Goals:
– optimizing an objective
– get forecasts
– ensure privacy of his data
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

16. Use Cases and Requirements
16
■ Use Cases and Requirements are described based on IEEE 830
■ A use case is defined by:
□ Id
□ Name
□ Summary
□ Roles
□ Preconditions
□ Postconditions
□ Wireframes
□ More optional attributes
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

17. Use Case Example C1 Design User
Interaction
17
■ Id: C1 Name: Design User Interaction
■ Summary: When a user interaction should be run like a newsletter or an item-to-item recommendation the
consultant has to do the following steps: …
■ Roles: Consultant
■ Preconditions
□ User is logged in
□ User has the Consultant role
□ At least one user interaction is implemented
□ At least one provider is associated with the consultant
□ The provider has the necessary data which is needed for the user interaction
■ Postconditions
□ Provider received an email for approving the user interaction
□ User interaction is created in the system
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

18. C1 Design User Interaction
18
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

19. C1 Design User Interaction
19
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

20. C1 Design User Interaction
20
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

21. C1 Design User Interaction
21
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

22. Implemented Architecture
22
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

23. Logical Modularization
23
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

24. Survey Module Entities
24
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

25. Survey Module Services
25
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

26. 26
Demo
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

27. Implemented User Interaction
chocStore
27
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

28. 28
Related Work
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

29. Related Work: Competition
29
■ NetFlix Grand Prize 2006 – 2009
□ 1.000.000 $ to make CineMatch 10% better
□ Lots research of papers
■ KDD Cup 2011 Recommending Music Items
based on the Yahoo! Music Dataset
■ ECML/PKDD’2007 DISCOVERY CHALLENGE
□ User 1 User’s behaviour prediction
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

30. Related Work: Platforms
30
■ GroupLens Research of University of Minnesota
□ MovieLens 1997 http://movielens.umn.edu/
■ RichRelevance RecLab 2011
□ RecLab: A System For eCommerce Recommender Research
with Real Data, Context and Feedback
■ Knowledge and Data Engineering Group of Uni Kassel
□ 2006 BibSonomy is a system for sharing bookmarks and lists
of literature.
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

31. Further Research
31
■ Implement more user interactions
□ Item-to-Item recommender
■ Prove that the platform is scalable
■ Run the platform for a long time and evaluate usage
■ Integrate more companies
■ Promote plattform in science and economics
■ Take part at research projects together with companies
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

32. 32
Conclusion
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

33. Conclusion
33
■ An enterprise ready platform was defined and implemented
■ Companies already applied for using
■ One example user interaction was implemented
□ chocStore
■ Statistical test can be applied to the data to give scientific results
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

34. Questions
34
Questions?
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

35. Backup: What is a recommender?
35
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

36. 36
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

37. 37
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

38. 38
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11

39. 39
Evaluate Recommender Systems in Real World Scenarios | Manuel Blechschmidt 18.04.11