Your SlideShare is downloading. ×
0
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Multiple objectives in Collaborative Filtering (RecSys 2010)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Multiple objectives in Collaborative Filtering (RecSys 2010)

877

Published on

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
877
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
16
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Background statement
    Explain what we mean by multiple objectives in this context
    Give two examples
    Motivated from the user point of view
    Motivated from the system point of view
    User point of view would improve the user experience
    System point of view would take into account other external factors
  • Background statement
    Explain what we mean by multiple objectives in this context
    Give two examples
    Motivated from the user point of view
    Motivated from the system point of view
    User point of view would improve the user experience
    System point of view would take into account other external factors
  • Improving the accuracy of the system does not necessarily equal to improving user experience
    Defined the recommender algorithm how good it performs on a metrics and how fast it can do its job
    But we need additional factors that would define the whole system
    For example if we take a VOD service
    User interface affects how user reads, understands find information including the recommendation
    How interesting the movies are that the user gets

    System related factors include
    The underlying hardware of the system
    How much, how fast an item can be delivered

    External factors include
    How much the company earns on an item, for example some items have a higher profit margin
    Personalised advertisement, recommending preferred items

  • User related and system related factors can directly improve user experience
    And we suggest that the combination of accuracy and other factors might improve user experience
    And the rest can help to maximize profit 
  • Consider accuracy as the main objective


  • Combine these into a single optimization framework.
    Take the prediction values of any baseline algorithm - r-hat.
    And add additional constraints that are equally important.
    Baseline algorithm to minimize errors.
    Higher weights represent higher importance
    Without constraints it returns the same as the original order

  • Recommendation is considered to be a ranking problem
    Top-N list
    It is a linear optimization problem, so global solution can be found.
    Biasing the original order by using constraints
  • Introducing two case studies to illustrate the use of the framework
    The first one is aim to promote items from the long tail.
    Assumption
    Current systems are biased towards popular items
    Picked the first 100 items that have received the highest ratings
    Where SVD is likely to place them
    Plot the distribution
    Figure show the probability that some popular are placed higher
    on the recommendation list by all the widely used recommender algorithms
  • Does that reflect user needs?
    We assume that discovering unknown items are more valuable
    We aim to identify users who would like alternative choices
    And recommend from the long tail for them
    Keep recommending popular items if the user has a more mainstream taste
  • We add this as an inequality constraint to the framework
    m is a vector that contains the mainstreamness value of each items in the recommended list
    m_u measures the mainstreamness of the user
    How these values are calculated are in the paper if you are interested
  • We added another extra bit to the framework to diversify items
  • Experimenting with diversification
    Promote item from the long tail that differ from each other
    Higher covariance
    That would reduce the risk of such an extension
  • Movielens 1m dataset
    Around 4000 movies, 6000 users
    Five-fold cross validation
  • Since we approach recommendation as a ranking problem
    We used the following IR measures
  • The probability that popular items ranked higher is significantly reduced
    Only 32% of the users have popular items in the top position
    The baseline is 45%
    We get reduction until position four
    Then it is slightly worse than the baseline
    This is the case when user studies would provide a better way to evaluate performance

  • Long tail constraint alone
    Long tail constraint with diversification
    Slight performance loss for all measures, except one.
  • The other case – system case
    Adding other non-user related factors to the system
    the availably of certain items
    The aim is to rank items lower if we are about to run out of stock
    But also minimize performance loss
  • The second scenario was evaluated by simulating the stock level
    Of an imaginary company for 50 days
    We presented a recommendation list to a random number of users each day
    The probability that a user took an item depended on the rank
    The cumulative probability up to the present point was based on
    a, How many times an item was show in the past
    B, And at which ranking position
    To evaluate the system we monitored the waiting list size
  • We introduce a cut-off point from which the system will start to reorganize the ranking list.
  • Adding it as a constraint to the system, in a same fashion
    s is a vector that represents the probability that the item is available at the given time
    For each item in the list
    s_u controls how cautious the system should be with stock level.
    e.g if s_u is higher the system starts penalizing items later
    As they are getting less available
  • The experiment is designed to run out of stock
    This graph shows the waiting list size for the first 20 days
    With respect to parameter c
    It controls when the system should start penalizing items as they are running out of stock
    e.g if c is set to 0.8 then
    after it is 80% likely that an item is taken the system starts penalizing
  • This table show the average and the maximum performance loss per day
    As you can see from c=0.8 and above there is only a slight performance loss of the system.
  • Two papers
    Optimizing algorithm from the user point of view
    One way is to identify different errors
    A general framework to handle multiple goals
    Two scenarios to illustrate that
  • Improving user experience might be validated using user studies
  • Transcript

    • 1. Multiple Objectives in Collaborative Filtering Tamas Jambor and Jun Wang University College London
    • 2. Structure of the talk • Motivation • Multiple objectives • User perspective – Promoting less popular items • System perspective – Stock management
    • 3. Motivation • In the RecSys community, many research efforts are focused on recommendation accuracy • And yet accuracy is not a only concern • Practical recommender systems might have multiple goals
    • 4. Improved Accuracy != Improved User experience Algorithm Additional factors Available resources Cost of delivery User interface Diverse choices Profitability per item Advertisement
    • 5. Improved user experience Available resources Cost of delivery User interface Diverse choices Profitability per item Advertisement Additional factors Accuracy Improved user experience
    • 6. Handling Multiple objectives • Accuracy is the main objective – Defined in the baseline algorithm • User perspective – Define and consider user satisfaction as priority • System perspective – Consider additional system related objectives • Objectives of the system might contradict
    • 7. Where to optimize? • In the objective function or as a post-filter? • Post-filters have the advantage to – Add to any baseline algorithm – Extend easily – Add multiple goals
    • 8. The proposed optimization framework (for each user) • Add additional constraints of w 0 11:tosubject ˆmax w w rw T T w 
    • 9. Properties of the framework • Linear optimization problem • Recommendation as a ranking problem • Constraints provide the means of biasing the ranking
    • 10. User case – Promoting the Long Tail Current systems are biased towards popular items 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 2 3 4 5 6 7 8 9 10 Probabilityofanyof100mostpopularitembeingat rankingposition Ranking Position SVD User-based Item-based Random Sample
    • 11. Promoting the Long Tail • Does that reflect real user needs? • Popular items might not be interesting for the user • Discovering unknown item could be more valuable • The aim is to reduce recommending popular items – if the user is likely to be an interested in alternative choices – keep recommending popular items otherwise
    • 12. Promoting the Long Tail • Extending the optimization framework 0 11 :tosubject ˆmax w w mwm rw T u T T w   
    • 13. Promoting the Long Tail and Diversification • Diversifying the results 0 11 :tosubject ˆmax w w mwm wwrw T u T TT w     
    • 14. Diversification • Increase the covariance between recommended items – Reduce the risk of expanding the system – Provide a wider range of choice
    • 15. Experimental setup • MovieLens 1m dataset • 3900 movies, 6040 users • Five-fold cross validation
    • 16. Evaluation metrics • Recommendation as a ranking problem • IR measures – Normalized discounted cumulative gain (NDCG) – Precision – Mean reciprocal rank (MRR) • Constraint specific measures
    • 17. Results: Promoting the Long Tail 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5 1 2 3 4 5 6 7 8 9 10 Probabilityofanyof100mostpopularitembeingat rankingposition Ranking Position Baseline (SVD) Long Tail Constraint Long Tail Constraint and Diversification (λ=6) Random Sample
    • 18. Results: Promoting the Long Tail Baseline (SVD) LTC LTC and Div (λ=6) NDCG@10 0.8808 0.8780 (-0.3%) 0.8715 (-1.0%) P@10 0.8204 0.8207 (+0.2%) 0.8177 (-0.3%) MRR 0.9518 0.9453 (-0.6%) 0.9349 (-1.7%)
    • 19. System case – Resource Constraint • Introducing external factors to the system • Stock availability of recommended items • The aim is to rank items lower, if less of them are available • Minimizing performance loss
    • 20. Simulation • Online DVD-Rental company – Operates a warehouse – Only a limited number of items are available • Recommend items that are in stock higher in the ranking list
    • 21. Simulation • User choice is based purely on recommendation • Simulating the stock level for 50 days – Present a list of items to a random number of users – The probability that the item is taken depends on the rank – Cumulative probability depends on how many times the item was shown and at which rank position
    • 22. Cut-off point • Threshold c controls the cut-off point from which the system starts re-ranking items       cp cps s ti titi ti , ,, , if0 if
    • 23. Resource Constraint • Extending the optimization framework 0 11 :tosubject ˆmax   w w sws rw T u T w  
    • 24. Evaluation: Monitoring the waiting list size • Waiting list – If item is not in stock, user puts it on their waiting list – When item returns, it goes out to the next user • Waiting list size represents how long a user has to wait to get their favourite items
    • 25. Results: Resource Constraint 0 20 40 60 80 100 120 140 160 180 0 5 10 15 20 Numberofitemsonthewaitinglist Time (days) baseline c=1.6 c=1.2 c=0.0
    • 26. Results: Resource Constraint • Trade-off between low waiting list size and good performance 0.75 0.77 0.79 0.81 0.83 0.85 0.87 0.89 0.91 0 5 10 15 20 NDCG@3 Time (days) baseline c=1.6 c=1.2 c=0.0
    • 27. Results: Resource Constraint c=0 c=0.4 c=0.8 c=1.2 c=1.6 NDCG@3(mean) -12.3% -4.32% -1.03% -0.43% -0.13% NDCG@3(max) -14.7% -5.12% -1.34% -0.56% -0.50% P@10(mean) -6.42% -3.37% -0.86% -0.06% -0.03% P@10(max) -8.42% -3.91% -1.11% -0.24% -0.18% Performance loss over 50 days
    • 28. Conclusion • Recommender systems have multiple objectives • Multiple optimization framework – Expand the system with minor performance loss – It is designed to add objectives flexibly – It can be added to any recommender system • Two scenarios that offer practical solutions – Long-tail items – Stock simulation
    • 29. Future plan • Personalized digital content delivery – Reduce delivery cost • Diversification and the long tail – Does recommendation kill diversity? • Evaluate improved user experience – User studies
    • 30. Thank you.
    • 31. References • Deshpande, M., Karypis, G.: Item-based top-N recommendation algorithms. ACM Trans. Inf. Syst. 22(1) (2004) • Herlocker, J.L., Konstan, J.A., Borchers, A., Riedl, J.: An algorithmic framework for performing collaborative filtering. In: SIGIR '99. (1999) • Koren, Y., Bell, R., Volinsky, C.: Matrix factorization techniques for recommender systems. Computer 42(8) (2009) • Wang, J., de Vries, A.P., Reinders, M.J.T.: Unifying user-based and item- based collaborative filtering approaches by similarity fusion. In: SIGIR '06: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, New York, NY, ACM Press

    ×