Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

- Conform highway noise barriers by nuform 392 views
- Precast Concrete Interiors by DRM Investments LTD 1330 views
- Noise barriers by vandana bharti 2288 views
- Underground water tunnel by Ratnesh Shinde 221 views
- Wet services water by Julian Swindell 2084 views
- Designing of noise barrier by Toastmaster Inter... 1632 views

2,068 views

Published on

No Downloads

Total views

2,068

On SlideShare

0

From Embeds

0

Number of Embeds

156

Shares

0

Downloads

13

Comments

0

Likes

3

No embeds

No notes for slide

- 1. Users and Noise: The Magic Barrier of Recommender Systems Alan Said, Brijnesh J. Jain, Sascha Narr, Till Plumbaum Competence Center Information Retrieval & Machine Learning @alansaid, @saschanarr, @matip
- 2. Outline► The Magic Barrier► Empirical Risk Minimization► Deriving the Magic Barrier► User Study► Conclusion 20 July 2012 The Magic Barrier 2
- 3. The Magic Barrier 20 July 2012 The Magic Barrier 3
- 4. The Magic Barrier► No magic involved....► Coined by Herlocker et al. in 2004 “...an algorithm cannot be more accurate than the variance in a user’s ratings for the same item.” The maximum level of prediction that a recommender algorithm can attain.► What does this mean? 20 July 2012 The Magic Barrier 4
- 5. The Magic Barrier 20 July 2012 The Magic Barrier 5
- 6. The Magic Barrier► Even a “perfect” recommender should not reach RMSE = 0 or Precision @ N = 1► Why? People are inconsistent and noisy in their ratings “perfect” accuracy is not perfect► So? Knowing the highest possible level of accuracy, we can stop optimizing our algorithms at “perfect” (before overfitting) 20 July 2012 The Magic Barrier 6
- 7. The Magic BarrierSo – how do we find the magic barrier?We employ the Empirical Risk Minimization principle and a statistical model for user inconsistencies 20 July 2012 The Magic Barrier 7
- 8. The Magic Barrier – User InconsistenciesAssumption: If a user were to re-rate all previously rated items, keeping in mind the inconsistency, the ratings would differ, i.e. 𝑟 𝑢𝑖 = 𝜇 𝑢𝑖 + 𝜀 𝑢𝑖 where 𝜇 𝑢𝑖 is the expected rating, and 𝜀 𝑢𝑖 the rating error (has zero mean) 20 July 2012 The Magic Barrier 8
- 9. Empirical Risk Minimization► … is a principle in statistical learning theory which defines a family of learning algorithms and is used to give theoretical bounds on the performance of learning algorithms.[Wikipedia] 20 July 2012 The Magic Barrier 9
- 10. Empirical Risk Minimization► We formulate our risk function as 𝑅 𝑓 = 𝑢,𝑖,𝑟 𝑝 𝑢, 𝑖, 𝑟 𝑓 𝑢, 𝑖 − 𝑟 2 The prediction error The probability of user u rating item i with score r► Keeping the assumption in mind, we formulate the risk for a true, unknown, rating function as the sum of the noise variance, i.e. 𝑅 𝑓∗ = 𝑢,𝑖 𝑝 𝑢, 𝑖 𝕍 𝜀 𝑢𝑖 where 𝕍 𝜀 𝑢𝑖 is the noise variance 20 July 2012 The Magic Barrier 10
- 11. Deriving the Magic Barrier► We want to express the risk function in terms of a magic barrier for RMSE – we take the root of the risk function ℬ 𝒰×ℐ = 𝑢,𝑖 𝑝 𝑢, 𝑖 𝕍 𝜀 𝑢𝑖 RMSE=0 iff 𝜀 𝑢𝑖 = 0 over all ratings users and items► In terms of RMSE we can express this as 𝐸 𝑅𝑀𝑆𝐸 𝑓 = ℬ 𝒰×ℐ + 𝐸 𝑓 > ℬ 𝒰×ℐ where 𝐸 𝑓 is the error 20 July 2012 The Magic Barrier 11
- 12. Estimating the Magic Barrier1. For each user-item pair in our population a) Sample ratings on a regular basis, i.e. re-ratings b) Estimate the expected value of ratings 𝑚 1 𝜇 𝑢𝑖 = 𝑟 𝑡 𝑢𝑖 𝑚 𝑡=1 c. Estimate the rating variance 𝑚 1 2 𝜀 𝑢𝑖 2 = 𝑚 𝜇 𝑢𝑖 − 𝑟𝑡 𝑢𝑖 𝑡=12. Estimate the magic barrier by taking the average 1 ℬ= 𝜀 𝑢𝑖 2 𝒳 𝑢𝑖 ∈𝒳 20 July 2012 The Magic Barrier 12
- 13. A real-world user study 20 July 2012 The Magic Barrier 13
- 14. A User Study► We teamed up with moviepilot.de Germany’s largest online movie recommendation community Ratings scale 1-10 stars (Netflix: 1-5 stars)► Created a re-rating UI Users were asked to re-rate at least 20 movies 1 new rating (so-called opinions) per movie Collected data: 306 users 6,299 new opinions 2,329 movies 20 July 2012 The Magic Barrier 14
- 15. A User Study User study moviepilot 20 July 2012 The Magic Barrier 15
- 16. A User Study ~4 ratings steps Room for improvement ~1 rating steps Predictions vs Ratings above Ratings below Ratings user’s average user’s average Overall Opinions above Opinions below Magic Barrier user’s average user’s average 20 July 2012 The Magic Barrier 16
- 17. Conclusion► We created a mathematical characterization of the magic barrier► We performed a user study on a commercial movie recommendation website and estimated its magic barrier► We concluded the commercial recommender engine still has room for improvement► No magic 20 July 2012 The Magic Barrier 17
- 18. More?► Estimating the Magic Barrier of Recommender Systems: A User Study SIGIR 2012► Magic Barrier explained http://irml.dailab.de► Movie rating and explanation user study http://j.mp/ratingexplain► Recommender Systems Wiki www.recsyswiki.com► Recommender Systems Challenge www.recsyschallenge.com 20 July 2012 The Magic Barrier 18
- 19. Questions?► Thank You for Listening! 20 July 2012 The Magic Barrier 19

No public clipboards found for this slide

Be the first to comment