The Science and the Magic of User Feedback for Recommender Systems

Xavier Amatriain
Xavier AmatriainCofounder/CTO at Curai
The Science and the Magic
       of User Feedback
  for Recommender Systems




      Xavier Amatriain




                         Bay Area, March '11
But first...




About Telefonica and Telefonica R&D
Telefonica is a fast-growing Telecom


                            1989                   2000                    2008
  Clients                        About 12          About 68          About 260
                                  million           million           million
                                subscribers       customers          customers
 Services                        Basic        Wireline and mobile    Integrated ICT
                            telephone and       voice, data and     solutions for all
                             data services     Internet services       customers
Geographies
                                                 Operations in      Operations in
                              Spain                                 25 countries
                                                 16 countries

   Staff
                     About 71,000                About 149,000         About 257,000
                     professionals                professionals         professionals

 Finances                   Rev: 4,273 M€       Rev: 28,485 M€      Rev: 57,946 M€
                            EPS(1): 0.45 €      EPS(1): 0.67 €        EPS: 1.63 €
            (1) EPS: Earnings per share
Currently among the largest in the world
       Telco sector worldwide ranking by market cap (US$ bn)




                                                         Source: Bloomberg, 06/12/09




       Just announced 2010 results: record net earnings, 
       first Spanish company ever to make > 10B €
Leader in South America

Data as of March ‘09



 1      2 Argentina: 20.9 million                                                            Wireline market rank 
 2      1 Brazil: 61.4 million                                                               Mobile market rank
        2 Central America: 6.1 million
 1      2 Colombia: 12.6 million
 1      1 Chile: 10.1 million
        2 Ecuador: 3.3 million
        2 Mexico: 15.7 million
 1      1 Peru: 15.2 million
        1
          Uruguay: 1.5 million
        2 Venezuela: 12.0 million
                                                                      Total Accesses (as of March ‘09)
                                                                                       159.5 million

Notes:
- Central America includes Guatemala, Panama, El Salvador and Nicaragua
- Total accesses figure includes Narrowband Internet accesses of Terra Brasil and Terra Colombia, and Broadband
Internet accesses of Terra Brasil, Telefónica de Argentina, Terra Guatemala and Terra México.
And a significant footprint in Europe

                                      Wireline market rank
                                      Mobile market rank
Data as of March ‘09



                             1 1   Spain: 47.2 million
                                 1 UK: 20.8 million
                                 4 Germany: 16.0 million
                                 2
                                   Ireland: 1.7 million
                                   Czech Republic: 7.7 million
                             1   2
                                   Slovakia: 0.4 million
                                 3


                                     Total Accesses (as of March ’09)
                                              93.8 million
Scientific Research
                  Mobile and Ubicomp
Multimedia Core                                    User Modelling &
                                                     Data Mining



                                            HCIR

                                                   DATA MINING



                                                        Wireless Systems
Content Distribution & P2P
                                Social Networks
Projects
      Recommendation 
      Algorithms
                             Tourist routes

                        Social 
                       contacts     Music


User Analysis                 Movies
& Modeling                                                 Contextual
                           The Wisdom of the 
          Noise in users’        Few
                                            Mobile
         ratings 
                            Tourist 
                           behavior        Microprofiles
           Implicit user 
          feedback
                                       Multiverse 
                                      Tensor 
            IPTV viewing habits
                                      Factorization
Projects
      Recommendation 
      Algorithms
                             Tourist routes

                        Social 
                       contacts     Music


User Analysis                 Movies
& Modeling                                                 Contextual
                           The Wisdom of the 
          Noise in users’        Few
                                            Mobile
         ratings 
                            Tourist 
                           behavior        Microprofiles
           Implicit user 
          feedback
                                       Multiverse 
                                      Tensor 
            IPTV viewing habits
                                      Factorization
And about the world we live in...
Information Overload
More is Less
                         W
                          or
                            se
                                 D
                                  ec
                                    is
         ns
                                      io
       io

                                        ns
     is
   ec
  D
   s
 es
L
Analysis Paralysis is making
         headlines
Search engines don’t always hold the answer
The Science and the Magic of User Feedback for Recommender Systems
What about discovery?
What about curiosity?
What about information to help take decisions?
The Age of Search has come to
                           an end

... long live the Age of Recommendation!
●


●
    Chris Anderson in “The Long Tail”
    ●
        “We are leaving the age of information and entering the age of
        recommendation”
●
    CNN Money, “The race to create a 'smart' Google”:
    ●
        “The Web, they say, is leaving the era of search and entering
        one of discovery. What's the difference? Search is what you do
        when you're looking for something. Discovery is when
        something wonderful that you didn't know existed, or didn't
        know how to ask for, finds you.”
Recommender
                Systems
            Recommendations



Read this

            Attend this conference
Data mining +
           all those other things
●   User Interface
●   User modeling
●   System requirements (efficiency, scalability,
    privacy....)
●   Business Logic
●   Serendipity
●   ....
Approaches to
                   Recommendation
Collaborative Filtering
●

    ●
        Recommend items based only on the users past behavior

Content-based
●

    ●
        Recommend based on features inherent to the items

Social recommendations (trust-based)
●
What works

●
    It depends on the domain and particular problem
    ●
        As a general rule, it is usually a good idea to combine:
        Hybrid Recommender Systems
●
 However, in the general case it has been
demonstrated that (currently) the best isolated
approach is CF.
    ●
        Item-based in general more efficient and better but
        mixing CF approaches can improve result
    ●
        Other approaches can improve results in specific
        cases (cold-start problem...)
The CF Ingredients

● List of m Users and a list of n Items
● Each user has a list of items with associated opinion

  ● Explicit opinion - a rating score (numerical scale)


  ● Implicit feedback – purchase records or listening

    history
● Active user for whom the prediction task is performed


● A metric for measuring similarity between users


● A method for selecting a subset of neighbors


● A method for predicting a rating for items not rated by

the active user.

                                                        24
The Netflix Prize

●   500K users x 17K movie
    titles = 100M ratings = $1M
    (if you “only” improve
    existing system by 10%!
    From 0.95 to 0.85 RMSE)
    ●   49K contestants on 40K teams from
        184 countries.
    ●   41K valid submissions from 5K
        teams; 64 submissions per day
    ●   Wining approach uses hundreds of
        predictors from several teams
But ...
User Feedback is Noisy




                        DID YOU HEAR WHAT
                            I LIKE??!!




...and limits Our Prediction
          Accuracy
The Magic Barrier

●   Magic Barrier = Limit on prediction accuracy
    due to noise in original data
●   Natural Noise = involuntary noise introduced by
    users when giving feedback
    ●   Due to (a) mistakes, and (b) lack of resolution in
        personal rating scale
●   Magic Barrier >= Natural Noise Threshold
    ●   Our prediction error cannot be smaller than the
        error in the original data
Our related research questions
X. Amatriain, J.M. Pujol, N. Oliver (2009) "I like It... I like It Not: Measuring Users
 Ratings Noise in Recommender Systems", in UMAP 09


    ●   Q1. Are users inconsistent when providing
        explicit feedback to Recommender Systems via
        the common Rating procedure?
    ●   Q2. How large is the prediction error due to
        these inconsistencies?
    ●   Q3. What factors affect user inconsistencies?
Experimental Setup

●   100 Movies selected from Netflix dataset doing
    a stratified random sampling on popularity
●   Ratings on a 1 to 5 star scale
    ●   Special “not seen” symbol.
●   Trial 1 and 3 = random order; trial 2 = ordered
    by popularity
User Feedback is Noisy

●   Users are inconsistent
●   Inconsistencies are not
    random and depend on
    many factors
User Feedback is Noisy

●   Users are inconsistent
●   Inconsistencies are not
    random and depend on
    many factors
    ●   More inconsistencies for mild
        opinions
User Feedback is Noisy

●   Users are inconsistent
●   Inconsistencies are not
    random and depend on
    many factors
    ●   More inconsistencies for mild
        opinions
    ●   More inconsistencies for
        negative opinions
User’s ratings are far from
           ground truth
                    #Ti       #Tj          #              RMSE


                                                                  
      T1, T2        2185      1961      1838     2308     0.573    0.707


      T1, T3        2185      1909      1774     2320     0.637    0.765


      T2, T3        1969      1909      1730     2140     0.557    0.694



Pairwise comparison between trials, RMSE is already > 0.55 or > 0.69 (Netflix Prize
  was to get below 0.85 !!!)
Algorithm Robustness to NN
Trial 2 is 
       Alg./Trial
consistently the       <T1       T2       T3      Tworst /Tbest
least noisy
       User            1.2011   1.1469   1.1945       4.7%
       Average
         Item          1.0555   1.0361   1.0776        4%
         Average
         User­based    0.9990   0.9640   1.0171       5.5%
         kNN
         Item­based    1.0429   1.0031   1.0417        4%
         kNN
         SVD           1.0244   0.9861   1.0285       4.3%


      RMSE for different Recommendation algorithms 
     ●


     when predicting each of the trials
Rate it Again
X. Amatriain et al. (2009)"Rate it Again: Increasing Recommendation
 Accuracy by User re-Rating", 2009 ACM RecSys
   ●
       Given that users are noisy… can we benefit from
       asking to rate the same movie more than once?

   ●
       We propose an algorithm to allow for multiple ratings of
       the same <user,item> tuple.
       ●
           The algorithm is subjected to two fairness conditions:
            – Algorithm should remove as few ratings as possible (i.e.
              only when there is some certainty that the rating is only
              adding noise)
            – Algorithm should not make up new ratings but decide on
              which of the existing ones are valid.
Re-rating Algorithm
• One source re­rating case:

                                             Examples:
                                             {3, 1}    →Ø
                                             {4}       →4
                                             {3, 4}    →3

                                             (2 source)
                                             {3, 4, 5}  →3


• Given the following milding function:   
Results

●    One-source re-rating (Denoised⊚Denoising)
                           T1⊚T2    ΔT1        T1⊚T3    ΔT1        T2⊚T3    ΔT2
    User­based kNN         0.8861   11.3%   0.8960      10.3%     0.8984    6.8%


    SVD                    0.9121   11.0%      0.9274   9.5%       0.9159   7.1%



●    Two-source re-rating (Denoising T1with the other 2)
          Datasets                  T1⊚(T2, T3)                  ΔT1
          User­based kNN              0.8647                    13.4%
          SVD                         0.8800                    14.1%
Rate it again

●   By asking users to rate items again we can
    remove noise in the dataset
    ●   Improvements of up to 14% in accuracy!
●   Because we don't want all users to re-rate all
    items we design ways to do partial denoising
    ●   Data-dependent: only denoise extreme ratings
    ●   User-dependent: detect “noisy” users
Denoising only noisy users




●    Improvement in RMSE when doing one­source as a function of 
the percentage of denoised ratings and users: selecting only noisy 
users and extreme ratings
The value or a re-rating




                Adding new ratings
                increases performance
                of the CF algorithm
The value or a re-rating



                But you are better off
                doing re-rating than
                new ratings !!
The value or a re-rating



         And much better if you
         know which ratings to
         re-rate!!
Let's recap

●   Users are inconsistent
●   Inconsistencies can depend on many things including
    how the items are presented
●   Inconsistencies produce natural noise
●   Natural noise reduces our prediction accuracy
    independently of the algorithm
●   By asking (some) users to re-rate (some) items again
    we can remove noise and improve accuracy
●   Having users repeat existing ratings may have more
    value than adding new ones
Crowds are not always wise




                           ●   Diversity of opinion
Conditions that are        ●   Independence
needed to guarantee the    ●   Decentralization
Wisdom in a Crowd          ●   Aggregation
Crowds are not always wise




                vs.




        Who  won?
The Wisdom of the Few
    X. Amatriain et al. "The wisdom of the few: a collaborative filtering
     approach based on expert opinions from the web", SIGIR '09
“It is really only experts 
who can reliably account 
   for their reactions”
Expert-based CF
●   expert = individual that we can trust to have produced
    thoughtful, consistent and reliable evaluations (ratings) of
    items in a given domain
●   Expert-based Collaborative Filtering
    ●   Find neighbors from a reduced set of experts instead of
        regular users.
         1. Identify domain experts with reliable ratings
         2. For each user, compute “expert neighbors”
         3. Compute recommendations similar to standard kNN CF
User Study
●   57 participants, only 14.5 ratings/participant
●   50% of the users consider Expert-based CF to be
    good or very good
●   Expert-based CF: only algorithm with an average
    rating over 3 (on a 0-4 scale)
Advantages of the Approach

●   Noise                          ●   Cold Start problem
    ●   Experts introduce less         ●   Experts rate items as
        natural noise                      soon as they are
●   Malicious Ratings                      available
    ●   Dataset can be monitored
                                   ●   Scalability
        to avoid shilling              ●   Dataset is several order of
●   Data Sparsity                          magnitudes smaller
    ●   Reduced set of domain
                                   ●   Privacy
        experts can be motivated       ●   Recommendations can be
        to rate items                      computed locally
So...

●   Can we generate meaningful and personalized
    recommendations ensuring 100% privacy?
    ●   YES!
●   Can we have a recommendation algorithm that
    is so efficient to run on a phone?
    ●   YES!
●   Can we have a recommender system that
    works even if there is only one user?
    ●   YES!
Architecture of the approach
Some implementations

●   A distributed Music Recommendation engine
Some implementations (II)

●   A geo-localized Mobile Movie Recommender
    iPhone App
Geo-localized Expert Movie
   Recommendations




                             0




       Powered by...
Expert CF...

●   Recreates the old paradigm of manually finding
    your favorite experts in magazines but in a fully
    automatic non-supervised manner.
What if we don't have ratings?


The fascinating world of implicit user feedback



 Examples of implicit feedback:
 ● Movies you watched

 ● Links you visited

 ● Songs you listened to

 ● Items you bought

 ● ....
Main features of implicit
               feedback
●   Our starting hypothesis are different from those
    in previous works:
    1.Implicit feedback can contain negative feedback –
      given the right granularity and diversity, low
      feedback = negative feedback
    2.Numerical value of implicit feedback can be
      mapped to preference given the appropriate
      mapping
    3.Once we have a trustworthy mapping, we can
      evaluate implicit feedback predictions same way as
      with explicit feedback.
Our questions

●   Q1. Is it possible to predict ratings a user would
    give to items given their implicit feedback?
●   Q2. Are there other variables that affect this
    mapping?
Experimental setup

●   Online user study on the music domain
●   Users required to have a music profile in lastfm
●   Goal: Compare explicit ratings with their
    listening history taking to account a number of
    controlled variables
Results. Do explicit ratings relate
      to implicit feedback?




                                Almost perfect linear 
                              relation between ratings 
                               and quantized implicit 
                                      feedback
Results. Do explicit ratings relate
          to implicit feedback?




Extreme ratings have clear 
  ascending/descending 
  trend, but mild ratings 
 respond more to changes 
      in one direction
Results. Do other variables affect?




             Albums listened to more 
              recently tend to receive 
               more positive ratings
Results. Do other variables affect?




       Contrary to our expectations, 
       global album popularity does 
             not affect ratings
Results. What about user
                  variables?

●   We obtained many demographic (age, sex, location...)
    and usage variables (hours of music per week,
    concerts, music magazines, ways of buying music...)
    in the study.
●   We performed an ANOVA analysis on the data to
    understand which variables explained some of its
    variance.
●   Only one of the usage variables, contributed (Sig.
    Value < 0.05) → “Listening Style” encoded whether the
    user listened preferably to tracks, full albums, or both.
Results. Regression
                   Analysis
           –   Model   1:   riu   =   β0   +   β1   ·   ifiu
           –   Model   2:   riu   =   β0   +   β1   ·   ifiu + β2 · reiu
           –   Model   3:   riu   =   β0   +   β1   ·   ifiu + β2 · reiu + β3 · gpi
           –   Model   4:   riu   =   β0   +   β1   ·   ifiu + β2 · reiu + β3 · ifiu · reiu

Model    R2                 F-value                         p-value        β0        β1       β2       β3

  1      0.125    F (1, 10120) = 1146                   < 2.2 · 10−16      2.726    0.499

  2     0.1358    F (2, 10019) = 794.8                  < 2.2 · 10−16      2.491    0.484     0.133

  3     0.1362    F (3, 10018) = 531.8                  < 2.2 · 10−16      2.435    0.486     0.134   0.0285

  4     0.1368    F (3, 10018) = 534.7                  < 2.2 · 10−16      2.677    0.379     0.038    0.053



All models meaningfully explain the data. Introducing “recentness” 
improves 10% but “global popularity” or interaction between variables do 
not make much difference
Results. Predictive power
               Model RMSE –
                     Excluding non-rated items


        User Average                    1.131

                   1                    1.026

                   2                    1.017

                   3                    1.016

                   4                    1.016




Error in predicting 20% of the ratings, having trained our 
regression model on the other 80%
Conclusions
●   Recommender systems and similar applications
    usually focus on having more data
●   But... many times is not about having more but rather
    better data
●   User feedback can not always be treated as ground
    truth and needs to be processed
●   Crowds are not always wise and sometimes we are
    better off using experts
●   Implicit feedback represents a good alternative to
    understand users but mapping is not trivial
Colleagues
●   Josep M. Pujol and Nuria Oliver (Telefonica)
    worked on Natural Noise and Wisdom of the
    Few projects
●   Nava Tintarev (Telefonica) worked on
    Natural Noise


          External Collaborators
●   Neal Lathia (UCL, London), Haewook Ahn
    (KAIST, Korea), Jaewook Ahn (Pittsburgh
    Univ.), and Josep Bachs (UPF, Barcelona)
    on Wisdom of the Few
●   Denis Parra (Pittsburgh Univ.) worked on
    implicit-explicit
Thanks!


        Questions?

    Xavier Amatriain
          xar@tid.es
      http://xavier.amatriain.net
http://technocalifornia.blogspot.com
            @xamat
1 of 71

Recommended

Recommender Systems by
Recommender SystemsRecommender Systems
Recommender SystemsGirish Khanzode
8.6K views96 slides
Recommender Systems by
Recommender SystemsRecommender Systems
Recommender SystemsCarlos Castillo (ChaTo)
2.8K views67 slides
Recommendation System Explained by
Recommendation System ExplainedRecommendation System Explained
Recommendation System ExplainedCrossing Minds
10.7K views64 slides
Recommender system introduction by
Recommender system   introductionRecommender system   introduction
Recommender system introductionLiang Xiang
12.2K views39 slides
Recommender system algorithm and architecture by
Recommender system algorithm and architectureRecommender system algorithm and architecture
Recommender system algorithm and architectureLiang Xiang
40.7K views39 slides
Recent advances in deep recommender systems by
Recent advances in deep recommender systemsRecent advances in deep recommender systems
Recent advances in deep recommender systemsNAVER Engineering
1.9K views70 slides

More Related Content

What's hot

Introduction to Recommendation Systems by
Introduction to Recommendation SystemsIntroduction to Recommendation Systems
Introduction to Recommendation SystemsTrieu Nguyen
11.8K views40 slides
An introduction to Recommender Systems by
An introduction to Recommender SystemsAn introduction to Recommender Systems
An introduction to Recommender SystemsDavid Zibriczky
2.3K views105 slides
Overview of recommender system by
Overview of recommender systemOverview of recommender system
Overview of recommender systemStanley Wang
6.9K views59 slides
Recommendation engines by
Recommendation enginesRecommendation engines
Recommendation enginesGeorgian Micsa
7.1K views20 slides
Deep Learning for Recommender Systems by
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender SystemsJustin Basilico
21K views35 slides
Deep Learning for Recommender Systems RecSys2017 Tutorial by
Deep Learning for Recommender Systems RecSys2017 Tutorial Deep Learning for Recommender Systems RecSys2017 Tutorial
Deep Learning for Recommender Systems RecSys2017 Tutorial Alexandros Karatzoglou
32.2K views80 slides

What's hot(20)

Introduction to Recommendation Systems by Trieu Nguyen
Introduction to Recommendation SystemsIntroduction to Recommendation Systems
Introduction to Recommendation Systems
Trieu Nguyen11.8K views
An introduction to Recommender Systems by David Zibriczky
An introduction to Recommender SystemsAn introduction to Recommender Systems
An introduction to Recommender Systems
David Zibriczky2.3K views
Overview of recommender system by Stanley Wang
Overview of recommender systemOverview of recommender system
Overview of recommender system
Stanley Wang6.9K views
Deep Learning for Recommender Systems by Justin Basilico
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
Justin Basilico21K views
Deep Learning for Recommender Systems RecSys2017 Tutorial by Alexandros Karatzoglou
Deep Learning for Recommender Systems RecSys2017 Tutorial Deep Learning for Recommender Systems RecSys2017 Tutorial
Deep Learning for Recommender Systems RecSys2017 Tutorial
Building a Recommendation Engine - An example of a product recommendation engine by NYC Predictive Analytics
Building a Recommendation Engine - An example of a product recommendation engineBuilding a Recommendation Engine - An example of a product recommendation engine
Building a Recommendation Engine - An example of a product recommendation engine
Recommender systems by Tamer Rezk
Recommender systemsRecommender systems
Recommender systems
Tamer Rezk997 views
How to build a recommender system? by blueace
How to build a recommender system?How to build a recommender system?
How to build a recommender system?
blueace25.6K views
Deep Learning for Recommender Systems by inovex GmbH
Deep Learning for Recommender SystemsDeep Learning for Recommender Systems
Deep Learning for Recommender Systems
inovex GmbH1.5K views
Recommendation Systems by Robin Reni
Recommendation SystemsRecommendation Systems
Recommendation Systems
Robin Reni904 views
Recommender systems using collaborative filtering by D Yogendra Rao
Recommender systems using collaborative filteringRecommender systems using collaborative filtering
Recommender systems using collaborative filtering
D Yogendra Rao3.4K views
Boston ML - Architecting Recommender Systems by James Kirk
Boston ML - Architecting Recommender SystemsBoston ML - Architecting Recommender Systems
Boston ML - Architecting Recommender Systems
James Kirk8.6K views
Recommendation at Netflix Scale by Justin Basilico
Recommendation at Netflix ScaleRecommendation at Netflix Scale
Recommendation at Netflix Scale
Justin Basilico21.6K views
Recommender Systems by T212
Recommender SystemsRecommender Systems
Recommender Systems
T21243.9K views
Recommender systems: Content-based and collaborative filtering by Viet-Trung TRAN
Recommender systems: Content-based and collaborative filteringRecommender systems: Content-based and collaborative filtering
Recommender systems: Content-based and collaborative filtering
Viet-Trung TRAN11.4K views

Viewers also liked

Discovery by
DiscoveryDiscovery
DiscoveryPat Ferrel
1.6K views23 slides
Key Lessons Learned Building Recommender Systems for Large-Scale Social Netw... by
 Key Lessons Learned Building Recommender Systems for Large-Scale Social Netw... Key Lessons Learned Building Recommender Systems for Large-Scale Social Netw...
Key Lessons Learned Building Recommender Systems for Large-Scale Social Netw...Christian Posse
3.3K views27 slides
BIG2016- Lessons Learned from building real-life user-focused Big Data systems by
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsXavier Amatriain
3.4K views48 slides
Machine Learning for Q&A Sites: The Quora Example by
Machine Learning for Q&A Sites: The Quora ExampleMachine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora ExampleXavier Amatriain
6.2K views47 slides
Past, present, and future of Recommender Systems: an industry perspective by
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspectiveXavier Amatriain
11K views71 slides
Past present and future of Recommender Systems: an Industry Perspective by
Past present and future of Recommender Systems: an Industry PerspectivePast present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry PerspectiveXavier Amatriain
7.2K views20 slides

Viewers also liked(20)

Discovery by Pat Ferrel
DiscoveryDiscovery
Discovery
Pat Ferrel1.6K views
Key Lessons Learned Building Recommender Systems for Large-Scale Social Netw... by Christian Posse
 Key Lessons Learned Building Recommender Systems for Large-Scale Social Netw... Key Lessons Learned Building Recommender Systems for Large-Scale Social Netw...
Key Lessons Learned Building Recommender Systems for Large-Scale Social Netw...
Christian Posse3.3K views
BIG2016- Lessons Learned from building real-life user-focused Big Data systems by Xavier Amatriain
BIG2016- Lessons Learned from building real-life user-focused Big Data systemsBIG2016- Lessons Learned from building real-life user-focused Big Data systems
BIG2016- Lessons Learned from building real-life user-focused Big Data systems
Xavier Amatriain3.4K views
Machine Learning for Q&A Sites: The Quora Example by Xavier Amatriain
Machine Learning for Q&A Sites: The Quora ExampleMachine Learning for Q&A Sites: The Quora Example
Machine Learning for Q&A Sites: The Quora Example
Xavier Amatriain6.2K views
Past, present, and future of Recommender Systems: an industry perspective by Xavier Amatriain
Past, present, and future of Recommender Systems: an industry perspectivePast, present, and future of Recommender Systems: an industry perspective
Past, present, and future of Recommender Systems: an industry perspective
Xavier Amatriain11K views
Past present and future of Recommender Systems: an Industry Perspective by Xavier Amatriain
Past present and future of Recommender Systems: an Industry PerspectivePast present and future of Recommender Systems: an Industry Perspective
Past present and future of Recommender Systems: an Industry Perspective
Xavier Amatriain7.2K views
Machine Learning to Grow the World's Knowledge by Xavier Amatriain
Machine Learning to Grow  the World's KnowledgeMachine Learning to Grow  the World's Knowledge
Machine Learning to Grow the World's Knowledge
Xavier Amatriain29.6K views
Building Large-scale Real-world Recommender Systems - Recsys2012 tutorial by Xavier Amatriain
Building Large-scale Real-world Recommender Systems - Recsys2012 tutorialBuilding Large-scale Real-world Recommender Systems - Recsys2012 tutorial
Building Large-scale Real-world Recommender Systems - Recsys2012 tutorial
Xavier Amatriain20K views
Setting up Kubernetes with tectonic by Vishal Biyani
Setting up Kubernetes with tectonicSetting up Kubernetes with tectonic
Setting up Kubernetes with tectonic
Vishal Biyani4.6K views
Kdd 2014 Tutorial - the recommender problem revisited by Xavier Amatriain
Kdd 2014 Tutorial -  the recommender problem revisitedKdd 2014 Tutorial -  the recommender problem revisited
Kdd 2014 Tutorial - the recommender problem revisited
Xavier Amatriain46.9K views
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys... by Xavier Amatriain
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Recsys 2016 tutorial: Lessons learned from building real-life recommender sys...
Xavier Amatriain16.5K views
Using AWS to Build a Graph-Based Product Recommendation System (BDT303) | AWS... by Amazon Web Services
Using AWS to Build a Graph-Based Product Recommendation System (BDT303) | AWS...Using AWS to Build a Graph-Based Product Recommendation System (BDT303) | AWS...
Using AWS to Build a Graph-Based Product Recommendation System (BDT303) | AWS...
Amazon Web Services12.9K views
Approximate nearest neighbor methods and vector models – NYC ML meetup by Erik Bernhardsson
Approximate nearest neighbor methods and vector models – NYC ML meetupApproximate nearest neighbor methods and vector models – NYC ML meetup
Approximate nearest neighbor methods and vector models – NYC ML meetup
Erik Bernhardsson22.2K views
Recommender Systems with Apache Spark's ALS Function by Will Johnson
Recommender Systems with Apache Spark's ALS FunctionRecommender Systems with Apache Spark's ALS Function
Recommender Systems with Apache Spark's ALS Function
Will Johnson24.1K views
Collaborative Filtering with Spark by Chris Johnson
Collaborative Filtering with SparkCollaborative Filtering with Spark
Collaborative Filtering with Spark
Chris Johnson46.7K views
The Universal Recommender by Pat Ferrel
The Universal RecommenderThe Universal Recommender
The Universal Recommender
Pat Ferrel33.9K views
Recsys 2014 Tutorial - The Recommender Problem Revisited by Xavier Amatriain
Recsys 2014 Tutorial - The Recommender Problem RevisitedRecsys 2014 Tutorial - The Recommender Problem Revisited
Recsys 2014 Tutorial - The Recommender Problem Revisited
Xavier Amatriain27K views
Music Recommendations at Scale with Spark by Chris Johnson
Music Recommendations at Scale with SparkMusic Recommendations at Scale with Spark
Music Recommendations at Scale with Spark
Chris Johnson58.6K views
How to Build a Recommendation Engine on Spark by Caserta
How to Build a Recommendation Engine on SparkHow to Build a Recommendation Engine on Spark
How to Build a Recommendation Engine on Spark
Caserta 37.5K views
Music recommendations @ MLConf 2014 by Erik Bernhardsson
Music recommendations @ MLConf 2014Music recommendations @ MLConf 2014
Music recommendations @ MLConf 2014
Erik Bernhardsson28.6K views

Similar to The Science and the Magic of User Feedback for Recommender Systems

Media, data, context... and the Holy Grail of User Taste Prediction by
Media, data, context... and the Holy Grail of User Taste PredictionMedia, data, context... and the Holy Grail of User Taste Prediction
Media, data, context... and the Holy Grail of User Taste PredictionXavier Amatriain
9.3K views76 slides
User-driven Approaches to Recsys by
User-driven Approaches to RecsysUser-driven Approaches to Recsys
User-driven Approaches to RecsysXavier Amatriain
1.5K views71 slides
Being Social by
Being SocialBeing Social
Being SocialXavier Amatriain
1.2K views46 slides
The Beauty of Computing with People by
The Beauty of Computing with PeopleThe Beauty of Computing with People
The Beauty of Computing with PeopleXavier Amatriain
976 views74 slides
Presentatie Big Data Forum 22 januari 2013 - Big Data en Big Society by
Presentatie Big Data Forum 22 januari 2013 - Big Data en Big SocietyPresentatie Big Data Forum 22 januari 2013 - Big Data en Big Society
Presentatie Big Data Forum 22 januari 2013 - Big Data en Big SocietySURFnet
849 views31 slides
Now & Next - Mediacom by
Now & Next - MediacomNow & Next - Mediacom
Now & Next - MediacomDigital Next Australia
412 views57 slides

Similar to The Science and the Magic of User Feedback for Recommender Systems(20)

Media, data, context... and the Holy Grail of User Taste Prediction by Xavier Amatriain
Media, data, context... and the Holy Grail of User Taste PredictionMedia, data, context... and the Holy Grail of User Taste Prediction
Media, data, context... and the Holy Grail of User Taste Prediction
Xavier Amatriain9.3K views
Presentatie Big Data Forum 22 januari 2013 - Big Data en Big Society by SURFnet
Presentatie Big Data Forum 22 januari 2013 - Big Data en Big SocietyPresentatie Big Data Forum 22 januari 2013 - Big Data en Big Society
Presentatie Big Data Forum 22 januari 2013 - Big Data en Big Society
SURFnet849 views
Girardin lift france10_notes by Fing
Girardin lift france10_notesGirardin lift france10_notes
Girardin lift france10_notes
Fing1.2K views
Beyond the Internet: Seamless Global Communication by Jerry Fishenden
Beyond the Internet: Seamless Global CommunicationBeyond the Internet: Seamless Global Communication
Beyond the Internet: Seamless Global Communication
Jerry Fishenden1K views
IT: Current Stage and Future Trend by Widy Widyawan
IT: Current Stage and Future Trend IT: Current Stage and Future Trend
IT: Current Stage and Future Trend
Widy Widyawan547 views
Internet of Things - The Tip of the Iceberg or The Tipping Point by Dr. Mazlan Abbas
Internet of Things - The Tip of the Iceberg or The Tipping PointInternet of Things - The Tip of the Iceberg or The Tipping Point
Internet of Things - The Tip of the Iceberg or The Tipping Point
Dr. Mazlan Abbas5.4K views
Introducing the Internet of Things: lecture @IULM University by Leandro Agro'
Introducing the Internet of Things: lecture @IULM UniversityIntroducing the Internet of Things: lecture @IULM University
Introducing the Internet of Things: lecture @IULM University
Leandro Agro'3.1K views
MICROSTRATEGY - Sessione introduttiva sulla piattaforma di Business Intelligence by Twinergy
MICROSTRATEGY - Sessione introduttiva sulla piattaforma di Business IntelligenceMICROSTRATEGY - Sessione introduttiva sulla piattaforma di Business Intelligence
MICROSTRATEGY - Sessione introduttiva sulla piattaforma di Business Intelligence
Twinergy578 views
Towards user co-creation of value on the Internet-of-Things (IoT) by trappenl
Towards user co-creation of value on the Internet-of-Things (IoT)Towards user co-creation of value on the Internet-of-Things (IoT)
Towards user co-creation of value on the Internet-of-Things (IoT)
trappenl1.3K views
PLA 2012: Librarian's Field Guide to Near Field Communication by Kristen Yarmey
PLA 2012: Librarian's Field Guide to Near Field CommunicationPLA 2012: Librarian's Field Guide to Near Field Communication
PLA 2012: Librarian's Field Guide to Near Field Communication
Kristen Yarmey2.8K views
Ross Rubin's Presentation by Mediabistro
Ross Rubin's PresentationRoss Rubin's Presentation
Ross Rubin's Presentation
Mediabistro360 views
ChaCha ThinkLA Mobile Breakfast Keynote - Scott A. Jones by ChaCha
ChaCha ThinkLA Mobile Breakfast Keynote - Scott A. JonesChaCha ThinkLA Mobile Breakfast Keynote - Scott A. Jones
ChaCha ThinkLA Mobile Breakfast Keynote - Scott A. Jones
ChaCha1K views

More from Xavier Amatriain

Data/AI driven product development: from video streaming to telehealth by
Data/AI driven product development: from video streaming to telehealthData/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealthXavier Amatriain
434 views50 slides
AI-driven product innovation: from Recommender Systems to COVID-19 by
AI-driven product innovation: from Recommender Systems to COVID-19AI-driven product innovation: from Recommender Systems to COVID-19
AI-driven product innovation: from Recommender Systems to COVID-19Xavier Amatriain
864 views77 slides
AI for COVID-19 - Q42020 update by
AI for COVID-19 - Q42020 updateAI for COVID-19 - Q42020 update
AI for COVID-19 - Q42020 updateXavier Amatriain
1.3K views29 slides
AI for COVID-19: An online virtual care approach by
AI for COVID-19: An online virtual care approachAI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approachXavier Amatriain
1.8K views15 slides
Lessons learned from building practical deep learning systems by
Lessons learned from building practical deep learning systemsLessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systemsXavier Amatriain
71.8K views61 slides
AI for healthcare: Scaling Access and Quality of Care for Everyone by
AI for healthcare: Scaling Access and Quality of Care for EveryoneAI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for EveryoneXavier Amatriain
2.5K views29 slides

More from Xavier Amatriain(20)

Data/AI driven product development: from video streaming to telehealth by Xavier Amatriain
Data/AI driven product development: from video streaming to telehealthData/AI driven product development: from video streaming to telehealth
Data/AI driven product development: from video streaming to telehealth
Xavier Amatriain434 views
AI-driven product innovation: from Recommender Systems to COVID-19 by Xavier Amatriain
AI-driven product innovation: from Recommender Systems to COVID-19AI-driven product innovation: from Recommender Systems to COVID-19
AI-driven product innovation: from Recommender Systems to COVID-19
Xavier Amatriain864 views
AI for COVID-19: An online virtual care approach by Xavier Amatriain
AI for COVID-19: An online virtual care approachAI for COVID-19: An online virtual care approach
AI for COVID-19: An online virtual care approach
Xavier Amatriain1.8K views
Lessons learned from building practical deep learning systems by Xavier Amatriain
Lessons learned from building practical deep learning systemsLessons learned from building practical deep learning systems
Lessons learned from building practical deep learning systems
Xavier Amatriain71.8K views
AI for healthcare: Scaling Access and Quality of Care for Everyone by Xavier Amatriain
AI for healthcare: Scaling Access and Quality of Care for EveryoneAI for healthcare: Scaling Access and Quality of Care for Everyone
AI for healthcare: Scaling Access and Quality of Care for Everyone
Xavier Amatriain2.5K views
Towards online universal quality healthcare through AI by Xavier Amatriain
Towards online universal quality healthcare through AITowards online universal quality healthcare through AI
Towards online universal quality healthcare through AI
Xavier Amatriain1.7K views
From one to zero: Going smaller as a growth strategy by Xavier Amatriain
From one to zero: Going smaller as a growth strategyFrom one to zero: Going smaller as a growth strategy
From one to zero: Going smaller as a growth strategy
Xavier Amatriain1.8K views
Medical advice as a Recommender System by Xavier Amatriain
Medical advice as a Recommender SystemMedical advice as a Recommender System
Medical advice as a Recommender System
Xavier Amatriain5.6K views
Staying Shallow & Lean in a Deep Learning World by Xavier Amatriain
Staying Shallow & Lean in a Deep Learning WorldStaying Shallow & Lean in a Deep Learning World
Staying Shallow & Lean in a Deep Learning World
Xavier Amatriain7.6K views
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems by Xavier Amatriain
Strata 2016 -  Lessons Learned from building real-life Machine Learning SystemsStrata 2016 -  Lessons Learned from building real-life Machine Learning Systems
Strata 2016 - Lessons Learned from building real-life Machine Learning Systems
Xavier Amatriain5.9K views
Barcelona ML Meetup - Lessons Learned by Xavier Amatriain
Barcelona ML Meetup - Lessons LearnedBarcelona ML Meetup - Lessons Learned
Barcelona ML Meetup - Lessons Learned
Xavier Amatriain3.2K views
10 more lessons learned from building Machine Learning systems - MLConf by Xavier Amatriain
10 more lessons learned from building Machine Learning systems - MLConf10 more lessons learned from building Machine Learning systems - MLConf
10 more lessons learned from building Machine Learning systems - MLConf
Xavier Amatriain375.2K views
10 more lessons learned from building Machine Learning systems by Xavier Amatriain
10 more lessons learned from building Machine Learning systems10 more lessons learned from building Machine Learning systems
10 more lessons learned from building Machine Learning systems
Xavier Amatriain180.1K views
Lean DevOps - Lessons Learned from Innovation-driven Companies by Xavier Amatriain
Lean DevOps - Lessons Learned from Innovation-driven CompaniesLean DevOps - Lessons Learned from Innovation-driven Companies
Lean DevOps - Lessons Learned from Innovation-driven Companies
Xavier Amatriain15.4K views
10 Lessons Learned from Building Machine Learning Systems by Xavier Amatriain
10 Lessons Learned from Building Machine Learning Systems10 Lessons Learned from Building Machine Learning Systems
10 Lessons Learned from Building Machine Learning Systems
Xavier Amatriain378K views

Recently uploaded

Tunable Laser (1).pptx by
Tunable Laser (1).pptxTunable Laser (1).pptx
Tunable Laser (1).pptxHajira Mahmood
21 views37 slides
The Importance of Cybersecurity for Digital Transformation by
The Importance of Cybersecurity for Digital TransformationThe Importance of Cybersecurity for Digital Transformation
The Importance of Cybersecurity for Digital TransformationNUS-ISS
25 views26 slides
Photowave Presentation Slides - 11.8.23.pptx by
Photowave Presentation Slides - 11.8.23.pptxPhotowave Presentation Slides - 11.8.23.pptx
Photowave Presentation Slides - 11.8.23.pptxCXL Forum
126 views16 slides
"How we switched to Kanban and how it integrates with product planning", Vady... by
"How we switched to Kanban and how it integrates with product planning", Vady..."How we switched to Kanban and how it integrates with product planning", Vady...
"How we switched to Kanban and how it integrates with product planning", Vady...Fwdays
61 views24 slides
Microchip: CXL Use Cases and Enabling Ecosystem by
Microchip: CXL Use Cases and Enabling EcosystemMicrochip: CXL Use Cases and Enabling Ecosystem
Microchip: CXL Use Cases and Enabling EcosystemCXL Forum
129 views12 slides
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum... by
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...NUS-ISS
28 views35 slides

Recently uploaded(20)

The Importance of Cybersecurity for Digital Transformation by NUS-ISS
The Importance of Cybersecurity for Digital TransformationThe Importance of Cybersecurity for Digital Transformation
The Importance of Cybersecurity for Digital Transformation
NUS-ISS25 views
Photowave Presentation Slides - 11.8.23.pptx by CXL Forum
Photowave Presentation Slides - 11.8.23.pptxPhotowave Presentation Slides - 11.8.23.pptx
Photowave Presentation Slides - 11.8.23.pptx
CXL Forum126 views
"How we switched to Kanban and how it integrates with product planning", Vady... by Fwdays
"How we switched to Kanban and how it integrates with product planning", Vady..."How we switched to Kanban and how it integrates with product planning", Vady...
"How we switched to Kanban and how it integrates with product planning", Vady...
Fwdays61 views
Microchip: CXL Use Cases and Enabling Ecosystem by CXL Forum
Microchip: CXL Use Cases and Enabling EcosystemMicrochip: CXL Use Cases and Enabling Ecosystem
Microchip: CXL Use Cases and Enabling Ecosystem
CXL Forum129 views
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum... by NUS-ISS
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...
NUS-ISS28 views
"AI Startup Growth from Idea to 1M ARR", Oleksandr Uspenskyi by Fwdays
"AI Startup Growth from Idea to 1M ARR", Oleksandr Uspenskyi"AI Startup Growth from Idea to 1M ARR", Oleksandr Uspenskyi
"AI Startup Growth from Idea to 1M ARR", Oleksandr Uspenskyi
Fwdays26 views
"Role of a CTO in software outsourcing company", Yuriy Nakonechnyy by Fwdays
"Role of a CTO in software outsourcing company", Yuriy Nakonechnyy"Role of a CTO in software outsourcing company", Yuriy Nakonechnyy
"Role of a CTO in software outsourcing company", Yuriy Nakonechnyy
Fwdays40 views
"Thriving Culture in a Product Company — Practical Story", Volodymyr Tsukur by Fwdays
"Thriving Culture in a Product Company — Practical Story", Volodymyr Tsukur"Thriving Culture in a Product Company — Practical Story", Volodymyr Tsukur
"Thriving Culture in a Product Company — Practical Story", Volodymyr Tsukur
Fwdays40 views
How to reduce cold starts for Java Serverless applications in AWS at JCON Wor... by Vadym Kazulkin
How to reduce cold starts for Java Serverless applications in AWS at JCON Wor...How to reduce cold starts for Java Serverless applications in AWS at JCON Wor...
How to reduce cold starts for Java Serverless applications in AWS at JCON Wor...
Vadym Kazulkin70 views
GigaIO: The March of Composability Onward to Memory with CXL by CXL Forum
GigaIO: The March of Composability Onward to Memory with CXLGigaIO: The March of Composability Onward to Memory with CXL
GigaIO: The March of Composability Onward to Memory with CXL
CXL Forum126 views
"Fast Start to Building on AWS", Igor Ivaniuk by Fwdays
"Fast Start to Building on AWS", Igor Ivaniuk"Fast Start to Building on AWS", Igor Ivaniuk
"Fast Start to Building on AWS", Igor Ivaniuk
Fwdays36 views
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen... by NUS-ISS
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...
NUS-ISS23 views
Liqid: Composable CXL Preview by CXL Forum
Liqid: Composable CXL PreviewLiqid: Composable CXL Preview
Liqid: Composable CXL Preview
CXL Forum121 views
"Quality Assurance: Achieving Excellence in startup without a Dedicated QA", ... by Fwdays
"Quality Assurance: Achieving Excellence in startup without a Dedicated QA", ..."Quality Assurance: Achieving Excellence in startup without a Dedicated QA", ...
"Quality Assurance: Achieving Excellence in startup without a Dedicated QA", ...
Fwdays33 views
Understanding GenAI/LLM and What is Google Offering - Felix Goh by NUS-ISS
Understanding GenAI/LLM and What is Google Offering - Felix GohUnderstanding GenAI/LLM and What is Google Offering - Felix Goh
Understanding GenAI/LLM and What is Google Offering - Felix Goh
NUS-ISS39 views
.conf Go 2023 - Data analysis as a routine by Splunk
.conf Go 2023 - Data analysis as a routine.conf Go 2023 - Data analysis as a routine
.conf Go 2023 - Data analysis as a routine
Splunk90 views
AI: mind, matter, meaning, metaphors, being, becoming, life values by Twain Liu 刘秋艳
AI: mind, matter, meaning, metaphors, being, becoming, life valuesAI: mind, matter, meaning, metaphors, being, becoming, life values
AI: mind, matter, meaning, metaphors, being, becoming, life values
Web Dev - 1 PPT.pdf by gdsczhcet
Web Dev - 1 PPT.pdfWeb Dev - 1 PPT.pdf
Web Dev - 1 PPT.pdf
gdsczhcet52 views
MemVerge: Gismo (Global IO-free Shared Memory Objects) by CXL Forum
MemVerge: Gismo (Global IO-free Shared Memory Objects)MemVerge: Gismo (Global IO-free Shared Memory Objects)
MemVerge: Gismo (Global IO-free Shared Memory Objects)
CXL Forum112 views

The Science and the Magic of User Feedback for Recommender Systems

  • 1. The Science and the Magic of User Feedback for Recommender Systems Xavier Amatriain Bay Area, March '11
  • 2. But first... About Telefonica and Telefonica R&D
  • 3. Telefonica is a fast-growing Telecom 1989 2000 2008 Clients About 12 About 68 About 260 million million million subscribers customers customers Services Basic Wireline and mobile Integrated ICT telephone and voice, data and solutions for all data services Internet services customers Geographies Operations in Operations in Spain 25 countries 16 countries Staff About 71,000 About 149,000 About 257,000 professionals professionals professionals Finances Rev: 4,273 M€ Rev: 28,485 M€ Rev: 57,946 M€ EPS(1): 0.45 € EPS(1): 0.67 € EPS: 1.63 € (1) EPS: Earnings per share
  • 4. Currently among the largest in the world Telco sector worldwide ranking by market cap (US$ bn) Source: Bloomberg, 06/12/09 Just announced 2010 results: record net earnings,  first Spanish company ever to make > 10B €
  • 5. Leader in South America Data as of March ‘09 1 2 Argentina: 20.9 million Wireline market rank  2 1 Brazil: 61.4 million Mobile market rank 2 Central America: 6.1 million 1 2 Colombia: 12.6 million 1 1 Chile: 10.1 million 2 Ecuador: 3.3 million 2 Mexico: 15.7 million 1 1 Peru: 15.2 million 1 Uruguay: 1.5 million 2 Venezuela: 12.0 million Total Accesses (as of March ‘09) 159.5 million Notes: - Central America includes Guatemala, Panama, El Salvador and Nicaragua - Total accesses figure includes Narrowband Internet accesses of Terra Brasil and Terra Colombia, and Broadband Internet accesses of Terra Brasil, Telefónica de Argentina, Terra Guatemala and Terra México.
  • 6. And a significant footprint in Europe Wireline market rank Mobile market rank Data as of March ‘09 1 1 Spain: 47.2 million 1 UK: 20.8 million 4 Germany: 16.0 million 2 Ireland: 1.7 million Czech Republic: 7.7 million 1 2 Slovakia: 0.4 million 3 Total Accesses (as of March ’09) 93.8 million
  • 7. Scientific Research Mobile and Ubicomp Multimedia Core User Modelling & Data Mining HCIR DATA MINING Wireless Systems Content Distribution & P2P Social Networks
  • 8. Projects Recommendation  Algorithms Tourist routes Social  contacts Music User Analysis  Movies & Modeling Contextual The Wisdom of the  Noise in users’  Few Mobile ratings  Tourist  behavior Microprofiles Implicit user  feedback Multiverse  Tensor  IPTV viewing habits Factorization
  • 9. Projects Recommendation  Algorithms Tourist routes Social  contacts Music User Analysis  Movies & Modeling Contextual The Wisdom of the  Noise in users’  Few Mobile ratings  Tourist  behavior Microprofiles Implicit user  feedback Multiverse  Tensor  IPTV viewing habits Factorization
  • 10. And about the world we live in...
  • 12. More is Less W or se D ec is ns io io ns is ec D s es L
  • 13. Analysis Paralysis is making headlines
  • 14. Search engines don’t always hold the answer
  • 18. What about information to help take decisions?
  • 19. The Age of Search has come to an end ... long live the Age of Recommendation! ● ● Chris Anderson in “The Long Tail” ● “We are leaving the age of information and entering the age of recommendation” ● CNN Money, “The race to create a 'smart' Google”: ● “The Web, they say, is leaving the era of search and entering one of discovery. What's the difference? Search is what you do when you're looking for something. Discovery is when something wonderful that you didn't know existed, or didn't know how to ask for, finds you.”
  • 20. Recommender Systems Recommendations Read this Attend this conference
  • 21. Data mining + all those other things ● User Interface ● User modeling ● System requirements (efficiency, scalability, privacy....) ● Business Logic ● Serendipity ● ....
  • 22. Approaches to Recommendation Collaborative Filtering ● ● Recommend items based only on the users past behavior Content-based ● ● Recommend based on features inherent to the items Social recommendations (trust-based) ●
  • 23. What works ● It depends on the domain and particular problem ● As a general rule, it is usually a good idea to combine: Hybrid Recommender Systems ● However, in the general case it has been demonstrated that (currently) the best isolated approach is CF. ● Item-based in general more efficient and better but mixing CF approaches can improve result ● Other approaches can improve results in specific cases (cold-start problem...)
  • 24. The CF Ingredients ● List of m Users and a list of n Items ● Each user has a list of items with associated opinion ● Explicit opinion - a rating score (numerical scale) ● Implicit feedback – purchase records or listening history ● Active user for whom the prediction task is performed ● A metric for measuring similarity between users ● A method for selecting a subset of neighbors ● A method for predicting a rating for items not rated by the active user. 24
  • 25. The Netflix Prize ● 500K users x 17K movie titles = 100M ratings = $1M (if you “only” improve existing system by 10%! From 0.95 to 0.85 RMSE) ● 49K contestants on 40K teams from 184 countries. ● 41K valid submissions from 5K teams; 64 submissions per day ● Wining approach uses hundreds of predictors from several teams
  • 27. User Feedback is Noisy DID YOU HEAR WHAT I LIKE??!! ...and limits Our Prediction Accuracy
  • 28. The Magic Barrier ● Magic Barrier = Limit on prediction accuracy due to noise in original data ● Natural Noise = involuntary noise introduced by users when giving feedback ● Due to (a) mistakes, and (b) lack of resolution in personal rating scale ● Magic Barrier >= Natural Noise Threshold ● Our prediction error cannot be smaller than the error in the original data
  • 29. Our related research questions X. Amatriain, J.M. Pujol, N. Oliver (2009) "I like It... I like It Not: Measuring Users Ratings Noise in Recommender Systems", in UMAP 09 ● Q1. Are users inconsistent when providing explicit feedback to Recommender Systems via the common Rating procedure? ● Q2. How large is the prediction error due to these inconsistencies? ● Q3. What factors affect user inconsistencies?
  • 30. Experimental Setup ● 100 Movies selected from Netflix dataset doing a stratified random sampling on popularity ● Ratings on a 1 to 5 star scale ● Special “not seen” symbol. ● Trial 1 and 3 = random order; trial 2 = ordered by popularity
  • 31. User Feedback is Noisy ● Users are inconsistent ● Inconsistencies are not random and depend on many factors
  • 32. User Feedback is Noisy ● Users are inconsistent ● Inconsistencies are not random and depend on many factors ● More inconsistencies for mild opinions
  • 33. User Feedback is Noisy ● Users are inconsistent ● Inconsistencies are not random and depend on many factors ● More inconsistencies for mild opinions ● More inconsistencies for negative opinions
  • 34. User’s ratings are far from ground truth #Ti #Tj # RMSE     T1, T2 2185 1961 1838 2308 0.573 0.707 T1, T3 2185 1909 1774 2320 0.637 0.765 T2, T3 1969 1909 1730 2140 0.557 0.694 Pairwise comparison between trials, RMSE is already > 0.55 or > 0.69 (Netflix Prize was to get below 0.85 !!!)
  • 35. Algorithm Robustness to NN Trial 2 is  Alg./Trial consistently the  <T1 T2 T3 Tworst /Tbest least noisy User  1.2011 1.1469 1.1945 4.7% Average Item  1.0555 1.0361 1.0776 4% Average User­based  0.9990 0.9640 1.0171 5.5% kNN Item­based  1.0429 1.0031 1.0417 4% kNN SVD 1.0244 0.9861 1.0285 4.3%  RMSE for different Recommendation algorithms  ● when predicting each of the trials
  • 36. Rate it Again X. Amatriain et al. (2009)"Rate it Again: Increasing Recommendation Accuracy by User re-Rating", 2009 ACM RecSys ● Given that users are noisy… can we benefit from asking to rate the same movie more than once? ● We propose an algorithm to allow for multiple ratings of the same <user,item> tuple. ● The algorithm is subjected to two fairness conditions: – Algorithm should remove as few ratings as possible (i.e. only when there is some certainty that the rating is only adding noise) – Algorithm should not make up new ratings but decide on which of the existing ones are valid.
  • 37. Re-rating Algorithm • One source re­rating case: Examples: {3, 1} →Ø {4} →4 {3, 4} →3 (2 source) {3, 4, 5} →3 • Given the following milding function:   
  • 38. Results ● One-source re-rating (Denoised⊚Denoising) T1⊚T2 ΔT1 T1⊚T3 ΔT1 T2⊚T3 ΔT2 User­based kNN 0.8861 11.3% 0.8960 10.3% 0.8984 6.8% SVD 0.9121 11.0% 0.9274 9.5% 0.9159 7.1% ● Two-source re-rating (Denoising T1with the other 2) Datasets T1⊚(T2, T3) ΔT1 User­based kNN 0.8647 13.4% SVD 0.8800 14.1%
  • 39. Rate it again ● By asking users to rate items again we can remove noise in the dataset ● Improvements of up to 14% in accuracy! ● Because we don't want all users to re-rate all items we design ways to do partial denoising ● Data-dependent: only denoise extreme ratings ● User-dependent: detect “noisy” users
  • 40. Denoising only noisy users ●  Improvement in RMSE when doing one­source as a function of  the percentage of denoised ratings and users: selecting only noisy  users and extreme ratings
  • 41. The value or a re-rating Adding new ratings increases performance of the CF algorithm
  • 42. The value or a re-rating But you are better off doing re-rating than new ratings !!
  • 43. The value or a re-rating And much better if you know which ratings to re-rate!!
  • 44. Let's recap ● Users are inconsistent ● Inconsistencies can depend on many things including how the items are presented ● Inconsistencies produce natural noise ● Natural noise reduces our prediction accuracy independently of the algorithm ● By asking (some) users to re-rate (some) items again we can remove noise and improve accuracy ● Having users repeat existing ratings may have more value than adding new ones
  • 45. Crowds are not always wise ● Diversity of opinion Conditions that are  ● Independence needed to guarantee the  ● Decentralization Wisdom in a Crowd ● Aggregation
  • 46. Crowds are not always wise vs. Who  won?
  • 47. The Wisdom of the Few X. Amatriain et al. "The wisdom of the few: a collaborative filtering approach based on expert opinions from the web", SIGIR '09
  • 49. Expert-based CF ● expert = individual that we can trust to have produced thoughtful, consistent and reliable evaluations (ratings) of items in a given domain ● Expert-based Collaborative Filtering ● Find neighbors from a reduced set of experts instead of regular users. 1. Identify domain experts with reliable ratings 2. For each user, compute “expert neighbors” 3. Compute recommendations similar to standard kNN CF
  • 50. User Study ● 57 participants, only 14.5 ratings/participant ● 50% of the users consider Expert-based CF to be good or very good ● Expert-based CF: only algorithm with an average rating over 3 (on a 0-4 scale)
  • 51. Advantages of the Approach ● Noise ● Cold Start problem ● Experts introduce less ● Experts rate items as natural noise soon as they are ● Malicious Ratings available ● Dataset can be monitored ● Scalability to avoid shilling ● Dataset is several order of ● Data Sparsity magnitudes smaller ● Reduced set of domain ● Privacy experts can be motivated ● Recommendations can be to rate items computed locally
  • 52. So... ● Can we generate meaningful and personalized recommendations ensuring 100% privacy? ● YES! ● Can we have a recommendation algorithm that is so efficient to run on a phone? ● YES! ● Can we have a recommender system that works even if there is only one user? ● YES!
  • 54. Some implementations ● A distributed Music Recommendation engine
  • 55. Some implementations (II) ● A geo-localized Mobile Movie Recommender iPhone App
  • 56. Geo-localized Expert Movie Recommendations 0 Powered by...
  • 57. Expert CF... ● Recreates the old paradigm of manually finding your favorite experts in magazines but in a fully automatic non-supervised manner.
  • 58. What if we don't have ratings? The fascinating world of implicit user feedback Examples of implicit feedback: ● Movies you watched ● Links you visited ● Songs you listened to ● Items you bought ● ....
  • 59. Main features of implicit feedback ● Our starting hypothesis are different from those in previous works: 1.Implicit feedback can contain negative feedback – given the right granularity and diversity, low feedback = negative feedback 2.Numerical value of implicit feedback can be mapped to preference given the appropriate mapping 3.Once we have a trustworthy mapping, we can evaluate implicit feedback predictions same way as with explicit feedback.
  • 60. Our questions ● Q1. Is it possible to predict ratings a user would give to items given their implicit feedback? ● Q2. Are there other variables that affect this mapping?
  • 61. Experimental setup ● Online user study on the music domain ● Users required to have a music profile in lastfm ● Goal: Compare explicit ratings with their listening history taking to account a number of controlled variables
  • 62. Results. Do explicit ratings relate to implicit feedback? Almost perfect linear  relation between ratings  and quantized implicit  feedback
  • 63. Results. Do explicit ratings relate to implicit feedback? Extreme ratings have clear  ascending/descending  trend, but mild ratings  respond more to changes  in one direction
  • 64. Results. Do other variables affect? Albums listened to more  recently tend to receive  more positive ratings
  • 65. Results. Do other variables affect? Contrary to our expectations,  global album popularity does  not affect ratings
  • 66. Results. What about user variables? ● We obtained many demographic (age, sex, location...) and usage variables (hours of music per week, concerts, music magazines, ways of buying music...) in the study. ● We performed an ANOVA analysis on the data to understand which variables explained some of its variance. ● Only one of the usage variables, contributed (Sig. Value < 0.05) → “Listening Style” encoded whether the user listened preferably to tracks, full albums, or both.
  • 67. Results. Regression Analysis – Model 1: riu = β0 + β1 · ifiu – Model 2: riu = β0 + β1 · ifiu + β2 · reiu – Model 3: riu = β0 + β1 · ifiu + β2 · reiu + β3 · gpi – Model 4: riu = β0 + β1 · ifiu + β2 · reiu + β3 · ifiu · reiu Model R2 F-value p-value β0 β1 β2 β3 1 0.125 F (1, 10120) = 1146 < 2.2 · 10−16 2.726 0.499 2 0.1358 F (2, 10019) = 794.8 < 2.2 · 10−16 2.491 0.484 0.133 3 0.1362 F (3, 10018) = 531.8 < 2.2 · 10−16 2.435 0.486 0.134 0.0285 4 0.1368 F (3, 10018) = 534.7 < 2.2 · 10−16 2.677 0.379 0.038 0.053 All models meaningfully explain the data. Introducing “recentness”  improves 10% but “global popularity” or interaction between variables do  not make much difference
  • 68. Results. Predictive power Model RMSE – Excluding non-rated items User Average 1.131 1 1.026 2 1.017 3 1.016 4 1.016 Error in predicting 20% of the ratings, having trained our  regression model on the other 80%
  • 69. Conclusions ● Recommender systems and similar applications usually focus on having more data ● But... many times is not about having more but rather better data ● User feedback can not always be treated as ground truth and needs to be processed ● Crowds are not always wise and sometimes we are better off using experts ● Implicit feedback represents a good alternative to understand users but mapping is not trivial
  • 70. Colleagues ● Josep M. Pujol and Nuria Oliver (Telefonica) worked on Natural Noise and Wisdom of the Few projects ● Nava Tintarev (Telefonica) worked on Natural Noise External Collaborators ● Neal Lathia (UCL, London), Haewook Ahn (KAIST, Korea), Jaewook Ahn (Pittsburgh Univ.), and Josep Bachs (UPF, Barcelona) on Wisdom of the Few ● Denis Parra (Pittsburgh Univ.) worked on implicit-explicit
  • 71. Thanks! Questions? Xavier Amatriain xar@tid.es http://xavier.amatriain.net http://technocalifornia.blogspot.com @xamat