TELEFÓNICA Research April 2008 Search and Recommendation: two sides of the same coin? a.k.a. Is the future of Search in Recommendation? © 2007 Telefónica Investigación y Desarrollo, S.A. Unipersonal Xavier Amatriain Researcher Where is the content?

01 Introduction 02 Recommender Systems 03 Search Engines 04 Are Search Engines going towards RS? 05 Conclusions Index

01 Introduction

02 Recommender Systems

03 Search Engines

04 Are Search Engines going towards RS?

05 Conclusions

Information overload “People read around 10 MB worth of material a day, hear 400 MB a day, and see one MB of information every second” The Economist, November 2006

The Age of Search has come to an end ... long live the Age of Recommendation! In an article published in CNN Money, entitled “The race to create a 'smart' Google”, Fortune magazine writer Jeffrey M. O'Brien, writes: “The Web, they say, is leaving the era of search and entering one of discovery. What's the difference? Search is what you do when you're looking for something. Discovery is when something wonderful that you didn't know existed, or didn't know how to ask for, finds you.” (Extracted from ACM Recsys 08 website)

... long live the Age of Recommendation!

In an article published in CNN Money, entitled “The race to create a 'smart' Google”, Fortune magazine writer Jeffrey M. O'Brien, writes:

“The Web, they say, is leaving the era of search and entering one of discovery. What's the difference? Search is what you do when you're looking for something. Discovery is when something wonderful that you didn't know existed, or didn't know how to ask for, finds you.”

(Extracted from ACM Recsys 08 website)

Recommender Systems in Telefonica Multimedia Entertainment E-commerce Social Networking News/Blogs/Portals Comunidades PLATFORM PRODUCTS AND SERVICES COMMERCIALIZATION Content Packaging and Design Devices Access Commercialization Customers Recommendation Systems

Access through PC for exploring the whole application features Browse, Search, Rate, Comment… Get recommended series Get recommended games Meet and interact with other people Use a little application for mobile devices Get recommendations What series I watch now? Browse across TuSerie with another devices and discover new user experiences Cross-platform recommendation systems

Social Networks and Recommendations Browse through user profile to check the compatibility with a specific user See the last events or the last interactions that your friends have made with the system Send messages and make this user a friend or discard him Check your compatibility with another user. View his ratings and compare them with yours. Search users

Domain-specific recommendation User profiling Implicit through log analysis Explicit asking for user feedback Integrate search and recommendation in user-friendly interfaces Usuario Genérico User Profile Global Search Content Based Collaborative

User profiling

Implicit through log analysis

Explicit asking for user feedback

Integrate search and recommendation in user-friendly interfaces

Investment in Recommendation Companies

02 Recommender Systems

The “Recommender problem” Estimate a utility function that is able to automatically predict how much a user will like an item that is unknown for her. Based on: Past behavior Relations to other users Item similarity Context ... Offline Online

Estimate a utility function that is able to automatically predict how much a user will like an item that is unknown for her. Based on:

Past behavior

Relations to other users

Item similarity

Context

...

Approaches to Recommendation Collaborative Filtering Recommend items based only on the users past behavior Similarity between users or items computed only from this User-based Find similar users to me and recommend what those users liked Item-based Find similar items to those that I have previously liked Content-based Recommend based on features inherent to the items

Collaborative Filtering

Recommend items based only on the users past behavior

Similarity between users or items computed only from this

User-based

Find similar users to me and recommend what those users liked

Item-based

Find similar items to those that I have previously liked

Content-based

Recommend based on features inherent to the items

Recommendation as a Datamining problem The core of the Recommendation Engine has been assimilated to a general datamining problem: However RS have attracted input from a large community IR, e-Commerce, HCI, Psychology...

The core of the Recommendation Engine has been assimilated to a general datamining problem:

However RS have attracted input from a large community

IR, e-Commerce, HCI, Psychology...

What works It depends on the domain: Domain-specific modeling However, in the general case it has been demonstrated that (currently) the best isolated approach is CF. Item-based in general more efficient and better but mixing CF approaches can improve result Other approaches can be hybridized to improve results in specific cases (cold-start problem...) What matters: Data preprocessing: outlier removal, denoising, removal of global effects (e.g. individual user's average) “Smart” dimensionality reduction using MF such as SVD Combining classifiers

It depends on the domain: Domain-specific modeling

However, in the general case it has been demonstrated that (currently) the best isolated approach is CF.

Item-based in general more efficient and better but mixing CF approaches can improve result

Other approaches can be hybridized to improve results in specific cases (cold-start problem...)

What matters:

Data preprocessing: outlier removal, denoising, removal of global effects (e.g. individual user's average)

“Smart” dimensionality reduction using MF such as SVD

Combining classifiers

Data mining + all those other things User Interface System requirements (efficiency, scalability, privacy....) and ....

User Interface

System requirements (efficiency, scalability, privacy....)

and ....

Serindipity Unsought finding Don't recommend items the user already knows or would have found anyway . Expand the user's taste into neighboring areas by improving the obvious Collaborative filtering can offer controllable serendipity (e.g. controlling how many neighbors to use in the recommendation)

Unsought finding

Don't recommend items the user already knows or would have found anyway .

Expand the user's taste into neighboring areas by improving the obvious

Collaborative filtering can offer controllable serendipity (e.g. controlling how many neighbors to use in the recommendation)

02 Search

Search Engines General process diagram of a search engine Offline Online

General process diagram of a search engine

Search vs. recommendation Is search a content-based “recommendation”? In the indexing and retrieval processes we are trying to “cluster” similar documents based exlusively on content (no user information) or a poor-man's approach to CF? Most ranking algorithms can be seen as a simplified collaborative filtering where we are recommended the opinion of the average user's (what most people link) or the authorities (e.g. Page Rank). To some extent we can say that web “structure” reflects past users' behavior

Is search a content-based “recommendation”?

In the indexing and retrieval processes we are trying to “cluster” similar documents based exlusively on content (no user information)

or a poor-man's approach to CF?

Most ranking algorithms can be seen as a simplified collaborative filtering where we are recommended the opinion of the average user's (what most people link) or the authorities (e.g. Page Rank).

To some extent we can say that web “structure” reflects past users' behavior

03 But, is Search going towards Recommendation?

Personalized Search Last year's presentation in this same Workshop is a good starting point (Paul-Alexandru Chirita, Current Approaches to Personalize Web Search) Overall trend -> Use personalized user profile in order to improve returned page ranking

Last year's presentation in this same Workshop is a good starting point (Paul-Alexandru Chirita, Current Approaches to Personalize Web Search)

Overall trend -> Use personalized user profile in order to improve returned page ranking

Recent advances in Personalized Search Interesting approaches: Automatic Identification of User Interest for Personalized Search (Qiu et al WWW06) Improve topic-sensitive page rank by inferring topic preference vector for the user. Very similar to content-based recommendation CubeSVD: A Novel Approach to Personalized Web search (Sun et al WWW05) LSI using HOSVD to find a score for webpages based on q,u pairs. Very similar to CF

Interesting approaches:

Automatic Identification of User Interest for Personalized Search (Qiu et al WWW06)

Improve topic-sensitive page rank by inferring topic preference vector for the user.

Very similar to content-based recommendation

CubeSVD: A Novel Approach to Personalized Web search (Sun et al WWW05)

LSI using HOSVD to find a score for webpages based on q,u pairs.

Very similar to CF

A hybrid Search-Recommender? == ? Recommendation Search

04 Conclusions The ever-growing amount of content makes searching difficult (time-consuming and unsatisfactory) Too much to search for, too many results Frustration from it not being adaptive Search is starting to take the user into account Is search something users want to do or just something they can do with the tools we offer? Are search and recommendation two sides of the same coin? Is search about retrieval and recommendation about ranking? Should they complement each other or become the same thing?

The ever-growing amount of content makes searching difficult (time-consuming and unsatisfactory)

Too much to search for, too many results

Frustration from it not being adaptive

Search is starting to take the user into account

Is search something users want to do or just something they can do with the tools we offer?

Are search and recommendation two sides of the same coin?

Is search about retrieval and recommendation about ranking?

Should they complement each other or become the same thing?

[email_address] http://www.tid.es We're hiring!