Recommendation as Search                                    Reflections on Symmetry©MapR Technologies - Confidential      ...
Company Background      MapR provides the industry’s best Hadoop Distribution       –    Combines the best of the Hadoop ...
What is Hadoop?     A new style of computation     A new style of combining computation and storage     Allows very lar...
Why Big Data?     Because we can     Because we can learn new things     Because new economics of computation favors la...
Recommendations     Often known as collaborative filtering                       “People who bought x also bought y”    ...
Examples     Customers buying books (Linden et al)     Web visitors rating music (Shardanand and Maes) or movies      (R...
Fundamental Algorithm     History matrix A has the shape of actors x items     Cooccurrence matrix K has the shape of it...
… Warning …©MapR Technologies - Confidential        8
… Warning …                                    Mathematics ahead©MapR Technologies - Confidential           9
Fundamental Algorithmic Structure     Cooccurrence                                    K=A A T                            ...
But Wait ...                                    Does it have to be that way?©MapR Technologies - Confidential            11
But why not ...                                      T                                    (A A)h©MapR Technologies - Confi...
But why not ...                                            T                                     (A A)h                   ...
But why not ...                                            T                                     (B A)h                   ...
But why not ...                                            T                                     (B A)h                   ...
For example     Users enter queries (A)       –   (actor = user, item=query)     Users view videos (B)       –   (actor ...
The punch-line     B’A recommends videos in response to a query       –   (isn’t that a search engine?)       –   (not qu...
Real-life example     Query: “Paco de Lucia”     Conventional meta-data search results:       –   “hombres del paco” tim...
Real-life example©MapR Technologies - Confidential   19
Real-life example©MapR Technologies - Confidential   20
Hypothetical Example     Want a navigational ontology?     Just put labels on a web page with traffic       –   This giv...
Resources     Me       tdunning@maprtech.com       @ted_dunning     Slides and such:       –   http://info.mapr.com/ted-...
Upcoming SlideShare
Loading in …5
×

Search as recommendation

2,334 views

Published on

When recommendation is described in mathematical terms as a matrix equation, a striking symmetry in the form of the equation becomes apparent.

Exploiting this symmetry allows us to build search engines that don't need meta-data and self-organizing web-sites.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,334
On SlideShare
0
From Embeds
0
Number of Embeds
35
Actions
Shares
0
Downloads
40
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • MapR combines the best of the open source technology with our own deep innovations to provide the most advanced distribution for Apache Hadoop.MapR’s team has a deep bench of enterprise software experience with proven success across storage, networking, virtualization, analytics, and open source technologies.Our CEO has driven multiple companies to successful outcomes in the analytic, storage, and virtualization spaces.Our CTO and co-founder M.C. Srivas was most recently at Google in BigTable. He understands the challenges of MapReduce at huge scale. Srivas was also the chief software architect at Spinnaker Networks which came out of stealth with the fastest NAS storage on the market and was acquired quickly by NetAppThe team includes experience with enterprise storage at Cisco, VmWare, IBM and EMC. Our VP of Engineering led emerging technologies and a 600 person for EMC’s NAS engineering team. We also have experience in Business Intelligence and Analytic companies and open source committers in Hadoop, Zookeeper and Mahout including PMC members.MapR is proven technology with installs by leading Hadoop installations across industries and OEM by EMC and Cisco.
  • Search as recommendation

    1. 1. Recommendation as Search Reflections on Symmetry©MapR Technologies - Confidential 1
    2. 2. Company Background MapR provides the industry’s best Hadoop Distribution – Combines the best of the Hadoop community contributions with significant internally financed infrastructure development Background of Team – Deep management bench with extensive analytic, storage, virtualization, and open source experience – Google, EMC, Cisco, VMWare, Network Appliance, IBM, Microsoft, Apache Foundation, Aster Data, Brio, ParAccel Proven – MapR used across industries (Financial Services, Media, Telcom, Health Care, Internet Services, Government) – Strategic OEM relationship with EMC and Cisco – Over 1,000 installs ©MapR Technologies - Confidential 2
    3. 3. What is Hadoop? A new style of computation A new style of combining computation and storage Allows very large computations Used by all large internet companies, many other industries Fundamentally changes the economics of large-scale computation©MapR Technologies - Confidential 3
    4. 4. Why Big Data? Because we can Because we can learn new things Because new economics of computation favors large scale Because big data can be simpler than small data©MapR Technologies - Confidential 4
    5. 5. Recommendations Often known as collaborative filtering “People who bought x also bought y” Actors (people) interact (bought) with items (x and y) – observe successful interaction We want to suggest additional successful interactions Observations are inherently very sparse©MapR Technologies - Confidential 5
    6. 6. Examples Customers buying books (Linden et al) Web visitors rating music (Shardanand and Maes) or movies (Riedl, et al), (Netflix) Internet radio listeners not skipping songs (Musicmatch) Internet video watchers watching >30 s (Veoh) iTunes song purchases or plays (Apple)©MapR Technologies - Confidential 6
    7. 7. Fundamental Algorithm History matrix A has the shape of actors x items Cooccurrence matrix K has the shape of items x items an actor interacted with both x and y sum over all actors A is also a linear operator K tells us “users who interacted with x also interacted with y”©MapR Technologies - Confidential 7
    8. 8. … Warning …©MapR Technologies - Confidential 8
    9. 9. … Warning … Mathematics ahead©MapR Technologies - Confidential 9
    10. 10. Fundamental Algorithmic Structure Cooccurrence K=A A T r = AT (Ah) = (AT A)h For very large data-sets r = sparsify(A A)h T©MapR Technologies - Confidential 10
    11. 11. But Wait ... Does it have to be that way?©MapR Technologies - Confidential 11
    12. 12. But why not ... T (A A)h©MapR Technologies - Confidential 12
    13. 13. But why not ... T (A A)h Why just dyadic learning?©MapR Technologies - Confidential 13
    14. 14. But why not ... T (B A)h Why just dyadic learning? Why not triadic learning?©MapR Technologies - Confidential 14
    15. 15. But why not ... T (B A)h Why just dyadic learning? Why not p-adic learning?©MapR Technologies - Confidential 15
    16. 16. For example Users enter queries (A) – (actor = user, item=query) Users view videos (B) – (actor = user, item=video) A’A gives query recommendation – “did you mean to ask for” B’B gives video recommendation – “you might like these videos”©MapR Technologies - Confidential 16
    17. 17. The punch-line B’A recommends videos in response to a query – (isn’t that a search engine?) – (not quite, it doesn’t look at content or meta-data)©MapR Technologies - Confidential 17
    18. 18. Real-life example Query: “Paco de Lucia” Conventional meta-data search results: – “hombres del paco” times 400 – not much else Recommendation based search: – Flamenco guitar and dancers – Spanish and classical guitar – Van Halen doing a classical/flamenco riff©MapR Technologies - Confidential 18
    19. 19. Real-life example©MapR Technologies - Confidential 19
    20. 20. Real-life example©MapR Technologies - Confidential 20
    21. 21. Hypothetical Example Want a navigational ontology? Just put labels on a web page with traffic – This gives A = users x label clicks Remember viewing history – This gives B = users x items Cross recommend – B’A = label to item mapping After several users click, results are whatever users think they should be©MapR Technologies - Confidential 21
    22. 22. Resources Me tdunning@maprtech.com @ted_dunning Slides and such: – http://info.mapr.com/ted-paris-05-2012 The original paper – Accurate Methods for the Statistics of Surprise and Coincidence – (check on citeseer) Source code – Mahout project – contact me©MapR Technologies - Confidential 22

    ×