• Save
Top 10 Lessons learned Developing, Deploying and Operating Real-World Recommender Systems
Upcoming SlideShare
Loading in...5
×
 

Top 10 Lessons learned Developing, Deploying and Operating Real-World Recommender Systems

on

  • 2,842 views

Strands develops products that help people find information online that they want and need. Strands offers production recommendation services for eCommerce, interactive tools for personal finance ...

Strands develops products that help people find information online that they want and need. Strands offers production recommendation services for eCommerce, interactive tools for personal finance management, and personal interest and lifestyle-oriented social discovery solutions. Strands also operates moneystrands.com, a personal finance management platform, and strands.com, a training log and information source for active people.

In this talk, Strands Chief Innovation Officer Marc Torrens, PhD, will discuss Strands' "Top 10 Lessons Learned" from our experience building recommender systems and interactions with customers deploying our systems. The lessons learned will range from customer relations and marketing ("It must make "strategic" sense), to business planning ("Don't wait too long to get ready to scale"), to technical ("Cold start? Be Creative"). As recommender technology becomes ubiquitous online, and even overshadows search in many commercials settings, Strands has found these "Top 10 Lessons Learned" continue to be valuable guidelines.

Statistics

Views

Total Views
2,842
Views on SlideShare
2,069
Embed Views
773

Actions

Likes
8
Downloads
0
Comments
0

8 Embeds 773

http://marctorrens.net 743
http://www.linkedin.com 19
https://www.linkedin.com 5
http://www.yatedo.com 2
url_unknown 1
http://fbweb-test.comoj.com 1
http://www.docseek.net 1
http://feeds.feedburner.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />
  • <br />

Top 10 Lessons learned Developing, Deploying and Operating Real-World Recommender Systems Top 10 Lessons learned Developing, Deploying and Operating Real-World Recommender Systems Presentation Transcript

  • Top 10 Lessons learned Developing, Deploying and Operating Real-World Recommender Systems based on Francisco J. Martin RecSys 2009 Keynote speech Marc Torrens Chief Innovation Officer 4th ACM Conference on Recommender Systems Barcelona :: September 30, 2010
  • Agenda About Strands What is a Recommender? The Business Perspective Top Lessons Learned 4th ACM Conference on Recommender Systems 2 Barcelona :: September 30, 2010
  • Agenda About Strands What is a Recommender? The Business Perspective Top Lessons Learned 4th ACM Conference on Recommender Systems 3 Barcelona :: September 30, 2010
  • About Strands Year 2003... • implicit preferences! What to synch? What to play? Provide Recommendation for the Music Space 4th ACM Conference on Recommender Systems 4 Barcelona :: September 30, 2010
  • About Strands Strands develops technologies to better understand people’s taste and help them discover things they like and didn’t know about. everything! training plans nances people people videos videos music music music 2003 2004 2005 2006 2007 2008 2009 2010 Same mission evolving through different domains 4th ACM Conference on Recommender Systems 5 Barcelona :: September 30, 2010
  • About Strands Understanding Understanding consumer Understanding consumer preferences habits at the commerce consumer behavior at in real-world activities (transaction) level the product level Highly-targeted Product Placement 4th ACM Conference on Recommender Systems 6 Barcelona :: September 30, 2010
  • About Strands • BBVA Tú Cuentas • ING Netherlands 4th ACM Conference on Recommender Systems 7 Barcelona :: September 30, 2010
  • About Strands 4th ACM Conference on Recommender Systems 8 Barcelona :: September 30, 2010
  • About Strands 4th ACM Conference on Recommender Systems 9 Barcelona :: September 30, 2010
  • About Strands 4th ACM Conference on Recommender Systems 10 Barcelona :: September 30, 2010
  • Agenda About Strands What is a Recommender? The Business Perspective Top Lessons Learned 4th ACM Conference on Recommender Systems 11 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender selects the product that if acquired by the buyer maximizes value of both buyer and seller at a given point in time 4th ACM Conference on Recommender Systems 12 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender processes information and transforms it into actionable knowledge 4th ACM Conference on Recommender Systems 13 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender processes information and transforms it into actionable knowledge Recommender Components 4th ACM Conference on Recommender Systems 13 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender processes information and transforms it into actionable knowledge Recommender Components Knowledge Base 4th ACM Conference on Recommender Systems 13 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender processes information and transforms it into actionable knowledge Recommender Components Knowledge Knowledge Processing Base Application 4th ACM Conference on Recommender Systems 13 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender processes information and transforms it into actionable knowledge Recommender Components Knowledge Business Knowledge Processing Control Base Application & Analytics 4th ACM Conference on Recommender Systems 13 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender processes information and transforms it into actionable knowledge Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 4th ACM Conference on Recommender Systems 13 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender processes information and transforms it into actionable knowledge Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 25% 25% 25% 25% 4th ACM Conference on Recommender Systems 13 Barcelona :: September 30, 2010
  • What is a Recommender? A Recommender processes information and transforms it into actionable knowledge Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 25% 25% 25% 25% It has a certain level of autonomy presenting recommendations to the end user 4th ACM Conference on Recommender Systems 13 Barcelona :: September 30, 2010
  • Agenda About Strands What is a Recommender? The Business Perspective Top Lessons Learned 4th ACM Conference on Recommender Systems 14 Barcelona :: September 30, 2010
  • The Business Perspective 4th ACM Conference on Recommender Systems 15 Barcelona :: September 30, 2010
  • The Business Perspective 1. Do I need a recommender? 4th ACM Conference on Recommender Systems 15 Barcelona :: September 30, 2010
  • The Business Perspective 1. Do I need a recommender? 2. How do I get one? 4th ACM Conference on Recommender Systems 15 Barcelona :: September 30, 2010
  • The Business Perspective 1. Do I need a recommender? 2. How do I get one? 3. Is it performing well? 4th ACM Conference on Recommender Systems 15 Barcelona :: September 30, 2010
  • The Business Perspective 1. Do I need a recommender? 2. How do I get one? 3. Is it performing well? 4. Was it a good idea after all? 4th ACM Conference on Recommender Systems 15 Barcelona :: September 30, 2010
  • Agenda About Strands What is a Recommender? The Business Perspective Top Lessons Learned 4th ACM Conference on Recommender Systems 16 Barcelona :: September 30, 2010
  • LESSON 1 Make sure it is needed. 4th ACM Conference on Recommender Systems 17 Barcelona :: September 30, 2010
  • LESSON 1 Make sure it is needed. 4th ACM Conference on Recommender Systems 18 Barcelona :: September 30, 2010
  • LESSON 1 Make sure it is needed. 4th ACM Conference on Recommender Systems 19 Barcelona :: September 30, 2010
  • LESSON 1 Make sure it is needed. Diversity of the Catalog X high OK medium X X X X low low medium high Diversity of the Customers 000s in both products and customers 4th ACM Conference on Recommender Systems 20 Barcelona :: September 30, 2010
  • LESSON 1 high medium Make sure it is needed. ROI low random top 10 recommender sophisticated even more recommender sophisticated recommender 4th ACM Conference on Recommender Systems 21 Barcelona :: September 30, 2010
  • LESSON 1 high medium Make sure it is needed. ROI low random top 10 recommender sophisticated even more recommender sophisticated recommender 4th ACM Conference on Recommender Systems 21 Barcelona :: September 30, 2010
  • LESSON 1 high medium Make sure it is needed. ROI low random top 10 recommender sophisticated even more recommender sophisticated recommender 4th ACM Conference on Recommender Systems 21 Barcelona :: September 30, 2010
  • LESSON 1 high medium Make sure it is needed. ROI low random top 10 recommender sophisticated even more recommender sophisticated recommender 4th ACM Conference on Recommender Systems 21 Barcelona :: September 30, 2010
  • LESSON 1 high medium Make sure it is needed. ROI low random top 10 recommender sophisticated even more recommender sophisticated recommender 4th ACM Conference on Recommender Systems 21 Barcelona :: September 30, 2010
  • LESSON 1 high medium Make sure it is needed. ROI low random top 10 recommender sophisticated even more recommender sophisticated recommender 4th ACM Conference on Recommender Systems 21 Barcelona :: September 30, 2010
  • LESSON 1 high medium Make sure it is needed. ROI low random top 10 recommender sophisticated even more recommender sophisticated recommender 4th ACM Conference on Recommender Systems 21 Barcelona :: September 30, 2010
  • LESSON 1 high Make sure it is needed. showing this impact is already challenging! medium ROI low random top 10 recommender sophisticated even more recommender sophisticated recommender 4th ACM Conference on Recommender Systems 21 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. 4th ACM Conference on Recommender Systems 22 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. Is the best recommendation for the customer the best recommendation for the business? 4th ACM Conference on Recommender Systems 23 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. customer perspective • Good vs Useful • Correctness is often too obvious to be useful • Riskier recommendations have less chances of being known business perspective • Short-term gain vs long term return 4th ACM Conference on Recommender Systems 24 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. 4th ACM Conference on Recommender Systems 25 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics ?% ?% 4th ACM Conference on Recommender Systems 25 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics ?% ?% 4th ACM Conference on Recommender Systems 25 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics ?% ?% • How much business logic goes into Recommender? 4th ACM Conference on Recommender Systems 25 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics ?% ?% • How much business logic goes into Recommender? • What’s the right level of autonomy a recommender must have? 4th ACM Conference on Recommender Systems 25 Barcelona :: September 30, 2010
  • LESSON 2 It must make “strategic” sense. Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics ?% ?% • How much business logic goes into Recommender? • What’s the right level of autonomy a recommender must have? • How can the business control recommendations? 4th ACM Conference on Recommender Systems 25 Barcelona :: September 30, 2010
  • The Business Perspective 1. Do I need a recommender? 2. How do I get one? 3. Is it performing well? 4. Was it a good idea after all? 4th ACM Conference on Recommender Systems 26 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. 4th ACM Conference on Recommender Systems 27 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. 4th ACM Conference on Recommender Systems 28 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. Small company Select a vendor 4th ACM Conference on Recommender Systems 28 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. Small company Select a vendor Hire a copule of RecSys Medium company students! 4th ACM Conference on Recommender Systems 28 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. Small company Select a vendor Hire a copule of RecSys Medium company students! Partner with an Large company experienced vendor 4th ACM Conference on Recommender Systems 28 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. Small company Select a vendor Hire a copule of RecSys Medium company students! Partner with an Large company experienced vendor Tech company Organize a contest! 4th ACM Conference on Recommender Systems 28 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 4th ACM Conference on Recommender Systems 29 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. Price 000s licensing + 000s integration $99 / month 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 4th ACM Conference on Recommender Systems 29 Barcelona :: September 30, 2010
  • LESSON 3 Choose the right partner. Price Vendors 000s licensing + 000s integration hundreds vendors $99 / month few vendors 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 4th ACM Conference on Recommender Systems 29 Barcelona :: September 30, 2010
  • LESSON 4 Cold start? Be creative! 4th ACM Conference on Recommender Systems 30 Barcelona :: September 30, 2010
  • LESSON 4 Cold start? Be creative! With the advent of the Internet the start for a Recommender isn’t so cold anymore 4th ACM Conference on Recommender Systems 31 Barcelona :: September 30, 2010
  • LESSON 5 Data and algorithms. 4th ACM Conference on Recommender Systems 32 Barcelona :: September 30, 2010
  • LESSON 5 Data and Algorithms. Which really makes the difference? Ingredients or Receipe? 4th ACM Conference on Recommender Systems 33 Barcelona :: September 30, 2010
  • LESSON 5 Data and Algorithms. Data Quantity and Quality OK good X X bad bad good Algorithm Performance 4th ACM Conference on Recommender Systems 34 Barcelona :: September 30, 2010
  • LESSON 6 Finding correlated items is easy, deciding what, how, and when to present to the user is hard. 4th ACM Conference on Recommender Systems 35 Barcelona :: September 30, 2010
  • LESSON 6 Finding correlated items is easy, deciding what, how, and when to present to the user is hard. Math Art 4th ACM Conference on Recommender Systems 36 Barcelona :: September 30, 2010
  • LESSON 7 Don’t wast time calculating nearest neighbours. 4th ACM Conference on Recommender Systems 37 Barcelona :: September 30, 2010
  • LESSON 7 Don’t waste time calculating nearest neighbours. Let people tell you! 4th ACM Conference on Recommender Systems 38 Barcelona :: September 30, 2010
  • LESSON 8 Don’t wait too long to get ready to scale. 4th ACM Conference on Recommender Systems 39 Barcelona :: September 30, 2010
  • LESSON 8 Don’t wait too long to get ready to scale. When is the right moment? • if you do too soon and recommendations don’t take off... • if you do too late and recommendations do take off... 4th ACM Conference on Recommender Systems 40 Barcelona :: September 30, 2010
  • LESSON 9 Choose the right feedback mechanism. 4th ACM Conference on Recommender Systems 41 Barcelona :: September 30, 2010
  • LESSON 9 Choose the right feedback mechanism. 4th ACM Conference on Recommender Systems 42 Barcelona :: September 30, 2010
  • LESSON 9 Choose the right feedback mechanism. 4th ACM Conference on Recommender Systems 43 Barcelona :: September 30, 2010
  • LESSON 9 Choose the right feedback mechanism. 4th ACM Conference on Recommender Systems 44 Barcelona :: September 30, 2010
  • LESSON 9 Choose the right feedback mechanism. • Implicit Ratings vs Explicit Ratings • Implicit Semantics vs Explicit Semantic of Ratings 4th ACM Conference on Recommender Systems 45 Barcelona :: September 30, 2010
  • LESSON 9 Choose the right feedback mechanism. The Ideal Rating system... feedback actions I have it (i knew it, i saw it) my friend has it (knew it, saw it) Good so-so* Bad Show it to me later Don’t show it any more *optional at it may help to con rm some implicit actions. 4th ACM Conference on Recommender Systems 46 Barcelona :: September 30, 2010
  • LESSON 9 Choose the right feedback mechanism. 4th ACM Conference on Recommender Systems 47 Barcelona :: September 30, 2010
  • LESSON 10 Measure everything. 4th ACM Conference on Recommender Systems 48 Barcelona :: September 30, 2010
  • LESSON 10 Measure everything. 4th ACM Conference on Recommender Systems 49 Barcelona :: September 30, 2010
  • LESSON 10 Measure everything. 4th ACM Conference on Recommender Systems 50 Barcelona :: September 30, 2010
  • So, what have we learned? Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 4th ACM Conference on Recommender Systems 51 Barcelona :: September 30, 2010
  • So, what have we learned? Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 50% 4th ACM Conference on Recommender Systems 51 Barcelona :: September 30, 2010
  • So, what have we learned? Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 20% 50% 4th ACM Conference on Recommender Systems 51 Barcelona :: September 30, 2010
  • So, what have we learned? Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 5% 20% 50% 4th ACM Conference on Recommender Systems 51 Barcelona :: September 30, 2010
  • So, what have we learned? Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 25% 5% 20% 50% 4th ACM Conference on Recommender Systems 51 Barcelona :: September 30, 2010
  • So, what have we learned? Recommender Components Knowledge Business Knowledge User Processing Control Base Interface Application & Analytics 25% 5% 20% 50% EU(t) A framework to analyze the expected utility of each recommendation and guide the recommender over time to get the most out of every customer 4th ACM Conference on Recommender Systems 51 Barcelona :: September 30, 2010
  • Questions? Thank you! 4th ACM Conference on Recommender Systems 52 Barcelona :: September 30, 2010