Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Anatomy of Relevance - From Data to Action: Presented by Saïd Radhouani, Yellow Pages Canada

736 views

Published on

Presented at Lucene/Solr Revolution 2014

Published in: Software
  • Be the first to comment

  • Be the first to like this

Anatomy of Relevance - From Data to Action: Presented by Saïd Radhouani, Yellow Pages Canada

  1. 1. Anatomy Of Relevance: From Data To Action Saïd Radhouani, Ph.D. radhouani Yellow Pages, Canada
  2. 2. Agenda • Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks • Measuring and Improving Relevance • Conclusion
  3. 3. Who we are • Leading media and marketing solutions company in Canada • Champion the neighbourhood economy by connecting businesses and consumers 300 000 Local and national advertisers 7.3 million Unduplicated unique visitors to our network of websites 6.5 million Downloads of our mobile apps 26% Internet users reached in Canada each month
  4. 4. Who I am • Entrepreneur • Ph.D. in Search and Knowledge Management • IT Director – Content, Search and Relevance • Noble cause: leverage data to help build smarter organizations
  5. 5. Agenda • Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks • Measuring and Improving Relevance • Conclusion
  6. 6. Local Search
  7. 7. The Importance Of Relevance Businesses Consumers Source: unknown
  8. 8. Agenda • Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks • Measuring and Improving Relevance • Conclusion
  9. 9. User Objective Understand the user intent to maximize their satisfaction Data • Who the user is (individual or group) • Where they come from (direct, SEs, …) • Their context (Location, TOD/DOW/Season, Device, …) • Their interactions history KPIs • Coverage: %tracked user, %tracked user interactions • Freshness: Daily updates of User profiles & interactions User
  10. 10. Content Objective Have accurate and “fresh” content to satisfy your audience needs Data • Enriched Directory Content (Merchants, Products, Deals, etc.) • Rich Editorial Content (Neighborhoods, Smart Tips, etc.) • Third-party content (social networks, etc. ) • User-Generated Content KPIs • Coverage: %Canadian businesses, %Neighborhoods • Completeness: e.g., for each merchant, have all required info • Freshness: Frequency of validation & updates Content User
  11. 11. Knowledge Objective Help to bridge the gap between user queries and content data Data • Linguistic (synonyms, acronyms, etc.) • Rich and “fresh” geo data (location names, polygons, etc.) • Multilingual KPIs • Coverage: %searched entities (keywords, location names, etc.) • Completeness: e.g., for each location: centroid, polygon, synonyms, relationships, etc. • Freshness: Frequency of validation & updates Content Knowledge User
  12. 12. Search Objective Action-driven, personalized and contextualized relevant search experience KPIs Offline • Recall & Precision Online • #returned results • Mean Reciprocal Rank (MRR) – the multiplicative inverse of the rank of the first correct answer. Perfect SE should have a MRR of 1 Content Knowledge Search User
  13. 13. Presentation Objective Allow users to access quickly to actionable content through optimal & dynamic search results presentation e.g., “Movie Theater” vs. “Restaurant” KPIs [that really matter] • Page Views • Unique Visitors • Visit Duration • Bounce Rate • Return Visits Frequency • (Useful) Interactions per Visits Search Presenta4on Content Knowledge User
  14. 14. Optimization UserSatisfaction improveRelevance() { While (alive) { Act; Measure; Learn; Grow; } } Search Presenta4on Content Op4miza4on Knowledge User
  15. 15. Agenda • Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks • Measuring and Improving Relevance • Conclusion
  16. 16. A problem well-­‐stated is half-­‐solved. Where To Start? Charles Ke2ering
  17. 17. Data-Driven Actions Measure Act • Implicit User Feedback (analytics & logs) • Stable Test Collection • Side-By-Side Comparison • A/B Testing • Explicit User Feedback (Survey) • Find gaps in data and improve them • Improve processes • Improve presentation • Improve algorithms • Confirm hypothesis KPIs
  18. 18. Know Your Data • Web & Mobile Analytics • Search logs Analysis • Build KPIs
  19. 19. From Data To Action Examples: • Missing geo-location • Missing businesses • Seasonal trends (adapt search algo) Manual or automa8c ac8ons
  20. 20. Test Collection KPIs: Recall & Precision • Predefined test collection built manually • Variety of use cases that cover most users needs • Evaluate new updates before going to production
  21. 21. Side-By-Side Comparison • Comparison of two (versions of the) Search Engines • Pre-defined set of real searches selected from logs • Pre-classified (keywords, business names, language, geo location, etc.) • “Blind” comparison • Significant number of testers per search
  22. 22. A/B Testing Principle • Experiment one or many ingredients at a time with real users • Some non-trivial amount of traffic is always being tested KPIs • Page Views • Unique Visitors • Visit Duration • Bounce Rate • Return Visits Frequency • Useful Interactions per Visits Simulation • Run new versions on a significant search logs sample • Estimate the impact prior deployment to prod and avoid bad surprises
  23. 23. Relevance In Action Valuable signals, NOT only for marketing! e.g., Bounce Rate – Why the user left quickly? • Who is the user? • Where they come from (direct vs. SE) • Their devices • Which content was provided? • Which page? • etc.
  24. 24. Agenda • Yellow Pages & Myself • The importance of relevance for our business • Relevance Building Blocks • Measuring and Improving Relevance • Conclusion
  25. 25. Conclusion Stop coding and make your data talk!
  26. 26. Anatomy Of Relevance: From Data To Action Saïd Radhouani, Ph.D. radhouani Yellow Pages, Canada

×