SlideShare a Scribd company logo
1 of 54
Download to read offline
Search Quality
A Business-Friendly Perspective
Peter Fries
Search Consultant
Search Quality
Successful Delivery of
Search Projects
Successful Delivery of
Search Projects
Search Project Quality
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
WHAT MADNESS?
Search Quality Anti-Patterns
● Search Quality is Bug Squashing
● Search Quality is Too Hard For You
● Search Quality is Off Limits
● Search Quality is a Feeling
● Search Quality is Relevance
● Search Quality is Mysterious
Search Quality Anti-Patterns
● Search Quality is Bug Squashing
● Search Quality is Too Hard For You
● Search Quality is Off Limits
● Search Quality is a Feeling
● Search Quality is Relevance
● Search Quality is Mysterious
Search Quality - A Business-Friendly Perspective
Search Quality - A Business-Friendly Perspective
Search Quality Anti-Patterns
● Search Quality is Bug Squashing
● Search Quality is Too Hard For You
● Search Quality is Off Limits
● Search Quality is a Feeling
● Search Quality is Relevance
● Search Quality is Mysterious
Search Quality - A Business-Friendly Perspective
Search Quality Anti-Patterns
● Search Quality is Bug Squashing
● Search Quality is Too Hard For You
● Search Quality is Off Limits
● Search Quality is a Feeling
● Search Quality is Relevance
● Search Quality is Mysterious
Search Quality - A Business-Friendly Perspective
Search Quality Anti-Patterns
● Search Quality is Bug Squashing
● Search Quality is Too Hard For You
● Search Quality is Off Limits
● Search Quality is a Feeling
● Search Quality is Relevance
● Search Quality is Mysterious
Search Quality - A Business-Friendly Perspective
Search Quality Anti-Patterns
● Search Quality is Bug Squashing
● Search Quality is Too Hard For You
● Search Quality is Off Limits
● Search Quality is a Feeling
● Search Quality is Relevance
● Search Quality is Mysterious
Search Quality - A Business-Friendly Perspective
Search Quality Anti-Patterns
● Search Quality is Bug Squashing
● Search Quality is Too Hard For You
● Search Quality is Off Limits
● Search Quality is a Feeling
● Search Quality is Relevance
● Search Quality is Mysterious
Search Quality Anti-Patterns
● Search Quality is Bug Squashing
● Search Quality is Too Hard For You
● Search Quality is Off Limits
● Search Quality is a Feeling
● Search Quality is Relevance
● Search Quality is Mysterious!
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
Search Quality - A Business-Friendly Perspective
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
DO YOUR HOMEWORK
GATHER PRELIMINARIES
TALK TO THE PEOPLE
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
● Big Picture
○ Use Cases, Domain, IR Vertical
● Classes of Queries
○ Navigational, Informational, Transactional, Research
○ Length, Category, Intent
● Distribution of Queries
○ Head/Tail Analysis
○ Over Tail, Over Time
○ Relative Performance, Outliers
○ Quantify Scale/Impact
● Connect to Revenue
● Points of Interest
Analysis
Search Quality - A Business-Friendly Perspective
Search Quality - A Business-Friendly Perspective
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
ROADMAPPING
Searching for People
Motivation: Many customers are
searching for movies by actor name.
~10% of random sample includes an
actor name in the query. These
queries have poor performance
relative to their peers as measured
by click-through.
We believe that recognizing named
entities (people names) will bring
performance in-line with peer
group.
IMPACT: HIGH
RISK: LOW
LOE: MEDIUM
SCOPE: Named Entities queries
SCALE: ~10%
MEASURE: Relative CTR
VALUE: $XX.XX
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
Iron Triangle of Search Quality
MEASUREMENT
OPTIMIZATIONDATA
Iron Triangle of Search Quality
MEASUREMENT
OPTIMIZATIONDATA
Iron Triangle of Search Quality
MEASUREMENT
OPTIMIZATIONDATA
Iron Triangle of Search Quality
MEASUREMENT
OPTIMIZATIONDATA
Search Evaluation Metrics
● MRR
● MAP
● Precision @K
● NDCG
● LGTM
RELEVANCE JUDGMENTS
● relevance is approximately query similarity
● relevance is binary
● judgment lists agree with users
● judgment lists are complete and consistent
Search Quality - A Business-Friendly Perspective
Search Quality
A Business-Friendly Perspective
What are we talking about?
1. Does it deliver value?
2. At what cost?
Who are we talking about?
1. Marketing
a. define offerings
b. attract and retain customers
2. Management
a. set goals
b. plan and allocate resources
HIERARCHY OF BUSINESS OBJECTIVES
REVENUE
ENGAGEMENT
CONVERSIONSRETENTION
MARKET
SHARE
CUSTOMER
SATISFACTION
CTR ABANDONMENTACQUISITION
Search Behaviors
● behaviors make sense
● behaviors are measurable
○ even without relevance data
● behaviors tell a story
● you can map behaviors to user tasks
HIERARCHY OF BUSINESS OBJECTIVES
REVENUE
ENGAGEMENT
CONVERSIONSRETENTION
MARKET
SHARE
CUSTOMER
SATISFACTION
CTR ABANDONMENTACQUISITION
SEARCH-EXITS
SERP-TO-CART THRASHING POGO-STICKING
PAGINATION
SEARCHES
PER-SESSION
REVENUE
PER-SEARCH
ZERO-CLICKS
DWELL-TIME
SEARCH-RATE
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
Development Framework for Search
LABOPERATIONS
INTEGRATION
CLICKSTREAM
A/B TESTING
EVALUATION
OFFLINE
RELEVANCE
ONLINE
AUTOMATED
TESTING
IRON TRIANGLE
1. DO YOUR HOMEWORK
2. GATHER PRELIMINARIES
3. INTERVIEWS
4. ANALYSIS
5. ROADMAPPING
6. PRIORITIZE & PLAN
7. EXECUTE ITERATIVELY
a. IRON TRIANGLE
b. FEEDBACK LOOPS
A METHOD TO THE MADNESS
THANK
YOU!

More Related Content

Recently uploaded

+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎️+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 

Recently uploaded (20)

Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024AXA XL - Insurer Innovation Award Americas 2024
AXA XL - Insurer Innovation Award Americas 2024
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
GenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdfGenAI Risks & Security Meetup 01052024.pdf
GenAI Risks & Security Meetup 01052024.pdf
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ..."I see eyes in my soup": How Delivery Hero implemented the safety system for ...
"I see eyes in my soup": How Delivery Hero implemented the safety system for ...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Data Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt RobisonData Cloud, More than a CDP by Matt Robison
Data Cloud, More than a CDP by Matt Robison
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 

Featured

Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
Kurio // The Social Media Age(ncy)
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Saba Software
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming Language
Simplilearn
 

Featured (20)

How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024How to Prepare For a Successful Job Search for 2024
How to Prepare For a Successful Job Search for 2024
 
Social Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie InsightsSocial Media Marketing Trends 2024 // The Global Indie Insights
Social Media Marketing Trends 2024 // The Global Indie Insights
 
Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024Trends In Paid Search: Navigating The Digital Landscape In 2024
Trends In Paid Search: Navigating The Digital Landscape In 2024
 
5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary5 Public speaking tips from TED - Visualized summary
5 Public speaking tips from TED - Visualized summary
 
ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd ChatGPT and the Future of Work - Clark Boyd
ChatGPT and the Future of Work - Clark Boyd
 
Getting into the tech field. what next
Getting into the tech field. what next Getting into the tech field. what next
Getting into the tech field. what next
 
Google's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search IntentGoogle's Just Not That Into You: Understanding Core Updates & Search Intent
Google's Just Not That Into You: Understanding Core Updates & Search Intent
 
How to have difficult conversations
How to have difficult conversations How to have difficult conversations
How to have difficult conversations
 
Introduction to Data Science
Introduction to Data ScienceIntroduction to Data Science
Introduction to Data Science
 
Time Management & Productivity - Best Practices
Time Management & Productivity -  Best PracticesTime Management & Productivity -  Best Practices
Time Management & Productivity - Best Practices
 
The six step guide to practical project management
The six step guide to practical project managementThe six step guide to practical project management
The six step guide to practical project management
 
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
Beginners Guide to TikTok for Search - Rachel Pearson - We are Tilt __ Bright...
 
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
Unlocking the Power of ChatGPT and AI in Testing - A Real-World Look, present...
 
12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work12 Ways to Increase Your Influence at Work
12 Ways to Increase Your Influence at Work
 
ChatGPT webinar slides
ChatGPT webinar slidesChatGPT webinar slides
ChatGPT webinar slides
 
More than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike RoutesMore than Just Lines on a Map: Best Practices for U.S Bike Routes
More than Just Lines on a Map: Best Practices for U.S Bike Routes
 
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
Ride the Storm: Navigating Through Unstable Periods / Katerina Rudko (Belka G...
 
Barbie - Brand Strategy Presentation
Barbie - Brand Strategy PresentationBarbie - Brand Strategy Presentation
Barbie - Brand Strategy Presentation
 
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them wellGood Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
Good Stuff Happens in 1:1 Meetings: Why you need them and how to do them well
 
Introduction to C Programming Language
Introduction to C Programming LanguageIntroduction to C Programming Language
Introduction to C Programming Language
 

Search Quality - A Business-Friendly Perspective

  • 7. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS
  • 9. Search Quality Anti-Patterns ● Search Quality is Bug Squashing ● Search Quality is Too Hard For You ● Search Quality is Off Limits ● Search Quality is a Feeling ● Search Quality is Relevance ● Search Quality is Mysterious
  • 10. Search Quality Anti-Patterns ● Search Quality is Bug Squashing ● Search Quality is Too Hard For You ● Search Quality is Off Limits ● Search Quality is a Feeling ● Search Quality is Relevance ● Search Quality is Mysterious
  • 13. Search Quality Anti-Patterns ● Search Quality is Bug Squashing ● Search Quality is Too Hard For You ● Search Quality is Off Limits ● Search Quality is a Feeling ● Search Quality is Relevance ● Search Quality is Mysterious
  • 15. Search Quality Anti-Patterns ● Search Quality is Bug Squashing ● Search Quality is Too Hard For You ● Search Quality is Off Limits ● Search Quality is a Feeling ● Search Quality is Relevance ● Search Quality is Mysterious
  • 17. Search Quality Anti-Patterns ● Search Quality is Bug Squashing ● Search Quality is Too Hard For You ● Search Quality is Off Limits ● Search Quality is a Feeling ● Search Quality is Relevance ● Search Quality is Mysterious
  • 19. Search Quality Anti-Patterns ● Search Quality is Bug Squashing ● Search Quality is Too Hard For You ● Search Quality is Off Limits ● Search Quality is a Feeling ● Search Quality is Relevance ● Search Quality is Mysterious
  • 21. Search Quality Anti-Patterns ● Search Quality is Bug Squashing ● Search Quality is Too Hard For You ● Search Quality is Off Limits ● Search Quality is a Feeling ● Search Quality is Relevance ● Search Quality is Mysterious
  • 22. Search Quality Anti-Patterns ● Search Quality is Bug Squashing ● Search Quality is Too Hard For You ● Search Quality is Off Limits ● Search Quality is a Feeling ● Search Quality is Relevance ● Search Quality is Mysterious!
  • 23. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS
  • 25. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS
  • 28. TALK TO THE PEOPLE
  • 29. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS
  • 30. ● Big Picture ○ Use Cases, Domain, IR Vertical ● Classes of Queries ○ Navigational, Informational, Transactional, Research ○ Length, Category, Intent ● Distribution of Queries ○ Head/Tail Analysis ○ Over Tail, Over Time ○ Relative Performance, Outliers ○ Quantify Scale/Impact ● Connect to Revenue ● Points of Interest Analysis
  • 33. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS
  • 35. Searching for People Motivation: Many customers are searching for movies by actor name. ~10% of random sample includes an actor name in the query. These queries have poor performance relative to their peers as measured by click-through. We believe that recognizing named entities (people names) will bring performance in-line with peer group. IMPACT: HIGH RISK: LOW LOE: MEDIUM SCOPE: Named Entities queries SCALE: ~10% MEASURE: Relative CTR VALUE: $XX.XX
  • 36. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS
  • 37. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS
  • 38. Iron Triangle of Search Quality MEASUREMENT OPTIMIZATIONDATA
  • 39. Iron Triangle of Search Quality MEASUREMENT OPTIMIZATIONDATA
  • 40. Iron Triangle of Search Quality MEASUREMENT OPTIMIZATIONDATA
  • 41. Iron Triangle of Search Quality MEASUREMENT OPTIMIZATIONDATA
  • 42. Search Evaluation Metrics ● MRR ● MAP ● Precision @K ● NDCG ● LGTM
  • 43. RELEVANCE JUDGMENTS ● relevance is approximately query similarity ● relevance is binary ● judgment lists agree with users ● judgment lists are complete and consistent
  • 46. What are we talking about? 1. Does it deliver value? 2. At what cost?
  • 47. Who are we talking about? 1. Marketing a. define offerings b. attract and retain customers 2. Management a. set goals b. plan and allocate resources
  • 48. HIERARCHY OF BUSINESS OBJECTIVES REVENUE ENGAGEMENT CONVERSIONSRETENTION MARKET SHARE CUSTOMER SATISFACTION CTR ABANDONMENTACQUISITION
  • 49. Search Behaviors ● behaviors make sense ● behaviors are measurable ○ even without relevance data ● behaviors tell a story ● you can map behaviors to user tasks
  • 50. HIERARCHY OF BUSINESS OBJECTIVES REVENUE ENGAGEMENT CONVERSIONSRETENTION MARKET SHARE CUSTOMER SATISFACTION CTR ABANDONMENTACQUISITION SEARCH-EXITS SERP-TO-CART THRASHING POGO-STICKING PAGINATION SEARCHES PER-SESSION REVENUE PER-SEARCH ZERO-CLICKS DWELL-TIME SEARCH-RATE
  • 51. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS
  • 52. Development Framework for Search LABOPERATIONS INTEGRATION CLICKSTREAM A/B TESTING EVALUATION OFFLINE RELEVANCE ONLINE AUTOMATED TESTING IRON TRIANGLE
  • 53. 1. DO YOUR HOMEWORK 2. GATHER PRELIMINARIES 3. INTERVIEWS 4. ANALYSIS 5. ROADMAPPING 6. PRIORITIZE & PLAN 7. EXECUTE ITERATIVELY a. IRON TRIANGLE b. FEEDBACK LOOPS A METHOD TO THE MADNESS