SlideShare a Scribd company logo
Establishing a relevance focused
culture in a large organization
Tom Burgmans
Search Scientist
April 25th 2019
1
2
Once upon a time
3
• Matching all- and only
the correct documents
• Tuning the scoring algorithm
• Accurate highlighting
• Attractive KWIC summaries
• Variety of facets / filters
• Predictive autocompleting
• Auto-correcting misspellings
• Suggesting spelling corrections
• Providing intuitive feedback
• Support grouping / collapsing
• Pro-active recommendations
• Direct answers
Improving Search Quality means…
=
4
When tuning relevance…
creating/maintaining judgment sets
feature development
query processing (vocabularies)
curating/enriching content
search engine configuration
tuning scoring algorithm
strategy meetings
analysis & research
bug fixing
What we ‘tune’: What we’re actually doing:
Search Engine
Content
Vocabularies
Product features
5
“A goal without a plan
is just a wish.”
― Antoine de Saint-Exupéry
6
Search Quality team
Boss / Product owner
Content architect / curator
Search specialist
Domain expert(s)
owns the tuning program
understands the content
understands the search engine
understands the user’s need
Project manager
Business Analyst
Solution architect
DevOps engineer
Controlled Vocabulary specialist
manages the tuning exercise
understands the business’ needs
architects the overall design
manages infrastructure
manages dictionaries
designs the user interface
UX developer / Front-end designer
Bill of materials
Documentation
space
Issue tracker Meeting schedule
Project essentials
Tuning environment
Full content set
Customer feedback
Usage logsAnalysis tools
Judgement
queries
https://youtu.be/4tcizH8b5pY
Mature version
product & content
Let’s go!
Search Engine
Content
Vocabularies
Product
features
Categorize issues
Strategize improvements
Study user feedback and logs
Analyze and experiment
Tune search
Domain expert(s)
Content
architect
Search specialist
Update tuning environment
Update production
Meetings Search Quality team
Boss
Solution architect
Project manager
DevOps engineer Controlled
Vocabulary
specialist
UX developer
Business
Analyst
Maintenance
Categorize issues
Strategize improvements
Study user feedback and logs
Analyze and experiment
Domain expert(s)
Content
architect
Search specialist
Update tuning environment
Update production
Meetings Search Quality team
Boss
Solution architect
Search Engine
Content
Vocabularies
Product
features
Tune search
“The whole is greater
than the sum of its parts.”
Aristotle
10
11
Relevance
metrics
Search Quality dashboard
DWH
Judgment set tests
Environments
Environments &
software statistics
User behavior
statistics
A/B tests
Change
metrics
Search
Quality
Team
query
autocomplete
semantic
suggestions
index
spelling
SaaS
Designing search in a large company
Search
Engine
Extremely flexible similarity class
https://youtu.be/BsyVsmuS50c
Highlighter
Query cooking
Content transformation & enrichment
Spellchecker
Autocompleter
Semantic query
suggester
Generic, re-usable, highly customizable,
loosely coupled components
Plugins
Search engine
query
User query
to AST
Query analysis
Query planning
Tree to Solr
syntax
?
term
q=…
Regular
expressions
Java
classes Vocabularies
Expand synonyms
Remove stop words
Quote concepts
Normalize dates
Apply auto filters
Detect concepts
& synonyms
Detect auto-filters
Detect known queries
Detect dates
& other patterns
Detect stop words
Flexible query cooking pipeline
15
Lessons learned
• Quality is value
• Have a plan
• Get involved early
• Work in a team with multiple disciplines
• Create a community
• Have decent tools
• Design generic, re-usable, customizable
• Don’t stop
The End

More Related Content

Similar to Haystack 2019 - Establishing a relevance focused culture in a large organization - Tom Burgmans

Search Analytics - Comperio
Search Analytics - ComperioSearch Analytics - Comperio
Search Analytics - Comperio
Comperio - Search Matters.
 
Enabling Enterprise SEO
Enabling Enterprise SEO Enabling Enterprise SEO
Enabling Enterprise SEO
Bill Hunt
 
Search Solutions 2015: Towards a new model of search relevance testing
Search Solutions 2015:  Towards a new model of search relevance testingSearch Solutions 2015:  Towards a new model of search relevance testing
Search Solutions 2015: Towards a new model of search relevance testing
Charlie Hull
 
Managing a Global SEO Program
Managing a Global SEO ProgramManaging a Global SEO Program
Managing a Global SEO Program
Bill Hunt
 
Enterprise Search by Bill Hunt at SEMPO Asia Tour
Enterprise Search by Bill Hunt at SEMPO Asia TourEnterprise Search by Bill Hunt at SEMPO Asia Tour
Enterprise Search by Bill Hunt at SEMPO Asia Tour
OMcareers Community
 
Vsts
VstsVsts
SEO in the Age of Artificial Intelligence | How AI influences Search
SEO in the Age of Artificial Intelligence | How AI influences SearchSEO in the Age of Artificial Intelligence | How AI influences Search
SEO in the Age of Artificial Intelligence | How AI influences Search
Philipp Klöckner
 
Agile testing
Agile testingAgile testing
Choosing The Right Tools For The Right Job
Choosing The Right Tools For The Right JobChoosing The Right Tools For The Right Job
Choosing The Right Tools For The Right Job
guest6159b2
 
Building an effective sharepoint team
Building an effective sharepoint teamBuilding an effective sharepoint team
Building an effective sharepoint team
Baris Bruce Tuncertan
 
"Unstoppable Traffic" SEO cheat sheets for you
"Unstoppable Traffic" SEO cheat sheets for you"Unstoppable Traffic" SEO cheat sheets for you
"Unstoppable Traffic" SEO cheat sheets for you
vidyamittal
 
Enterprise Search in SharePoint 2013
Enterprise Search in SharePoint 2013Enterprise Search in SharePoint 2013
Enterprise Search in SharePoint 2013
Findwise
 
Do's & Don'ts of Developing Corporate Hispanic Websites in the Age of Social ...
Do's & Don'ts of Developing Corporate Hispanic Websites in the Age of Social ...Do's & Don'ts of Developing Corporate Hispanic Websites in the Age of Social ...
Do's & Don'ts of Developing Corporate Hispanic Websites in the Age of Social ...
Stone Crossing Solutions
 
Lifecycle of a FAST Search Implementation
Lifecycle of a FAST Search ImplementationLifecycle of a FAST Search Implementation
Lifecycle of a FAST Search Implementation
Perficient, Inc.
 
SEARCH ENGINE OPTIMIZATION
SEARCH ENGINE OPTIMIZATIONSEARCH ENGINE OPTIMIZATION
SEARCH ENGINE OPTIMIZATION
netultimateemp
 
Search Quality Management
Search Quality ManagementSearch Quality Management
Search Quality Management
Agnes Molnar
 
SharePoint Jumpstart #2 Making Basic SharePoint Search Work
SharePoint Jumpstart #2 Making Basic SharePoint Search WorkSharePoint Jumpstart #2 Making Basic SharePoint Search Work
SharePoint Jumpstart #2 Making Basic SharePoint Search Work
Earley Information Science
 
Increase Contact Center Performance with Google Search and Salesforce Service...
Increase Contact Center Performance with Google Search and Salesforce Service...Increase Contact Center Performance with Google Search and Salesforce Service...
Increase Contact Center Performance with Google Search and Salesforce Service...
Perficient, Inc.
 
Search Engine Optimization
Search Engine OptimizationSearch Engine Optimization
Search Engine Optimization
SD Sharma
 
Best Practices for Enterprise Search
Best Practices for Enterprise SearchBest Practices for Enterprise Search
Best Practices for Enterprise Search
Chris Risner
 

Similar to Haystack 2019 - Establishing a relevance focused culture in a large organization - Tom Burgmans (20)

Search Analytics - Comperio
Search Analytics - ComperioSearch Analytics - Comperio
Search Analytics - Comperio
 
Enabling Enterprise SEO
Enabling Enterprise SEO Enabling Enterprise SEO
Enabling Enterprise SEO
 
Search Solutions 2015: Towards a new model of search relevance testing
Search Solutions 2015:  Towards a new model of search relevance testingSearch Solutions 2015:  Towards a new model of search relevance testing
Search Solutions 2015: Towards a new model of search relevance testing
 
Managing a Global SEO Program
Managing a Global SEO ProgramManaging a Global SEO Program
Managing a Global SEO Program
 
Enterprise Search by Bill Hunt at SEMPO Asia Tour
Enterprise Search by Bill Hunt at SEMPO Asia TourEnterprise Search by Bill Hunt at SEMPO Asia Tour
Enterprise Search by Bill Hunt at SEMPO Asia Tour
 
Vsts
VstsVsts
Vsts
 
SEO in the Age of Artificial Intelligence | How AI influences Search
SEO in the Age of Artificial Intelligence | How AI influences SearchSEO in the Age of Artificial Intelligence | How AI influences Search
SEO in the Age of Artificial Intelligence | How AI influences Search
 
Agile testing
Agile testingAgile testing
Agile testing
 
Choosing The Right Tools For The Right Job
Choosing The Right Tools For The Right JobChoosing The Right Tools For The Right Job
Choosing The Right Tools For The Right Job
 
Building an effective sharepoint team
Building an effective sharepoint teamBuilding an effective sharepoint team
Building an effective sharepoint team
 
"Unstoppable Traffic" SEO cheat sheets for you
"Unstoppable Traffic" SEO cheat sheets for you"Unstoppable Traffic" SEO cheat sheets for you
"Unstoppable Traffic" SEO cheat sheets for you
 
Enterprise Search in SharePoint 2013
Enterprise Search in SharePoint 2013Enterprise Search in SharePoint 2013
Enterprise Search in SharePoint 2013
 
Do's & Don'ts of Developing Corporate Hispanic Websites in the Age of Social ...
Do's & Don'ts of Developing Corporate Hispanic Websites in the Age of Social ...Do's & Don'ts of Developing Corporate Hispanic Websites in the Age of Social ...
Do's & Don'ts of Developing Corporate Hispanic Websites in the Age of Social ...
 
Lifecycle of a FAST Search Implementation
Lifecycle of a FAST Search ImplementationLifecycle of a FAST Search Implementation
Lifecycle of a FAST Search Implementation
 
SEARCH ENGINE OPTIMIZATION
SEARCH ENGINE OPTIMIZATIONSEARCH ENGINE OPTIMIZATION
SEARCH ENGINE OPTIMIZATION
 
Search Quality Management
Search Quality ManagementSearch Quality Management
Search Quality Management
 
SharePoint Jumpstart #2 Making Basic SharePoint Search Work
SharePoint Jumpstart #2 Making Basic SharePoint Search WorkSharePoint Jumpstart #2 Making Basic SharePoint Search Work
SharePoint Jumpstart #2 Making Basic SharePoint Search Work
 
Increase Contact Center Performance with Google Search and Salesforce Service...
Increase Contact Center Performance with Google Search and Salesforce Service...Increase Contact Center Performance with Google Search and Salesforce Service...
Increase Contact Center Performance with Google Search and Salesforce Service...
 
Search Engine Optimization
Search Engine OptimizationSearch Engine Optimization
Search Engine Optimization
 
Best Practices for Enterprise Search
Best Practices for Enterprise SearchBest Practices for Enterprise Search
Best Practices for Enterprise Search
 

More from OpenSource Connections

Encores
EncoresEncores
Test driven relevancy
Test driven relevancyTest driven relevancy
Test driven relevancy
OpenSource Connections
 
How To Structure Your Search Team for Success
How To Structure Your Search Team for SuccessHow To Structure Your Search Team for Success
How To Structure Your Search Team for Success
OpenSource Connections
 
The right path to making search relevant - Taxonomy Bootcamp London 2019
The right path to making search relevant  - Taxonomy Bootcamp London 2019The right path to making search relevant  - Taxonomy Bootcamp London 2019
The right path to making search relevant - Taxonomy Bootcamp London 2019
OpenSource Connections
 
Payloads and OCR with Solr
Payloads and OCR with SolrPayloads and OCR with Solr
Payloads and OCR with Solr
OpenSource Connections
 
Haystack 2019 Lightning Talk - The Future of Quepid - Charlie Hull
Haystack 2019 Lightning Talk - The Future of Quepid - Charlie HullHaystack 2019 Lightning Talk - The Future of Quepid - Charlie Hull
Haystack 2019 Lightning Talk - The Future of Quepid - Charlie Hull
OpenSource Connections
 
Haystack 2019 Lightning Talk - State of Apache Tika - Tim Allison
Haystack 2019 Lightning Talk - State of Apache Tika - Tim AllisonHaystack 2019 Lightning Talk - State of Apache Tika - Tim Allison
Haystack 2019 Lightning Talk - State of Apache Tika - Tim Allison
OpenSource Connections
 
Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...
Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...
Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...
OpenSource Connections
 
Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj Bharadwaj
Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj BharadwajHaystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj Bharadwaj
Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj Bharadwaj
OpenSource Connections
 
Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...
Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...
Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...
OpenSource Connections
 
Haystack 2019 - Search-based recommendations at Politico - Ryan Kohl
Haystack 2019 - Search-based recommendations at Politico - Ryan KohlHaystack 2019 - Search-based recommendations at Politico - Ryan Kohl
Haystack 2019 - Search-based recommendations at Politico - Ryan Kohl
OpenSource Connections
 
Haystack 2019 - Search with Vectors - Simon Hughes
Haystack 2019 - Search with Vectors - Simon HughesHaystack 2019 - Search with Vectors - Simon Hughes
Haystack 2019 - Search with Vectors - Simon Hughes
OpenSource Connections
 
Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey Grainger
Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey GraingerHaystack 2019 - Natural Language Search with Knowledge Graphs - Trey Grainger
Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey Grainger
OpenSource Connections
 
Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...
Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...
Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...
OpenSource Connections
 
Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...
Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...
Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...
OpenSource Connections
 
Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...
Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...
Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...
OpenSource Connections
 
Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...
Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...
Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...
OpenSource Connections
 
2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via
2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via
2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via
OpenSource Connections
 
Haystack 2019 - Addressing variance in AB tests: Interleaved evaluation of ra...
Haystack 2019 - Addressing variance in AB tests: Interleaved evaluation of ra...Haystack 2019 - Addressing variance in AB tests: Interleaved evaluation of ra...
Haystack 2019 - Addressing variance in AB tests: Interleaved evaluation of ra...
OpenSource Connections
 
Haystack 2019 - Beyond The Search Engine: Improving Relevancy through Query E...
Haystack 2019 - Beyond The Search Engine: Improving Relevancy through Query E...Haystack 2019 - Beyond The Search Engine: Improving Relevancy through Query E...
Haystack 2019 - Beyond The Search Engine: Improving Relevancy through Query E...
OpenSource Connections
 

More from OpenSource Connections (20)

Encores
EncoresEncores
Encores
 
Test driven relevancy
Test driven relevancyTest driven relevancy
Test driven relevancy
 
How To Structure Your Search Team for Success
How To Structure Your Search Team for SuccessHow To Structure Your Search Team for Success
How To Structure Your Search Team for Success
 
The right path to making search relevant - Taxonomy Bootcamp London 2019
The right path to making search relevant  - Taxonomy Bootcamp London 2019The right path to making search relevant  - Taxonomy Bootcamp London 2019
The right path to making search relevant - Taxonomy Bootcamp London 2019
 
Payloads and OCR with Solr
Payloads and OCR with SolrPayloads and OCR with Solr
Payloads and OCR with Solr
 
Haystack 2019 Lightning Talk - The Future of Quepid - Charlie Hull
Haystack 2019 Lightning Talk - The Future of Quepid - Charlie HullHaystack 2019 Lightning Talk - The Future of Quepid - Charlie Hull
Haystack 2019 Lightning Talk - The Future of Quepid - Charlie Hull
 
Haystack 2019 Lightning Talk - State of Apache Tika - Tim Allison
Haystack 2019 Lightning Talk - State of Apache Tika - Tim AllisonHaystack 2019 Lightning Talk - State of Apache Tika - Tim Allison
Haystack 2019 Lightning Talk - State of Apache Tika - Tim Allison
 
Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...
Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...
Haystack 2019 Lightning Talk - Relevance on 17 million full text documents - ...
 
Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj Bharadwaj
Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj BharadwajHaystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj Bharadwaj
Haystack 2019 Lightning Talk - Solr Cloud on Kubernetes - Manoj Bharadwaj
 
Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...
Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...
Haystack 2019 Lightning Talk - Quaerite a Search relevance evaluation toolkit...
 
Haystack 2019 - Search-based recommendations at Politico - Ryan Kohl
Haystack 2019 - Search-based recommendations at Politico - Ryan KohlHaystack 2019 - Search-based recommendations at Politico - Ryan Kohl
Haystack 2019 - Search-based recommendations at Politico - Ryan Kohl
 
Haystack 2019 - Search with Vectors - Simon Hughes
Haystack 2019 - Search with Vectors - Simon HughesHaystack 2019 - Search with Vectors - Simon Hughes
Haystack 2019 - Search with Vectors - Simon Hughes
 
Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey Grainger
Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey GraingerHaystack 2019 - Natural Language Search with Knowledge Graphs - Trey Grainger
Haystack 2019 - Natural Language Search with Knowledge Graphs - Trey Grainger
 
Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...
Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...
Haystack 2019 - Search Logs + Machine Learning = Auto-Tagging Inventory - Joh...
 
Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...
Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...
Haystack 2019 - Improving Search Relevance with Numeric Features in Elasticse...
 
Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...
Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...
Haystack 2019 - Custom Solr Query Parser Design Option, and Pros & Cons - Ber...
 
Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...
Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...
Haystack 2019 - Solving for Satisfaction: Introduction to Click Models - Eliz...
 
2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via
2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via
2019 Haystack - How The New York Times Tackles Relevance - Jeremiah Via
 
Haystack 2019 - Addressing variance in AB tests: Interleaved evaluation of ra...
Haystack 2019 - Addressing variance in AB tests: Interleaved evaluation of ra...Haystack 2019 - Addressing variance in AB tests: Interleaved evaluation of ra...
Haystack 2019 - Addressing variance in AB tests: Interleaved evaluation of ra...
 
Haystack 2019 - Beyond The Search Engine: Improving Relevancy through Query E...
Haystack 2019 - Beyond The Search Engine: Improving Relevancy through Query E...Haystack 2019 - Beyond The Search Engine: Improving Relevancy through Query E...
Haystack 2019 - Beyond The Search Engine: Improving Relevancy through Query E...
 

Recently uploaded

A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
AlessioFois2
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
Sm321
 
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
sameer shah
 
"Financial Odyssey: Navigating Past Performance Through Diverse Analytical Lens"
"Financial Odyssey: Navigating Past Performance Through Diverse Analytical Lens""Financial Odyssey: Navigating Past Performance Through Diverse Analytical Lens"
"Financial Odyssey: Navigating Past Performance Through Diverse Analytical Lens"
sameer shah
 
一比一原版巴斯大学毕业证(Bath毕业证书)学历如何办理
一比一原版巴斯大学毕业证(Bath毕业证书)学历如何办理一比一原版巴斯大学毕业证(Bath毕业证书)学历如何办理
一比一原版巴斯大学毕业证(Bath毕业证书)学历如何办理
y3i0qsdzb
 
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docxDATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
SaffaIbrahim1
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
Social Samosa
 
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
taqyea
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
v7oacc3l
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
Bill641377
 
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google CloudBuild applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
Márton Kodok
 
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
mkkikqvo
 
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performanceAnalysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
roli9797
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
aqzctr7x
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
Timothy Spann
 
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
ElizabethGarrettChri
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
Sachin Paul
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Kiwi Creative
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
manishkhaire30
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Lars Albertsson
 

Recently uploaded (20)

A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
 
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
STATATHON: Unleashing the Power of Statistics in a 48-Hour Knowledge Extravag...
 
"Financial Odyssey: Navigating Past Performance Through Diverse Analytical Lens"
"Financial Odyssey: Navigating Past Performance Through Diverse Analytical Lens""Financial Odyssey: Navigating Past Performance Through Diverse Analytical Lens"
"Financial Odyssey: Navigating Past Performance Through Diverse Analytical Lens"
 
一比一原版巴斯大学毕业证(Bath毕业证书)学历如何办理
一比一原版巴斯大学毕业证(Bath毕业证书)学历如何办理一比一原版巴斯大学毕业证(Bath毕业证书)学历如何办理
一比一原版巴斯大学毕业证(Bath毕业证书)学历如何办理
 
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docxDATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
DATA COMMS-NETWORKS YR2 lecture 08 NAT & CLOUD.docx
 
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...
 
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(harvard毕业证书)哈佛大学毕业证如何办理
 
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
在线办理(英国UCA毕业证书)创意艺术大学毕业证在读证明一模一样
 
Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...Population Growth in Bataan: The effects of population growth around rural pl...
Population Growth in Bataan: The effects of population growth around rural pl...
 
Build applications with generative AI on Google Cloud
Build applications with generative AI on Google CloudBuild applications with generative AI on Google Cloud
Build applications with generative AI on Google Cloud
 
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
原版一比一多伦多大学毕业证(UofT毕业证书)如何办理
 
Analysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performanceAnalysis insight about a Flyball dog competition team's performance
Analysis insight about a Flyball dog competition team's performance
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 
DSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelinesDSSML24_tspann_CodelessGenerativeAIPipelines
DSSML24_tspann_CodelessGenerativeAIPipelines
 
Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024Open Source Contributions to Postgres: The Basics POSETTE 2024
Open Source Contributions to Postgres: The Basics POSETTE 2024
 
Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......Palo Alto Cortex XDR presentation .......
Palo Alto Cortex XDR presentation .......
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
 

Haystack 2019 - Establishing a relevance focused culture in a large organization - Tom Burgmans

  • 1. Establishing a relevance focused culture in a large organization Tom Burgmans Search Scientist April 25th 2019 1
  • 3. 3 • Matching all- and only the correct documents • Tuning the scoring algorithm • Accurate highlighting • Attractive KWIC summaries • Variety of facets / filters • Predictive autocompleting • Auto-correcting misspellings • Suggesting spelling corrections • Providing intuitive feedback • Support grouping / collapsing • Pro-active recommendations • Direct answers Improving Search Quality means… =
  • 4. 4 When tuning relevance… creating/maintaining judgment sets feature development query processing (vocabularies) curating/enriching content search engine configuration tuning scoring algorithm strategy meetings analysis & research bug fixing What we ‘tune’: What we’re actually doing: Search Engine Content Vocabularies Product features
  • 5. 5 “A goal without a plan is just a wish.” ― Antoine de Saint-Exupéry
  • 6. 6 Search Quality team Boss / Product owner Content architect / curator Search specialist Domain expert(s) owns the tuning program understands the content understands the search engine understands the user’s need Project manager Business Analyst Solution architect DevOps engineer Controlled Vocabulary specialist manages the tuning exercise understands the business’ needs architects the overall design manages infrastructure manages dictionaries designs the user interface UX developer / Front-end designer Bill of materials Documentation space Issue tracker Meeting schedule Project essentials Tuning environment Full content set Customer feedback Usage logsAnalysis tools Judgement queries https://youtu.be/4tcizH8b5pY Mature version product & content
  • 7. Let’s go! Search Engine Content Vocabularies Product features Categorize issues Strategize improvements Study user feedback and logs Analyze and experiment Tune search Domain expert(s) Content architect Search specialist Update tuning environment Update production Meetings Search Quality team Boss Solution architect Project manager DevOps engineer Controlled Vocabulary specialist UX developer Business Analyst
  • 8. Maintenance Categorize issues Strategize improvements Study user feedback and logs Analyze and experiment Domain expert(s) Content architect Search specialist Update tuning environment Update production Meetings Search Quality team Boss Solution architect Search Engine Content Vocabularies Product features Tune search
  • 9. “The whole is greater than the sum of its parts.” Aristotle
  • 10. 10
  • 11. 11
  • 12. Relevance metrics Search Quality dashboard DWH Judgment set tests Environments Environments & software statistics User behavior statistics A/B tests Change metrics Search Quality Team
  • 13. query autocomplete semantic suggestions index spelling SaaS Designing search in a large company Search Engine Extremely flexible similarity class https://youtu.be/BsyVsmuS50c Highlighter Query cooking Content transformation & enrichment Spellchecker Autocompleter Semantic query suggester Generic, re-usable, highly customizable, loosely coupled components Plugins
  • 14. Search engine query User query to AST Query analysis Query planning Tree to Solr syntax ? term q=… Regular expressions Java classes Vocabularies Expand synonyms Remove stop words Quote concepts Normalize dates Apply auto filters Detect concepts & synonyms Detect auto-filters Detect known queries Detect dates & other patterns Detect stop words Flexible query cooking pipeline
  • 15. 15 Lessons learned • Quality is value • Have a plan • Get involved early • Work in a team with multiple disciplines • Create a community • Have decent tools • Design generic, re-usable, customizable • Don’t stop