1.
Establishing a relevance focused
culture in a large organization
Tom Burgmans
Search Scientist
April 25th 2019
1

2.
2
Once upon a time

3.
3
• Matching all- and only
the correct documents
• Tuning the scoring algorithm
• Accurate highlighting
• Attractive KWIC summaries
• Variety of facets / filters
• Predictive autocompleting
• Auto-correcting misspellings
• Suggesting spelling corrections
• Providing intuitive feedback
• Support grouping / collapsing
• Pro-active recommendations
• Direct answers
Improving Search Quality means…
=

4.
4
When tuning relevance…
creating/maintaining judgment sets
feature development
query processing (vocabularies)
curating/enriching content
search engine configuration
tuning scoring algorithm
strategy meetings
analysis & research
bug fixing
What we ‘tune’: What we’re actually doing:
Search Engine
Content
Vocabularies
Product features

5.
5
“A goal without a plan
is just a wish.”
― Antoine de Saint-Exupéry

6.
6
Search Quality team
Boss / Product owner
Content architect / curator
Search specialist
Domain expert(s)
owns the tuning program
understands the content
understands the search engine
understands the user’s need
Project manager
Business Analyst
Solution architect
DevOps engineer
Controlled Vocabulary specialist
manages the tuning exercise
understands the business’ needs
architects the overall design
manages infrastructure
manages dictionaries
designs the user interface
UX developer / Front-end designer
Bill of materials
Documentation
space
Issue tracker Meeting schedule
Project essentials
Tuning environment
Full content set
Customer feedback
Usage logsAnalysis tools
Judgement
queries
https://youtu.be/4tcizH8b5pY
Mature version
product & content

7.
Let’s go!
Search Engine
Content
Vocabularies
Product
features
Categorize issues
Strategize improvements
Study user feedback and logs
Analyze and experiment
Tune search
Domain expert(s)
Content
architect
Search specialist
Update tuning environment
Update production
Meetings Search Quality team
Boss
Solution architect
Project manager
DevOps engineer Controlled
Vocabulary
specialist
UX developer
Business
Analyst

8.
Maintenance
Categorize issues
Strategize improvements
Study user feedback and logs
Analyze and experiment
Domain expert(s)
Content
architect
Search specialist
Update tuning environment
Update production
Meetings Search Quality team
Boss
Solution architect
Search Engine
Content
Vocabularies
Product
features
Tune search

9.
“The whole is greater
than the sum of its parts.”
Aristotle

10.
10

11.
11

12.
Relevance
metrics
Search Quality dashboard
DWH
Judgment set tests
Environments
Environments &
software statistics
User behavior
statistics
A/B tests
Change
metrics
Search
Quality
Team

13.
query
autocomplete
semantic
suggestions
index
spelling
SaaS
Designing search in a large company
Search
Engine
Extremely flexible similarity class
https://youtu.be/BsyVsmuS50c
Highlighter
Query cooking
Content transformation & enrichment
Spellchecker
Autocompleter
Semantic query
suggester
Generic, re-usable, highly customizable,
loosely coupled components
Plugins

14.
Search engine
query
User query
to AST
Query analysis
Query planning
Tree to Solr
syntax
?
term
q=…
Regular
expressions
Java
classes Vocabularies
Expand synonyms
Remove stop words
Quote concepts
Normalize dates
Apply auto filters
Detect concepts
& synonyms
Detect auto-filters
Detect known queries
Detect dates
& other patterns
Detect stop words
Flexible query cooking pipeline

15.
15
Lessons learned
• Quality is value
• Have a plan
• Get involved early
• Work in a team with multiple disciplines
• Create a community
• Have decent tools
• Design generic, re-usable, customizable
• Don’t stop

16.
The End