For a relevance engineer one of the most difficult tasks in the tuning process is to convince others in the organization that this is a joint effort. Even the brightest search guru doesn't get very far when working in isolation, so establishing cross-collaboration through the organization is essential. But how to get there?
On top of that, in a large organization a relevance engineer often works on multiple seemingly unrelated search projects. The challenge is not to get drowned in building custom solutions for each project, but to design generic and re-usable strategies which solve many problems at once.
In this session we'll discuss how to build a widely supported basis for search quality improvements in an organization. It is full of practical tips and examples which could help you in establishing a cross-functional culture that is optimal for relevance tuning. It also zooms in on an holistic approach of solving multiple equivalent search issues at once.
6. 6
Search Quality team
Boss / Product owner
Content architect / curator
Search specialist
Domain expert(s)
owns the tuning program
understands the content
understands the search engine
understands the user’s need
Project manager
Business Analyst
Solution architect
DevOps engineer
Controlled Vocabulary specialist
manages the tuning exercise
understands the business’ needs
architects the overall design
manages infrastructure
manages dictionaries
designs the user interface
UX developer / Front-end designer
Bill of materials
Documentation
space
Issue tracker Meeting schedule
Project essentials
Tuning environment
Full content set
Customer feedback
Usage logsAnalysis tools
Judgement
queries
https://youtu.be/4tcizH8b5pY
Mature version
product & content
7. Let’s go!
Search Engine
Content
Vocabularies
Product
features
Categorize issues
Strategize improvements
Study user feedback and logs
Analyze and experiment
Tune search
Domain expert(s)
Content
architect
Search specialist
Update tuning environment
Update production
Meetings Search Quality team
Boss
Solution architect
Project manager
DevOps engineer Controlled
Vocabulary
specialist
UX developer
Business
Analyst
8. Maintenance
Categorize issues
Strategize improvements
Study user feedback and logs
Analyze and experiment
Domain expert(s)
Content
architect
Search specialist
Update tuning environment
Update production
Meetings Search Quality team
Boss
Solution architect
Search Engine
Content
Vocabularies
Product
features
Tune search
9. “The whole is greater
than the sum of its parts.”
Aristotle
13. query
autocomplete
semantic
suggestions
index
spelling
SaaS
Designing search in a large company
Search
Engine
Extremely flexible similarity class
https://youtu.be/BsyVsmuS50c
Highlighter
Query cooking
Content transformation & enrichment
Spellchecker
Autocompleter
Semantic query
suggester
Generic, re-usable, highly customizable,
loosely coupled components
Plugins
14. Search engine
query
User query
to AST
Query analysis
Query planning
Tree to Solr
syntax
?
term
q=…
Regular
expressions
Java
classes Vocabularies
Expand synonyms
Remove stop words
Quote concepts
Normalize dates
Apply auto filters
Detect concepts
& synonyms
Detect auto-filters
Detect known queries
Detect dates
& other patterns
Detect stop words
Flexible query cooking pipeline
15. 15
Lessons learned
• Quality is value
• Have a plan
• Get involved early
• Work in a team with multiple disciplines
• Create a community
• Have decent tools
• Design generic, re-usable, customizable
• Don’t stop