This document introduces Quaerite, a search relevance toolkit for testing search relevance parameters offline. It allows running experiments to test different combinations of tokenizers, filters, scoring models and other parameters to evaluate search relevance without live user queries. The toolkit supports experimenting with all parameter permutations using grid search or random search, and also incorporates a genetic algorithm with cross-fold validation. It currently supports Apache Solr and plans to add support for ElasticSearch. The goal is to help optimize search relevance through offline testing of parameter configurations.

1. © 2019 The MITRE Corporation. All rights reserved.
Quaerite – Search Relevance Toolkit
Tim Allison
tallison@apache.org, @_tallison
April 24, 2019
Haystack Conference
Approved for Public Release;
Distribution Unlimited. Case
Number 18-3138-5

2. | 2 |
© 2019 The MITRE Corporation. All rights reserved.
Debt of Gratitude
▪ Thank you Doug Turnbull, John Berryman and Open Source
Connections for the inspiration/examples/training with tmdb and for
sharing your ground truth set!

3. | 3 |
© 2019 The MITRE Corporation. All rights reserved.
Yet Another Toolkit? Why!?
▪ How many parameters do we have?
▪ How many permutations of those parameters are available?

4. | 4 |
© 2019 The MITRE Corporation. All rights reserved.
Available Parameters
▪ 14 tokenizers https://lucene.apache.org/solr/guide/7_1/tokenizers.html
▪ ~45 token filters (not including language-specific token filters – see next slide)
https://lucene.apache.org/solr/guide/7_1/filter-descriptions.html
▪ Query parsers
▪ Query operators, minimum should match, should, must, not
▪ Token/field based scoring – best_fields, most_fields, cross_fields
▪ Field boosting
▪ Phrasal boosting/shingling
▪ Synonym lists, taxonomies
▪ Similarity scoring parameters (with BM25)
▪ Elevate
▪ External signal enrichment
– manual or automatic (NLP – entity extraction, categorization, etc.)
▪ Reranking via machine learning (Learning to Rank)
| 4 |
© 2019 The MITRE Corporation. All rights reserved. For internal MITRE use

5. | 5 |
© 2019 The MITRE Corporation. All rights reserved.
Each Token Filter Can Have Many Parameters
<filter class="solr.WordDelimiterFilterFactory"
protected="protwords.txt"
generateWordParts="1"
generateNumberParts="1"
catenateWords="1"
catenateNumbers="1"
catenateAll="0"
splitOnCaseChange="0"
preserveOriginal="1"/>
| 5 |
© 2019 The MITRE Corporation. All rights reserved. For internal MITRE use

6. | 6 |
© 2019 The MITRE Corporation. All rights reserved.
Overview – Offline testing toolkit
Prerequisites:
1. Reliable, generalizable ground truth
2. Reliable, useful underlying data
3. Offline metric has to have some connection to KPIs
4. Expertise – you still have to know what you’re doing!!!

7. | 7 |
© 2019 The MITRE Corporation. All rights reserved.
Main Tools
1. Run Experiments
2. Generate Experiments
▪ All permutations (grid search)
▪ Random experiments (random search)
3. Genetic Algorithm
▪ Cross-fold validation!!!
▪ Complementary to LTR -- main diff is algorithm and in running offline to tune general settings rather
than as reranking top n

8. | 8 |
© 2019 The MITRE Corporation. All rights reserved.
Odds and Ends
▪ Analyzer Comparison over (mostly) the index
▪ Significant Terms (yawn…for archaic versions of Solr)…and planning to
add these as parameters in “generate experiments”

9. | 9 |
© 2019 The MITRE Corporation. All rights reserved.
Adding Porter Stemming: create account
creat
created: 709
create: 551
creating: 269
creates: 153
creat: 1
account
account: 3244
accounts: 1924
accounting: 1548
accountants: 340
accountant: 176
accounted: 134
accountability: 74
accountable: 74
accountancy: 65
account's: 7
accountant's: 7

10. | 10 |
© 2019 The MITRE Corporation. All rights reserved.
Status
▪ Alpha release 3/22/2019 (Solr only)
▪ Beta1 release this week (?)
– This will include support for ElasticSearch
▪ Dream
– Incorporate experiment generation/GA into Rated Ranking Evaluator (RRE)
– Apache Incubator -> Top Level Project (TLP)

11. | 11 |
© 2019 The MITRE Corporation. All rights reserved.
Links
▪ Main site: https://github.com/mitre/quaerite
▪ Examples: https://github.com/mitre/quaerite/blob/master/quaerite-
examples/README.md
▪ Contact
– tallison@apache.org
– @_tallison