1. © OpenSource Connections, 2018
The Solr Synonyms Maze
Haystack - The Search Relevance Conference
April 10-11, 2018
1
Bertrand Rigaldies
Search Consultant
OpenSource Connections
brigaldies@o19s.com
Linkedin: bertrandrigaldies
Multi-Terms Of Graphs

2. © OpenSource Connections, 2018
Agenda
1. Analysis Chain, Tokens, and Tokens Graph
Explained
2. Issue #1: Index-Time “Sausagization”
3. Issue #2: Graphs Interactions
4. Workarounds
5. Recommendations Recap
2

3. © OpenSource Connections, 2018
Tokens, and Tokens Graph
The “analysis chain” of content to index, or a
search string, produces a “tokens graph”
3
Synonyms:
scifi, sci fi, science fiction
End-user’s search/query:
q=scifi movies

4. © OpenSource Connections, 2018
4
Analysis
Chain,
and
Tokens
Separate terms (Tokenization) → tokens
Remove stop words
Lower case
Remove possessive form (‘s)
Protect selected terms against stemming
Stemming
Expand With Synonyms:
scifi, sci fi, science fiction

5. © OpenSource Connections, 2018
“Tokens Graph”
5
1 2 3 4 (End)
X --sci------>X --fi--------------------->X
X --science-----------------X--fiction--->X
X --scifi-------------------------------->X
4
(End)

6. © OpenSource Connections, 2018
6
Powered by Jupyter notebook
Query-Time Synonyms
Synonyms: scifi, sci fi, science fiction
q=sci fi movies
👍
Document: … science fiction …
defType=edismax
*** sow=false *** (Query-time: Split on whitespace)
qf=overview_query_syn
+(DisjunctionMaxQuery(((
(+overview_query_syn:sci +overview_query_syn:fi)
(+overview_query_syn:science
+overview_query_syn:fiction)
overview_query_syn:scifi)))
DisjunctionMaxQuery((overview_query_syn:movies)))
qf=overview_query_syn_auto_phrase
autoGeneratePhraseQueries=”true”
+(DisjunctionMaxQuery(((
overview_query_syn_auto_phrase:"sci fi"
overview_query_syn_auto_phrase:"science fiction"
overview_query_syn_auto_phrase:scifi)))
DisjunctionMaxQuery((overview_query_syn_auto_phrase:
movies)))

7. © OpenSource Connections, 2018
7Powered by Jupyter notebook
Query vs Index
Synonyms: scifi, sci fi, science fiction
q=sci fi movies
👍
Unexpected phrase matches!
“scifi fiction movies”
“scifi fi movies”
Unexpected not matching!
“scifi movies”
Query-Time Index-Time
All graph edges are forced
to the next node!
Document: …
scifi movies …
👍

8. © OpenSource Connections, 2018
Issue #1: Index-Time Multi-Terms
Synonyms “Sausage”
● First documented by Michael McCandless
blog in 2012:
○ “the indexer completely ignores PositionLength
attribute; ... This means the indexer acts as if all
arcs always arrive at the very next position”
● Also, nicely explained in this LucidWorks
2017 blog by Steve Rowe.
8

9. © OpenSource Connections, 2018
Issue #1 Workaround (1/2): Avoid Indexing
Multi-Terms Synonyms
9

10. © OpenSource Connections, 2018
10
Split the synonyms into separate files for single- and multi-terms:
<fieldType name="text_syn_split" …
autoGeneratePhraseQueries=”true”>
<analyzer type="index">
…
<filter … synonyms="synonyms_single.txt"/>
…
</analyzer>
<analyzer type="query">
…
<filter … synonyms="synonyms_multi.txt"/>
…
</analyzer>
</fieldType>
Issue #1 Workaround (1/2): Configuration
👍
+DisjunctionMaxQuery((((
overview_query_syn_auto_phrase:"sci fi"
overview_query_syn_auto_phrase:"science fiction"
overview_query_syn_auto_phrase:scifi))))

11. © OpenSource Connections, 2018
Issue #1 Workaround (2/2):
Semantic Unit Injection
syn1, syn2, … synN => __semantic_unit__
● Query-Time Injection:
Synonyms declaration: scifi, sci fi, science fiction => __science_fiction__
Query sci fi movies is analyzed as __science_fiction__ movies
The parsed query is
+(DisjunctionMaxQuery((overview_syn_sem_unit:__science_fiction__))
DisjunctionMaxQuery((overview_syn_sem_unit:movies)))
● Index-Time Injection:
Document: sci fi movie on the invasion of earth is analyzed as
__science_fiction__ movie on the invasion of earth
11

12. © OpenSource Connections, 2018
12
ATTENTION: Semantic units must be
injected at both Query and Index
times so that any one of the
synonyms in the search string and the
index do match!
Issue #1 Workaround (2/2 cont’d): Graph
q=”scifi movies”
q=”sci fi movies”
q=”science fiction movies”
👍
Document: scifi movies
Document: sci fi movies
Document: science fiction movies

13. © OpenSource Connections, 2018
Issue #1 Workaround (2b/2): Taxonomies
term1 term2 term => __hyponym_semantic_unit__, hypernym
● Query-Time Conversion:
Synonyms declaration: scifi, scifi, science fiction => __science_fiction__,
__fiction__
Query sci fi movies is analyzed as (__science_fiction__ __fiction__) movies
The parsed query is
+(DisjunctionMaxQuery((Synonym(overview_index_taxonomy:__fiction__
overview_index_taxonomy:__science_fiction__)))
DisjunctionMaxQuery((overview_index_taxonomy:movies)))
● Index-Time Conversion:
Document: sci fi movie on the invasion of earth is analyzed as
(__science_fiction__ __fiction__) movie ...
Document: political fiction movie on the rise of fascism is analyzed as
(__political_fiction__ __fiction__) movie ...
13

14. © OpenSource Connections, 2018
14
Issue #1 Workaround (2b/2 cont’d): Graph
q=sci fi movies
+(
DisjunctionMaxQuery((
Synonym(
overview_index_taxonomy:__fiction__
overview_index_taxonomy:__science_fiction__)
))
DisjunctionMaxQuery((
Overview_index_taxonomy:movies
))
)
Document: political fiction movie
Relevancy impact?
DF(__fiction__) >= DF(__science_fiction__)
Then, other movies only matching on __science__
have a lower score than the science fiction movies
matching on both __science__ and
__science_fiction__

15. © OpenSource Connections, 2018
Issue #2:
Intra-Analysis Graphs Interactions
Issue when chaining several graph-producing filters such as
SynonymGraphFilter and WordDelimiterGraphFilter in a field analysis
chain.
Not very well documented, or explained:
● Solr WordDelimiterGraphFilter documentation: “Note: although this
filter produces correct token graphs, it cannot consume an input token graph
correctly.” Not mentioned in the documentation of
SynonymGraphFilter!?
● Bottom of JIRA ticket LUCENE-6664: “This new syn filter still cannot
consume an arbitrary graph.”
● SynonymGraphFilter.java source code: “this cannot consume an incoming
graph; results will be undefined.”
15

16. © OpenSource Connections, 2018
Issue #2: Graph-Producing Filters Interaction
● Use Case #1: Several synonyms filters
<analyzer type="query">
…
<filter … synonyms="synonyms_multi.txt"/>
<filter … synonyms="synonyms_multi_2.txt"/>
…
</analyzer>
Synonyms_multi.txt:
scifi, sci fi, science fiction
Synonyms_multi_2.txt:
fiction, story telling
16
+DisjunctionMaxQuery(((((+overview_syn_split:sci +overview_syn_split:fi +overview_syn_split:telling)
(+overview_syn_split:science +overview_syn_split:fiction) (+overview_syn_split:science +overview_syn_split:story
+overview_syn_split:telling) (+overview_syn_split:scifi +overview_syn_split:telling)))))
👍
Terms overlap!

17. © OpenSource Connections, 2018
Issue #2: Graph-Producing Filters Interaction
17
● Use Case #1 Workaround:
✅ Do not chain several Synonyms-producing graph filters: Gather all multi-
terms synonyms in a single file.
✅ Be careful with any combos of synonyms files:
<filter … synonyms="synonyms_single.txt"/>
<filter … synonyms="synonyms_multi.txt"/>
fiction, fable
scifi, sci fi, science fiction
Avoid terms overlap!

18. © OpenSource Connections, 2018
Issue #2: Graph-Producing Filters Interaction
● Use Case #2: Word Delimiter + Synonyms Combo
<analyzer type="query">
…
<filter class="solr.WordDelimiterGraphFilterFactory"
protected="content_wdf_protected_terms.txt"
generateWordParts="1"
generateNumberParts="1"
catenateWords="1"
catenateNumbers="1"
catenateAll="1"
splitOnCaseChange="1"
stemEnglishPossessive="1"
preserveOriginal="1"/>
…
<filter … synonyms="synonyms_multi.txt"/>
…
</analyzer>
18
scifi, sci fi, science fiction
Sci-fi →
1 2 3
--sci-fi------------>
--scifi------------->
--sci------>--fi---->

19. © OpenSource Connections, 2018
Issue #2: Graph-Producing Filters Interaction
Use Case #2 (Cont’d)
19
+DisjunctionMaxQuery(((((+overview_wdf_query_syn:sci +overview_wdf_query_syn:fi +overview_wdf_query_syn:sci +overview_wdf_query_syn:fi)
(+overview_wdf_query_syn:sci +overview_wdf_query_syn:fi +overview_wdf_query_syn:science +overview_wdf_query_syn:fiction)
(+overview_wdf_query_syn:sci +overview_wdf_query_syn:fi +overview_wdf_query_syn:scifi) (+overview_wdf_query_syn:science
+overview_wdf_query_syn:fiction +overview_wdf_query_syn:sci +overview_wdf_query_syn:fi) (+overview_wdf_query_syn:science
+overview_wdf_query_syn:fiction +overview_wdf_query_syn:science +overview_wdf_query_syn:fiction) (+overview_wdf_query_syn:science
+overview_wdf_query_syn:fiction +overview_wdf_query_syn:scifi) (+overview_wdf_query_syn:science +overview_wdf_query_syn:sci-fi
+overview_wdf_query_syn:fi) (+overview_wdf_query_syn:scifi +overview_wdf_query_syn:sci +overview_wdf_query_syn:fi)
Word Delimiter-produced
Graph
Synonym-produced Graph
next next
q=sci-fi
👍

20. © OpenSource Connections, 2018
Issue #2: Graph-Producing Filters Interaction
20
● Use Case #2 Workaround: Configure the Word Delimiter Graph filter to NOT
produce a graph:
<analyzer type="query">
…
<filter class="solr.WordDelimiterGraphFilterFactory"
protected="content_wdf_protected_terms.txt"
generateWordParts="1"
generateNumberParts="1"
catenateWords="0"
catenateNumbers="0"
catenateAll="0"
splitOnCaseChange="1"
stemEnglishPossessive="1"
preserveOriginal="0"/>
…
<filter … synonyms="synonyms_multi.txt"/>
…
</analyzer>
Sci-fi →
1 2 3
--sci------>--fi---->

21. © OpenSource Connections, 2018
Issue #2: Graph-Producing Filters Interaction
Use Case #2 Workaround Graphs
21
q=sci-fi
next next next
+DisjunctionMaxQuery(((((+overview_wdf_no_concat_query_syn:sci
+overview_wdf_no_concat_query_syn:fi) (+overview_wdf_no_concat_query_syn:science
+overview_wdf_no_concat_query_syn:fiction) overview_wdf_no_concat_query_syn:scifi))))
👍👍

22. © OpenSource Connections, 2018
What if you need all parts produced by the Word Delimiter, not just the split parts?
For example, let’s say we have 3 documents titled:
1. Lowcost Appliances Repair
2. Low Cost Appliances Parts
3. Affordable Appliances Fixing
End-users search with: q=low-cost, lowcost, or low cost
We have the synonyms: low cost, affordable
Our title analysis chain used the White Space tokenizer, Word Delimiter with split-parts only, and a Synonyms filter.
Then, q=lowcost matches document #1 only.
lowcost → [WT] lowcost → [WDGF] lowcost → [SGF] lowcost
q=low cost matches document #2 and #3 only!
low cost → [WT] low | cost → [WDGF] low | cost → [SGF] (low affordable) | cost
q=low-cost matches document #2 and #3 only!
low-cost → [WT] low-cost → [WDGF] low | cost → [SGF] (low affordable) | cost
22
Issue #2: Graph-Producing Filters Interaction
Use Case #2 Workaround Variation

23. © OpenSource Connections, 2018
● Generally, we like the decompounding done by Word Delimiter.
● So, we’ll let it generate all split and concatenated parts, e.g., for “a-b”:
○ (a ab a-b) | b
● But, for those terms also present in synonyms, we’ll protect them from
decompounding in the Word Delimiter (see the protected parameter),
and do the decompounding in the synonyms file:
○ low-cost, lowcost, low cost, affordable
23
Issue #2: Graph-Producing Filters Interaction
Use Case #2 Workaround Variation (Cont’d)
Search string WT WDGF SGF
lowcost lowcost lowcost (lowcost low-cost low affordable) | cost
low cost low | cost low | cost (lowcost low-cost low affordable) | cost
low-cost low-cost low-cost
(protected)
(lowcost low-cost low affordable) | cost

24. © OpenSource Connections, 2018
The “Flatten Graph” Filter
24
<analyzer type="index">
…
<filter class="solr.WordDelimiterGraphFilterFactory" … />
<filter class="solr.FlattenGraphFilterFactory"/>
…
<filter class="solr.SynonymGraphFilterFactory" … synonyms="synonyms_multi_sem_unit.txt"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.SynonymGraphFilterFactory" … synonyms="synonyms_single.txt"/>
<filter class="solr.FlattenGraphFilterFactory"/>
…
</analyzer>

25. © OpenSource Connections, 2018
The “Flatten Graph” Filter (Cont’d)
25
Synonyms Expansion Graph Flattening
next
Matching phrases: “scifi fi”
“scifi fiction”
“sci fiction”
“science fi”

26. © OpenSource Connections, 2018
Multi-terms Synonyms: Recommendations
● Split single- and multi-terms synonyms in separate files for configuration
flexibility.
○ Index-time single-terms synonyms
○ Index-time semantic units-based synonyms
○ Query-time multi-terms synonyms (sow=false!)
● Watch out for combinations of filters that produce non-flat graphs that
overlap for one or more terms:
○ Two or more SynonymGraphFilter
○ WordDelimiterFilter (Split parts only) + SynonymGraphFilter
26

27. © OpenSource Connections, 2018
A Working Configuration<analyzer type="index">
<charFilter … />
<tokenizer
class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter
class="solr.WordDelimiterGraphFilterFactory"
protected="wdf_protected_terms.txt"
generateWordParts="1"
generateNumberParts="1"
catenateWords="0"
catenateNumbers="0"
catenateAll="0"
splitOnCaseChange="1"
stemEnglishPossessive="1"
preserveOriginal="0"/>
<filter class="solr.FlattenGraphFilterFactory"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter
class="solr.EnglishMinimalStemFilterFactory"/>
<filter … synonyms="synonyms_single.txt"/>
<filter … synonyms="synonyms_sem_units.txt"/>
<filter class="solr.FlattenGraphFilterFactory"/>
… 27
<analyzer type="query">
<charFilter … />
<tokenizer
class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter
class="solr.WordDelimiterGraphFilterFactory"
protected="wdf_protected_terms.txt"
generateWordParts="1"
generateNumberParts="1"
catenateWords="0"
catenateNumbers="0"
catenateAll="0"
splitOnCaseChange="1"
stemEnglishPossessive="1"
preserveOriginal="0"/>
<filter class="solr.ASCIIFoldingFilterFactory"/>
<filter class="solr.KeywordMarkerFilterFactory"
protected="protwords.txt"/>
<filter
class="solr.EnglishMinimalStemFilterFactory"/>
<filter … synonyms="synonyms_multi.txt"/>
…
</analyzer>

28. © OpenSource Connections, 2018
Thank You
28