Art and Science Come Together When Mastering Relevance Ranking - Tom Burgmans, Wolters Kluwer

Art and Science come together
when mastering Relevance Ranking
Tom Burgmans

01
2
01
Our purpose: Wolters Kluwer provides information, software, and services to
legal, business, tax, accounting, ﬁnance, audit, risk, compliance, and healthcare professionals
Employees: 19,000 Employees with operations in over 40 countries
Revenue: € 4.3 billion (2016)
Tom Burgmans
Search Specialist
tom.burgmans@wolterskluwer.com

Agenda
•  Create a customizable scoring algorithm
•  Best practice examples of tuning elements of this algorithm

6
01
Ingredients for baking a relevance cake
Understand of what relevance means
Prepare the content
Control the scoring algorithm
A process & culture in the organization to structurally improve relevance
Domain experts
Judgement lists
User behavioral data
Entity detection
Classification
Vocabulary management
Construct the query
Single field / multi-field search
Term-centric vs field-centric
Grammar parsing
(External) query processing
Re-shape and tune the scoring algorithm:
• Customized similarity
• Function queries
Understand the EXPLAIN feedback:
• Querylog tool
Prepare the Solr configuration
How to tokenize
Balance recall vs precision via the analyzers
Feature modeling
Institutionalize engineering of relevance through the organization
Measure expectation vs reality via a diff with Golden Answers
Automated relevance tests
Work iterative

01
Score (ClassicSimilarity) =
Multiplicative function query *
coord(q,d) *
queryNorm(q) *
(
Additive function query +
∑(for each term)
(
TermBoost(t) *
FieldBoost(d) *
TF(t in d) *
IDF(t)2 *
Norm(t,d) *
Payload(t)
)
)
Score (BM25Similarity) =
(
∑(for each term)
(
TermBoost(t) *
FieldBoost(d) *
TF(t in d, Norm) *
IDF(t)2 *
Payload(t)
)
)
>= Solr 6.0< Solr 6.0
1/
√ 𝑠𝑢𝑚𝑂𝑓
𝑆𝑞𝑢𝑎𝑟𝑒𝑑
𝑊𝑒𝑖𝑔ℎ𝑡𝑠 

𝑜𝑣𝑒𝑟
𝑙𝑎𝑝/
𝑀𝑎𝑥
𝑂𝑣𝑒
𝑟𝑙𝑎𝑝 
√⁠ 𝑓 𝑟𝑒𝑞/
(1+ 𝑑𝑖𝑠𝑡
𝑎𝑛𝑐𝑒)  
1+ ln⁠(
𝑛𝑢𝑚𝐷𝑜
𝑐𝑠/
𝑑𝑜𝑐𝐹𝑟𝑒

≈ 𝑖𝑛𝑑𝑒𝑥𝑏𝑜
𝑓 𝑟𝑒𝑞/
(1+ 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒) 
∗( 𝑘1+1) / 𝑓 𝑟𝑒𝑞/
ln(
𝑛𝑢𝑚𝐷𝑜𝑐𝑠
− 𝑑𝑜𝑐𝐹𝑟𝑒𝑞
+0.5/
𝑑𝑜𝑐𝐹𝑟𝑒𝑞
+0.5 )⁠ 
≈⁠1/
𝑛𝑢𝑚
𝑇𝑒𝑟𝑚

01
Create your own customizable Similarity class
MySimilarityFactory.java
MySimilarity.java
Make jar
Reference jar from solrconfig.xml
Extends SchemaSimilarityFactory
Creates MySimilarity instance. Passes parameters from schema
Extends BM25Similarity
Customizations to the algorithm happens here
Add on bottom schema.xml
<similarity class="solr.SchemaSimilarityFactory“ />
Add inside fieldType definition
<fieldType ...>
<analyzer > ... </analyzer>
<similarity class="com.mycompany.MySimilarityFactory">
<str name="tuneIDF">1.0</str> 
<str name="tuneSloppyFreq">0</str> 
<str name="tunePayload">3</str> 
<str name="k1">1.5</str> 
<str name="b">1.0</str> 
</similarity>
</fieldType>
Delegate similarity to per-field setting
Set tuning parameters

TF
Classic TF = √⁠ 𝒇 𝒓𝒆𝒒 = 𝒇 𝒓𝒆𝒒↑ 𝟎.𝟓 
ClassicSimilarity.java
public float tf(float freq) {
return (float)Math.sqrt(freq);
}
Classic TF =
tunable 𝒇 𝒓𝒆𝒒↑𝒕𝒖𝒏𝒆𝑻𝑭 
𝟎 ← 𝒕𝒖𝒏𝒆𝑻
public float tf(float freq) {
return (float)Math.pow(freq,tuneTF);
}
MySimilarity.java
Tune in the schema

01
𝟎 ← 𝒌 𝟏 → >𝟎
𝟎 ← 𝒃 →𝟏
BM25 TF = 𝒇 𝒓𝒆𝒒∗( 𝒌 𝟏+𝟏) /𝒇𝒓𝒆𝒒+(𝒌 𝟏∗( 𝟏− 𝒃+ 𝒃
L=≈ 𝑓𝑖𝑒𝑙𝑑𝐿𝑒𝑛𝑔𝑡ℎ/
tunable
Tune term-frequency saturation
Tune sensitivity saturation
towards field length
TF

𝟎 ← 𝒌 𝟏 𝒂 → >𝟎
BM25 TF = 𝒇 𝒓𝒆𝒒∗( 𝒌 𝟏 𝒂+𝟏) /𝒇𝒓𝒆𝒒+(𝒌 𝟏 𝒃∗( 𝟏− 𝒃+ 𝒃∗ 𝑳))
L=≈ 𝑓𝑖𝑒𝑙𝑑𝐿𝑒𝑛𝑔𝑡ℎ/𝑎𝑣𝑔𝐹𝑖𝑒
Tune TF impact
independent from saturation
better tunable
𝟎 ← 𝒌 𝟏 𝒃 → >𝟎
Tune term-frequency saturation
independent from TF impact
TF

01
𝒍 𝒏⁠(
𝒏𝒖𝒎𝑫𝒐𝒄𝒔/
( 𝒅𝒐𝒄𝑭𝒓𝒆𝒒
+𝟏) )+𝟏 
Classic IDF =
BM25 IDF =
𝒍 𝒏⁠( 𝒏 𝒖𝒎𝑫𝒐𝒄𝒔
− 𝒅𝒐𝒄𝑭𝒓𝒆𝒒
+𝟎.𝟓/
𝒍 𝒏⁠( 𝒏 𝒖𝒎𝑫𝒐𝒄𝒔/( 𝒅𝒐𝒄𝑭𝒓𝒆𝒒+𝟏Classic IDF =
BM25 IDF =
tunable
𝒍 𝒏⁠( 𝒏 𝒖𝒎𝑫𝒐𝒄𝒔 − 𝒅𝒐𝒄𝑭𝒓𝒆𝒒+𝟎tunable
𝟎 ← 𝒕𝒖𝒏𝒆𝑰𝑫
https://www.desmos.com/calculator/wfpvne0z1c
IDF
In this example: numDocs = 100

01
• Tune the impact of queryNorm and coord (in case of ClassicSimilarity)
Or just set them to a hard 1, like I did
• Tune the impact of distance on TF
In case of a phrase + slop query
• Tune the curve & mapping of Payloads
This is a hard 1 by default
• Tune the 255 entries in the Norm byte
Small range for small fields, big range for big fields => higher precision for each case
Other items you might want to make tunable via MySimilarity

01
coord(q,d) *
queryNorm(q) *
(
∑(for each term)
(
TermBoost(t) *
FieldBoost(d) *
TF(t in d) *
IDF(t)2 *
Norm(t,d) *
Payload(t)
)
)
(
∑(for each term)
(
TermBoost(t) *
FieldBoost(d) *
TF(t in d, Norm) *
IDF(t)2 *
Payload(t)
)
)
>= Solr 6.0< Solr 6.0
i.e.
Ratings
Pagerank
Proﬁtability
Popularity
Document type
Page views
Geo-distance
Freshness
i.e.
Implicit proximity
Exact title match
Shingle match
Author name in author ﬁeld
Extending
the terms score
Amplifying
the terms score

01
Function queries: index time boost
<field
name="docboost"
type="int"
indexed="false"
stored="false"
docValues="true"
/>

<add>

<doc>

<field
name="id">id1</field>

<field
name="body">Life
is
like
riding
a
bicycle.</field>

<field
name="docboost">2</field>

</doc>

<doc>

<field
name="id">id2</field>

<field
name="body">To
keep
your
balance,
you
must
keep
moving.</field>

<field
name="docboost">5</field>

</doc>

</add>

q=*:*&boost=ﬁeld(docboost)

q=*:*&bf=ﬁeld(docboost)

additive
multiplicative
doctype
docboost

Archived
0

Form
1

Pending-‐legisla?on
2

Case-‐law
3

Ruling
4

Annota?on
5

News
6

Prac?ce-‐aid
7

Regula?on
8

Law
9

Explana?ons
10

domain
expert

Tuning multiplicative boost?
tuneBoost = 0
Scale up or down by summing?
boost=sum(field(docboost),$tuneBoost)
Tuning additive boost
bf=product(field(docboost),$tuneBoost)
nope
bigger effect?
tuneBoost: -5 ß 0
Scale up or down by multiplying
smaller effect
tuneBoost: 0 à 5
but not consistently…
tuneBoost = 1
smaller effect
tuneBoost: 1 à 0.2
bigger effect
tuneBoost: 1 à 5

Relative difference between multiplicative boost values
docboost

0

1

2

3

4

5

6

7

8

9

10

inﬁnite
100%
differenc
e
50%
33%
25%
20%
17%
14%
12%
11%
docboost
‘translated
’
0
1

1
2

2
4

3
8

4
16

5
32

6
64

7
128

8
256

9
512

10
1024

docboost
‘translated
’ & tuned
0
1,00

1
1,15

2
1,32

3
1,52

4
1,74

5
2,00

6
2,30

7
2,64

8
3,03

9
3,48

10
4,00

100%
100%
differenc
e
100%
100%
100%
100%
100%
100%
100%
100%
2↑( 𝑑𝑜𝑐𝑏𝑜𝑜𝑠𝑡) 
01
15%
15%
differenc
e
15%
15%
15%
15%
15%
15%
15%
15%
2↑( 𝑑𝑜𝑐𝑏𝑜𝑜𝑠
𝒕𝒖𝒏
𝒆𝑩
𝒐𝒐𝒔
𝒕

01
Tuning multiplicative boost
tuneBoost = 0.2
bigger effectsmaller effect
tuneBoost: 0.2 à 0 tuneBoost: 0.2 à 0.4
boost=pow(2,product(field(docboost),$tuneBoost)))
boost =2↑( 𝑑𝑜𝑐𝑏𝑜𝑜𝑠𝑡 ∗ 𝒕𝒖𝒏𝒆𝑩𝒐𝒐𝒔𝒕)

01
docid
docboost

doc0
0

doc1
1

doc2
2

doc3
3

doc4
4

doc5
6

doc6
8

doc7
9

doc8
99

doc9
999

doc10
9.999

doc11
99.999

doc12
999.999

Index time boost with extreme values tamed and tunable
‘translated’

1

1,07

1,12

1,15

1,17

1,21

1,25

1,27

1,59

2,00

2,51

3,16

3,98
boost=pow(field(docboost),$tuneBoost)
( 𝑑𝑜𝑐𝑏𝑜𝑜𝑠𝑡
+1)↑𝒕𝒖𝒏𝒆𝑩𝒐𝒐𝒔𝒕 
tuneBoost = 0.1

01
docid
docboost
‘translated’

doc0
0
1

doc1
1
1,07

doc2
2
1,12

doc3
3
1,15

doc4
4
1,17

doc5
6
1,21

doc6
8
1,25

doc7
9
1,27

doc8
99
1,59

doc9
999
2,00

doc10
9.999
2,51

doc11
99.999
3,16

doc12
999.999
3,98
boost=pow(field(docboost),$tuneBoost)
+1)↑𝒕𝒖𝒏𝒆𝑩𝒐𝒐𝒔𝒕 
Index time boost with extreme values tamed and tunable
smaller effect
bigger effect
tuneBoos
t = 0.14
tuneBoost = 0.1
tuneBoos
t = 0.05
26%
26%
26%
26%
26%

01
coord(q,d) *
queryNorm(q) *
(
∑(for each term)
(
TermBoost(t) *
FieldBoost(d) *
TF(t in d) *
IDF(t)2 *
Norm(t,d) *
Payload(t)
)
)
(
∑(for each term)
(
TermBoost(t) *
FieldBoost(d) *
TF(t in d, Norm) *
IDF(t)2 *
Payload(t)
)
)
>= Solr 6.0< Solr 6.0
1/
√ 𝑠𝑢𝑚𝑂𝑓𝑆
𝑞𝑢𝑎𝑟𝑒𝑑𝑊𝑒
𝑖𝑔ℎ𝑡𝑠  OR 1

𝑜𝑣𝑒𝑟𝑙
𝑎𝑝/
𝑀𝑎𝑥𝑂
𝑣𝑒𝑟𝑙𝑎
𝑝 OR 1
( 𝑓 𝑟𝑒𝑞/
1+ 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒∗
𝒕𝒖𝒏𝒆𝑫𝒊𝒔𝒕𝒂𝒏
𝒄𝒆 )↑𝒕𝒖𝒏𝒆𝑻𝑭 
1+ ln⁠(
𝑛𝑢𝑚𝐷𝑜𝑐𝑠/
𝑑𝑜𝑐𝐹𝑟𝑒𝑞
+1 )∗ 𝒕𝒖𝒏𝒆𝑰
≈⁠1/
𝑛𝑢𝑚𝑇𝑒𝑟𝑚𝑠
𝑓 𝑟𝑒𝑞/
(1+ 𝑑𝑖𝑠𝑡𝑎𝑛𝑐𝑒∗ 𝒕𝒖𝒏𝒆𝑫𝒊𝒔𝒕𝒂
𝒏𝒄𝒆) ∗( 𝒌 𝟏 𝒂+1) / 𝑓 𝑟𝑒𝑞/
ln( 𝑛 𝑢𝑚𝐷𝑜𝑐𝑠
− 𝑑𝑜𝑐𝐹𝑟𝑒𝑞+0.5/
𝑑𝑜𝑐𝐹𝑟𝑒𝑞+0.5 )⁠
∗ 𝒕𝒖𝒏𝒆𝑰𝑫𝑭
+1− 𝒕𝒖𝒏𝒆𝑰𝑫𝑭 
≈⁠1/
𝑛𝑢𝑚
𝑇𝑒𝑟
𝑚𝑠↑
𝒕𝒖𝒏𝒆
𝑵𝒐𝒓𝒎
𝒃𝒚𝒕𝒆
𝒑𝒐𝒑𝒖
𝒍𝒂𝒕𝒊𝒐
𝒏

2↑( 𝑑𝑜𝑐𝑏𝑜
𝑜𝑠𝑡 ∗
𝒕𝒖𝒏𝒆𝑩𝒐𝒐
𝒔𝒕) 
+1)↑𝒕𝒖𝒏𝒆𝑩
𝒐𝒐𝒔𝒕 
𝑑𝑜𝑐𝑏𝑜𝑜𝑠
𝑡
∗ 𝒕𝒖𝒏𝒆𝑩
𝒐𝒐𝒔𝒕

01
After a certain high level of technical skill is achieved,
science and art tend to coalesce in esthetics, plasticity,
and form.
The greatest scientists are artists as well.
Albert Einstein

01
One more thing…
Expected soon:
Preview?
contact
tom.burgmans@wolterskluwer.com

Art and Science Come Together When Mastering Relevance Ranking - Tom Burgmans, Wolters Kluwer

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Art and Science Come Together When Mastering Relevance Ranking - Tom Burgmans, Wolters Kluwer

Similar to Art and Science Come Together When Mastering Relevance Ranking - Tom Burgmans, Wolters Kluwer (20)

More from Lucidworks

More from Lucidworks (20)

Recently uploaded

Recently uploaded (20)

Art and Science Come Together When Mastering Relevance Ranking - Tom Burgmans, Wolters Kluwer