This talk was given during Activate Conference 2019. Lucene has a lot of options for configuring similarity, and Solr inherits them. Similarity makes the base of your relevancy score: how similar is this document to the query? The default similarity (BM25) is a good start, but you may need to tweak it for your use-case. In this session, you will learn how BM25 works and how you may want to change its parameters. Then, we'll move to other similarity classes: DFR, DFI, IB and LM. You will learn the thinking behind them, how that thinking translates to the similarity score, and which parameters allow you to tweak how score evolves based on things like term frequency or document length. By the end, you’ll have a good understanding of which similarity options are likely to work well for your use-case. You'll know which tunables are available and whether you need to implement a custom similarity class. As an example, we’ll focus on E-commerce, where you often end up ignoring term frequency altogether. Key Takeaway 1) What are the built-in Lucene/Solr similarities and what they do 2) Which similarity to use for which use-case 3) How to use a custom similarity class in Solr Learn more about search relevance and similarity: sematext.com/blog/search-relevance-solr-elasticsearch-similarity

2. Tweaking the Base Score:
Lucene/Solr Similarities Explained
Demo: github.com/sematext/activate/tree/master/2019
More info: sematext.com/blog/search-relevance-solr-elasticsearch-similarity
Radu
Gheorghe
Rafał
Kuć
www.sematext.com

3. Agenda
BM25 - Best Match: the default
DFR - Divergence From Randomness framework
DFI - Divergence From Independence
IB - Information-Based models
LM - Language Models
Custom similarity
Putting it all together

4. TF*IDF
You know, for historical reasons

5. BM25 - the TF part
freq / (freq + k1 * (1 - b + b * dl / avgdl))
Best for Most 😁

6. BM25 tunables
freq / (freq + k1 * (1 - b + b * dl / avgdl))
k1 - raise or lower ceiling

7. BM25 tunables
freq / (freq + k1 * (1 - b + b * dl / avgdl))
doc length normalization

8. BM25 demo
yes, that’s how we look
when we give demos

9. BM25
Good default. You can
tune the weight of freq
and docLength.

10. Divergence From Randomness
Basic Model
G, I(n), I(ne), I(F)
After Effect
L, B
Normalization
H1, H2, H3, Z, none

11. tf * c * avgFieldLength / docFieldLength
Divergence From Randomness - H1

12. Divergence From Randomness - H1
No normalization, and H1 with c == 1, 3, 5, 7

13. tf * log2
(1 + c * (avgFieldLength / docFieldLength))
Divergence From Randomness - H2

14. Divergence From Randomness - H2
No normalization, and H2 with c == 1, 3, 5, 7

15. tf * (avgFieldLength / docFieldLength)Z
Divergence From Randomness - Z

16. Divergence From Randomness - Z
No normalization, and Z with z == 0.1, 0.2, 0.3, 0.4

17. (tf * mu * ((totalTermFreq + 1) / (#fieldTokens + 1)))
(docFieldLength + mu) * mu
Divergence From Randomness - H3

18. Divergence From Randomness - H3
No normalization, and H3 with mu == 1, 3, 5, 7

19. DFR demo
Only one, I promise

20. DFR
Framework. Tunable:
choose algorithm and
tune parameters for
both IDF* and
docLength.
* generic name for importance
of this term

21. Divergence From Independence
expected frequency

22. Divergence From Independence
docLength*totalTermFrequency/numberOfFieldTokens
expected frequency

23. DFI: Standardized
(actual - expected)/sqrt(expected)

24. DFI demo
Oh, but don’t remove
stopwords*!
1) arbitrarily chops field length
2) stopwords aren’t always
stopwords ;)

25. DFI
Simple. Parameterless.
Flexible: works well
with various datasets.

26. Information Based
how much information we get from this term?

27. Information Based
Distribution
Log-Logistic, Smoothed Power-Law
Lambda
DF, TTF
Normalization
H1, H2, H3, Z, none

28. Information Based - Log-Logistic
log( tfn / (lambda + 1) )

29. Information Based - Log-Logistic
lambda: 0.1 (red), 0.3 (black), 0.8 (blue)

30. Information Based - Retrieval Function
the average of the document information brought
by each query term

31. Information Based - Retrieval Function - DF
number of matching documents
(docFrequency + 1) / (numberOfDocuments + 1)

32. Information Based - Retrieval Function - TTF
total number of term occurrences
(totalTermFrequency + 1) / (numberOfDocuments + 1)

33. IB demo

34. IB
Framework. like DFR.
Even has the same
normalization options.
But newer and, in the
paper, better.

35. Language Models
probability of a term being our term

36. Language Models
totalTermFreq/totalFieldTokens
probability of a term being our term

37. Language Models: Jelinek-Mercer
log(
(1-λ)*
tf
)docLength
λ * probability

38. LM demo
feat. Jelinek-Mercer

39. LM
Two probabilistic
models. Similar
approach to DFI, but
tunable.

40. Custom Similarity
compute a similarity score using custom code

41. Custom Similarity - Activate Similarity Factory
public class ActivateSimilarityFactory extends SimilarityFactory {
private volatile Similarity similarity;
public void init(SolrParams params) {
super.init(params);
}
public Similarity getSimilarity() {
if (similarity == null) {
similarity = new ActivateSimilarity();
}
return similarity;
}
}

42. Custom Similarity - Activate Similarity Factory
public class ActivateSimilarityFactory extends SimilarityFactory {
private volatile Similarity similarity;
public void init(SolrParams params) {
super.init(params);
}
public Similarity getSimilarity() {
if (similarity == null) {
similarity = new ActivateSimilarity();
}
return similarity;
}
}

43. Custom Similarity - Activate Similarity Factory
public class ActivateSimilarityFactory extends SimilarityFactory {
private volatile Similarity similarity;
public void init(SolrParams params) {
super.init(params);
}
public Similarity getSimilarity() {
if (similarity == null) {
similarity = new ActivateSimilarity();
}
return similarity;
}
}

44. Custom Similarity - Similarity
public class ActivateSimilarity extends Similarity {
public ActivateSimilarity() {}
public long computeNorm(FieldInvertState state) { return 1; }
public Similarity.SimScorer scorer(float boost,
CollectionStatistics collectionStats, TermStatistics... termStats) {
return new ActivateSimScorer();
}
}

45. Custom Similarity - Similarity
public class ActivateSimilarity extends Similarity {
public ActivateSimilarity() {}
public long computeNorm(FieldInvertState state) { return 1; }
public Similarity.SimScorer scorer(float boost,
CollectionStatistics collectionStats, TermStatistics... termStats) {
return new ActivateSimScorer();
}
}

46. Custom Similarity - Similarity
public class ActivateSimilarity extends Similarity {
public ActivateSimilarity() {}
public long computeNorm(FieldInvertState state) { return 1; }
public Similarity.SimScorer scorer(float boost,
CollectionStatistics collectionStats, TermStatistics... termStats) {
return new ActivateSimScorer();
}
}

47. Custom Similarity - SimScorer
public class ActivateSimScorer extends Similarity.SimScorer {
public float score(float freq, long norm) {
return freq;
}
}

48. Custom Similarity - SimScorer
public class ActivateSimScorer extends Similarity.SimScorer {
public float score(float freq, long norm) {
return freq;
}
}

49. Custom
Similarity
demo

50. Custom
When you need
something special, like
disregarding term
frequency.

51. Multiple
similarities
demo

53. THANK YOU