This document analyzes smoothing methods used in pseudo-relevance feedback (PRF) models for language retrieval. It shows that collection-based smoothing causes PRF models to select very common words, while additive smoothing prevents this by adding pseudo-counts. Through axiomatic analysis and empirical experiments on three PRF models, the document demonstrates that additive smoothing outperforms collection-based smoothing by supporting an inverse document frequency effect and increasing retrieval performance and term discrimination.

1. 10/1/2015
Axiomatic Analysis of Smoothing Methods in
Language Models for Pseudo-Relevance Feedback
HUSSEIN HAZIMEH AND CHENGXIANG ZHAI
UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN
1

2. Pseudo Relevance Feedback
Judgments:
d1 +
d2 -
d3 +
…
dk -
...
Query Retrieval
Engine
Results:
d1 3.5
d2 2.4
…
dk 0.5
...
User
Document
collection
Judgments:
d1 +
d2 +
d3 +
…
dk -
...
top 10
Pseudo feedback
Assume top 10 docs
are relevant
Relevance feedback User judges documents
New
Query
Feedback
Learn from
Examples
2

3. Pseudo-Relevance Feedback
It’s blind!
Good for high recall information needs
A Blind Superhero. Courtesy of iStock
3

4. Collection-based Smoothing
Collection-based smoothing is generally used for LM-based retrieval
functions and for PRF models
A commonly used collection-based smoothing scheme is Dirichlet
prior smoothing:
Dirichlet Prior
(Smoothing Parameter)
Document Length
Count of Word in
Document
4

5. Study of Smoothing Methods in PRF
We will establish both analytically and empirically that collection-
based smoothing is not a good choice for PRF:
◦ It forces PRF models to select very common words
Additive smoothing will be shown to outperform the collection-
based counterpart
5

6. How Do LM PRF Models Work?
D1
Dn
… Averaging
Function: 𝑨
Scoring
Function: 𝒇
𝑃(𝑤|𝜃1)
𝑃(𝑤|𝜃 𝑛)
𝑃(𝑤|𝜃 𝐶)
𝑃(𝑤|𝜃 𝐹)
6

7. How Do LM PRF Models Work?
 The feedback LM, 𝜃 𝐹, would generally have the following form:
𝐴: ℝ 𝑛
→ ℝ is an averaging function, e.g. geometric mean
𝑓: ℝ2 → ℝ is a function increasing in the first argument and
decreasing in the second
Rewards common
words in feedback set
Penalizes common
words in collection
7

8. Problem!
The first argument rewards common words in the collection while the
second penalizes them. The analysis shows that the first argument
usually “wins”!
Rewards common
words in feedback set
and collection
Penalizes common
words in collection
Proportional to
𝑃(𝑤|Θ 𝐶)
8

9. Overview of the Analysis
We considered three PRF models in the study:
◦ Divergence Minimization Model
◦ Relevance Model
◦ Geometric Relevance Model
Next, we will briefly discuss how the DMM and GRM work and then
give an overview of the axiomatic analysis.
The analysis of the RM is very similar to the GRM and the same
results apply
9

10. Divergence Minimization Model
(Zhai and Lafferty, 2001)
The DMM solves the following optimization problem:
The solution has a closed form and is given by:
10

11. Geometric Relevance Model
(Seo and Croft, 2010)
An enhanced form of the Relevance Model (RM) that replaces the
arithmetic mean used in RM by the geometric mean:
Note that the function above is not is not affected by 𝑃(𝑤|𝜃𝑐), i.e.,
the model is not designed to penalize common words.
11

12. Main Axiom: IDF Effect
(Clinchant and Gaussier, 2013)
 Rationale: A PRF model is expected to penalize common words in
the collection in order to select high quality discriminative terms.
 Given any two words 𝑤1and 𝑤2 from the feedback set 𝐷1, 𝐷2,
12

13. DMM with Collection-based smoothing: IDF
Effect
Study the sign of:
Not straightforward. Strategy:
◦ Find an attainable lower bound on the expression above
◦ Study the sign of the lower bound
◦ If the lower bound is strictly positive, then DMM supports the IDF effect
13

14. DMM: Results of Analysis
Conclusion: Using collection-based smoothing the DMM will be either
consistently reward common terms or will select only one feedback term
14

15. GRM with Collection-based smoothing: IDF
Effect
The GRM cannot support the IDF effect:
It consistently rewards favors common words in the collection
15

16. Proposed Solution: Additive Smoothing
 Words get additional pseudo-counts:
 Next, we show how additive smoothing prevents the models from
rewarding common terms
16

17. DMM with Additive Smoothing: IDF Effect
The DMM unconditionally supports the IDF Effect:
Now it is performing the intended objective!
17

18. DMM: Empirical Validation
Query: “Computer”
18

19. GRM with Additive Smoothing: IDF Effect
Although the IDF effect is still not supported:
However, common terms are no longer being rewarded!
19

20. GRM: Empirical Validation
Query: “Computer”
20

21. Empirical Evaluation: Retrieval Measures
21

22. Empirical Evaluation: Robustness of
Additive Smoothing
22

23. Measuring the Discrimination of PRF Models
In previous studies, the average of the IDF of the top terms was used
as an indicator of how discriminative the terms selected by a PRF
method are
Such a measure might not work well in some cases
We propose the Discrimination Measure (DM):
≈ Expected Document Frequency
Constant
23

24. Empirical Evaluation: Discrimination Measure
A several-fold decrease in the expected document frequency
24

25. Conclusion
Collection-based smoothing forces PRF models to select very
common terms
◦ The same problem might exist in other applications where LMs are aggregated
Additive smoothing prevents PRF models from rewarding common
terms and increases the retrieval performance significantly
A new measure for quantifying PRF Discrimination
25

26. Future Work
Should PRF models penalize common words?
Analysis of other smoothing methods such as topic-based smoothing
Inspect areas, other than PRF, where collection-based smoothing is
used in aggregating language models
26

27. Thanks to SIGIR for the Student Travel Grant!
Thank you for Listening!
27