Divergence from randomness, one of the very first models, is one type of probabilistic model. It is basically used to test the amount of information carried in the documents. It is based on Harter's 2-Poisson indexing-model. The 2-Poisson model has a hypothesis that the level of the documents is related to a set of documents which contains words occur relatively greater than the rest of the documents

1. Information Retrieval : 20
Divergence from Randomness
Prof Neeraj Bhargava
Vaibhav Khanna
Department of Computer Science
School of Engineering and Systems Sciences
Maharshi Dayanand Saraswati University Ajmer

2. Divergence from Randomness
• A distinct probabilistic model has been
proposed by Amati and Rijsbergen
• The idea is to compute term weights by
measuring the divergence between a term
distribution produced by a random process
and the actual term distribution
• Thus, the name divergence from randomness
• The model is based on two fundamental
assumptions, as follows.

3. First assumption:
• Not all words are equally important for describing
the content of the documents
• Words that carry little information are assumed to
be randomly distributed over the whole
document collection C
• Given a term ki, its probability distribution over
the whole collection is referred to as P(ki|C)
• The amount of information associated with this
distribution is given by −log P(ki|C)
• By modifying this probability function, we can
implement distinct notions of term randomness

4. Second assumption
• A complementary term distribution can be obtained by
considering just the subset of documents that contain
term ki
• This subset is referred to as the elite set
• The corresponding probability distribution, computed
with regard to document dj , is referred to as P(ki|dj)
• Smaller the probability of observing a term ki in a
document dj , more rare and important is the term
considered to be
• Thus, the amount of information associated with the
term in the elite set is defined as 1 − P(ki|dj)

5. Divergence from Randomness

6. Random Distribution
• To compute the distribution of terms in the collection,
distinct probability models can be considered
• For instance, consider that Bernoulli trials are used to
model the occurrences of a term in the collection
• To illustrate, consider a collection with 1,000 documents
and a term ki that occurs 10 times in the collection
• Then, the probability of observing 4 occurrences of term
ki in a document is given by

7. Random Distribution

8. Random Distribution
• Under these conditions, we can aproximate
the binomial distribution by a Poisson process,
which yields

9. Distribution over the Elite Set

10. Normalization

11. Normalization

12. Assignment
• Explain the Information Retrieval Model of
Divergence from Randomness