Term weighting assigns a weight to terms in documents to quantify their importance in describing the document's contents. Weights are higher for terms that occur frequently in a document but rarely in other documents. Term frequency in a document and inverse document frequency are used to calculate TF-IDF weights. Term occurrences may be correlated, so term weights should reflect their correlation. For example, terms like "computer" and "network" often appear together in documents about computer networks.

1. Information Retrieval : 8
Term Weighting
Prof Neeraj Bhargava
Vaibhav Khanna
Department of Computer Science
School of Engineering and Systems Sciences
Maharshi Dayanand Saraswati University Ajmer

2. Term Weighting
• The terms of a document are not equally useful
for describing the document contents
• In fact, there are index terms which are simply
vaguer than others
• There are properties of an index term which are
useful for evaluating the importance of the term
in a document
– For instance, a word which appears in all documents
of a collection is completely useless for retrieval tasks

3. Term Weighting
• To characterize term importance, we associate a
weight wi,j > 0 with each term ki that occurs in the
document dj
– If ki that does not appear in the document dj , then
wi,j = 0.
• The weight wi,j quantifies the importance of the
index term ki for describing the contents of
document dj
• These weights are useful to compute a rank for
each document in the collection with regard to a
given query

4. Term Weighting

5. Term Weighting
• The weights wi,j can be computed using the frequencies of
occurrence of the terms within documents
• Let fi,j be the frequency of occurrence of index term ki in
• the document dj
• The total frequency of occurrence Fi of term ki in the
collection is defined as
• where N is the number of documents in the collection

6. Term Weighting
• The document frequency ni of a term ki is the number of
documents in which it occurs
• Notice that ni < Fi or ni = Fi.
• For instance, in the document collection below, the values fi,j ,
Fi and ni associated with the term do are

7. Term-term correlation matrix
• For classic information retrieval models, the index term
weights are assumed to be mutually independent
– This means that wi,j tells us nothing about wi+1,j
• This is clearly a simplification because occurrences of
index terms in a document are not uncorrelated
• For instance, the terms computer and network tend to
appear together in a document about computer
networks
– In this document, the appearance of one of these terms
attracts the appearance of the other
• Thus, they are correlated and their weights should
reflect this correlation.

8. Term-term correlation matrix

9. Term-term correlation matrix

10. TF-IDF Weights
• TF-IDF term weighting scheme:
– Term frequency (TF)
– Inverse document frequency (IDF)
– Foundations of the most popular term weighting
scheme in IR

11. Assignment
• Discuss in detail the concept of Term
Weighting and Term Correlation matrix