3. Information Retrieval
• Information Retrieval - Given a set of terms and a set of document terms select
only the most relevant document (precision), and preferably all the relevant ones
(recall)
• Goal - find documents relevant to an information need from a large document set
– Mostly textual information ( text/document retrieval)
– documents, images, videos, data, services, audio
– XML, RDF, html, txt, PDF
• Large collections on internet with billions of pages
• Information retrieval problem: locating relevant documents based on user input,
such as keywords or example documents
4. Information Retrieval / Data Retrieval
Information Retrieval Data Retrieval
Matching vague exact
Model probabilistic deterministic
Query language natural artificial
Query specification incomplete complete
Items wanted relevant all (matching)
Error handling insensitive sensitive
7. Tasks
• Clustering - Group documents into clusters based on their contents
• Classification - Given a topics set and a new doc D, decide which topics
the documents D belongs to (spam / no-spam…)
• Information Extraction - Find all snippets dealing with a given topic (like
company merger)
• Question Answering - Handle wide range of question types like fact, list,
definition, how, why, hypothetical, semantically constrained, and cross-
lingual questions
• Opinion Mining - Analyze / summarize sentiment in a text
8. Terminology
• Searching - seek specific information within a body of information to get result of a search -
a set of hits
• Browsing - unstructured exploration of a body of information
• Linking - moving from one item to another following links like citations, references
• Query - a string of text, describing the information that user seeks. Each word of the query
is called a search term or keyword
• A query can be a single search term, a string of terms, a phrase in natural language or a
stylized expression using special symbols
• Full text searching - methods that compare the query with every word in the text, without
distinguishing the function (meaning, position) of the various words
• Fielded searching - methods that search on specific bibliographic or structural fields, such as
author or heading
10. Zipf's Law
• Distribution of word frequencies is similar for different texts (natural
language) of significantly large size
• Zipf's law holds even for different languages
11. Luhn's Hypothesis
• Frequency of words is a measurement of word significance - A measurement of
the power of a word to discriminate documents by their content ...
• Resolving/Discriminating
power of words
• Optimal power half way
between the cut-offs
12. Techniques
• IR Models - Governs how a document and a query are represented and how
the relevance of a document to a user query is defined
– Boolean Model
– Vector Model
– Probabilistic Model
• Index Terms (attribute) Selection
– Stop list
– Word stem
– Index terms weighting methods
• Terms Documents Frequency Matrices
13. Indexing Based IR
• Simple queries
– composed of two or three,
perhaps of dozen of keywords
– web retrieval
• Boolean queries
– `Database AND computer’
– online catalog and patent search
• Context queries
– proximity search, phrase queries
14. Sorting & Ranking
• User sends a query to search system which returns a set of hits
• For a large documents collection this set could be large
• The value of results depends on the order in which the hits are presented
• Ranking algorithms are at the core of information retrieval systems
(predicting which documents are relevant and which are not)
• Ranking methods
– Sorting the hits (by date…)
– Ranking the hits by similarity between query and document
– Ranking the hits by the importance of the documents
15. Bag of Words Model
• The most common way to represent documents in IR
• How to weight a word within a document
– Boolean: 1 is the word i is in doc j, 0 otherwise
– Tf*idf and others - the weight is a function of the word frequency in the
document and of the frequency of documents with that word
• What is a word
– Single, inflected word (going)
– Lemmatised word (going, go, gone go)
– Multi-word, proper nouns, numbers, dates (board of directors, John Stack, April,
2012)
– Meaning: plan,project,design PLAN#03
16. Bag of Words Model
• Treats all the words in a document as index
terms
• Assigns a weight to each term based on
importance (or presence/absence of word)
• Disregards order, structure, meaning of the
words
• Simple but effective
• Assumptions
– Term occurrence is independent
– Document relevance is independent
– Words are well-defined
• Consider three documents
– John likes to watch movies.
– Mary likes movies too.
– John also likes football
• The bag of words is shown below
17. Document Parsing
• Format and language of each document
– What format is it in?
– PDF / word / excel / html?
– What language is it in?
– What character set is in use?
• Each of these is a classification problem
• These tasks often performed heuristically
Sec. 2.1
18. Parsing Challenges
• Documents being indexed can be from different languages
– A single index may contain terms of several languages
• A document / components can contain multiple languages / formats
– French email with a German PDF attachment
• What is a unit document?
– A file?
– An email?
– An email with 5 attachments?
– A group of files (PPT or LaTeX as HTML pages)
Sec. 2.1
19. Tokenization
• Token - instance of a sequence of characters
• Each token is a candidate for an index entry after further processing
• Input: Customers Suppliers and Factory
• Output: Tokens
– Customers
– Suppliers
– Factory
Sec. 2.2.1
20. Tokenization Issues
• Finland’s capital → Finland? Finlands? Finland’s?
• Hewlett-Packard → Hewlett and Packard as two tokens?
– state-of-the-art - break up hyphenated sequence
– co-education
– lowercase, lower-case, lower case ?
• San Francisco - one token or two?
– How to decide if it is one token?
– Cheap San Francisco-Los Angeles fares
Sec. 2.2.1
21. Stop Words
• Many of the most frequently used words in English are useless in IR and text mining
• Those words are called stop words
– the, a, and, to, be , of, in, about, with …
– Little semantic value
– Stop words account for 20-30% of total word counts
• Stop list contains stop-words that should not to be used as index
– Prepositions
– Articles
– Pronouns
– Some adverbs and adjectives
– Some frequent words (e.g. document)
Sec. 2.2.2
22. Stop Words
• Removal of stop-words improves
efficiency and effectiveness of
searches
• A few standard stop-lists are
commonly used
• Reduces indexing data file sizes
Sec. 2.2.2
23. Stop Words - New Trend
• Stop words need very small space for storage due to good compression
techniques
• Query time is not affected due to stop words because of good query
optimization techniques
• Stop words are necessary for
– Phrase queries - King of Spain
– Various song titles.. - Let it be, To be or not to be
– Relational queries - flights to London
24. Normalization - Terms
• Normalization of words in indexed text and query into same form
• A term is a normalized word type, which is a single entry in IR system
dictionary
• Implicitly defines equivalent classes of terms by
– Deleting periods to form a term
• U.S.A., USA USA
– Deleting hyphens to form a term
• anti-discriminatory, antidiscriminatory antidiscriminatory
– Synonyms
• Car, Automobile
Sec. 2.2.3
25. Case Folding
• Reduces all letters to lower case
• Exception: upper case words in mid-sentence
• General Motors
• Fed vs. fed
• MIT vs. mit
• It is best to lowercase everything since users will often type search
queries in lowercase regardless of interested information
• Google example
– When query is C.A.T. -> #1 result is for “cat” (Wikipedia) but not Caterpillar
Inc.
Sec. 2.2.3
26. Synonyms and Homonyms
• Synonyms
– Document - motorcycle repair - motorcycle maintenance
– maintenance and repair are synonyms
– System can extend query as motorcycle and (repair or maintenance)
• Homonyms
– Object has different meanings as noun/verb
– Can disambiguate meanings to some extent from the context
• Extending queries automatically using synonyms can be problematic
– Need to understand intended meaning in order to infer synonyms
• Or verify synonyms with user
– Synonyms may have other meanings as well
27. Normalization - Synonyms
• Handling Synonyms and Homonyms
– Hand-constructed equivalence classes
• car = automobile color = colour
– Rewrite words to form equivalence-class terms
• When a document contains automobile, index it under car-automobile (and vice-versa)
– Expand a query
• When a query contains automobile, look under car as well
• Spelling mistakes
– Soundex - a phonetic algorithm for equivalent classes of words based on phonetic
heuristics
• Google Googol
28. Stemming
• Techniques used to reduce words of variant form to a stem or root form
before indexing
• Stemming – Remove endings of word
– Computer
– Compute
– Computes
– Computing
– Computed
– Computation
Sec. 2.2.4
comput
29. Stemming
• As a result if query is house plans, the results will also include all pages
containing variation of that term
– House plan
– House planer
– House planning
• Increases recall at the expense of precision
• Improves effectiveness of IR and text mining
– Matching similar words
– Reducing indexing size
– Combing words with same roots may reduce indexing size as much as 40-50%.
• Produced by Stemmers
Sec. 2.2.4
30. Lemmatization
• Transform to standard dictionary form lemma, according to syntactic
category
– verb + ing → verb, noun + s → noun
• More accurate than stemming but consumes more resources
• Balance noise Vs. recognition rate
• Compromise between precision and recall
• Increased recall without hurting precision
• Produced by lemmatizers
• the boy's cars are different colors → the boy car be different color
31. Porter’s Algorithm
• The most common algorithm for stemming English and one that has
repeatedly been shown to be empirically very effective - suffix stripping
• Conventions + 5 phases of reductions
– phases applied sequentially
– each phase consists of a set of commands
– sample convention: of the rules in a compound command, select the one that
applies to the longest suffix
Sec. 2.2.4
32. Porter Algorithm Steps
1. Plurals and past participles
SSES -> SS caresses -> caress
(*v*) ING -> motoring -> motor
1. adj->n, n->v, n->adj, …
(m>0) OUSNESS -> OUS callousness -> callous
(m>0) ATIONAL -> ATE relational -> relate
1. (m>0) ICATE -> IC triplicate -> triplic
2. (m>1) AL -> revival -> reviv
(m>1) ANCE -> allowance -> allow
1. (m>1) E -> probate -> probat
2. (m > 1 and *d and *L) -> single letter controll -> control
33. Stemmers Comparison
• Sample text: Such an analysis can reveal features that are not easily visible from the
variations in the individual genes and can lead to a picture of expression that is more
biologically transparent and accessible to interpretation
• Porter’s: such an analysi can reveal featur that ar not easili visibl from the variat in the
individu gene and can lead to pictur of express that is more biolog transpar and access
to interpret
• Lovins’s: such an analys can reve featur that ar not eas vis from th vari in th individu
gen and can lead to a pictur of expres that is mor biolog transpar and acces to interpres
• Paice’s : such an analys can rev feat that are not easy vis from the vary in the individ
gen and can lead to a pict of express that is mor biolog transp and access to interpret
34. Deep Analysis
• Detailed Natural Language Processing (NLP) algorithms
• Semantic disambiguation, phrase indexing (board of directors), named
entities (President Monti = Mario Monti)...
• Standard search engines use deeper techniques (Google)
35. Document Indexing
• Store an index to optimize speed and performance
• Useful in finding relevant documents for a search query
• Reduces time and CPU usage
• Without an index, search engine will scan every document in
the corpus
• An index of 10,000 documents is queried in milliseconds
• A sequential scan of every word in 10,000 documents takes
much more time
• Each document is represented by a set of weighted
keywords known as terms
– D1 → {(t1, w1), (t2,w2), …}
• D1 → {(comput, 0.2), (architect, 0.3), …}
• D2 → {(comput, 0.1), (network, 0.5), …}
• Inverted file - used in retrieval for higher efficiency
– comput → {(D1,0.2), (D2,0.1), …}
36. Boolean Model
• Query terms are combined logically using the
Boolean operators
– AND, OR, NOT
– ((asthma AND exercise) AND (NOT cardiac))
– Views each document as a set of words
– Precise: document matches a condition or not
• Retrieval
– Given a Boolean query, system retrieves each
document that makes the query logically true
– Called exact match
• No Rank - A document is judged to be relevant if
the terms in the document satisfies the logical
expression of the query
37. Inverted index
• A data structure that attaches each distinctive term with a list of all
documents that contains the term in a document collection
• The list is called posting list
Sec. 1.2
39. What Goes in Inverted File
• Boolean retrieval
– Just the document number
• Ranked Retrieval
– Document number and term weight (TF, IDF, TF*IDF, ...)
• Proximity operators
– Word offsets for each occurrence of the term
– Example : t17 (doc1,49) (doc1,70) (doc2,3)
40. Inverted File Size
• Very compact for Boolean retrieval
– About 10% of the size of the documents
– If an aggressive stop word list is used
• Not much larger for ranked retrieval
– Perhaps 20%
• Enormous for proximity operators
– Sometimes larger than the documents
– But access is fast - you know where to look
41. Inverted Index Construction
Sec. 1.2
Friends Romans Countrymen
friend roman countryman
friend
roman
countryman
2 4
2
13 16
1
Friends, Romans, Countrymen
42. Inverted Index – Search Steps
• Given a query q
– Vocabulary search - find each term/word from q in the inverted index
– Results merging - Merge results to find documents that contain all or some of
the words/terms in q
– Rank score computation - To rank the resulting documents/pages, using
• Content-based ranking
• Link-based ranking
43. Inverted Index - Boolean Retrieval
11
11
11
11
11
11
1 2 3
11
11
11
4
blueblue
catcat
eggegg
fishfish
greengreen
hamham
hathat
oneone
3
4
1
4
4
3
2
1
blue
cat
egg
fish
green
ham
hat
one
2
11redred
11twotwo
2red
1two
Document
IDs
44. Boolean Retrieval
• To execute a Boolean query
– Build query syntax tree
– For each clause, look up postings
– Traverse postings and apply Boolean operator
• Efficiency analysis
– Postings traversal is linear (assuming sorted postings)
– Start with shortest posting first
( blue AND fish ) OR ham
1
2blue
fish 2
45. Query Processing - AND
• Consider processing the query - Brutus AND Caesar
– Locate Brutus in the Dictionary
• Retrieve its postings
– Locate Caesar in the Dictionary
• Retrieve its postings
– Merge the two postings
128
34
Brutus
Caesar
Sec. 1.3
46. The Merge
• Walk through the two postings simultaneously, in time linear in the total
number of postings entries
• These postings are sorted by docID
34
1282 4 8 16 32 64
1 2 3 5 8 13 21
128
34
Brutus
Caesar
2 8
If the list lengths are x and y, the merge takes O(x+y) operations
51. 13
Optimization of Index Search
• What is the best order of words for query processing?
• Consider a query that is an AND of n terms
• Process words in order of increasing freq
– start with smallest set, then keep cutting further
– This is why we kept document freq. in dictionary
• For each of the n terms, get its postings, then AND them together
Brutus
Caesar
Calpurnia
Query: Brutus AND Calpurnia AND Caesar
Sec. 1.3
1 2 3 5 8 16 21 34
2 4 8 16 32 64 128
16
52. More General Optimization
• (madding OR crowd) AND (ignoble OR strife)
• Get document frequencies for all terms
• Conservative - estimate the size of each OR by the sum of its doc
frequencies
• Process in increasing order of OR sizes
Sec. 1.3
53. Query Optimization
Term Frequency
eyes 213312
kaleidoscope 87009
marmalade 107913
skies 271658
tangerine 46653
trees 316812
Recommend a query processing order for –
(tangerine OR trees) AND (marmalade OR
skies) AND (kaleidoscope OR eyes)
300321
379571
363465
(kaleydoscopeOReyes) AND (tangerineORtrees)AND(marmaladeORskies)
54. Skip Pointers
• Intersection is the most important operation when for search engines
• This is because in web search, most queries are implicitly intersections
• car repairs, Britney spears songs translates into – car AND repairs, Britney
AND spears AND songs, which means it will be intersecting 2 or more
postings lists in order to return a result
• Because intersection is so crucial, search engines try to speed it up in any
way possible. One such way is to use skip pointers
55. Optimized Skip Pointers
• Augment Postings with skip pointers (at indexing time)
• Why? - To skip postings that will not figure in the search results.
• Where do we place skip pointers?
1282 4 8 41 48 64
311 2 3 8 11 17 21
3111
41 128
Sec. 2.3
56. Query Processing with Skip Pointers
• Start using the normal intersection algorithm
• Continue until the match 12 and advance to the next item in each list. At this point
the "car" list is on 48 and the "repairs" list is on 13, but 13 has a skip pointer
• Check the value the skip pointer is pointing at (i.e. 29) and if this value is less than
the current value of the "car" list (which it is), we follow our skip pointer and jump
to this value in the list
57. Where to Place Skips - Tradeoff
• More skips → shorter skip spans ⇒ more likely to skip
– But lots of comparisons to skip pointers.
• Fewer skips → few pointer comparison, but then long skip spans ⇒
few successful skips
58. Placing Skips
• Simple heuristic - for postings of length L, use √L evenly-spaced skip
pointers
• This ignores the distribution of query terms
• Easy if the index is relatively static
• Harder if L keeps changing because of updates
• How much do skip pointers help?
– CPUs were slow, they used to help a lot
– Today’s CPUs are fast and disk is slow, so reducing disk postings list size
dominates
59. Strengths and Weaknesses
• Strengths
– Very precise queries can be specified
– Easy to implement in the simple form
• Weaknesses
– Retrieval results are poor since term frequency is not considered - No index
term weighting
– Specifying the query may be difficult for casual users
– Result might be 1 or 0 (unordered set of documents)
60. Similarity Based Retrieval
• Retrieve documents that are similar to a given document
– Similarity may be defined on the basis of common words
– Find k terms in A with highest TF (d, t ) / n (t ) and use these terms to find
relevance of other documents
• Relevance feedback
– Similarity can be used to refine answer set to keyword query
– User selects a few relevant documents from those retrieved by keyword
query and system finds other documents similar to these
61. Similarity Based Retrieval - Vector Space Model
• Define an n-dimensional space, where n is the number of words in the
document set
• Vector for document d goes from origin to a point whose ith
coordinate is
TF (d,t ) / n (t )
• The cosine of the angle between vectors of two documents is used as a
measure of their similarity
62. Vector Space Model
• Assumption - Documents that are close together in vector space talk
about same things
• Hence retrieve documents depending on closeness to the query
(similarity ~ closeness)
63. Vector Space Model
• Documents are treated as a bag of words or terms
• Documents are presented in high dimensional space
• Each document is represented as a vector
• Implemented by forming term-document matrix
• Dimension of space depends on number of indexing terms which are chosen to be
relevant for the collection
• Rank according to the similarity metric (e.g. cosine) between the query and document
• The smaller the angle between the document and query the more similar they are
believed to be
– Documents are represented by a term vector
– Queries are represented by a similar vector
64. Vector Space Model
• Term weights are not pure 0 or 1
• Each weight is computed based on some variations of TF or TF-IDF
scheme
• Query has the same shape as document (m dimensional vector)
– Cosine is commonly used in text clustering
• Measure of similarity between query q and a document dj is a cosine of
angle between these vectors
• Ad-hoc weightings (term frequency x inverse document frequency ) used
• No optimal ranking
65. Vector Space Model
• Vector space = all the keywords encountered <t1, t2, t3, …, tn>
• Document D = < a1, a2, a3, …, an> where ai = weight of ti in D
• Query Q = < b1, b2, b3, …, bn> where bi = weight of ti in Q
• R(D,Q) = Sim(D,Q)
• Consider Query q
– Relevance of di to q - Compare similarity of query q and document di
– Cosine similarity (the cosine of the angle between the two vectors)
67. Term-Document Matrix
• A collection of n documents can
be represented in vector space
model using this matrix
• A m × n matrix where m is
number of terms and n is number
of documents
• Term - Row of term-document
matrix
• Document - Column of term-
document matrix
=
↓↓↓
aaa
aaa
aaa
A
ddd
mnmm
n
n
n
21
22221
11211
21
t
t
t
m
←
←
←
2
1
69. Similarity Formulae
• Dot Product
• Cosine
• Dice
• Jaccard ∑ ∑ ∑
∑
∑ ∑
∑
∑ ∑
∑
∑
−+
=
+
=
=
=
i i i
iiii
i
ii
i i
ii
i
ii
i i
ii
i
ii
ii
baba
ba
QDSim
ba
ba
QDSim
ba
ba
QDSim
baQDSim
)*(
)*(
),(
)*(2
),(
*
)*(
),(
)*(),(
22
22
22
t1
t2
D
Q
70. Index Storage
• Term-Document matrix is very sparse
• A few 100 terms for a document and a few terms for a query, while the
term space is large (~100k)
• Stored as
D1 → {(t1, a1), (t2,a2), …}
t1 → {(D1,a1), …}
71. Implementation
• Implementation of Vector Space Model using dot product
– Naïve implementation: O(m*n)
– Implementation using inverted file
• Given a query = {(t1,b1), (t2,b2)}
1. Find the sets of related documents through inverted file for t1 and t2
2. Calculate the score of the documents for each weighted term (t1,b1) →
{(D1,a1 *b1), …}
3. Combine the sets and sum the weights (∑)
4. O(|Q|*n)
72. Similarity Calculation
• Consider two documents D1, D2 and a query Q
– D1 = (0.5, 0.8, 0.3), D2 = (0.9, 0.4, 0.2), Q = (1.5, 1.0, 0)
73. Ranked Retrieval
• Documents are ranked based on their score
• Advantages
– Query is easy to specify
– Output is ranked based on the estimated relevance of the documents to the
query
– A wide variety of theoretical models exist
• Disadvantages
– Query less precise (although weighting can be used)
74. Example Query
• A document space is defined by three terms – computer, application, users
• A set of documents defined as
– A1=(1, 0, 0), A2=(0, 1, 0), A3=(0, 0, 1)
– A4=(1, 1, 0), A5=(1, 0, 1), A6=(0, 1, 1)
– A7=(1, 1, 1), A8=(1, 0, 1). A9=(0, 1, 1)
• Query is computer and application
• What documents should be retrieved?
75. Example Query
• In Boolean query matching
– AND - document A4, A7 will be retrieved
– OR - retrieved: A1, A2, A4, A5, A6, A7, A8, A9
• In similarity matching (cosine)
– q=(1, 1, 0)
– S(q, A1)=0.71, S(q, A2)=0.71, S(q, A3)=0
– S(q, A4)=1, S(q, A5)=0.5, S(q, A6)=0.5
– S(q, A7)=0.82, S(q, A8)=0.5, S(q, A9)=0.5
– Document retrieved set (with ranking)
• {A4, A7, A1, A2, A5, A6, A8, A9}
77. Binary Weights
• Only the presence 1 or absence 0 of a term is included in the vector
docs t1 t2 t3 RSV=Q.Di
D1 1 0 1 4
D2 1 0 0 1
D3 0 1 1 5
D4 1 0 0 1
D5 1 1 1 6
D6 1 1 0 3
D7 0 1 0 2
D8 0 1 0 2
D9 0 0 1 3
D10 0 1 1 5
D11 1 0 1 3
Q 1 2 3
q1 q2 q3
78. Raw Term Weights
• The frequency of occurrence for the term in each document is included in
the vector
docs t1 t2 t3
D1 2 0 3
D2 1 0 0
D3 0 4 7
D4 3 0 0
D5 1 6 3
D6 3 5 0
D7 0 8 0
D8 0 10 0
D9 0 0 1
D10 0 3 5
D11 4 0 1
79. TF.IDF - Term Weighting
• Assigns a tf * idf weight to each term in each document
• Term weights components
– Local - How important is the term in this document?
– Global - How important is the term in the collection?
• Logic
– Terms that appear often in a document should get high weights
– Terms that appear in many documents should get low weights
• Mathematical Capturing
– Term Frequency (local)
– Inverse Document Frequency (global)
80. TF.IDF - Term Weighting
• tf = Term Frequency
– Frequency of a term/keyword in a document
– The higher the tf, the higher the importance (weight) for the doc.
• df = document frequency
– Number of documents containing the term
– Distribution of the term
• idf = Inverse Document Frequency
– Unevenness of term distribution in the corpus
– Specificity of term to a document
• The more the term is distributed evenly, the less it is specific to a document:
weight(t, D) = tf(t ,D) * idf(t)
81. Term Weighting
• Based on common sense, but adjusted/engineered following experiments
• Terms that occur in only a few documents are often more valuable than
ones that occur in many - IDF
• The more often a term occurs in a document, the more likely it is to be
important for that document - TF
• A term that occurs for same number of times in a short and a long
document is likely to be more valuable for the short document
82. • Word occurrence frequency is a measure of the significance of terms and
their discriminatory power
Word
frequency
Too frequent:
useless discriminators
Significant terms
Too rare: no significant
contribution to
the content of the document
Term Significance
83. TF.IDF Weight
• Term frequency weight measures importance in document:
• Inverse document frequency measures importance in collection:
• Some heuristic modifications
84. Relevance Ranking Using Terms
• TF-IDF (Term frequency/Inverse Document frequency) ranking
– Let n(d) = number of terms in the document d
– n(d, t) = number of occurrences of term t in the document d.
– Relevance of a document d to a term t
• The log factor is to avoid excessive weight to frequent terms
– Relevance of document to query Q
nn((dd))
nn((dd,, tt))
1 +1 +TFTF ((dd,, tt) =) = loglog
rr ((dd,, QQ) =) = ∑∑ TFTF ((dd,, tt))
nn((tt))tt∈∈QQ
85. Inverse Document Frequency
• IDF provides high values for rare words and low values for common words
4
1
10000
log
698.2
20
10000
log
301.0
5000
10000
log
0
10000
10000
log
=
=
=
=
For a collection
of 10000
documents
86. TF.IDF Normalization
• Normalize the term weights
– So longer documents are not unfairly given more weight
– Normalize usually means force all values to fall within a certain range, usually
between 0 and 1, inclusive.
∑=
=
t
k kik
kik
ik
nNtf
nNtf
w
1
22
)]/[log()(
)/log(
87. Relevance Ranking Using Terms
• Ranking of documents on the basis of estimated relevance to a query
• Relevance ranking is based on factors
– Term frequency
• Frequency of occurrence of query keyword in document
– Inverse document frequency
• How many documents the query keyword occurs in
– Fewer give more importance to keyword
– Hyperlinks to documents
• More links to a document document is more important
88. Relevance Ranking Using Terms
• Documents are returned in decreasing order of relevance score
– Usually only top few documents are returned
• Most systems improvise above model
– Common words like a, an, the, it etc are eliminated
• Called stop words
– Words that occur in title, author list, section headings are given greater
importance
– Words whose first occurrence is late in the document are given lower importance
– Proximity: if keywords in query occur close together in the document, the
document has higher importance than if they are far apart
89. Relevance Using Hyperlinks
• Number of documents relevant to a query can be enormous if only term
frequencies are taken into account
• Using term frequencies makes spamming easy
– A fitness center could add many occurrences of the words like weights to its
page to make its rank very high
• Most of the time people are looking for pages from popular sites
• Idea - use popularity of web site to rank site pages that match given
keywords
• Problem - hard to find actual popularity of site
90. Relevance Using Hyperlinks
• Solution: use number of hyperlinks to a site as a measure of the popularity or
prestige of the site
– Count only one hyperlink from each site
– Popularity measure is for site, not for individual page
• But most hyperlinks point to root of site
• Concept of site is difficult to define since a URL prefix like cs.yale.edu contains many
unrelated pages of varying popularity
• Refinements
– When computing prestige based on links to a site, give more weight to links from
sites that themselves have higher prestige
• Definition is circular
• Set up and solve system of simultaneous linear equations
– Above idea is the basis of the Google Page Rank ranking mechanism
91. Relevance Using Hyperlinks
• Connections in social networking
– Ranks prestige of people
– Someone known by multiple prestigious people has higher prestige
• Hub and authority based ranking
– A hub is a page that stores links to many pages (on a topic)
– An authority is a page that contains actual information on a topic
– Each page gets a hub prestige based on prestige of authorities that it points to
– Each page gets an authority prestige based on prestige of hubs that point to it
– Prestige definitions are cyclic and can be ontained by solving linear equations
– Use authority prestige when ranking answers to a query
92. Probability Ranking Principle
• Given a user query q and a document d, estimate the probability that the
user will find d relevant
• Robertson (1977)
• If a reference retrieval system’s response to each request is a ranking of
the documents in the collection in order of decreasing probability of
relevance to the user who submitted the request
• Probabilities are estimated as accurately as possible on the basis of
whatever data have been made available to the system for this purpose,
• Overall effectiveness of the system to its user will be the best that is
obtainable on the basis of those data
94. Bayes Classifier
• Bayes Decision Rule
– A document D is relevant if P(R|D) > P(NR|D)
• Estimating probabilities
– Use Bayes Rule
– Classify a document as relevant if
– Left hand side is likelihood ratio
95. Estimating P(D|R)
• Assume independence
• Binary independence model
– based on information related to presence and absence of terms in relevant and non-relevant
documents
– information acquired through relevance feedback process
• user stating which of the retrieved documents are relevant / non-relevant
– Based on the probability ranking principle, which ensures an optimal ranking
– document represented by a vector of binary features indicating term occurrence (or non-
occurrence)
– pi is probability that term i occurs (has value 1) in relevant document, si is probability of
occurrence in non-relevant document
97. Binary Independence Model
• Scoring function is
• Query provides information about relevant documents
• If we assume pi constant, si approximated by entire collection, get idf-like
weight
99. BM25 Ranking Algorithm
• Popular and effective ranking algorithm based on binary independence
model
– adds document and query term weights
– k1, k2 and K are parameters whose values are set empirically
– dl is doc length
– Typical TREC value for k1 is 1.2, k2 varies from 0 to 1000, b = 0.75
100. BM25 Example
• Query with two terms player Pele, (qf = 1)
• No relevance information (r and R are zero)
• N = 500,000 documents
• player occurs in 40,000 documents (n1 = 40, 000)
• Pele occurs in 300 documents (n2 = 300)
• Player occurs 15 times in doc (f1 = 15)
• Pele occurs 25 times (f2 = 25)
• Document length is 90% of the average length (dl/avdl = .9)
• k1 = 1.2, b = 0.75, and k2 = 100
• K = 1.2 · (0.25 + 0.75 · 0.9) = 1.11
102. BM25 Example - Effect of Term Frequencies
Frequency of player Frequency of Pele BM25 Score
15 25 20.66
15 1 12.74
15 0 5.00
1 25 18.2
0 25 16.66
103. Relevance Feedback Process – Iterative Cycle
1. User presented with a list of retrieved documents
2. User marks those which are relevant (or not relevant)
1. In practice top 10-20 ranked documents are examined
2. Incremental: one document after the other
3. The relevance feedback algorithm selects important terms from documents assessed relevant by users
4. The relevance feedback algorithm emphasises the importance of these terms in a new query in the following ways
1. Query expansion - add these terms to the query
2. Term reweighing - modify the term weights in the query
3. Query expansion + term reweighing
5. The updated query is submitted to the system
6. If the user is satisfied with the new set of retrieved documents, then feedback process stops, otherwise go to step 2
• Approaches
– Approach 1: Add/Remove/Change query terms
– Approach 2: Re-weight query terms
104. Relevance Feedback Issues
• Relevance feedback
– Often users are not reliable in making relevance assessments, or do not make
relevance assessments
– Implicit relevance feedback by looking at what users access
• clicks in web logs
• works well - “wisdom of the crowd"
– Positive, negative
– Partial relevance assessments (very relevant or partially relevant)?
– Why is a document relevant?
• Interactive query expansion (as opposed to automatic)
– Users choose the terms to be added
105. Latent Semantic Indexing - LSI
• Term document matrices are very large but people talk about few things
• So how to represent term document by a lower dimensional latent space?
• Latent Semantic Analysis is the analysis of latent i.e. hidden semantics in a corpora of
text
• LSI transforms the original data in a different space so that two documents/words
about the same concept are mapped close (so that they have higher cosine similarity)
• LSI achieves this by Singular Value Decomposition (SVD) of term-document matrix
• Maps documents and terms to a low dimensional representation
• Design a mapping such that the low dimensional space reflects semantic association
• Compute document similarity based on the inner product in this latent semantic space
107. LSI
• For LSI truncated SVD is used
• Where
– Ukis m×k matrix whose columns are first k left singular vectors of A
– Σk is k×k diagonal matrix whose diagonal is formed by k leading singular
values of A
– Vkis n×k matrix whose columns are first k right singular vectors of A
• Rows of Uk= terms
• Rows of Vk= documents
• SVD can be used to compute optimal low rank approximations
VUA
T
kkkk Σ=
108. LSI
• In truncated LSI, first k independent linear components of A (singular vectors and values) are
included
• Documents are projected in means of least squares on space spread by first k singular
vectors of A (LSI space)
• First k components capture the major associational structure in in the term-document
matrix and throw out the noise
• Minor differences in terminology used in documents are ignored
• Closeness of objects (queries and documents) is determined by overall pattern of term
usage, so it is context based
• Documents which contain synonyms are closer in LSI space than in original space;
• Documents which contain polysemy (a word having different meaning in different contexts )
in different context are more far in LSI space than in original space
109. Concept Indexing (CI)
• Lexical focused relevance estimation is less effective than semantic
focussed one
• Indexing using concept decomposition (CD) instead of SVD like in LSI
• Concept decomposition was introduced in 2001
• I. S. Dhillon, D.S. Modha: Concept decomposition for large sparse text
data using clustering, Machine Learning, 42:1, 2001, pp. 143-175
110. Concept Decomposition
• Cluster documents in term-document matrix A on k groups
• Clustering algorithms
– Spherical k-means algorithm
– Fuzzy k-means algorithm
• Spherical k-means algorithm is a variant of k-means algorithm which uses the fact that
vectors of documents are of the unit norm
• Centroids of groups = concept vectors
• Concept matrix is matrix whose columns are centroids of groups
cj – centroid of j-th group
[ ]cccC kk 21
=
111. Concept Decomposition
• Next step: calculate concept decomposition
• Concept decomposition Dkof term-document matrix A is least squares
approximation of A on the space of concept vectors
where Z is solution of the least squares problem
• Rows of Ck = terms
• Columns of Z = documents
ZCD kk =
( ) ACCCZ T
kk
T
k
1−
=
112. System Evaluation
• There are many retrieval models/ algorithms/ systems
• Which one is the best?
– How much effective a system is capable of retrieving relevant documents?
• What is the best component for
– Ranking function (dot-product, cosine, …)
– Term selection (stop word removal, stemming…)
– Term weighting (TF, TF-IDF,…)
• How far down the ranked list will a user need to look to find some / all
relevant documents?
113. Effectiveness
• Goal of an IR system is to retrieve as many relevant documents as
possible and as few non-relevant documents as possible.
• Evaluating the above consists of a comparative evaluation of technical
• performance of IR system(s)
– In traditional IR, technical performance means the effectiveness of the IR
system: the ability of the IR system to retrieve relevant documents and
suppress non-relevant documents
• Effectiveness is measured by the combination of recall and precision
114. Measuring Query Retrieval Effectiveness
• Information-retrieval systems save space by using index structures that
support only approximate retrieval
• May result in
– False negative (false drop) - some relevant documents may not be retrieved
– False positive - some irrelevant documents may be retrieved
– For many applications a good index should not permit any false drops, but
may permit a few false positives
115. Precision and Recall
Retrieved &
relevant
Not retrieved but
relevant
Retrieved &
irrelevant
Not retrieved &
irrelevant
retrieved not retrieved
relevantirrelevant
Relevant
Relevant
&
Retrieved
Retrieved
All
Documents
116. Precision and Recall
• In the ideal case, the set of retrieved documents is equal to
the set of relevant documents. However, in most cases, the
two sets will be different
• This difference is formally measured with precision and
recall
• Precision - what percentage of the retrieved documents are
relevant to the query
• Recall - what percentage of the documents relevant to the
query were retrieved
documentsrelevantofnumberTotal
retrieveddocumentsrelevantofNumber
recall =
retrieveddocumentsofnumberTotal
retrieveddocumentsrelevantofNumber
precision =
117. Measuring Retrieval Effectiveness
• The above two measures do not take into account where the relevant documents are retrieved, this is, at
which rank (crucial since the output of most IR systems is a ranked list of documents).
• This is very important because an effective IR system should not only retrieve as many relevant
documents as possible and as few non-relevant documents as possible, but also it should retrieve
relevant documents before the non-relevant ones.
• Recall vs. Precision tradeoff
– Can increase recall by retrieving many documents (down to a low level of relevance ranking), but many
irrelevant documents would be fetched, reducing precision
• Measures of retrieval effectiveness
– Recall as a function of number of documents fetched or
– Precision as a function of recall
• Equivalently, as a function of number of documents fetched
– Example - precision of 75% at recall of 50%, and 60% at a recall of 75%
• Problem: which documents are actually relevant, and which are not
118. Systems Evaluation - Challenges
• Effectiveness is related to the relevancy of retrieved items
• Relevancy is not typically binary but continuous
• Even if relevancy is binary, it can be a difficult judgment to make
• Relevancy from a human standpoint
– Subjective: Depends upon a specific user’s judgment
– Situational: Relates to user’s current needs
– Cognitive: Depends on human perception and behavior
– Dynamic: Changes over time
• Total number of relevant items is sometimes not available
– Sample across the database and perform relevance judgment on these items
– Apply different retrieval algorithms to the same database for the same query. The aggregate of
relevant items is taken as the total relevant set
120. Recall / Precision
• Let us assume that for a given query, the following documents are relevant
• fd3, d5, d9, d25, d39, d44, d56, d71, d89, d123g
• Now suppose that the following documents are retrieved for that query
• For each relevant document (in red bold), we calculate the precision value and the recall value
• For example, for d56, we have 3 retrieved documents, and 2 among them are relevant, so the
precision is 2/3. We have 2 of the relevant documents so far retrieved (the total number of relevant
documents being 10), so recall is 2/10.
121. Recall / Precision
• For each query, we obtain pairs of recall and
precision values
• In our example, we would obtain (1/10, 1/1)
(2/10, 2/3) (3/10, 3/6) (4/10, 4/10) (5/10, 5/14) . .
. which are usually expressed in % (10%,
100%) (20%, 66.66%) (30%, 50%) (40%, 40%)
(50%, 35.71%) . . .
• This can be read for instance: at 20% recall,
we have 66.66% precision; at 50% recall, we
have 35.71% precision
• The pairs of values are plotted into a graph,
which has the following curve
122. The complete methodology
• For each IR system / IR system version
– For each query in the test collection
• We first run the query against the system to obtain a ranked list of retrieved
documents
• We use the ranking and relevance judgements to calculate recall/precision pairs
– Then we average recall / precision values across all queries, to obtain an
overall measure of the effectiveness
124. Computing Recall / Precision Points
• For a given query, produce the ranked list of retrievals
• Adjusting a threshold on this ranked list produces different sets of
retrieved documents, and therefore different recall/precision measures
• Mark each document in the ranked list that is relevant according to the
gold standard
• Compute a recall/precision pair for each position in the ranked list that
contains a relevant document
125. Computing Recall / Precision Points
• Let total # of relevant docs = 6
• Check each new recall point
Missing one
relevant document.
Never reach
100% recall
n doc # relevant
1 588 X R=1/6=0.167; P=1/1=1
2 589 X R=2/6=0.333; P=2/2=1
3 576
4 590 X R=3/6=0.5; P=3/4=0.75
5 986
6 592 X R=4/6=0.667; P=4/6=0.667
7 984
8 988
9 578
10 985
11 103
12 591
13 772 X R=5/6=0.833; p=5/13=0.38
14 990
126. Computing Recall / Precision Points
• Let total # of relevant docs = 6
• Check each new recall point
n doc # relevant
1 588 X R=1/6=0.167; P=1/1=1
2 576
3 589 X R=2/6=0.333; P=2/3=0.667
4 342
5 590 X R=3/6=0.5; P=3/5=0.6
6 717
7 984
8 772 X R=4/6=0.667; P=4/8=0.5
9 312 X R=5/6=0.833; P=5/9=0.556
10 498
11 113
12 628
13 772
14 592 X R=6/6=1.0; p=6/14=0.429
127. Average Recall / Precision Curve
• Typically average performance over a large set of queries
• Compute average precision at each standard recall level across all queries
• Plot average precision/recall curves to evaluate overall system
performance on a document/query corpus
128. Compare Systems
• The curve closest to the upper right-hand corner of the graph indicates
the best performance
129. R- Precision
• It is the precision at the R position in the
ranking of results for a query
• R = relevant documents
• R-precision places lower emphasis on the
exact ranking of the relevant documents
returned
• This can be useful when a topic has a large
number of judged relevant documents
• Or when an evaluator is more interested in
measuring aggregate performance as
opposed to the fine-grained quality of the
ranking provided by the system.
n doc # relevant
1 588 x
2 589 x
3 576
4 590 x
5 986
6 592 x
7 984
8 988
9 578
10 985
11 103
12 591
13 772 x
14 990
R = # of relevant docs = 6
R-Precision = 4/6 = 0.67
130. F-Measure
• A measure of performance that considers both recall and precision
• Harmonic mean of recall and precision
• Compared to arithmetic mean, both need to be high for harmonic mean
to be high
PR
RP
PR
F 11
22
+
=
+
=
131. E Measure - Parameterized F Measure
• A variant of F measure that allows weighting emphasis on precision over
recall
• Value of β controls trade-off
– β = 1: Equally weight precision and recall (E=F)
– β > 1: Weight recall more
– β < 1: Weight precision more
PR
RP
PR
E 1
2
2
2
2
)1()1(
+
+
=
+
+
= β
β
β
β
132. Mean Average Precision (MAP)
• Average Precision: Average of the precision values at the points at which
each relevant document is retrieved
– Ex1: (1 + 1 + 0.75 + 0.667 + 0.38 + 0)/6 = 0.633
– Ex2: (1 + 0.667 + 0.6 + 0.5 + 0.556 + 0.429)/6 = 0.625
• Mean Average Precision: Average of the average precision value for a set
of queries
• Provides a single number value to decide better algorithm
133. Non-Binary Relevance
• Documents are rarely entirely relevant or non-relevant to a query
• Many sources of graded relevance judgments
– Relevance judgments on a 5-point scale
– Multiple judges
– Click distribution and deviation from expected levels (but click-through !=
relevance judgments)
134. A/B Testing
• Exploits existing user base to provide useful feedback
• Randomly send a small fraction (1−10%) of incoming users to a variant of
the system that includes a single change
• Judge effectiveness by measuring change in click-through
• The percentage of users that click on the top result (or any result on the
first page)
135. References
1. Baeza-Yates, R. and Ribeiro-Neto, B. (2011) - Modern Information Retrieval - the concepts and technology behind search -Addison Wesley
2. Grossman, D. A. and Frieder, O. (2004) - Information Retrieval. Algorithms and Heuristics, 2nd ed., volume 15 of The Information Retrieval Series
– Springer
3. Manning, C. D., Raghavan, P., and Schuetze, H., editors (2008) - Introduction to Information Retrieval - Cambridge University Press
4. Roelleke, T., Tsikrika, T., and Kazai, G. (2006) - A general matrix framework for modelling information retrieval - Journal on Information
Processing & Management (IP&M), Special Issue on Theory in Information Retrieval, 42(1)
5. Zaragoza, H., Hiemstra, D., and Tipping, M. (2003) - Bayesian extension to the language model for ad hoc information retrieval - In SIGIR '03:
Proceedings of the 26th annual international ACM SIGIR conference on research and development in information retrieval, pages 4-9, New York,
NY, USA. ACM Press
6. Roelleke, T., Tsikrika, T., and Kazai, G. (2006) - A general matrix framework for modelling information retrieval -Journal on Information Processing
& Management (IP&M), Special Issue on Theory in Information Retrieval, 42(1).
7. Roelleke, T. and Wang, J. (2006) - A parallel derivation of probabilistic information retrieval models - In ACM SIGIR, pages 107-114, Seattle, USA
8. Roelleke, T. and Wang, J. (2008) - TF-IDF uncovered: A study of theories and probabilities -In ACM SIGIR, pages 435 - 442, Singapore
9. Robertson, S. (2004) - Understanding inverse document frequency: On theoretical arguments for idf - Journal of Documentation, 60:503-520
10. Metzler, D. and Croft, W. B. (2004) - Combining the language model and inference network approaches to retrieval - Information Processing &
Management, 40(5):735-750
136. Thank You
Check Out My LinkedIn Profile at
https://in.linkedin.com/in/girishkhanzode
