Presentation by Tomasz Korzeniowski from Oredev conference in Malmoe 2007.

2. Tomasz Korzeniowski
tomek@polarrose.com

3. Information Retrieval

7. Retrieval strategies
• Vector Space Model
• Latent Semantic Indexing
• Probabilistic Retrieval Strategies
• Language Models
• Inference Networks
• Extended Boolean Retrieval
• Neural Networks
• Genetic Algorithms
• Fuzzy Set Retrieval

8. Vector space model

9. Text retrieval

10. Analysis

11. Tokenization

12. Stop-words

13. Stemming
Lemmatization

14. http://tartarus.org/~martin/
PorterStemmer/

15. Document
Term

17. Term frequency

18. r boost for a query on ferrari than the
get from a query on insurance.
entInversionof a term used to sca
frequency df document
total number of documents in a corpu
frequency follows:
frequency (idf) of a term t as
N
idft = log .
dft
rare term is high, whereas the idf of a
ure 6.4 gives an example of idf’s in a co

19. g scheme assigns to term
tf-idft,d = tft,d × idft .
ssigns to term t a weigh

20. Search

21. 7 Vector space re
6
v(q)
v(d2 )
B
¨
¨
¨¨ v(d2 )
I
¨
¨
¨¨
¨¨
¨
¨
-
¨
Cosine similarity illustrated.
igure 7.1

23. Q: “gold silver truck”
D1: “Shipment of gold damaged in a
fire”
D2: “Delivery of silver arrived in a
silver truck”
D3: “Shipment of gold arrived in a
truck”

24. TF
a arrived damaged delivery fire gold in of shipment silver truck
D1 1 1 1 1 11 1
0 0 0 0
D2 1 1 1 11 2
0 0 0 0 0
D3 1 1 1 11 1 1
0 0 0 0
1 1 1
Q 0 0 0 0 0 00 0

25. N
idft = log .
dft
• • of
area term is high, whereas 0the idf of
0
log 3/3 = log 3/3 =
• arrived • silver
re 6.4 gives0.176 example of idf’s in a
an 0.477
log 3/2 = log 3/1 =
• damaged • shipment
ample logarithms are to the base 10.
0.477 0.176
log 3/1 = log 3/2 =
• delivery • truck
0.477 0.176
log 3/1 = log 3/2 =
• fire • gold
0.477 0.176
log 3/1 = log 3/2 =
always finite?
• in 0
log 3/3 =

26. a arrived damaged delivery fire gold in of shipment silver truck
0.477 0.477 0.176 0 0 0.176
D1 0 0 0 0 0
0.176 0.477 0.954 0.176
D2 0 0 0 0 00 0
0.176 0.176 0 0 0.176 0.176
D3 0 0 0 0 0
0.176 0 0 0.477 0.176
Q 0 0 0 0 0 0

27. SC(Q,D1) = (0)(0)+(0)(0)+(0)(0.477)+(0)
(0)+(0)(0.477)+(0.176)(0.176)+(0)(0)+(0)
(0)+(0)(0.176)+(0.477)(0)+(0.176)(0)=
(0.176)(0.176) ⋲ 0.031

28. SC(Q,D2)=(0.954)(0.477)+(0.176)(0.176) ⋲ 0.486
SC(Q,D3)=(0.176)(0.176)+(0.176)(0.176) ⋲ 0.062

29. Inverted index

30. term - 1 (dn,1) (d10,1)
term - 2 (dn,5) (dn,3)
term - 3 (d2,11) (d10,1)
term - 4 (dn,1) (d2,1)
term - 5 (dn,2) (d4,3)
term - n (d6,1) (d7,3)

31. Lucene

32. Analysis

35. Lucene includes several built-in analyzers. The primary ones are shown in table 4.2.
We’ll leave discussion of the two language-specific analyzers, RussianAnalyzer
and GermanAnalyzer, to section 4.8.2 and the special per-field analyzer wrapper,
PerFieldAnalyzerWrapper, to section 4.4.
Table 4.2 Primary analyzers available in Lucene
Analyzer Steps taken
Splits tokens at whitespace
WhitespaceAnalyzer
Divides text at nonletter characters and lowercases
SimpleAnalyzer
Divides text at nonletter characters, lowercases, and removes stop words
StopAnalyzer
Tokenizes based on a sophisticated grammar that recognizes e-mail
StandardAnalyzer
addresses, acronyms, Chinese-Japanese-Korean characters,
alphanumerics, and more; lowercases; and removes stop words
The built-in analyzers we discuss in this section—WhitespaceAnalyzer, Simple-
Analyzer, StopAnalyzer, and StandardAnalyzer—are designed to work with text in
almost any Western (European-based) language. You can see the effect of each of
these analyzers in the output in section 4.2.3. WhitespaceAnalyzer and Simple-
Analyzer are both trivial and we don’t cover them in more detail here. We explore
the StopAnalyzer and StandardAnalyzer in more depth because they have non-

36. Index

37. Index
• IndexWriter
• Directory
• Analyzer
• Document
• Field

38. ex options: store
store
Value Description
:no Don’t store field
:yes Store field in its original format.
Use this value if you want to highlight
matches or print match excerpts a la Google
search.
:compressed Store field in compressed format.

39. index
Index options: index
Value Description
:no Do not make this field searchable.
:yes Make this field searchable and tok-
enize its contents.
:untokenized Make this field searchable but do not
tokenize its contents. Use this value
for fields you wish to sort by.
:omit norms Same as :yes except omit the norms
file. The norms file can be omit-
ted if you don’t boost any fields and
you don’t need scoring based on field
length.
:untokenized omit norms Same as :untokenized except omit the
norms file.
Ruby Day Kraków: Full Text Search with Ferret

40. term_vector
Index options: term vector
Value Description
:no Don’t store term-vectors
:yes Store term-vectors without storing positions
or offsets.
:with positions Store term-vectors with positions.
:with offsets Store term-vectors with offsets.
:with positions ofssets Store term-vectors with positions and off-
sets.
Ruby Day Kraków: Full Text Search with Ferret

42. Search

43. Search
• IndexSearcher
• Term
• Query
• Hits

44. Query

45. Query
• API
• new TermQuery(new Term(“name”,”Tomek”));
• Lucene QueryParser
• queryParser.parse(“name:Tomekquot;);

46. TermQuery
name:Tomek

47. BooleanQuery
ramobo OR ninja
+rambo +ninja –name:rocky

48. PhraseQuery
“ninja java” –name:rocky

49. SloppyPhraseQuery
“red-faced politicians”~3

50. RangeQuery
releaseDate:[2000 TO 2007]

51. WildcardQuery
sup?r, su*r, super*

52. FuzzyQuery
color~
colour, collor, colro

53. http://en.wikipedia.org/wiki/Levenshtein_distance
color colour - 1
colour coller - 2

54. Equation 1. Levenstein Distance Score
This means that an exact match will h
corresponding letters will have a score

55. Boost
title:Spring^10