Improving Top-K Retrieval Algorithms Using Dynamic Programming and ...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

London Information Retrieval Meetup
19 Feb 2019
Improving Top-K Retrieval
Algorithms Using Dynamic
Programming and Longer
...

Introduction
●Top-k retrieval and inverted index
●Introduction to early termination techniques
●Block max wand
Faster Bloc...

Queries over search engines

Inverted index
Documents
term1 term2
term3 term4
term5

Inverted index compression
We compressed posting lists with
partitioned Elias-Fano.
Giuseppe Ottaviano and Rossano Venturi...

Top-K Retrieval
We are interested only in the first
K documents, with k small.

Tf-Idf
In details, we use OKAPI BM25
Term frequency
Inverse document frequency
1 2 5 7 12 13 14 20Doc-id
3 2 1 8 2 4 6 2Fr...

Inverted list iterator operations
next() Find the next document Id
nextGEQ(k) Find the next document id in the list with i...

Ranked-Or
Doc-Id
T1
T2
T3
T4
T5

Early termination techniques
●It is not necessary to compute the score function on all the
postings.
●Max score
●Wand
●Blo...

Wand
Doc-Id
T1
T2
T3
T4
T5
𝜭 = 15.8
Ms = 5.4
Ms = 5.0
Ms = 4.2
Ms = 4.3
Ms = 2.3
Pivot
List
Sum = 5.4
Sum = 14.6
Sum = 9.6...

Block-Max-Wand
Doc-Id
T1
T2
T3
T4
T5
𝜭 = 15.8
3.2 4.2
4.5 5.4 2.1
2.1 3.2
2.1 2.3 0.8
3.5 1.4
4.1
2.7
Pivot
List
Block max...

Block Max Wand
1.Pivot selection as in wand.
2.Compute block max contributions (blockmaxsum) of the pivot doc-id
3.If bloc...

Partitioning
Fixed size blocks Variable size blocks
∑
𝑏∈𝐵
(max(𝑠 ∈ 𝑏) 𝑏 −
∑
𝑠 ∈ 𝑏
𝑠 )
𝑏
min
∑
𝑏∈𝐵
(max(𝑠 ∈ 𝑏) 𝑏 ) +...

Shortest Path Problem
• V postings sorted by their position in the list
• E every possible block in the list
• C(i,j) is t...

Approximation algorithm
● Monotonicity: Quasi-subaddictivity:
𝑂( 𝑛2
) → 𝑂(𝑛 log 𝑈 ) 𝑂( 𝑛log𝑈) → 𝑂(𝑛)
C(i, j) ≤ C(i, j + 1)...

Experimental analysis
Collection Size
Size after
compression
# lists # postings # documents
Gov2 44 GiB 4.4 GiB 35 million...

Choosing block size
Block size Block size

Block-Max-Wand Compression
Maximum impact
element
Boundary doc-id

Block-Max-Wand Compression (score quantization)
Uniform partitioning
Opt partitioning
Sort

Compression algorithms comparisons

Gov2 Clueweb09
Trec2005 Trec2006 Trec2005 Trec2006
Wand 7.06 (1.92x) 12.92 (1.55x) 28.85 (2.25x) 37.55 (1.40x)
MaxScore 6....

Longer skipping
We can do better than skip
at the block boundary.
Ls-boundaryBoundary
Iterate over the
blocks at runtime
A...

Block-Max-Wand
Doc-Id
T1
T2
T3
T4
T5
𝜭 = 15.8
3.2 4.2
4.5 5.4 2.1
2.1 3.2
2.1 2.3 0.8
3.5 1.4
4.1
2.7
Pivot
List
Block max...

Longer skipping
2 3 4 5 6+
VBMW 3.17 (1.45x) 6.39 (1.13x) 8.92 (1.04x) 14.46 (1.00x) 32.04 (1.03x)
VBMW LS 2.18 5.66 8.57 ...

Thank you

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1066413 views
Storytelling For The Web: Integrate... by Chiara Aliotta 986196 views
Artificial Intelligence, Data and C... by OECD Directorate ... 930965 views
How to Leverage AI to Boost Employe... by SocialHRCamp 396116 views
2024 State of Marketing Report – by... by Marius Sescu 223818 views
Everything You Need To Know About C... by Expeed Software 238978 views

View on Slideshare

1 of 26 Ad

View on Slideshare

1 of 26 Ad