Retrieval and Feedback Models for Blog Feed Search SIGIR 2008 Singapore Jonathan Elsas, Jaime Arguello, Jamie Callan & Jaime Carbonell LTI/SCS/CMU

Outline The task Overview of Blogs & Blog Search Challenges in Blog Search Our approach Retrieval Models Query Expansion Models Conclusion

The task

Overview of Blogs & Blog Search

Challenges in Blog Search

Our approach

Retrieval Models

Query Expansion Models

Conclusion

Background

What is a Blog?

What is a Feed? <xml> <feed> <entry> <author>Peter …</> <title>Good, Evil…</> <content>I’ve said…</> </entry> <entry> <author>Peter …</> <title>Agreeing…</> <content>Some peo…</> </entry> …

Blog-Feed Correspondence Blog Feed Post Entry HTML XML

Why are Blogs important? Technorati currently tracking > 112.8 Million Blogs > 175,000 new Blogs per day > 1.6 Million posts per day [http://www.technorati.com/about/]

Technorati currently tracking > 112.8 Million Blogs > 175,000 new Blogs per day > 1.6 Million posts per day

The Task

Feed Search at TREC Ranking Blogs/Feeds (collections of posts) in response to a user’s query, [X] “ A relevant feed should have a principle and recurring interest in X ” — TREC 2007 Blog Track (a.k.a. Blog Distillation)

Ranking Blogs/Feeds (collections of posts) in response to a user’s query, [X]

“ A relevant feed should have a principle and recurring interest in X ”

— TREC 2007 Blog Track

Feed Search at TREC [Gardening] [Apple iPod] [Violence in Sudan] [Gun Control] [Food] [Wine] Represent Ongoing Information Needs Frequently Very General

[Gardening]

[Apple iPod]

[Violence in Sudan]

[Gun Control]

[Food]

[Wine]

Challenges in Feed Search

Challenges in Feed Search A feed is a collection of documents entries time feed

A feed is a collection of documents

A feed is a collection of documents How does relevance at the entry level correspond to relevance at the feed level? Challenges in Feed Search entries time feed

A feed is a collection of documents

How does relevance at the entry level correspond to relevance at the feed level?

Challenges in Feed Search 2. Even a topical feed is topically diverse time Space Exploration topic NASA China’s plans for the moon shuttle launch My dog Mars rover Boeing

2. Even a topical feed is topically diverse

Challenges in Feed Search 2. Even a topical feed is topically diverse Can we favor entries close to the central topic of the feed? Space Exploration time topic

2. Even a topical feed is topically diverse

Can we favor entries close to the central topic of the feed?

Challenges in Feed Search 3. Feeds are noisy Spam blogs, Spam & off topic comments time

3. Feeds are noisy

Spam blogs, Spam & off topic comments

Challenges in Feed Search 4. General & Ongoing Information Needs [Mac] [Music] [Food] [Wine] … post regularly about new products , features , or application software of Apple Mac computers. … describing songs , biographies of musicians, musical styles and their influences of music on people are discussed. … such as tastings , reviews , food matching or pairing , and oenophile news and events . … describing experiences eating cuisines, culinary delights , recipes , nutrition plans .

4. General & Ongoing Information Needs

Our Approach

Feeds: Topically Diverse Noisy Collections Information Needs: General & Ongoing Challenges Our Approach Retrieval Models Feedback Models

Topically Diverse

Noisy

Collections

Retrieval Models Challenge: ranking topically diverse collections Representation: feed vs. entry Model topical relationship between entries

Challenge: ranking topically diverse collections

Representation: feed vs. entry

Model topical relationship between entries

Large Document (Feed) Model [Q] <?xml… … </…> `<?xml… … </…> <?xml… … </…> <?xml… <feed> <entry> <entry> <entry> <entry> <entry> … </…> <?xml… … </…> <?xml… … </…> <?xml… … </…> <?xml… <feed> <entry> <entry> <entry> <entry> <entry> … </…> Feed Document Collection Ranked Feeds Rank by Indri’s standard retrieval model [Metzler and Croft, 2004; 2005]

Large Document (Feed) Model Advantages: A straightforward application of existing retrieval techniques Potential Pitfalls: Large entries dominate a feed’s language model Ignores relationship among entries Feed Entry E E Entry Entry E

Advantages:

A straightforward application of existing retrieval techniques

Potential Pitfalls:

Large entries dominate a feed’s language model

Ignores relationship among entries

Small Document (Entry) Model Ranked Entries [Q] <entry> <entry> <entry> <entry> <?xml… <entry> Entry Document Collection <entry> <entry> <entry> <entry> <?xml… <entry> <entry> <entry> <entry> <entry> <?xml… <entry> <entry> <entry> <entry> <entry> <?xml… <entry> <entry> <entry> <entry> <entry> <?xml… <entry> <entry> <entry> <entry> <entry> <?xml… <entry> <entry> <entry> <entry> <entry> <?xml… <entry> Ranked Feeds document = entry Apply some rank aggregation function Rank By

Small Document (Entry) Model Query Likelihood Entry Centrality Feed Prior: favors longer feeds ReDDE Federated Search Algortihm [Si & Callan, 2003]

Query Likelihood

Entry Centrality

Feed Prior: favors longer feeds

Entry Centrality Uniform : Geometric Mean : time topic

Uniform :

Geometric Mean :

Small Document (Entry) Model Advantages: Controls for differing entry length Models topical relationship among entries Disadvantages: Centrality computation is slow(er) Not only improves speed, Also performance Q

Advantages:

Controls for differing entry length

Models topical relationship among entries

Disadvantages:

Centrality computation is slow(er)

Retrieval Model Results

Retrieval Model Results 45 Queries from the TREC 2007 Blog Distillation Task BLOG06 test collection, XML feeds only 5-Fold Cross Validation for all retrieval model smoothing parameters

45 Queries from the TREC 2007 Blog Distillation Task

BLOG06 test collection, XML feeds only

5-Fold Cross Validation for all retrieval model smoothing parameters

Retrieval Model Results Mean Average Precision Large Document (Feed) Model Small Document (Entry) Models

Retrieval Model Results Mean Average Precision Uniform Log(Feed Length) Uniform Log Prior Map 0.188

Retrieval Model Results Mean Average Precision Uniform Log(Feed Length) Uniform n/a

Feedback Models Challenge: Noisy collection with general & ongoing information needs Use a cleaner external collection for query expansion (Wikipedia) With an expansion technique designed to identify multiple query facets

Challenge: Noisy collection with general & ongoing information needs

Use a cleaner external collection for query expansion (Wikipedia)

With an expansion technique designed to identify multiple query facets

Query Expansion (PRF) [Q] BLOG06 Collection Related Terms from top K documents [Q + Terms] [Lavrenko & Croft, 2001]

Query Expansion Example Ideal digital photography depth of field photographic film photojournalism cinematography [Photography] PRF photography nude erotic art girl free teen fashion women

Ideal

digital photography

depth of field

photographic film

photojournalism

cinematography

Feedback Model Results Mean Average Precision None PRF

Query Expansion (Wikipedia PRF) [Q] BLOG06 Collection [Q + Terms] [Lavrenko & Croft, 2001] Wikipedia [Diaz & Metzler, 2006] Related Terms from top K documents

Query Expansion Example Ideal digital photography depth of field photographic film photojournalism cinematography [Photography] PRF photography nude erotic art girl free teen fashion women Wikipedia PRF photography director special film art camera music cinematographer photographic

Ideal

digital photography

depth of field

photographic film

photojournalism

cinematography

Feedback Model Results Mean Average Precision None PRF Wiki. PRF

Query Expansion (Wikipedia Link) [Q] BLOG06 Collection [Q + Terms] Wikipedia Related Terms from link structure

Wikipedia Link-Based Query Expansion

Wikipedia Link-Based Expansion Wikipedia … Q

Wikipedia Link-Based Expansion … Relevance Set, Top R = 100 Working Set, Top W = 1000 Q Wikipedia

Wikipedia Link-Based Expansion … Wikipedia Q Relevance Set, Top R = 100 Working Set, Top W = 1000

Wikipedia Link-Based Expansion Relevance Set, Top R = 100 Working Set, Top W = 1000 … Wikipedia Extract anchor text from Working Set that link to the Relevance Set . Q

Wikipedia Link-Based Expansion Relevance Set, Top R = 500 Working Set, Top W = 1000 … Wikipedia Extract anchor text from Working Set that link to the Relevance Set . Q Combines relevance and popularity Relevance: An anchor phrase that links to a high ranked article gets a high score Popularity: An anchor phrase that links many times to a mid-ranked articles also gets high score

Query Expansion Example Wikipedia Link-Based photography photographer digital photography photographic depth of field feature photography film photographic film photojournalism [Photography] PRF photography nude erotic art girl free teen fashion women Ideal digital photography depth of field photographic film photojournalism cinematography

Wikipedia Link-Based

photography

photographer

digital photography

photographic

depth of field

feature photography

film

photographic film

photojournalism

Feedback Model Results Mean Average Precision None PRF Wiki. PRF Wiki. Link

Conclusion Feed Search Challenges: Feeds are topically diverse, noisy collections Ranked against ongoing & general information needs Novel Retrieval Models: Ranking collections, sensitive to topical relationship among entries Novel Feedback Models: Discover multiple query facets & robust to collection noise

Feed Search Challenges:

Feeds are topically diverse, noisy collections

Ranked against ongoing & general information needs

Novel Retrieval Models:

Ranking collections, sensitive to topical relationship among entries

Novel Feedback Models:

Discover multiple query facets & robust to collection noise

Thank You! Student Travel Grant funding from: ACM SIGIR, Amit Singhal, Microsoft Research

Entry Centrality GM Derivation where Entry Generation Likelihood: |E|

Query Expansion Examples Wikipedia Expansion Music Folk music Electronic music Folk Music video World music Ambient Electronic Country music [Music] PRF Music Country Download Free MP3 Mp3andmore Lyric Listen Song

Wikipedia Expansion

Music

Folk music

Electronic music

Folk

Music video

World music

Ambient

Electronic

Country music

Query Expansion Examples Wikipedia Expansion scotland scottish parliament scottish scottish national party wars of scottish independence scottish independence william wallace glasgow scottish socialist party [Scottish Independence] PRF scotland independence party convention politics snp national people scot

Wikipedia Expansion

scotland

scottish parliament

scottish

scottish national party

wars of scottish independence

scottish independence

william wallace

glasgow

scottish socialist party

Query Expansion Examples Wikipedia Expansion machine learning learning artificial intelligence turing machine machine gun neural network support vector machine supervised learning artificial neural network [Machine Learning] PRF learn machine credit card karaoke journal sex model sew

Wikipedia Expansion

machine learning

learning

artificial intelligence

turing machine

machine gun

neural network

support vector machine

supervised learning

artificial neural network

Query Generality Characteristics Query Length: BLOG: 1.9 words TB04: 3.2 words TB05: 3.0 words ODP Depth BLOG: 4.7 levels TB04: 5.2 levels TB05: 5.3 levels

Query Length:

BLOG: 1.9 words

TB04: 3.2 words

TB05: 3.0 words

ODP Depth

BLOG: 4.7 levels

TB04: 5.2 levels

TB05: 5.3 levels

Relevance Set Cohesiveness … Relevance Set, Top R = 100 Wikipedia Cohesiveness = | L in | | L in U L out |

Relevant Set Cohesiveness

Is it the Queries? Feed Search Queries ≠ TB Adhoc Queries But, none of these measures predict whether wikipedia expansions helps…

Feed Search Queries

≠

TB Adhoc Queries