News Article Ranking : Leveraging the Wisdom of Bloggers
Upcoming SlideShare
Loading in...5
×
 

News Article Ranking : Leveraging the Wisdom of Bloggers

on

  • 700 views

Paper presented at RIAO 2010 by Richard McCreadie entitled 'News Article Ranking: Leveraging the Wisdom of Bloggers'

Paper presented at RIAO 2010 by Richard McCreadie entitled 'News Article Ranking: Leveraging the Wisdom of Bloggers'

Statistics

Views

Total Views
700
Views on SlideShare
696
Embed Views
4

Actions

Likes
0
Downloads
6
Comments
0

1 Embed 4

http://www.slideshare.net 4

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • More blog posts the more important the news articleApproximate editor ranking
  • Displays perrformanceGreen trec best systemsBlue votes spprach
  • sumarise

News Article Ranking : Leveraging the Wisdom of Bloggers News Article Ranking : Leveraging the Wisdom of Bloggers Presentation Transcript

  • News Article Ranking:Leveraging the Wisdom of Bloggers
    Richard McCreadie, Craig Macdonald & IadhOunis
  • Introduction
    Background:
    • Bloggers react to news as it happens
    • Thelwall explored how bloggers reacted to the London bombings
    • 30% of bloggersblog on news-related topics (Technorati poll 2008)
    • Hence, the blogosphere is valuable as a source of news-related information
    • Kȍniget al. & Sayyadiet al. have exploited the blogosphere for event detection
    Obama Victory
    Number of blog posts
    Day (November 2008)
    M. Thelwall WWW’06
    Kȍnig et al. SIGIR’09
    Sayyadi et al. ICWSM’09
  • Introduction
    • Editorial News:
    • Every day newspaper editors select articles for placement within their newspapers.
    • This can be seen as a ranking problem.
    • Rank articles by readership interest
    Front
    Page
    Page
    2
    Newspaper
    Editor
    . . .
    We investigate how such a ranking can be approximated using evidence from the blogosphere
    • Introduction
    • The News Article Ranking Problem
    • The Votes Approach
    • Evaluating Votes
    • Temporal Promotion
    • News Article Representation
    • Conclusions
    Talk Outline
  • News Article Ranking
    Problem Definition:
    • Rank news articles by their inherent importance.
    • Given a day of interest dQ we wish to score each news article a by its predicted importance, score(a,dQ) using evidence from the blogosphere.
    =29
    Day dQ
    =23
    =14
    =13
    News Article
    Ranker
    =4
    =4
    Importance
    Scores
  • Idea:
    • The more blog posts about an article the more important the subject must be.
    • Score by blog post volume
    Approach
    Two Stages:
    Score each news article a for all days d based on related blog post volume for day d.
    News articles are represented by their headlines
    Given a query day dQ rank A based on the score for each news article on day dQ, i.e. score(a, dQ)
    -> a voting process
    The Votes Approach
  • Votes Approach : Stage 1
    Stage 1: Score days for each news story
    1
    1
    2
    3
    4
    2
    3
    4
    Ranking of
    days for a
    blog post
    ranking
    4) Rank days by votes received
    2) Select the top 1000 blog posts for a
    3) Each post votes for a day
    Days
    votes = 2
    votes = 1
    votes = 2
    votes = 2
    For each
    news articlea
    1) Use its representation (headline) as a query
    votes = 0
    votes = 1
    votes = 2
    votes = 0
    Terrier
    Votes
    Voting Model : Count
    * Craig Macdonald PhD thesis 2009
  • Votes Approach : Stage 2
    Stage 2: Rank news articles for day dQ
    votes = 2
    2
    Stage 1
    votes = 2
    votes = 2
    4
    2
    votes = 1
    votes = 2
    News article a
    News article a
    News article a
    1
    4
    1
    2
    3
    votes = 0
    votes = 1
    3
    1
    votes = 0
    3
    votes = 6
    4
    votes = 2
    votes = 6
    3
    4
    Query
    Day 2
    votes = 3
    votes = 2
    News article a
    1
    3
    2
    votes = 1
    votes = 3
    2
    1
    votes = 1
    2
    votes = 9
    1
    votes = 7
    votes = 9
    3
    1
    votes = 5
    votes = 7
    2
    News article a
    3
    votes = 0
    3
    votes = 5
    4
    2
    votes = 0
    4
    Ranking of Articles
    • Introduction
    • The News Article Ranking Problem
    • The Votes Approach
    • Evaluating Votes
    • Temporal Promotion
    • News Article Representation
    • Conclusions
    Talk Outline
  • Hypothesis:
    • The volume of relevant blog posts published on a news article is a strong indicator of that articles importance (from an editors perspective).
    Research Questions:
    • Can the number of related blog posts to a news article published on day dQ provide a comparative ranking to that which an editor might make?
    Evaluating Votes
  • Task
    TREC 2009:
    • Blog Track : top news stories identification task
    • Rank news articles by predicted importance
    • Evidence mined Blogs08
    • 100k Articles provided by the New York Times
    • e.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map’
    Baselines:
    • Random ranking
    • Inlinks (hyperlink evidence vs Votes textual evidence)
    • TREC best systems
  • Setup :
    • TREC 2009 Blog track top news stories identification task
    • 100k news headlines from the New York Times to represent articles
    • E.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map’
    • Uses blog posts from the Blogs08 blog post corpus (28 million posts)
    • Judgments for 50 days of interest (dQ’s)
    • E.g. 2008-05-22 : headline1 headline34 headline35 headline38
    Evaluation:
    • Mean Average Precision (MAP)
    Experimental Setup
    dQ
    Important headlines on dQ
  • Evaluating Votes
    Evaluation:
    • TREC’2009 Blog Track
    • Top stories identification task
    • Blogs08 blog post corpus
    • News Stories from New York Times
    Judgements
    • Participants were asked to label queries as being important or not
    • Criteria:
    • Timing : Favour stories that cover ‘live’ events
    • Significance : Favour stories that effect many people
    • Proximity : Favour stories that are local to the reader (USA)
    • Prominence : Favour stories about celebrities, politicians etc.
    TREC Task:
    • Each participating system needs to rank a set of news articles for a day dQ based upon evidence from the Blogs08 collection.
    • Ranking performance is measured in terms of Mean Average Precision (MAP).
  • Indexing & Retrieval:
    • Indexed Blogs08 using Terrier (stemming, stopwords)
    • Secondary index holds blog post -> day relations
    • Retrieve 1000 blogposts for headlines.
    • DPH (DFR)
    • BM25
    Baselines:
    • Random ranking : average over 10 runs
    • Inlinks : hyperlink evidence
    • TREC 2009 best systems
    Experimental Setup
  • Votes Performance
    Better performance than TREC 2009 best systems
    Results:
    BM25<DPH (DFR)
    Votes + extras
    Hyperlink evidence is of less value than textual evidence
    Votes Approach
    TREC 2009 Best Systems
  • Conclusions:
    • Blog post volume is a decent indicator of editorial importance
    • Can be effectively leveraged to rank news articles by their importance
    • However, still room for improvement (0.17 map)
    Votes Performance
    How can we improve Votes performance?
    • Introduction
    • The News Article Ranking Problem
    • The Votes Approach
    • Evaluating Votes
    • Temporal Promotion
    • News Article Representation
    • Conclusions
    Talk Outline
  • Idea
    • Re-score for each news article using evidence from days before and after dQ.
    Intuition
    • Important stories will be discussed before or after the event
    E.g. Run up to an election
    Temporal Promotion
    Both articles receive the same score for dQ under Votes
    dQ
    Num
    Votes
    Days
  • Hypothesis:
    • An article which is highly blogged about either before or after dQ should be scored more highly than one which is not.
    Approach:
    • Promote articles which were highly blogged before or after dQ
    • Two Techniques
    • NDayBoost
    • GaussBoost
    Temporal Promotion
  • Approach
    • Linearly combines the scores for day dQ with the n days before or after dQ.
    NDayBoost
    dQ
    N = -2
    Num
    Votes
    Score=11
    Score=6
    Days
  • Idea:
    • Evidence will weaken as the distance from dQ increases
    • NDayBoost might over-estimate the importance of days distant from dQ
    Approach:
    • Linearly combine scores as with NDayBoost, but weight each day by its distance from dQ using a Gaussian curve.
    GaussBoost
    Distance of days ∆d
    Weight
  • GaussBoost
    Weight
    ∆d
    Weighting
    • The weight for each article is calculated as :
    • ∆d is the distance (in days) from dQ
    • w is the width of the gaussian curve
    • Controls the score decay as ∆d increases
  • GaussBoost
    Example:
    • n = -2, w = 1
    • Weights downward the scores for each day dependent on w.
    ScoreGaussBoost(B,4)
    = (1*4)+(0.79*1)+(0.18*1)
    = 4.970
    ScoreGaussBoost(A,4)
    = (1*4)+(0.79*4)+(0.18*3)
    = 7.700
    dQ
    N = -2
    Score
    =7.700
    Num
    Votes
    Score=11
    Score
    =4.970
    Score=6
    Days
  • Hypothesis:
    • An article which is highly blogged about either before or after dQ is more likely to be important than one which is not.
    Research Questions:
    • Can the promotion of articles which are highly blogged about before or after dQ improve article ranking performance?
    • Does the quality of evidence decrease as distance from dQ increases?
    • Is historical or future (before or after dQ) blog post evidence more useful?
    Research Questions
  • GaussBoost
    Approach
    • Linearly combine scores as with NDayBoost, but weight each day by its distance from d using a Gaussian curve.
    • The parameter w determines the width of the Gaussian curve, and as such, the weights ∆d for the days.
    ( n = -2, w = 0.5 )
    ScoreGaussBoost(A,4)
    = (1*4)+(0.38*4)+(0.01*3) = 4.608
    ScoreGaussBoost(B,4)
    = (1*4)+(0.38*1)+(0.01*1) = 4.390
    ( n = -2, w = 1 )
    ScoreGaussBoost(A,4)
    = (1*4)+(0.79*4)+(0.18*3) = 7.700
    ScoreGaussBoost(B,4)
    = (1*4)+(0.79*1)+(0.18*1) = 4.970
    Temporal Promotion
  • NDayBoost Performance
    Future blog postings does provide useful evidence
    Baseline DPH+Votes
    MAP
    Historical evidence is not useful for NDayBoost
    n value (days)
  • GaussBoost Performance
    Future blog postings provide stronger evidence than historical postings
    Historical blog postings are useful for days close to dQ
    Baseline DPH+Votes
    MAP
    w value (not days!)
    • Conclusions
    • Both historical and future evidence is useful to improve Votes ranking performance
    • Can use this evidence to generate a better ranking for editors if the data is available
    • Future evidence is more powerful than historical evidence
    • Not too useful if we want to rank in real-time though
    • NDayBoost is only effective for future evidence
    • GaussBoost is effective for both future and historical evidence
    • The most effective of the techniques
    • Does not over emphasise evidence from days distant from dQ
    Temporal Promotion
    • Introduction
    • The News Article Ranking Problem
    • The Votes Approach
    • Evaluating Votes
    • Temporal Promotion
    • News Article Representation
    • Conclusions
    Talk Outline
    Can we improve upon the news article representation?
  • Issue:
    • News articles are represented with headlines
    • e.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map
    • Headlines are a sparse representation of an article
    • Many headlines are not `news-worthy’
    • Editors don’t even consider these
    • e.g. paid death notices
    Approach:
    • Enrich the headlines using related terms extracted from blog posts and Wikipedia.
    • Prune headlines less likely to be news-worthy
    Improving the Article Representation
  • News Article Enrichment
    Idea:
    • Improve the news-article representation (headline)
    • Add related terms (counter sparsity)
    Approach:
    Select retrieve top 3 blog posts from:
    Blogs08
    (query expansion , K. L. Kwok and M. S. Chan. SIGIR 1998)
    Wikipedia
    (collection enrichment, F. Diaz and D. Metzler. SIGIR 2006)
    using DPH (DFR)
    Expand query with the top 10 terms identified using Bo1 (G. Amati, Thesis 2003) from those documents.
    a
    Terrier
    Top
    Terms
    DPH
    Bo1
    Blogs08/Wikipedia
    Query expansion/External Query expansion/Collection Enrichment
  • Related but generic terms
    Case specific terms
  • Article Enrichment:
    • News headlines while being good quality representations are still ambigious
    • Collection enrichment helps find the blog posts that are related.
    Article Improvement Performance
    Collection enrichment with Wikipedia significantly increases performance
    MAP
  • Article Pruning
    Idea:
    • Editors have lots of latent knowledge to draw upon
    • Try simulating this within the system
    • Prune away articles that an editor would not even consider
    Non-stories:
    • Remove news articles which follow editorially defined patterns
    Noisy headlines:
    • Remove misleading dates
    • Remove uppercase category terms
    Patterns List: New York Times
    • Paid Notice
    • Corrections for the Record
    • Comments of the Week
    • Inside the Times
    • Best Sellers
    • The Week Ahead
    • Movie Review
    • Arts Briefly
    • The Listings
    • Dance Review
    • Whats on Today
    • Critics Choice
    • Book of the Times
    • Music Review
    E.g. ‘Inside the Times, November 6, 2008’
    E.g. ‘N.F.L. ROUNDUP; Giants Shut Down
    Tyree for Season; Raiders Cut Hall’
  • Article Pruning:
    • Removing non-news-worthy articles makes the ranking of articles easier.
    Article Pruning Performance
    Dates and Uppercase further increase performance when combined.
    Patterns significantly increase performance over Votes alone
    MAP
  • Additive Results?
    Idea:
    • Combine
    • Temporal promotion (GaussBoost)
    • Headline pruning (All Heuristics)
    • Headline enrichment (Collection Enrichment)
    Results:
    • Significant increase in performance over
    • DPH+Votes
    • DPH+Votes + Single techniques
  • Votes:
    • The volume of blog posts about a news story is a useful measure for the importance from an editorial perspective
    • Can be used to automatically rank news stories for a newspaper editor
    • The Voting model provides strong baseline ranking performace
    Temporal Promotion:
    • Can be beneficial to look at blog post volume either before or after the day of interest
    • More useful to look at tomorrows blog posts than yesterdays blog posts
    • Evidence diminishes as we look further from the day of interest, evidence should be weighted accordingly
    Article representation Improvements
    • Editorshold much in the way of latent knowledge that we need to simulate
    • i.e. they can disregard whole classes of articles as not being news-worthy
    • By pruning away such articlesapriori, ranking performance is improved
    • Headlines are sparse representations of news articles
    • Enrichment with terms from Wikipedia can help find more representative blog posts
    Conclusions
  • TREC 2010:
    • Blog track top stories identification task is running again in 2010
    • Focus on real-time ranking of news (no future evidence)
    • Uses a larger news article collection from Reuters
    Future Work
    Questions?