News Article Ranking : Leveraging the Wisdom of Bloggers


Published on

Paper presented at RIAO 2010 by Richard McCreadie entitled 'News Article Ranking: Leveraging the Wisdom of Bloggers'

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • More blog posts the more important the news articleApproximate editor ranking
  • Displays perrformanceGreen trec best systemsBlue votes spprach
  • sumarise
  • News Article Ranking : Leveraging the Wisdom of Bloggers

    1. 1. News Article Ranking:Leveraging the Wisdom of Bloggers<br />Richard McCreadie, Craig Macdonald & IadhOunis<br />
    2. 2. Introduction<br />Background:<br /><ul><li>Bloggers react to news as it happens
    3. 3. Thelwall explored how bloggers reacted to the London bombings
    4. 4. 30% of bloggersblog on news-related topics (Technorati poll 2008)
    5. 5. Hence, the blogosphere is valuable as a source of news-related information
    6. 6. Kȍniget al. & Sayyadiet al. have exploited the blogosphere for event detection</li></ul>Obama Victory<br />Number of blog posts<br />Day (November 2008)<br />M. Thelwall WWW’06<br />Kȍnig et al. SIGIR’09<br />Sayyadi et al. ICWSM’09<br />
    7. 7. Introduction<br /><ul><li>Editorial News:
    8. 8. Every day newspaper editors select articles for placement within their newspapers.
    9. 9. This can be seen as a ranking problem.
    10. 10. Rank articles by readership interest</li></ul>Front<br />Page<br />Page<br />2<br />Newspaper<br />Editor<br /> . . .<br />We investigate how such a ranking can be approximated using evidence from the blogosphere<br />
    11. 11. <ul><li>Introduction
    12. 12. The News Article Ranking Problem
    13. 13. The Votes Approach
    14. 14. Evaluating Votes
    15. 15. Temporal Promotion
    16. 16. News Article Representation
    17. 17. Conclusions</li></ul>Talk Outline<br />
    18. 18. News Article Ranking<br />Problem Definition:<br /><ul><li>Rank news articles by their inherent importance.
    19. 19. Given a day of interest dQ we wish to score each news article a by its predicted importance, score(a,dQ) using evidence from the blogosphere.</li></ul>=29<br />Day dQ<br />=23<br />=14<br />=13<br />News Article<br />Ranker<br />=4<br />=4<br />Importance<br />Scores<br />
    20. 20. Idea:<br /><ul><li>The more blog posts about an article the more important the subject must be.
    21. 21. Score by blog post volume</li></ul>Approach<br />Two Stages:<br />Score each news article a for all days d based on related blog post volume for day d.<br /> News articles are represented by their headlines<br />Given a query day dQ rank A based on the score for each news article on day dQ, i.e. score(a, dQ)<br />-> a voting process<br />The Votes Approach<br />
    22. 22. Votes Approach : Stage 1<br />Stage 1: Score days for each news story<br />1<br />1<br />2<br />3<br />4<br />2<br />3<br />4<br />Ranking of <br />days for a<br />blog post<br />ranking<br />4) Rank days by votes received<br />2) Select the top 1000 blog posts for a<br />3) Each post votes for a day<br />Days<br />votes = 2<br />votes = 1<br />votes = 2<br />votes = 2<br />For each<br />news articlea<br />1) Use its representation (headline) as a query<br />votes = 0<br />votes = 1<br />votes = 2<br />votes = 0<br />Terrier<br />Votes<br />Voting Model : Count<br />* Craig Macdonald PhD thesis 2009 <br />
    23. 23. Votes Approach : Stage 2<br />Stage 2: Rank news articles for day dQ<br />votes = 2<br />2<br />Stage 1<br />votes = 2<br />votes = 2<br />4<br />2<br />votes = 1<br />votes = 2<br />News article a<br />News article a<br />News article a<br />1<br />4<br />1<br />2<br />3<br />votes = 0<br />votes = 1<br />3<br />1<br />votes = 0<br />3<br />votes = 6<br />4<br />votes = 2<br />votes = 6<br />3<br />4<br />Query<br />Day 2<br />votes = 3<br />votes = 2<br />News article a<br />1<br />3<br />2<br />votes = 1<br />votes = 3<br />2<br />1<br />votes = 1<br />2<br />votes = 9<br />1<br />votes = 7<br />votes = 9<br />3<br />1<br />votes = 5<br />votes = 7<br />2<br />News article a<br />3<br />votes = 0<br />3<br />votes = 5<br />4<br />2<br />votes = 0<br />4<br />Ranking of Articles<br />
    24. 24. <ul><li>Introduction
    25. 25. The News Article Ranking Problem
    26. 26. The Votes Approach
    27. 27. Evaluating Votes
    28. 28. Temporal Promotion
    29. 29. News Article Representation
    30. 30. Conclusions</li></ul>Talk Outline<br />
    31. 31. Hypothesis:<br /><ul><li>The volume of relevant blog posts published on a news article is a strong indicator of that articles importance (from an editors perspective).</li></ul>Research Questions:<br /><ul><li>Can the number of related blog posts to a news article published on day dQ provide a comparative ranking to that which an editor might make?</li></ul>Evaluating Votes<br />
    32. 32. Task<br />TREC 2009:<br /><ul><li>Blog Track : top news stories identification task
    33. 33. Rank news articles by predicted importance
    34. 34. Evidence mined Blogs08
    35. 35. 100k Articles provided by the New York Times
    36. 36. e.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map’</li></ul>Baselines:<br /><ul><li>Random ranking
    37. 37. Inlinks (hyperlink evidence vs Votes textual evidence)
    38. 38. TREC best systems</li></li></ul><li>Setup :<br /><ul><li>TREC 2009 Blog track top news stories identification task
    39. 39. 100k news headlines from the New York Times to represent articles
    40. 40. E.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map’
    41. 41. Uses blog posts from the Blogs08 blog post corpus (28 million posts)
    42. 42. Judgments for 50 days of interest (dQ’s)
    43. 43. E.g. 2008-05-22 : headline1 headline34 headline35 headline38</li></ul>Evaluation:<br /><ul><li>Mean Average Precision (MAP)</li></ul>Experimental Setup<br />dQ<br />Important headlines on dQ<br />
    44. 44. Evaluating Votes<br />Evaluation:<br /><ul><li>TREC’2009 Blog Track
    45. 45. Top stories identification task
    46. 46. Blogs08 blog post corpus
    47. 47. News Stories from New York Times</li></ul>Judgements<br /><ul><li>Participants were asked to label queries as being important or not
    48. 48. Criteria:
    49. 49. Timing : Favour stories that cover ‘live’ events
    50. 50. Significance : Favour stories that effect many people
    51. 51. Proximity : Favour stories that are local to the reader (USA)
    52. 52. Prominence : Favour stories about celebrities, politicians etc.</li></ul>TREC Task:<br /><ul><li>Each participating system needs to rank a set of news articles for a day dQ based upon evidence from the Blogs08 collection.
    53. 53. Ranking performance is measured in terms of Mean Average Precision (MAP).</li></li></ul><li>Indexing & Retrieval:<br /><ul><li>Indexed Blogs08 using Terrier (stemming, stopwords)
    54. 54. Secondary index holds blog post -> day relations
    55. 55. Retrieve 1000 blogposts for headlines.
    56. 56. DPH (DFR)
    57. 57. BM25</li></ul>Baselines:<br /><ul><li>Random ranking : average over 10 runs
    58. 58. Inlinks : hyperlink evidence
    59. 59. TREC 2009 best systems</li></ul>Experimental Setup<br />
    60. 60. Votes Performance<br />Better performance than TREC 2009 best systems<br />Results:<br />BM25<DPH (DFR)<br />Votes + extras<br />Hyperlink evidence is of less value than textual evidence<br />Votes Approach<br />TREC 2009 Best Systems<br />
    61. 61. Conclusions:<br /><ul><li>Blog post volume is a decent indicator of editorial importance
    62. 62. Can be effectively leveraged to rank news articles by their importance
    63. 63. However, still room for improvement (0.17 map)</li></ul>Votes Performance<br />How can we improve Votes performance?<br />
    64. 64. <ul><li>Introduction
    65. 65. The News Article Ranking Problem
    66. 66. The Votes Approach
    67. 67. Evaluating Votes
    68. 68. Temporal Promotion
    69. 69. News Article Representation
    70. 70. Conclusions</li></ul>Talk Outline<br />
    71. 71. Idea<br /><ul><li>Re-score for each news article using evidence from days before and after dQ.</li></ul>Intuition<br /><ul><li>Important stories will be discussed before or after the event</li></ul>E.g. Run up to an election<br />Temporal Promotion<br />Both articles receive the same score for dQ under Votes<br />dQ<br />Num<br />Votes<br />Days<br />
    72. 72. Hypothesis:<br /><ul><li>An article which is highly blogged about either before or after dQ should be scored more highly than one which is not.</li></ul>Approach:<br /><ul><li>Promote articles which were highly blogged before or after dQ
    73. 73. Two Techniques
    74. 74. NDayBoost
    75. 75. GaussBoost</li></ul>Temporal Promotion<br />
    76. 76. Approach<br /><ul><li>Linearly combines the scores for day dQ with the n days before or after dQ.</li></ul>NDayBoost<br />dQ<br />N = -2<br />Num<br />Votes<br />Score=11<br />Score=6<br />Days<br />
    77. 77. Idea:<br /><ul><li>Evidence will weaken as the distance from dQ increases
    78. 78. NDayBoost might over-estimate the importance of days distant from dQ</li></ul>Approach:<br /><ul><li>Linearly combine scores as with NDayBoost, but weight each day by its distance from dQ using a Gaussian curve.</li></ul>GaussBoost<br />Distance of days ∆d <br />Weight<br />
    79. 79. GaussBoost<br />Weight<br />∆d<br />Weighting<br /><ul><li>The weight for each article is calculated as :
    80. 80. ∆d is the distance (in days) from dQ
    81. 81. w is the width of the gaussian curve
    82. 82. Controls the score decay as ∆d increases</li></li></ul><li>GaussBoost<br />Example:<br /><ul><li>n = -2, w = 1
    83. 83. Weights downward the scores for each day dependent on w.</li></ul>ScoreGaussBoost(B,4)<br /> = (1*4)+(0.79*1)+(0.18*1)<br /> = 4.970<br />ScoreGaussBoost(A,4)<br /> = (1*4)+(0.79*4)+(0.18*3)<br /> = 7.700<br />dQ<br />N = -2<br />Score<br />=7.700<br />Num<br />Votes<br />Score=11<br />Score<br />=4.970<br />Score=6<br />Days<br />
    84. 84. Hypothesis:<br /><ul><li>An article which is highly blogged about either before or after dQ is more likely to be important than one which is not.</li></ul>Research Questions:<br /><ul><li>Can the promotion of articles which are highly blogged about before or after dQ improve article ranking performance?
    85. 85. Does the quality of evidence decrease as distance from dQ increases?
    86. 86. Is historical or future (before or after dQ) blog post evidence more useful?</li></ul>Research Questions<br />
    87. 87. GaussBoost<br />Approach<br /><ul><li>Linearly combine scores as with NDayBoost, but weight each day by its distance from d using a Gaussian curve.
    88. 88. The parameter w determines the width of the Gaussian curve, and as such, the weights ∆d for the days.</li></ul>( n = -2, w = 0.5 )<br />ScoreGaussBoost(A,4)<br /> = (1*4)+(0.38*4)+(0.01*3) = 4.608<br />ScoreGaussBoost(B,4)<br /> = (1*4)+(0.38*1)+(0.01*1) = 4.390<br />( n = -2, w = 1 )<br />ScoreGaussBoost(A,4)<br /> = (1*4)+(0.79*4)+(0.18*3) = 7.700<br />ScoreGaussBoost(B,4)<br /> = (1*4)+(0.79*1)+(0.18*1) = 4.970<br />Temporal Promotion<br />
    89. 89. NDayBoost Performance<br />Future blog postings does provide useful evidence<br />Baseline DPH+Votes<br />MAP<br />Historical evidence is not useful for NDayBoost<br />n value (days)<br />
    90. 90. GaussBoost Performance<br />Future blog postings provide stronger evidence than historical postings<br />Historical blog postings are useful for days close to dQ<br />Baseline DPH+Votes<br />MAP<br />w value (not days!)<br />
    91. 91. <ul><li>Conclusions
    92. 92. Both historical and future evidence is useful to improve Votes ranking performance
    93. 93. Can use this evidence to generate a better ranking for editors if the data is available
    94. 94. Future evidence is more powerful than historical evidence
    95. 95. Not too useful if we want to rank in real-time though
    96. 96. NDayBoost is only effective for future evidence
    97. 97. GaussBoost is effective for both future and historical evidence
    98. 98. The most effective of the techniques
    99. 99. Does not over emphasise evidence from days distant from dQ</li></ul>Temporal Promotion<br />
    100. 100. <ul><li>Introduction
    101. 101. The News Article Ranking Problem
    102. 102. The Votes Approach
    103. 103. Evaluating Votes
    104. 104. Temporal Promotion
    105. 105. News Article Representation
    106. 106. Conclusions</li></ul>Talk Outline<br />Can we improve upon the news article representation?<br />
    107. 107. Issue:<br /><ul><li>News articles are represented with headlines
    108. 108. e.g. ‘In a Decisive Victory, Obama Reshapes the Electoral Map
    109. 109. Headlines are a sparse representation of an article
    110. 110. Many headlines are not `news-worthy’
    111. 111. Editors don’t even consider these
    112. 112. e.g. paid death notices</li></ul>Approach:<br /><ul><li>Enrich the headlines using related terms extracted from blog posts and Wikipedia.
    113. 113. Prune headlines less likely to be news-worthy</li></ul>Improving the Article Representation<br />
    114. 114. News Article Enrichment<br />Idea:<br /><ul><li>Improve the news-article representation (headline)
    115. 115. Add related terms (counter sparsity)</li></ul>Approach:<br />Select retrieve top 3 blog posts from: <br />Blogs08 <br />(query expansion , K. L. Kwok and M. S. Chan. SIGIR 1998)<br />Wikipedia<br />(collection enrichment, F. Diaz and D. Metzler. SIGIR 2006) <br />using DPH (DFR)<br />Expand query with the top 10 terms identified using Bo1 (G. Amati, Thesis 2003) from those documents.<br />a<br />Terrier<br />Top<br />Terms<br />DPH<br />Bo1<br />Blogs08/Wikipedia<br />Query expansion/External Query expansion/Collection Enrichment<br />
    116. 116. Related but generic terms<br />Case specific terms<br />
    117. 117. Article Enrichment:<br /><ul><li>News headlines while being good quality representations are still ambigious
    118. 118. Collection enrichment helps find the blog posts that are related.</li></ul>Article Improvement Performance<br />Collection enrichment with Wikipedia significantly increases performance<br />MAP<br />
    119. 119. Article Pruning<br />Idea:<br /><ul><li>Editors have lots of latent knowledge to draw upon
    120. 120. Try simulating this within the system
    121. 121. Prune away articles that an editor would not even consider</li></ul>Non-stories:<br /><ul><li>Remove news articles which follow editorially defined patterns</li></ul>Noisy headlines:<br /><ul><li>Remove misleading dates
    122. 122. Remove uppercase category terms</li></ul>Patterns List: New York Times<br /><ul><li>Paid Notice
    123. 123. Corrections for the Record
    124. 124. Comments of the Week
    125. 125. Inside the Times
    126. 126. Best Sellers
    127. 127. The Week Ahead
    128. 128. Movie Review
    129. 129. Arts Briefly
    130. 130. The Listings
    131. 131. Dance Review
    132. 132. Whats on Today
    133. 133. Critics Choice
    134. 134. Book of the Times
    135. 135. Music Review</li></ul>E.g. ‘Inside the Times, November 6, 2008’<br />E.g. ‘N.F.L. ROUNDUP; Giants Shut Down<br /> Tyree for Season; Raiders Cut Hall’<br />
    136. 136. Article Pruning:<br /><ul><li>Removing non-news-worthy articles makes the ranking of articles easier.</li></ul>Article Pruning Performance<br />Dates and Uppercase further increase performance when combined.<br />Patterns significantly increase performance over Votes alone<br />MAP<br />
    137. 137. Additive Results?<br />Idea:<br /><ul><li>Combine
    138. 138. Temporal promotion (GaussBoost)
    139. 139. Headline pruning (All Heuristics)
    140. 140. Headline enrichment (Collection Enrichment)</li></ul>Results:<br /><ul><li>Significant increase in performance over
    141. 141. DPH+Votes
    142. 142. DPH+Votes + Single techniques</li></li></ul><li>Votes:<br /><ul><li>The volume of blog posts about a news story is a useful measure for the importance from an editorial perspective
    143. 143. Can be used to automatically rank news stories for a newspaper editor
    144. 144. The Voting model provides strong baseline ranking performace</li></ul>Temporal Promotion:<br /><ul><li>Can be beneficial to look at blog post volume either before or after the day of interest
    145. 145. More useful to look at tomorrows blog posts than yesterdays blog posts
    146. 146. Evidence diminishes as we look further from the day of interest, evidence should be weighted accordingly</li></ul>Article representation Improvements<br /><ul><li>Editorshold much in the way of latent knowledge that we need to simulate
    147. 147. i.e. they can disregard whole classes of articles as not being news-worthy
    148. 148. By pruning away such articlesapriori, ranking performance is improved
    149. 149. Headlines are sparse representations of news articles
    150. 150. Enrichment with terms from Wikipedia can help find more representative blog posts</li></ul>Conclusions<br />
    151. 151. TREC 2010:<br /><ul><li>Blog track top stories identification task is running again in 2010
    152. 152. Focus on real-time ranking of news (no future evidence)
    153. 153. Uses a larger news article collection from Reuters</li></ul>Future Work<br />Questions?<br />