The use of the Opinionated facet allowed to leverage past track work on opinion-finding 9 } binary
Topics One appropriate facet added to each topic <query> hugo chavez </query> <desc> I am looking for blogs that talk about Venezuelan president Hugo Chavez and his politics. </desc> <facet> indepth </facet> <narr>I want to follow blogs that talk about Hugo Chavez, the president of Venezuela. Blogs that follow his role in Venezuelan politics are relevant, as well as those that discuss non-political stories and activities. I am more interested in blogs about Chavez than blogs about Venezuelan politics generally.</narr> 50 new topics were created by TREC assessors:
Task complexity, early-stage techniques, or difficult topics? 14
Baseline runs results: 39 topics; Top 5 Groups; Title-only (ranked by MAP) Topic relevance model and expansion using terms from <desc> and <narr> topic fields. Blog posts ranked using BM25, then scores aggregated to blogs Fuzzy aggregation methods to combine regularized blog posts scores into blog scores.
Most of the groups indexed only the Permalinks components of Blogs08
Almost all deployed retrieval techniques scored a blog based on the scores of its corresponding relevant posts
Faceted blog search runs results: 39 topics; Top 5 Groups; Ranked by ALL (MAP) Indepth facet: posts scored using Cross Entropy. For other facets: Mutual Information is used to weight terms in posts, using various lexicons. Did not attempt faceted search. Post scores are altered using temporal information before being aggregated into blog scores. Learned a classifier for the Indepth facet. For other facets, they used heuristics to score blog posts before aggregation.
Faceted search proved to be particularly challenging
For all groups, and in almost all cases: Applying faceted search leads to a decrease in performance viz. the faceted performance of the baseline ranking
Outline Blog Track: Background TREC Blog Track 2009 Overview
Top Stories Identification Task Many blog search engine queries are news-related New task’s main research question:How well does the blogosphere respond to real-world events? Facilitated by the Blogs08 test collection – 54 weeks in length, including
Runs and Assessments A run consists of a ranking of 100 headlines, each supported by up to 10 diverse blog posts
Runs use the SUPPORTing run format developed for the Enterprise track expert search task
25 runs by 7 groups: pooled top 20 headlines from each run
Two phases of participant community judging:
Top news story judging: Identify important news stories for each day
Blog post judging: Identify relevant and diverse blog posts for relevant headlines
Phase 1: Top News Story Judging We asked assessors to take the role of a newspaper editor
What stories would they put on the front page of a newspaper or news website?
Assess whether the headline actually occurred on the query day, and judge each headline story as “Important” or “Not Important”
Could consider their own recollection of events, or refer to external Web resources
Editorial factors to consider: Timing, Significance, Prominence, Human Interest, Proximity Interface provided pool of headlines to judge, headline and snippet of story, and link to actual NYT news article 23
Phase 2: Blog Post Judging Once headlines were judged, important ones were sampled for which to perform blog post judging
2-phase judging avoids judging blog posts at the same time as judging headline
Assessors only have to read blog posts for judged important headlines
Blog posts were judged “Relevant” or “Not Relevant” to the headline When judging, assessors defined “aspects” to group relevant blog posts
e.g. for a headline on the Oscars, the assessor defined aspects such as “liveblogs”, “factual”, “opinionated”, “accuracy of predictions”
Task was fairly difficult: retrieval performances were rather low
Identifying Top News Stories: Runs Voting Model: Number of blog posts mentioning a headline. Probabilistic: Combination of query generating headline probability and headline prior calculated from time- or term-based evidence Two probabilistic approaches: news to blogs or blogs to news.
All groups indexed only the Permalinks component of Blogs08 (exceptions are UAms & USI)