Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Proposal for Evaluating Answer Distillation from Web Data

440 views

Published on

Answer Distillation presentation from SIGIR 2016 WebQA Workshop.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

A Proposal for Evaluating Answer Distillation from Web Data

  1. 1. A Proposal for Evaluating Answer Distillation from Web Data Bhaskar Mitra, Grady Simon, Jianfeng Gao, Nick Craswell & Li Deng
  2. 2. Answer distillation task Given query and passage containing answer, summarize answer for presentation, on- screen or read-out. Answers are query-biased summaries: • Single entities or phrases (e.g., “Rome” for the query “Italy capital”) • Multi-sentence (e.g., for the query “how to get a passport”) • Need not be spans from passage • Might combine multiple passages
  3. 3. commissioner of the nba Source passages: Query: Answer? Adam Silver
  4. 4. http://www.kdnuggets.com/2016/05/datasets-over-algorithms.html Datasets let algorithms shine
  5. 5. Questions are not search queries Questions are well-formed & curated Single entity / phrase answers Multiple-choice answers Data E.g., TREC-QA, MCTest, CBT, WikiQA, SQuAD Designed for matching short responses, or Poorly correlate with human judgments, or Human in the loop (non-repeatable) Metric E.g., P/R, BLEU, METEOR + Existing QA Datasets
  6. 6. Sample queries from Bing logs Editorially curated reference answers Many reference answers per query Data Phrasing Aware (pa-) metrics Modified versions of BLEU / METEOR Metric+ Our proposal
  7. 7. Towards variance reduction Use single reference passage set to reduce variance from conflicting information at source Get many reference answers to model the natural variance in answer phrasing Extend existing metrics to take better advantage of the large number of available reference answers The law requires all children traveling in the front or rear seat of any car, van or goods vehicle must use the correct child car seat until they are either 135cm in height or 12 years old (which ever they reach first). After this they must use an adult seat belt. There are very few exceptions. law for ages for children allowed to sit in front seatQuery Passages Children under the age of 12 and less than 135cm tall need a child car seat when traveling in the front or the rear seat of a car. Distilled answers Children of any age can travel in the front or the rear seat of a car. They need a child seat if under the age of 12. Children under the age of 12 need a child seat, unless more than 135cm tall. … A child seat is necessary for children under 12. Otherwise an adult seat belt must be worn.
  8. 8. Generating the dataset Sample queries • Randomly sample from Bing logs • Remove PII • Remove navigational, transactional queries • Remove queries with no deterministic answers (E.g., “holiday recipes”) Retrieve candidate passages • Retrieve top-N candidate passages per query • Typically retrieved from many different documents Select minimal passage set • Editors select the minimal but sufficient passage set • If multiple passages are selected then information across passages should not conflict Curate reference answers • Editors curate minimal but complete answer for ach query • Answers can be single entity or phrase, or multi- sentence passage
  9. 9. Phrasing Aware Metrics Score candidate answer based on average similarity with all available reference answers Each reference answer is importance weighted based on agreement with other reference answers Metrics like BLEU (or METEROR) can be used as similarity metric
  10. 10. Request For Comments We want to make the proposed Answer Distillation dataset and corresponding metrics publicly available for academic research We need YOUR feedback to build the right evaluation framework https://gitter.im/ProjectDistillery/Distillery

×