Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Learning event extractors from knowledge base revisions

270 views

Published on

Broad-coverage knowledge bases (KBs) such as Wikipedia, Freebase, Microsoft's Satori and Google's Knowledge Graph contain structured data describing real-world entities.
These data sources have become increasingly important for a wide range of intelligent systems: from information retrieval and question answering, to Facebook's Graph Search, IBM's Watson, and more.
Previous work on learning to populate knowledge bases from text has, for the most part, made the simplifying assumption that facts remain constant over time.
But this is inaccurate - we live in a rapidly changing world.
Knowledge should not be viewed as a static snapshot, but instead a rapidly evolving set of facts that must change as the world changes.

In this paper we demonstrate the feasibility of accurately identifying entity-transition-events, from real-time news and social media text streams, that drive changes to a knowledge base.
We use Wikipedia's edit history as distant supervision to learn event extractors, and evaluate the extractors based on their ability to predict online updates.
Our weakly supervised event extractors are able to predict 10 KB revisions per month at 0.8 precision. By lowering our confidence threshold, we can suggest 34.3 correct edits per month at 0.4 precision.
64% of predicted edits were detected before they were added to Wikipedia. The average lead time of our forecasted knowledge revisions over Wikipedia's editors is 40 days, demonstrating the utility of our method for suggesting edits that can be quickly verified and added to the knowledge graph.

Published in: Technology
  • Be the first to comment

Learning event extractors from knowledge base revisions

  1. 1. Learning to Extract Events from Knowledge Base Revisions Alexander Konovalov Ohio State University konovalov.2@osu.edu Benjamin Strauss Ohio State University strauss.105@osu.edu Alan Ritter Ohio State University ritter.1492@osu.edu Brendan O'Connor University of Amherst brenocon@cs.umass.edu
  2. 2. Knowledge Bases Some notable examples: ● Google Knowledge Graph ● Microsoft Satori KB ● Wolfram Alpha ● Wikidata ● DBPedia ● Wikipedia Infoboxes ● Freebase
  3. 3. Knowledge Bases Some notable examples: ● Google Knowledge Graph ● Microsoft Satori KB ● Wolfram Alpha ● Wikidata ● DBPedia ● Wikipedia Infoboxes ● Freebase
  4. 4. Extracting KBs from Text Prior work assumes static text corpora and knowledge bases: ● NELL [AAAI 2015] ● Mintz et al. [ACL 2009] ● DeepDive [VLDS 2012] ● Knowledge Vault [KDD 2014]
  5. 5. Real-time Knowledge Bases Knowledge Bases are not static snapshots! Events are constantly changing the world.
  6. 6. Real-time Knowledge Bases Knowledge Bases are not static snapshots! Events are constantly changing the world.
  7. 7. Wikipedia: a dynamically evolving knowledge base
  8. 8. Wikipedia: a dynamically evolving knowledge base
  9. 9. Wikipedia: a dynamically evolving knowledge base
  10. 10. Wikipedia: a dynamically evolving knowledge base
  11. 11. 50% of deaths updated within a couple of days Manual Editing…
  12. 12. 50% of deaths updated within a couple of days ... but for scientists take 31 days to reach 50% coverage!* Manual Editing… can not Scale Up! * https://research.googleblog.com/2013/05/distributing-edit-history-of-wikipedia.html Goal: extract KB revisions from text as soon as public knowledge is available
  13. 13. Infobox Revisions News Event extraction pipeline Training Social Media In a nutshell
  14. 14. Prediction Infobox Revisions News Event extraction pipeline Social Media In a nutshell
  15. 15. In a nutshell
  16. 16. X welcomes Y as our new mayor X sworn in as Y mayor In a nutshell
  17. 17. X welcomes Y as our new mayor X sworn in as Y mayor In a nutshell
  18. 18. In a nutshell X welcomes Y as our new mayor X sworn in as Y mayor{{infobox Settlement | name=Pittsburgh | Leader=Bill Peduto
  19. 19. In a nutshell: formally.
  20. 20. In a nutshell: formally. ● An event is a quadruple of a predicate, two entities, and a time.
  21. 21. In a nutshell: formally. ● Features are collected from tweets written before time t. Prediction time Feature window
  22. 22. In a nutshell: formally. ● A bag of Mintz et al. like features. State arg1 wins in arg2 cnn projects arg1 wins IN arg2 0.333 0.115 CurrentTeam arg1 to arg2 arg1 joins arg2 arg1 signs...with arg2 1.291 0.513 0.419 DeathPlace arg1 found dead in arg2 arg1 found JJ IN arg2 arg1 killed in arg2 arg1 memorial...in arg2 0.269 0.260 0.195 0.125
  23. 23. In a nutshell: formally. ● L₂ regularized log-linear model. ● Optimize conditional likelihood with respect to θ.
  24. 24. Challenge: Non-Semantic Edits | spouse = Soumaya Domit (1967–1999; her death); 6 children | spouse = Soumaya Domit <small>(1967-1999; her death)</small>
  25. 25. Challenge: Vandalism |leader_name = Chuck Norris|leader_name = Dave Paulekas
  26. 26. Revision as of 17:27, 7 April 2015 ➕ | death_date = {{death date and age |2015|02|03|1978|05|29}}} ➕ | death_place = [[Valhalla, New York|Valhalla, NY]], U.S. Challenge: Edits After Event
  27. 27. Experiments: datasets. ● Twitter (Social Media) ○ Training: Sep. 2008 - Jun. 2011 ○ Test: Jun. 2011 - Jan. 2012 ○ 10% firehose. ● Annotated Gigaword (News) ○ Training: Jun. 2004 - Jun. 2009 ○ Test: Jun 2009 - Jan. 2011 ○ [Napoles et al. 2012]
  28. 28. Experiments: datasets. ● Twitter: ○ langid.py [IJCNLP 2011] ○ Domain-specific Twitter NLP [Ritter et al. 2011] ○ Down-sampling Twitter Firehose langid.py Tokens, NE, POS Down sampling Annotated Samples
  29. 29. Experiments: datasets. ● Annotated Gigaword ○ POS, NER, Syntactic trees, etc... ○ Each sentence is a single data-point. Annotated Gigaword Split into sentences Annotated Samples
  30. 30. Attributes Six Wikipedia infobox attributes: ● CurrentTeam ● LeaderName ● StateRepresentative ● Spouse ● Predecessor ● DeathPlace
  31. 31. Attributes Six Wikipedia infobox attributes: ● CurrentTeam ● LeaderName ● StateRepresentative ● Spouse ● Predecessor ● DeathPlace
  32. 32. Attributes Six Wikipedia infobox attributes: ● CurrentTeam ● LeaderName ● StateRepresentative ● Spouse ● Predecessor ● DeathPlace
  33. 33. Experiments: datasets. ● Intuition: Samples near the time of a revision are likely to mention an event that causes the change. Annotated Samples Matched Samples Match NE Aligned Samples Use intuition! Unaligned Samples
  34. 34. Experiments: datasets. ● Intuition: Samples near the time… Use temporal alignment (-10...+3 days). Annotated Samples Matched Samples Match NE Aligned Samples Unaligned Samples temporal alignment
  35. 35. Experiments: baseline. ● Baseline: a traditional relation extraction system that uses no temporal information. Annotated Samples Matched Samples Match NE Aligned Samples Unaligned Samples temporal alignment
  36. 36. Experiments: automatic evaluation. ● Evaluate assuming that events = edits (positive). ● Mix 90% randomly sampled as negative samples.
  37. 37. Experiments: automatic evaluation. Twitter Gigaword Attribute Our System Baseline Our System Baseline Current Team 0.53 0.28 0.03 0.03 Leader Name 0.50 0.12 0.27 0.05 Spouse 0.27 0.22 0.08 0.05 State Representative 0.01 0.002 0.43 0.29 Predecessor 0.02 0.09 0.24 0.17 Death Place 0.49 0.63 0.41 0.35
  38. 38. Experiments: human evaluation. Automatic evaluation: ● Noise - underestimates precision & recall. ● Missing edits - overestimates recall. ● Unrealistic ratio of signal / noise.
  39. 39. Experiments: human evaluation. ● Split the predictions into month-long bins. ● Annotate the top predictions from both systems.
  40. 40. Experiments: evaluation. ● Human judgement via Amazon Mechanical Turk, 7 workers.
  41. 41. Experiments: evaluation. ● Inter-Annotator Agreement (Fleiss-Kappa): ○ 0.64 on Twitter ○ 0.30 on Gigaword Context matters: Much of Wednesday was about pomp and circumstance for Rand and Ron Paul. They were sworn in, [...] But the newly minted senator from Kentucky also tended to some business [...]
  42. 42. Experiments: All attributes. Predictions per month Precision Twitter Gigaword
  43. 43. Experiments: All attributes. Predictions per month Precision Twitter Gigaword Our system Baseline
  44. 44. Experiments: All attributes. Predictions per month Precision Twitter Gigaword
  45. 45. Conclusions ● Revisions - a source for distant supervision for event extraction.
  46. 46. Conclusions ● Revisions - a source for distant supervision for event extraction. ● Use temporal and surface matching to identify reliable infobox edits corresponding to real world events.
  47. 47. Conclusions ● Revisions - a source for distant supervision for event extraction. ● Use temporal and surface matching to identify reliable infobox edits corresponding to real world events. ● Automate KB updates by learning predictors from past KB updates.
  48. 48. Conclusions ● We generate on average 34.3 edits per month with high precision for 6 attributes. ● Often beat human KB contributors in terms of recall (64%⇑ before) and lead time (40 days).
  49. 49. Future Work ● Database revisions are not always well-aligned:
  50. 50. Future Work ● Align edits to the actual events using latent variable alignment model. With an offset
  51. 51. Q & A Learning to Extract Events from Knowledge Base Revisions You can reach me via konovalov.2@osu.edu. Code and data will be available soon at github.com/alexknvl/dsup.

×