Extreme Extraction - Machine Reading in a Week

21,254 views
21,943 views

Published on

Published in: Technology, Education
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
21,254
On SlideShare
0
From Embeds
0
Number of Embeds
13,212
Actions
Shares
0
Downloads
15
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Extreme Extraction - Machine Reading in a Week

  1. 1. [Freedman+ EMNLP11] ExtremeExtraction – Machine Reading in a Week 23 Dec 2011 Nakatani Shuyo @ Cybozu labs, Inc twitter : @shuyo
  2. 2. Abstract• Target: – Rapid construction of concept and relation extraction system• Method: – Extend an existing ACE system for new relation – in short time with minimum training data • in a Week (<50 person hours) with <20 example pairs – Evaluate by question answering task
  3. 3. Phases1. Ontology and resources2. Extending system for new ontology3. Extracting relations4. Evaluation
  4. 4. 1. Ontology and resources• possibleTreatment( Substance, Condition ) – SSRIs(S) are effective treatments for depression(C)• expectedDateOnMarket( Substance , Date ) – More drugs for type 2(S) expected on market soon(D)• responsibleForTreatment( Substance, Agent ) – Officials(A) Responsible for Treatment of War Dead(S)• studiesDisease( Agent , Condition ) not sure – cancer(C) researcher Dr. Henri Joyeux(A)• hasSideEffect( Substance, Condition )
  5. 5. 2. Extending system for new ontology• Add new relation/class detectors into “our” extraction system for ACE task – Details of the system are not clear... • Class detectors with unsupervised word clustering • Bootstrap relation learner with a template and seeds • Pattern learning for relation extraction• Annotate words for 4 classes• Coreference
  6. 6. Bootstrap relation learner• DAP(Double-Anchored Pattern) (Kozareva+ 08) – Web search with a query based on “<CLASS> such as <SEED> and *” – Add words at the position “*” in snippet into the class member as new seeds – Repeat “the bootstraping loop” while seeds are available
  7. 7. Relation detection with DAP• CLASS = disease / SEED = cold• Web search = “disease such as cold and”
  8. 8. Relation detection with DAP• CLASS = disease / SEED = cold• Web search = “disease such as cold and” – disease such as cold and flu (9). ... – disease such as cold and heat, external ... – disease such as cold and pneumonia. ... – disease (such as cold and hot diseases), ... – disease such as cold and flu viruses. ... – disease such as cold and food poisoning. ...
  9. 9. Four classes to annotate• Substance-Name – medicine name• Substance-Description – e.g. “new drags”• Condition-Name – name of disease• Condition-Description – e.g. “the illness”
  10. 10. Annotation• Name tagging with active learning(Miller+ 04) – Unsupervised word clustering on binary tree (Brown+ 90) – Tagging with clustering information • Averaged Perceptron (Collins 02) – Request annotation for selected sentence based on “confidence score” • score = (highest perceptron score) - (second one) !?
  11. 11. Results of Class Detection What’s GS(GoldStandard)? from [Freedman+ 11]• substances & conditions – -Name / -Description respectively• without/with lists of known substances and conditions
  12. 12. Coreference• It took the most time(20 of 43 hours)• But its detail is not clear... – domain independent heuristics – appositive linking
  13. 13. 3. Extracting relations• Learned Patterns vs. Handwritten Patterns from [Freedman+ 11]
  14. 14. from [Freedman+ 11]
  15. 15. 4. Evaluation• Question Answering with extracted information• Query examples – Find possible treatments for diabetes – What is expected date to market for Abilify?
  16. 16. Answer Example• ACME produces a wide range of drugs including treatments for malaria and athletes foot – responsibleForTreatment(drugs, ACME) – possibleTreatment(drugs, malaria) – possibleTreatment(drugs, athletes foot)
  17. 17. from [Freedman+ 11]• useful = answering complex query
  18. 18. When non-useful answers are removed from [Freedman+ 11]• annotator’s recall (A)• using combining both (C)• using only handwritten rules (H, HW)• using only learned patterns (L)
  19. 19. from [Freedman+ 11]
  20. 20. Discussion from [Freedman+ 11]
  21. 21. Conclusions• The combination system can achieve F1 of 0.51 in a new domain in a week.• It requires so little training data.• The effectiveness of learning algorithms is still not competitive with handwritten patterns.
  22. 22. References• [Freedman+ 11] Extreme Extraction – Machine Reading in a Week• [Kozareva+ 08] Semantic Class Learning from the Web with Hyponym Pattern Linkage• [Miller+ 04] Name Tagging with Word Cluster and Discriminative Training – [Brown+ 90] Class-based n-gram models of natural language – [Collins 02] Discriminative Training Methods for Hidden Markov Models: Theory and Experiments with Perceptron Algorithm

×