1. The document describes a method for rapidly constructing a concept and relation extraction system in a new domain within a week with minimal training data. It extends an existing ACE system to a new ontology and evaluates the system using a question answering task. 2. The method involves developing an ontology, extending the extraction system to the new ontology, extracting relations using learned and handwritten patterns, and evaluating the system through question answering. 3. The results show the combined system achieved an F1 score of 0.51 in the new domain within a week using very little training data, though handwritten patterns were still more effective than learned patterns.

1. [Freedman+ EMNLP11] Extreme
Extraction – Machine Reading in a
Week
23 Dec 2011
Nakatani Shuyo @ Cybozu labs, Inc
twitter : @shuyo

2. Abstract
• Target:
– Rapid construction of concept and relation
extraction system
• Method:
– Extend an existing ACE system for new relation
– in short time with minimum training data
• in a Week (<50 person hours) with <20 example pairs
– Evaluate by question answering task

3. Phases
1. Ontology and resources
2. Extending system for new ontology
3. Extracting relations
4. Evaluation

4. 1. Ontology and resources
• possibleTreatment( Substance, Condition )
– SSRIs(S) are effective treatments for depression(C)
• expectedDateOnMarket( Substance , Date )
– More drugs for type 2(S) expected on market soon(D)
• responsibleForTreatment( Substance, Agent )
– Officials(A) Responsible for Treatment of War Dead(S)
• studiesDisease( Agent , Condition ) not
sure
– cancer(C) researcher Dr. Henri Joyeux(A)
• hasSideEffect( Substance, Condition )

5. 2. Extending system for new
ontology
• Add new relation/class detectors into “our”
extraction system for ACE task
– Details of the system are not clear...
• Class detectors with unsupervised word clustering
• Bootstrap relation learner with a template and seeds
• Pattern learning for relation extraction
• Annotate words for 4 classes
• Coreference

6. Bootstrap relation learner
• DAP(Double-Anchored Pattern) (Kozareva+ 08)
– Web search with a query based on “<CLASS>
such as <SEED> and *”
– Add words at the position “*” in snippet into the
class member as new seeds
– Repeat “the bootstraping loop” while seeds are
available

7. Relation detection with DAP
• CLASS = disease / SEED = cold
• Web search = “disease such as cold and”

8. Relation detection with DAP
• CLASS = disease / SEED = cold
• Web search = “disease such as cold and”
– disease such as cold and flu (9). ...
– disease such as cold and heat, external ...
– disease such as cold and pneumonia. ...
– disease (such as cold and hot diseases), ...
– disease such as cold and flu viruses. ...
– disease such as cold and food poisoning. ...

9. Four classes to annotate
• Substance-Name
– medicine name
• Substance-Description
– e.g. “new drags”
• Condition-Name
– name of disease
• Condition-Description
– e.g. “the illness”

10. Annotation
• Name tagging with active learning(Miller+ 04)
– Unsupervised word clustering on binary tree
(Brown+ 90)
– Tagging with clustering information
• Averaged Perceptron (Collins 02)
– Request annotation for selected sentence based on
“confidence score”
• score = (highest perceptron score) - (second one)
!?

11. Results of Class Detection
What’s
GS(GoldStandard)?
from [Freedman+ 11]
• substances & conditions
– -Name / -Description respectively
• without/with lists of known substances and conditions

12. Coreference
• It took the most time(20 of 43 hours)
• But its detail is not clear...
– domain independent heuristics
– appositive linking

13. 3. Extracting relations
• Learned Patterns vs. Handwritten Patterns
from [Freedman+ 11]

14. from [Freedman+ 11]

15. 4. Evaluation
• Question Answering with extracted
information
• Query examples
– Find possible treatments for diabetes
– What is expected date to market for Abilify?

16. Answer Example
• ACME produces a wide range of drugs
including treatments for malaria and
athletes foot
– responsibleForTreatment(drugs, ACME)
– possibleTreatment(drugs, malaria)
– possibleTreatment(drugs, athletes foot)

17. from [Freedman+ 11]
• useful = answering complex query

18. When non-useful answers are removed
from [Freedman+ 11]
• annotator’s recall (A)
• using combining both (C)
• using only handwritten rules (H, HW)
• using only learned patterns (L)

19. from [Freedman+ 11]

20. Discussion
from [Freedman+ 11]

21. Conclusions
• The combination system can achieve
F1 of 0.51 in a new domain in a week.
• It requires so little training data.
• The effectiveness of learning algorithms is
still not competitive with handwritten
patterns.

22. References
• [Freedman+ 11] Extreme Extraction – Machine
Reading in a Week
• [Kozareva+ 08] Semantic Class Learning from the
Web with Hyponym Pattern Linkage
• [Miller+ 04] Name Tagging with Word Cluster and
Discriminative Training
– [Brown+ 90] Class-based n-gram models of natural
language
– [Collins 02] Discriminative Training Methods for Hidden
Markov Models: Theory and Experiments with Perceptron
Algorithm