Watson & Natural Language Processing meet-up talk

1. ★

2. •
–
…
–
•
Anca Dumitrache

3. •
•
•
•
Anca Dumitrache

4. •
•
Anca Dumitrache

5. •
•
•
•
•
•
•
Anca Dumitrache

6. Anca Dumitrache

7. Which is the mood most appropriate
for each song?
★

8. METHYLERGOMETRINE is a blood vessel constrictor most
commonly used to control excessive BLEEDING.
What is the relation between the
highlighted terms?
★

9. •
•
•
•
•
Anca Dumitrache

10. ANTIBIOTICS are the first line treatment for indications of
TYPHUS.
With ANTIBIOTICS in short supply, DDT was used during World
War II to control the insect vectors of TYPHUS.
Is the TREATS relation expressed
between the highlighted terms?
★

11. •
•
•
•
Anca Dumitrache

12. Which mood cluster is
most appropriate for a song?
★

13. agents are useful for patients with renal
impairment, but in patients with severe renal failure
requiring dialysis it presents a risk of nephrogenic
systemic .
☒ ☑
What is the relation between the
highlighted terms?
★

14. •
•
–
–
Anca Dumitrache

15. Does each sentence express the TREAT relation?
ANTIBIOTICS are the first line treatment for indications of TYPHUS.
→ agreement 95%
Patients with TYPHUS who were given ANTIBIOTICS exhibited side-
effects.
→ agreement 80%
With ANTIBIOTICS in short supply, DDT was used during WWII to control
the insect vectors of TYPHUS.
→ agreement 50%
★

16. •
•
•
Anca Dumitrache

17. ★

18. Does each sentence express the TREAT relation?
ANTIBIOTICS are the first line treatment for indications of
TYPHUS.
is not a reliable cure for .
★

19. •
•
–
•
•
Anca Dumitrache

20. What is the relation between the highlighted terms?
He was the first physician to identify the relationship
between and
★

21. What is the (medical) relation between the
highlighted (medical) terms?
• 91% of expert annotations covered by the crowd
• expert annotators reach agreement only in 30%
• most popular crowd vote covers 95% of this
expert annotation agreement
★

22. •
•
–
–
Anca Dumitrache

23. Which are mentions of terrorists in this sentence?
used money from his own
construction company to support the in
Afghanistan against Soviet forces.
★

24. • Annotator disagreement
is signal, not noise.
• It is indicative of the
variation in human
semantic interpretation of
signs
• It can indicate ambiguity,
vagueness, similarity, over-
generality, etc,
as well as quality
Anca Dumitrache

25. • methodology
• disagreement-aware
crowdsourcing to collect gold
standard data
• metrics to capture
disagreement
• software
• online platform for
crowdsourcing task workflows
and data analytics
• ground truth collection
• medical relation extraction
• salience in news and tweets
• sound annotation
http://CrowdTruth.org

26. CrowdTruth.org
github.com/CrowdTruth
data.CrowdTruth.org

27. • Goals:
○ collecting a relation extraction
gold standard
○ improve the performance of a
relation extraction classifier
http://CrowdTruth.org

28. TASK
→

29. 1 1 1

30. 1 1 1
1 1
1
1 1
1 1
1 1
1
1
1
0 1 1 0 0 4 3 0 0 5 1 0

33. 1
0 1 1 0 0 4 3 0 0 5 1 0
Unit vector for
relation R6
Sentence
Vector
Cosine = .55

36. • methodology
• disagreement-aware
crowdsourcing to collect gold
standard data
• metrics to capture
disagreement
• software
• online platform for
crowdsourcing task workflows
and data analytics
• ground truth collection
• medical relation extraction
• salience in news and tweets
• sound annotation
http://CrowdTruth.org

37. http://crowdtruth.org
http://github.com/CrowdTruth
★