CrowdTruth Tutorial: Using the Crowd to Understand Ambiguity

1. CrowdTruth: Using the Crowd to Understand Ambiguity Anca Dumitrache, Lora Aroyo, Chris Welty http://crowdtruth.org

2. http://crowdtruth.org

3. 1759: computing date of Halley’s Comet return 1892: for the first time appears as term in NYT http://crowdtruth.org

4. • collective decisions of large groups of people • a group of error-prone decision-makers can be surprisingly good at picking the best choice. • for a binary decision, odds that majory of people in a crowd will pick the right answer is greater than the odds that any one of them will pick it on their own. • performance gets better as the size grows 1785 Marquis de Condorcet statistics for democracy “wisdom of crowds” http://crowdtruth.org

5. • asked 787 people to guess the weight of an ox • none got the right answer • their collective guess was almost perfect “wisdom of crowds” Sir Francis Galton http://crowdtruth.org

6. WWII Math Rosies 1942: Ballistics calculations and flight trajectories http://crowdtruth.org

7. “Who Wants to Be a Millionaire?” •help from individual expert or audience poll •majority of the audience right 91% of the time •individuals right only 65% of the time “wisdom of crowds” http://crowdtruth.org

8. humans solving computational problems too hard for computers alone Human Computing http://crowdtruth.org

9. “we treat human brains as processors in a distributed system each performing a small part of a massive computation” Human Computing Luis von Ahn http://crowdtruth.org

10. • 1979: amateur naturalists Don McCrimmon & Cal Smith first applied the concept of "citizen science” • 1980: Rick Bonney coined the term "citizen science” http://crowdtruth.org

11. • 2005: term “human computation” by Luis von Ahn • 2005: term “crowdsourcing” by Jeff Howe & Mark Robinson (Wired) • 2006: Games with the purpose by Luis von Ahn • 2006: “The Rise of the Crowdsourcing”, by Jeff Howe (Wired) • 2007: reCaptcha (company) • 2011: Duolingo, Luis von Ahn 2005 http://crowdtruth.org

12. 2009 http://crowdtruth.org

13. 2013 http://crowdtruth.org

14. Crowds Contributing in Cultural Heritage

15. Accurator ask the right crowd, enrich your collection

16. http://waisda.nl http://www.prestoprime.org/ @waisda

17. http://crowdtruth.org

18. today’s complex problem solving requires multiple perspectives http://crowdtruth.org

19. Crowdsourcing Myths http://crowdtruth.org

20. • Human annotators with domain knowledge provide better annotated data, e.g – if you want medical texts annotated for medical relations you need medical experts • But experts are expensive & don’t scale • Multiple perspectives on data can be useful, beyond what experts believe is salient or correct Crowdsourcing Myth: Ask the Expert What if the CROWD IS BETTER? http://CrowdTruth.org

21. What is the relation between the highlighted terms? He was the first physician to identify the relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY. Experts Better? Crowd reads text literally - provide better examples to machine experts: cause crowd: no relation http://CrowdTruth.org

22. • rather than accepting disagreement as a natural property of semantic interpretation • traditionally, disagreement is considered a measure of poor quality in the annotation task because: – task is poorly defined or – annotators lack training This makes the elimination of disagreement a goal Crowdsourcing Myth: Disagreement is Bad What if it is GOOD? http://CrowdTruth.org

23. Disagreement Bad? Does each sentence express the TREAT relation? ANTIBIOTICS are the first line treatment for indications of TYPHUS. → agreement 95% Patients with TYPHUS who were given ANTIBIOTICS exhibited side-effects. → agreement 80% With ANTIBIOTICS in short supply, DDT was used during WWII to control the insect vectors of TYPHUS. → agreement 50% Disagreement can reflect the degree of clarity in a sentence http://CrowdTruth.org

24. Do gold standards capture all this? http://crowdtruth.org

25. http://crowdtruth.org ● single human experts → never perfectly correct ● gold standard is measured in inter-annotator agreement → what if disagreeing annotators are both right? ● gold standards → do not account for alternative interpretations & clarity

26. http://crowdtruth.org ● single human experts → never perfectly correct ● gold standard is measured in inter-annotator agreement → what if disagreeing annotators are both right? ● gold standards → do not account for alternative interpretations & clarity The fallacy of the “one truth” assumption that pervades computational semantics

27. “best collective decisions are result of disagreement, not consensus or compromise” James Surowiecki http://crowdtruth.org

28. Position disagreement is a signal and not noise can we harness it?

29. • Annotator disagreement is signal, not noise. • It is indicative of the variation in human semantic interpretation of signs • It can indicate ambiguity, vagueness, similarity, over-generality, etc, as well as quality CrowdTruth http://CrowdTruth.org

30. RelExTASKinCrowdFlower Patients with ACUTE FEVER and nausea could be suffering from INFLUENZA AH1N1 Is ACUTE FEVER – related to → INFLUENZA AH1N1? http://crowdtruth.org

31. 1 1 1 Worker Vector http://crowdtruth.org

32. 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 1 1 0 0 4 3 0 0 5 1 0 Sentence Vector http://crowdtruth.org

33. Unclear relationship between the two arguments reflected in the disagreement http://crowdtruth.org What’s in a sentence vector?

34. Clearly expressed relation between the two arguments reflected in the agreement http://crowdtruth.org What’s in a sentence vector?

35. Patients with ACUTE FEVER and nausea could be suffering from INFLUENZA AH1N1. measures how clearly a relation is expressed in a sentence Sentence-Relation Score http://crowdtruth.org Sentence vector # Workers 10 15 0 15 3 15 8 15 . . . causes treats partof sym ptom

36. Disagreement is a property of the entire system! How to model CrowdTruth? http://crowdtruth.org Sentence Quality Relation Quality Worker Quality

37. measures the clarity of one sentence Worker i vector Worker j vector AVG (Cosine) ∀ workers0 1 1 0 0 1 1 0 0 1 1 0 0 4 3 0 0 5 1 0 0 1 1 0 Sentence Quality http://crowdtruth.org 1 10

38. measures the clarity of one sentence weighted by worker & relation quality Worker i vector Worker j vector AVG (Cosine) ∀ workers0 1 1 0 0 1 1 0 0 1 1 0 0 4 3 0 0 5 1 0 0 1 1 0 Sentence Quality http://crowdtruth.org 1 10 weighted by relation quality weighted by worker quality

39. measures the performance of one worker Worker Quality (worker i) = Worker - Worker Agreement (worker i) ٠ Worker - Sentence Agreement (worker i) Worker Quality http://crowdtruth.org

40. measures the agreement between one worker & all other workers Worker i vector Worker j vector AVG (Cosine) ∀ worker j, j ≠ i 0 1 1 0 0 1 1 0 0 1 1 0 0 4 3 0 0 5 1 0 0 1 1 0 Worker-Worker Agreement http://crowdtruth.org 1 10

41. measures the agreement between one worker & all other workers weighted by sentence, relation & other workers quality Worker i vector Worker j vector AVG (Cosine) ∀ worker j, j ≠ i 0 1 1 0 0 1 1 0 0 1 1 0 0 4 3 0 0 5 1 0 0 1 1 0 Worker-Worker Agreement http://crowdtruth.org 1 10 weighted by relation quality weighted by other workers quality and sentence quality

42. measures the agreement between a worker & all sentences 0 1 1 0 0 4 3 0 0 5 1 0 Worker vector Sentence vector AVG (Cosine) ∀ sentences 0 1 1 0 0 1 1 0 0 1 1 0 Worker-Sentence Agreement http://crowdtruth.org

43. measures the agreement between a worker & all sentences weighted by sentence & relation quality 0 1 1 0 0 4 3 0 0 5 1 0 Worker’s sentence vector Sentence Vector AVG (Cosine) ∀ sentences 0 1 1 0 0 1 1 0 0 1 1 0 Worker-Sentence Agreement http://crowdtruth.org weighted by relation quality weighted by sentence quality

44. • measures the conditional probability that if a worker picks the relation in a sentence, then another worker will also pick it • weighted by sentence & worker quality Relation Quality http://crowdtruth.org Relation Quality (relation r) = P( worker i annotates relation r in sentence s | worker j annotates relation r in sentence s) ∀ workers i,j, ∀ sentences s

45. Get the metrics at: https://git.io/v5iTB

46. • Goals: ○ collecting a medical relation extraction gold standard ○ improve the performance of a relation extraction classifier • Approach: ○ crowdsource 900 medical sentences ○ measure disagreement with CrowdTruth metrics ○ train & evaluate classifier with CrowdTruth score CrowdTruth for medical relation extraction in practice http://crowdtruth.org

47. Medical RelEx Classifier Trained with Expert vs. CrowdTruth Sentence-Relation Score 0.642, p = 0:016 0.638 http://crowdtruth.org

48. Medical RelEx Classifier Trained with Expert vs. CrowdTruth Sentence-Relation Score 0.642, p = 0:016 0.638 crowd provides training data that is at least as good if not better than experts http://crowdtruth.org

49. # of Workers: Impact on Sentence-Relation Score http://crowdtruth.org

50. # of Workers: Impact on RelEx Model Performance only 54 sent. had 15 or more workers http://crowdtruth.org

51. • crowd performs just as well as medical experts • crowd is also cheaper • crowd is always available • using only a few annotators for ground truth is faulty • min 10 workers/sentence are needed for highest quality annotations • CrowdTruth = a solution to Clinical NLP Challenge: • lack of ground truth for training & benchmarking In summary: CrowdTruth for Medical Relation Extraction http://crowdtruth.org

52. • Annotator disagreement is signal, not noise. • It is indicative of the variation in human semantic interpretation of signs • It can indicate ambiguity, vagueness, similarity, over-generality, etc, as well as quality CrowdTruth http://CrowdTruth.org

53. crowdtruth.org metrics: git.io/v5iTB datasets: data.CrowdTruth.org @anca_dmtrch

54. How to model CrowdTruth? http://crowdtruth.org

55. How to model CrowdTruth? (Ogden & Richards, 1923) http://crowdtruth.org

58. Disagreement is useful How to model CrowdTruth? http://crowdtruth.org