Your SlideShare is downloading. ×
0
Crowdsourcing for Watson                                               Lora Aroyo                          Croudwsourcing ...
Observation                                                 language is ambiguous & vague                                 ...
our goal is ...                               not to enforce agreement                            to capture different view...
so we need to ...      understand the range of disagreements by creating a        space of possibilities with frequencies ...
Harnessing the Crowd                          not just to save moneyFlickr: AmyJanelleMonday, December 10, 12
Monday, December 10, 12
Monday, December 10, 12
Worker	  #1	             MICRO-­‐TASK   ?                               Answer                          ?                 ...
The Task Answers                                             Lora AroyoMonday, December 10, 12
The Task Answers                                             Lora AroyoMonday, December 10, 12
The Task Answers                                             Lora AroyoMonday, December 10, 12
The Task Answers                                             Lora AroyoMonday, December 10, 12
The Task Answers                                             Lora AroyoMonday, December 10, 12
OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food      and Drug Administration (FDA) for the treatment of ...
OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food      and Drug Administration (FDA) for the treatment of ...
He was the first physician to identify the relationship between      HEMOPHILIA and HEMOPHILIC ARTHROPATHY.                ...
Sentence-Relation Distribution                                  Lora AroyoMonday, December 10, 12
Expert Annotators                                              Lora AroyoMonday, December 10, 12
Crowd vs. Experts                                              Lora AroyoMonday, December 10, 12
Crowd vs. Experts                                  x                                              Lora AroyoMonday, Decemb...
The Dark Side of Crowdsourcing                    Disagreement      • disagreement is beautiful, except when it results fr...
Position       Artificially                          restricting                                   humans d                ...
JOIN our team           • Dr. Watson                         choose                  Medical Synonyms Game           Your ...
Upcoming SlideShare
Loading in...5
×

Crowdsourcing for Watson

1,491

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,491
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
14
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Crowdsourcing for Watson"

  1. 1. Crowdsourcing for Watson Lora Aroyo Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoMonday, December 10, 12
  2. 2. Observation language is ambiguous & vague people interpret it from different hu points of view am m big an uit dis y o agr r v ee ag me ue nt ne m ss ay of sig lan na gu l ag e Flickr: elkabongMonday, December 10, 12
  3. 3. our goal is ... not to enforce agreement to capture different view points adapt to different annotation tasks Flickr: elkabongMonday, December 10, 12
  4. 4. so we need to ... understand the range of disagreements by creating a space of possibilities with frequencies & similarities teach machines to reason in the disagreement space score their output based on where it falls in this space Flickr: elkabongMonday, December 10, 12
  5. 5. Harnessing the Crowd not just to save moneyFlickr: AmyJanelleMonday, December 10, 12
  6. 6. Monday, December 10, 12
  7. 7. Monday, December 10, 12
  8. 8. Worker  #1   MICRO-­‐TASK ? Answer ? ? Worker  #2   Answer ? Worker  #3   Answer Worker  #N   AnswerMonday, December 10, 12
  9. 9. The Task Answers Lora AroyoMonday, December 10, 12
  10. 10. The Task Answers Lora AroyoMonday, December 10, 12
  11. 11. The Task Answers Lora AroyoMonday, December 10, 12
  12. 12. The Task Answers Lora AroyoMonday, December 10, 12
  13. 13. The Task Answers Lora AroyoMonday, December 10, 12
  14. 14. OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food and Drug Administration (FDA) for the treatment of  SCHIZOPHRENIA and bipolar disorder. Is SCHIZOPHRENIA related to OLANZAPINE ? treated_by treats may_treat RESPIRATORY ALKALOSIS is a medical condition in which increased respiration (HYPERVENTILATION) elevates the blood pH (a condition generally called alkalosis). RESPIRATORY Is HYPERVENTILATION related to ALKALOSIS ? diagnosed_by cause_of cause symptom_of Lora AroyoMonday, December 10, 12
  15. 15. OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food and Drug Administration (FDA) for the treatment of  SCHIZOPHRENIA and bipolar disorder. Is SCHIZOPHRENIA related to OLANZAPINE ? treated_by treats may_treat RESPIRATORY ALKALOSIS is a medical condition in which increased respiration (HYPERVENTILATION) elevates the blood pH (a condition generally called alkalosis). RESPIRATORY Is HYPERVENTILATION related to ALKALOSIS ? diagnosed_by cause_of cause may_cause symptom_of Lora AroyoMonday, December 10, 12
  16. 16. He was the first physician to identify the relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY. HEMOPHILIC Is HEMOPHILIA related to ? ARTHROPATHY other treated_by cause_of cause may_cause symptom_of has_manifestation It just says there is a relation between the two but gives no specifics about what the relation is There is a relationship between the two disorders but the sentence does not indicate what that relationship is. identify the relationship between Lora AroyoMonday, December 10, 12
  17. 17. Sentence-Relation Distribution Lora AroyoMonday, December 10, 12
  18. 18. Expert Annotators Lora AroyoMonday, December 10, 12
  19. 19. Crowd vs. Experts Lora AroyoMonday, December 10, 12
  20. 20. Crowd vs. Experts x Lora AroyoMonday, December 10, 12
  21. 21. The Dark Side of Crowdsourcing Disagreement • disagreement is beautiful, except when it results from spamming • crowdsourcing has to account for people that want to get paid for not doing any work • spammers generate disagreement for the wrong reasons • most spam detection requires gold standardMonday, December 10, 12
  22. 22. Position Artificially restricting humans d Machines oes not h will learn elp mach from dive ines to le arn. rsity Flickr: elkabongMonday, December 10, 12
  23. 23. JOIN our team • Dr. Watson choose Medical Synonyms Game Your • Medical Detective Thesis Gamification of Medical project Diagnosis • To CATCH a Thief SPAM detection in call now: lora.aroyo crowdsourcing with no Gold Standard @vu.nlMonday, December 10, 12
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×