• Like
Crowdsourcing for NLP Ground Truth Data
Upcoming SlideShare
Loading in...5
×

Crowdsourcing for NLP Ground Truth Data

  • 826 views
Uploaded on

Part II of my talk at Columbia University, 11 Oct 2012

Part II of my talk at Columbia University, 11 Oct 2012

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
826
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
9
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. From Crowd Knowledge to Machine Knowledge gather annotation of types, events, relations, coref Lora Aroyo and Chris Welty T e x t Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo TextWednesday, October 17, 12 1
  • 2. Position are par t of the t & vag ue ne s s disag reemen mantics The human t & relations se even Flickr: elkabong Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 2
  • 3. Position Artificially restricting humans d Machines oes not h will learn elp mach from dive ines to le arn. rsity Flickr: elkabong Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 3
  • 4. Disagreement Framework • ontology: disagreements on the basic status of events themselves as referents of linguistic utterances, e.g. are people events or do events exist at all. • granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc. • interpretation: disagreements that result from (non- granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September. Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 4
  • 5. Disagreement Framework • ontology: disagreements on the basic status of events themselves as referents of linguistic utterances, e.g. are people events or do events exist at all. • granularity: disagreements that result from issues of granularity, e.g. the location being a country, region, or city, the time being a day, week, month, etc. • interpretation: disagreements that result from (non- granular) ambiguity, differences in perspective, or error in interpreting an expression, e.g. classifying a person as a terrorist/hero, ”October Revolution” took place in September. Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 4
  • 6. Approach Principles 1. tolerate, capture & exploit disagreement 2. understand the disagreement by creating a space of possibilities (frequencies & similarities) 3. score the machine output based on where it falls in this space 4. adaptable to new annotation tasks Flickr: auroille Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 5
  • 7. Event Extraction crowdsourcing ground truth data Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo Lora AroyoWednesday, October 17, 12 6
  • 8. Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 7
  • 9. Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 7
  • 10. Event Participants Disagreement Israeli Prime minister 10% 50% Government Benjamin Netanyahu Israeli Cabinet 15% his Cabinet 15% 35% Benjamin {TOLD} Netanyahu Benjamin Israeli Prime Netanyahu’s 5% 15% minister Cabinet Cabinet 45% Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 8
  • 11. Temporal Disagreement Prime minister 50% Benjamin 50% Sunday Netanyahu March 1, 1998 25% 35% Benjamin {TOLD} March 1998 15% Netanyahu Spring 1998 5% Israeli Prime 15% minister Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 9
  • 12. Spatial Disagreement Southern 35%30% Israel Lebanon {WILLING TO WITHDRAW} Lebanon 45%65% Israels Northern Frontier Middle East 10% Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 10
  • 13. it  seems  to  refer  to  an   Top  Israeli  officials  SENT  strong   does not inference  or   new  SIGNALS  Sunday  that  Israel   refer to communicated  feeling   wants  to  withdraw  from  southern   an event more  than  specific   Lebanon,  ... event. a  group  of  people  did   refers to something  specific  at  a   an event specific  point  in  6me. the  actors  in  ques6on   (top  Israeli  officials)   refers to performed  an  ac6on   an event during  a  specified  6me   (Sunday). it  refers  to  what  the   israelis  did  on  sunday,   a  specific  6me. Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 11
  • 14. it  is  not  a  par6cular   That  1978  resolu6on  calls  for   movement  that  has  or  is   Israels  uncondi6onal   does not going  on  but  a  request  that   WITHDRAWAL  from  the  self-­‐ refer to the  country  of  Israel   declared  security  zone  it   an event remove  their  forces  from   occupies  in  south  Lebanon,  ... the  zone  they  occupy. does not refer to an event the  sentence  is  speaking  of   a  demand  for  a  withdrawal   that  had  not  yet  occurred. refers to an event Because  it  is  describing  a   historical  issue  concerning   the  resolu6on  of  1978 Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 12
  • 15. Relation Extraction crowdsourcing ground truth data Croudwsourcing for gathering NLP Ground Truth Data Lora Aroyo Lora AroyoWednesday, October 17, 12 13
  • 16. 6 experiments • 2 professional • 30 CFworkers per annotators per sentence sentence @20sentences @20sentences • 30 CFworkers per • 10 CFworkers per sentence @10sentences sentence @20sentences + relations definitions • 20 CFworkers per • 10 CFworkers per sentence @20sentences sentence @20sentence explanation validation Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 14
  • 17. The Task Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 15
  • 18. The Steps: Example Sentence (1) Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 16
  • 19. The Steps: Example Sentence (2) Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 17
  • 20. OLANZAPINE is an atypical antipsychotic, approved by the U.S.Food and Drug Administration (FDA) for the treatment of  SCHIZOPHRENIA and bipolar disorder. Is SCHIZOPHRENIA related to OLANZAPINE ? treated_by treats may_treat RESPIRATORY ALKALOSIS is a medical condition in which increased respiration (HYPERVENTILATION) elevates the blood pH (a condition generally called alkalosis). RESPIRATORY Is HYPERVENTILATION related to ALKALOSIS ? diagnosed_by cause_of cause may_cause symptom_of Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 18
  • 21. He was the first physician to identify the relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY. HEMOPHILIC Is HEMOPHILIA related to ? ARTHROPATHY other treated_by cause_of cause may_cause symptom_of has_manifestation relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY It just says there is a relation between the two but gives no specifics about what the relation is There is a relationship between the two disorders but the sentence does not indicate what that relationship is. identify the relationship between Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 19
  • 22. Distributions Compared 140" 120" 100" 80" 60" 40" 20" 0" sTB" sT" sMT" sPB" sP" sMP" sDB" sD" sMD" sCO" sC" sMC" sLO" sHM" sDF" sSS" sOTH" sNO" 10"workers" 20"workers" 30"workers" Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 20
  • 23. Sentence-Relation Distribution Professional Annotators Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 21
  • 24. Sentence-Relation Distribution 10w x 20s Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 22
  • 25. Sentence-Relation Distribution 10w x 20s is differentiate d from is_related caused_by caused_by x caused_by is_type_of, is_a has_type, includes is_related to maybe related to Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 22
  • 26. Sentence-Relation Distribution 20w x 20s is differentiate d from is_related caused_by caused_by x caused_by is_type_of, is_a has_type, includes is_related to maybe related to Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 23
  • 27. Sentence-Relation Distribution 30w x 20s is differentiate d from is_related caused_by caused_by x caused_by is_type_of, is_a has_type, includes is_related to maybe related to Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 24
  • 28. Sentence-Relation Distribution 30w x 10s with relation explanations Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 25
  • 29. The Task Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 26
  • 30. The Steps: Example Sentence Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 27
  • 31. The Dark Side of Crowdsourcing Disagreement • disagreement is beautiful, except when it results from spamming • crowdsourcing has to account for people that want to get paid for not doing any work • spammers generate disagreement for the wrong reasons • most spam detection requires gold standard Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 28
  • 32. • a new way of measuring Extraction of ground truth • Putative Events a new set of semantic input: putative events Manual selection of Gold Questions features for learning in input: output of A event extraction Phase I: Phase I: A. Collect event input: annotations + output of A C. Filtering spam motivations event annotations input: input: input: list of events list of events list of events Phase III: Phase IV: Phase II: A. Collect event A. Collect event A. Collect event types modalities + + motivations role fillers + motivations motivations input: input: input: output of A output of A output of A input: input: input: output of A Manual output of A Manual output of A Manual selection of selection of selection of Gold Questions Gold Questions Gold Questions Phase II: Phase IV: Phase III: B. Filtering spam B. Filtering spam B. Filtering spam event types event modalities event role fillers Croudwsourcing for gathering NLP Ground Truth Data Lora AroyoWednesday, October 17, 12 29
  • 33. Questions? @laroyo http://lora-aroyo.org Truth Data Croudwsourcing for gathering NLP Ground Lora AroyoWednesday, October 17, 12 30