0
CrowdTruth
What is disagreement & why
does it make more sense than
agreement?
Lora Aroyo & Chris Welty
problem
cognitive computing systems need
annotated data for training, testing,
evaluation
solution
human annotation through
crowdsourcing augmented with
machine processing
What's wrong with the gold
standard?
●  algorithmic performance is measured on test sets vetted by
human experts → never p...
One Truth?
Does each sentence express the TREAT relation?
ANTIBIOTICS are the first line treatment for indications of TYPH...
One Truth?
Does each sentence express the TREAT relation?
ANTIBIOTICS are the first line treatment for indications of TYPH...
What is the relation between the highlighted terms?
GADOLINIUM agents are useful for patients with renal
impairment, but i...
Does each sentence express the TREAT relation?
ANTIBIOTICS are the first line treatment for indications of TYPHUS.
QUININE...
Why Disagreement Happens
Why Disagreement Happens
Why Disagreement Happens
Why Disagreement Happens
Why Disagreement Happens
Disagreement is usefull
CrowdTruth
Annotator disagreement is signal, not noise.
It is indicative of the variation in human
semantic interpretation...
CrowdTruth
is the response to the current reality of
cognitive computing systems - driven by data
analytics & elevated by ...
CrowdTruth
captures and represents human
semantics & thus helps extending the
capabilities of cognitive computing
systems
crowdtruth.org
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Upcoming SlideShare
Loading in...5
×

Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)

1,006

Published on

CrowdTruth:
What is disagreement & why does it make more sense than agreement?
http://crowdtruth.org

Published in: Technology, News & Politics
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,006
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
138
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Transcript of "Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)"

  1. 1. CrowdTruth What is disagreement & why does it make more sense than agreement? Lora Aroyo & Chris Welty
  2. 2. problem cognitive computing systems need annotated data for training, testing, evaluation
  3. 3. solution human annotation through crowdsourcing augmented with machine processing
  4. 4. What's wrong with the gold standard? ●  algorithmic performance is measured on test sets vetted by human experts → never perfectly correct ●  gold standards are created assuming that for each annotated instance there is a single right answer → doesn’t account for alternative interpretations & clarity ●  gold standard quality is measured in inter-annotator agreement → what happens if disagreeing annotators are both right? The fallacy of the “one truth” assumption that pervades computational semantics
  5. 5. One Truth? Does each sentence express the TREAT relation? ANTIBIOTICS are the first line treatment for indications of TYPHUS. à agreement 95% Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. à agreement 80% With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. à agreement 50%
  6. 6. One Truth? Does each sentence express the TREAT relation? ANTIBIOTICS are the first line treatment for indications of TYPHUS. à agreement 95% Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. à agreement 80% With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. à agreement 50% Disagreement can reflect lack of clarity in a sentence
  7. 7. What is the relation between the highlighted terms? GADOLINIUM agents are useful for patients with renal impairment, but in patients with severe renal failure requiring dialysis it presents a risk of nephrogenic systemic FIBROSIS. One Interpretation? Disagreement can indicate alternative interpretations of relations cause or side effect ?
  8. 8. Does each sentence express the TREAT relation? ANTIBIOTICS are the first line treatment for indications of TYPHUS. QUININE is not a reliable cure for MALARIA. Disagreement can indicate low quality workers One Quality?
  9. 9. Why Disagreement Happens
  10. 10. Why Disagreement Happens
  11. 11. Why Disagreement Happens
  12. 12. Why Disagreement Happens
  13. 13. Why Disagreement Happens Disagreement is usefull
  14. 14. CrowdTruth Annotator disagreement is signal, not noise. It is indicative of the variation in human semantic interpretation of signs It can indicate ambiguity, vagueness, similarity, over-generality, etc, as well as quality
  15. 15. CrowdTruth is the response to the current reality of cognitive computing systems - driven by data analytics & elevated by interpretation. it supports the need to bring the human semantics, representing the dynamics of opinions and perspectives, into machine readable form
  16. 16. CrowdTruth captures and represents human semantics & thus helps extending the capabilities of cognitive computing systems
  17. 17. crowdtruth.org
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×