Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014

Lora Aroyo
Lora AroyoProfessor Human Computer Interaction at Vrije Universiteit Amsterdam / VU
Truth 
is 
a 
Lie 
CrowdTruth: 
The 
7 
Myths 
of 
Human 
Annota9on 
Lora 
Aroyo
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Human 
annota9on 
of 
seman)c 
interpreta)on 
tasks 
as 
cri)cal 
part 
of 
cogni)ve 
systems 
engineering 
– standard 
prac)ce 
based 
on 
an9quated 
ideal 
of 
a 
single 
correct 
truth 
– 7 
myths 
of 
human 
annota)on 
– new 
theory 
of 
truth: 
CrowdTruth 
Take 
Home 
Message 
Lora Aroyo
I 
amar 
prestar 
aen... 
• amount 
of 
data 
& 
scale 
of 
computa9on 
available 
have 
increased 
by 
a 
previously 
inconceivable 
amount 
• CS 
& 
AI 
moved 
out 
of 
thought 
problems 
to 
empirical 
science 
• current 
methods 
pre-­‐date 
this 
fundamental 
shi? 
• the 
ideal 
of 
“one 
truth” 
is 
a 
lie 
• crowdsourcing 
& 
seman9cs 
together 
correct 
the 
fallacy 
and 
improve 
analy)c 
systems 
The 
world 
has 
changed: 
there 
is 
a 
need 
to 
form 
a 
new 
theory 
of 
truth 
-­‐ 
appropriate 
to 
cogni)ve 
systems 
Lora Aroyo
Seman)c 
interpreta)on 
is 
needed 
in 
all 
sciences 
– Data 
abstracted 
into 
categories 
– PaIerns, 
correla9ons, 
associa9ons 
& 
implica9ons 
are 
extracted 
Seman9c 
Interpreta9on 
Cogni9ve 
Compu9ng: 
providing 
some 
way 
of 
scalable 
seman)c 
interpreta)on 
Lora Aroyo
• Humans 
analyze 
examples: 
annota)ons 
for 
ground 
truth 
= 
the 
correct 
output 
for 
each 
example 
• Machines 
learn 
from 
the 
examples 
• Ground 
Truth 
Quality: 
– measured 
by 
inter-­‐annotator 
agreement 
– founded 
on 
ideal 
for 
single, 
universally 
constant 
truth 
– high 
agreement 
= 
high 
quality 
– disagreement 
must 
be 
eliminated 
Tradi9onal 
Human 
Annota9on 
Lora Aroyo 
Current 
gold 
standard 
acquisi9on 
& 
quality 
evalua9on 
are 
outdated
• Cogni)ve 
Compu)ng 
increases 
the 
need 
for 
machines 
to 
handle 
the 
scale 
of 
data 
• Results 
in 
increasing 
need 
for 
new 
gold 
standards 
able 
to 
measure 
machine 
performance 
on 
tasks 
that 
require 
seman)c 
interpreta)on 
Need 
for 
Change 
Lora Aroyo 
The 
New 
Ground 
Truth 
is 
CrowdTruth
• One 
truth: 
data 
collec)on 
efforts 
assume 
one 
correct 
interpreta)on 
for 
every 
example 
• All 
examples 
are 
created 
equal: 
ground 
truth 
treats 
all 
examples 
the 
same 
– 
either 
match 
the 
correct 
result 
or 
not 
• Detailed 
guidelines 
help: 
if 
examples 
cause 
disagreement 
-­‐ 
add 
instruc)ons 
to 
limit 
interpreta)ons 
• Disagreement 
is 
bad: 
increase 
quality 
of 
annota)on 
data 
by 
reducing 
disagreement 
among 
the 
annotators 
• One 
is 
enough: 
most 
of 
the 
annotated 
examples 
are 
evaluated 
by 
one 
person 
• Experts 
are 
beIer: 
annotators 
with 
domain 
knowledge 
provide 
beIer 
annota)ons 
• Once 
done, 
forever 
valid: 
annota)ons 
are 
not 
updated; 
new 
data 
not 
aligned 
with 
previous 
7 
Myths 
myths 
directly 
influence 
the 
prac)ce 
of 
collec)ng 
human 
annotated 
data; 
Need 
to 
be 
revisited 
in 
the 
context 
of 
new 
changing 
world 
& 
in 
the 
face 
of 
a 
new 
theory 
of 
truth 
(CrowdTruth) 
Lora Aroyo
current 
ground 
truth 
collec)on 
efforts 
assume 
one 
correct 
interpreta)on 
for 
every 
example 
the 
ideal 
of 
truth 
is 
a 
fallacy 
for 
seman9c 
interpreta9on 
and 
needs 
to 
be 
changed 
1. 
One 
Truth 
What 
if 
there 
are 
MORE? 
Lora Aroyo
Which is the mood most appropriate 
Cluster 
1 
Cluster 
2 
Cluster 
3 
Cluster 
4 
Cluster 
5 
Other 
passionate, 
rollicking, 
literate, 
humorous, 
silly, 
aggressive, 
fiery, 
does 
not 
fit 
into 
rousing, 
cheerful, 
fun, 
poignant, 
wis9ul, 
campy, 
quirky, 
tense, 
anxious, 
any 
of 
the 
5 
confident, 
sweet, 
amiable, 
bi>ersweet, 
whimsical, 
wi>y, 
intense, 
vola?le, 
clusters 
boisterous, 
good-­‐natured 
autumnal, 
wry 
visceral 
rowdy 
brooding 
Lora Aroyo 
Choose 
one: 
for each song? 
one 
truth? 
Results 
in: 
(Lee 
and 
Hu 
2012)
• typically 
annotators 
are 
asked 
whether 
a 
binary 
property 
holds 
for 
each 
example 
• o?en 
not 
given 
a 
chance 
to 
say 
that 
the 
property 
may 
par9ally 
hold, 
or 
holds 
but 
is 
not 
clearly 
expressed 
• mathema9cs 
of 
using 
ground 
truth 
treats 
every 
example 
the 
same 
– 
either 
match 
correct 
result 
or 
not 
• poor 
quality 
examples 
tend 
to 
generate 
high 
disagreement 
disagreement 
allows 
us 
to 
weight 
sentences 
= 
the 
ability 
to 
train 
& 
evaluate 
a 
machine 
more 
flexibly 
2. 
All 
Examples 
Are 
Created 
Equal 
What 
if 
they 
are 
DIFFERENT? 
Lora Aroyo
Is TREAT relation expressed between 
the highlighted terms? 
ANTIBIOTICS are the first line treatment for indications of 
TYPHUS. 
clearly 
With ANTIBIOTICS in short supply, DDT was used during World 
War II to control the insect vectors of TYPHUS. 
treats 
less 
clear 
treats 
equal 
training 
data? 
disagreement 
can 
indicate 
vagueness 
& 
ambiguity 
of 
sentences 
Lora Aroyo
• Perfuming 
agreement 
scores 
by 
forcing 
annotators 
to 
make 
choices 
they 
may 
think 
are 
not 
valid 
• Low 
annotator 
agreement 
is 
addressed 
by 
detailed 
guidelines 
for 
annotators 
to 
consistently 
handle 
the 
cases 
that 
generate 
disagreement 
• Remove 
poten9al 
signal 
on 
examples 
that 
are 
ambiguous 
precise 
annota)on 
guidelines 
do 
eliminate 
disagreement 
but 
do 
not 
increase 
quality 
3. 
Detailed 
Guidelines 
Help 
What 
if 
they 
HURT? 
Lora Aroyo
Which mood cluster is 
most appropriate for a song? 
Instruc9ons 
Your 
task 
is 
to 
listen 
to 
the 
following 
30 
second 
music 
clips 
and 
select 
disagreement 
can 
indicate 
problems 
with 
the 
task 
the 
most 
appropriate 
mood 
cluster 
that 
represents 
the 
mood 
of 
the 
music. 
Try 
to 
think 
about 
the 
mood 
carried 
by 
the 
music 
and 
please 
try 
to 
ignore 
any 
lyrics. 
If 
you 
feel 
the 
music 
does 
not 
fit 
into 
any 
of 
the 
5 
clusters 
please 
select 
“Other”. 
The 
descrip)ons 
of 
the 
clusters 
are 
provided 
in 
the 
panel 
at 
the 
top 
of 
the 
page 
for 
your 
reference. 
Answer 
the 
ques)ons 
carefully. 
Your 
work 
will 
not 
be 
accepted 
if 
your 
answers 
are 
inconsistent 
and/or 
incomplete. 
restric2ng 
guidelines 
help? 
(Lee 
and 
Hu 
2012) 
Lora Aroyo
• rather 
than 
accep)ng 
disagreement 
as 
a 
natural 
property 
of 
seman)c 
interpreta)on 
• tradi)onally, 
disagreement 
is 
considered 
a 
measure 
of 
poor 
quality 
because: 
– task 
is 
poorly 
defined 
or 
– annotators 
lack 
training 
this 
makes 
the 
elimina9on 
of 
disagreement 
the 
GOAL 
4. 
Disagreement 
is 
Bad 
What 
if 
it 
is 
GOOD? 
Lora Aroyo
Does each sentence express 
the TREAT relation? 
ANTIBIOTICS are the first line treatment for indications of TYPHUS. 
à agreement 95% 
Patients with TYPHUS who were given ANTIBIOTICS exhibited side-effects. 
à agreement 80% 
With ANTIBIOTICS in short supply, DDT was used during WWII to control 
the insect vectors of TYPHUS. 
à agreement 50% 
disagreement 
bad? 
disagreement 
can 
reflect 
the 
degree 
of 
clarity 
in 
a 
sentence 
Lora Aroyo
• over 
90% 
of 
annotated 
examples 
– 
seen 
by 
1-­‐2 
annotators 
• small 
number 
overlap 
– 
to 
measure 
agreement 
five 
or 
six 
popular 
interpreta9ons 
can’t 
be 
captured 
by 
one 
or 
two 
people 
5. 
One 
is 
Enough 
What 
if 
it 
is 
NOT 
ENOUGH? 
Lora Aroyo
One 
Quality? 
accumulated 
results 
for 
each 
rela)on 
across 
all 
the 
sentences 
20 
workers/sentence 
(and 
higher) 
yields 
same 
rela9ve 
disagreement 
Lora Aroyo
• conven9onal 
wisdom: 
human 
annotators 
with 
domain 
knowledge 
provide 
beIer 
annotated 
data, 
e.g 
– medical 
texts 
should 
be 
annotated 
by 
medical 
experts 
• but 
experts 
are 
expensive 
& 
don’t 
scale 
mul9ple 
perspec9ves 
on 
data 
can 
be 
useful, 
beyond 
what 
experts 
believe 
is 
salient 
or 
correct 
6. 
Experts 
Are 
BeIer 
What 
if 
the 
CROWD 
IS 
BETTER? 
Lora Aroyo
What is the (medical) relation between 
the highlighted (medical) terms? 
• 91% of expert annotations covered by the crowd 
• expert annotators reach agreement only in 30% 
• most popular crowd vote covers 95% of this 
expert annotation agreement 
experts 
beIer 
than 
crowd? 
Lora Aroyo
• perspec9ves 
change 
over 
9me 
– 
old 
training 
data 
might 
contain 
examples 
that 
are 
not 
valid 
or 
only 
par)ally 
valid 
later 
• con9nuous 
collec9on 
of 
training 
data 
over 
)me 
allows 
the 
adapta)on 
of 
gold 
standards 
to 
changing 
)mes 
– popularity 
of 
music 
– levels 
of 
educa)on 
7. 
Once 
Done, 
Forever 
Valid 
What 
if 
VALIDITY 
CHANGES?
Which are mentions of terrorists 
in this sentence? 
OSAMA 
BIN 
LADEN used money from his own 
construction company to support the MUHAJADEEN in 
Afghanistan against Soviet forces. 
forever 
valid? 
1990: 
hero 
2011: 
terrorist 
both 
types 
should 
be 
valid 
-­‐ 
two 
roles 
for 
same 
en9ty 
-­‐ 
adapta9on 
of 
gold 
standards 
to 
changing 
9mes 
Lora Aroyo
crowdtruth.org 
Jean-­‐Marc 
Côté, 
1899
• annotator disagreement is signal, not noise. 
• it is indicative of the variation in human 
semantic interpretation of signs 
• it can indicate ambiguity, vagueness, 
similarity, over-generality, as well as quality 
crowdtruth.org
crowdtruth.org
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
The 
Team 
2013 
hIp://crowd-­‐watson.nl
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
The Crew 2014
The 
(almost 
complete) 
Team 
2014
lora-aroyo.org 
slideshare.com/laroyo 
@laroyo 
crowdtruth.org
1 of 34

Recommended

WebSci2013 Harnessing Disagreement in Crowdsourcing by
WebSci2013 Harnessing Disagreement in CrowdsourcingWebSci2013 Harnessing Disagreement in Crowdsourcing
WebSci2013 Harnessing Disagreement in CrowdsourcingLora Aroyo
5.8K views12 slides
Harnessing diversity in crowds and machines for better ner performance by
Harnessing diversity in crowds and machines for better ner performanceHarnessing diversity in crowds and machines for better ner performance
Harnessing diversity in crowds and machines for better ner performanceoanainel
4.5K views25 slides
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 by
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 Lora Aroyo
5.1K views20 slides
Exploiting disagreement through open ended tasks for capturing interpretation... by
Exploiting disagreement through open ended tasks for capturing interpretation...Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Benjamin Timmermans
5.2K views32 slides
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr... by
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Lora Aroyo
28.6K views40 slides
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017 by
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Lora Aroyo
8.8K views32 slides

More Related Content

What's hot

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone by
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneMy ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneLora Aroyo
2.8K views84 slides
CrowdTruth: Machine-Human Computation for Harnessing Disagreement in Semantic... by
CrowdTruth: Machine-Human Computation for Harnessing Disagreement in Semantic...CrowdTruth: Machine-Human Computation for Harnessing Disagreement in Semantic...
CrowdTruth: Machine-Human Computation for Harnessing Disagreement in Semantic...Lora Aroyo
668 views11 slides
CCCT University of Amsterdam Seminars 2013: Crowdsourcing Session by
CCCT University of Amsterdam Seminars 2013: Crowdsourcing SessionCCCT University of Amsterdam Seminars 2013: Crowdsourcing Session
CCCT University of Amsterdam Seminars 2013: Crowdsourcing SessionLora Aroyo
752 views52 slides
Observational studies in social media by
Observational studies in social mediaObservational studies in social media
Observational studies in social mediaCarlos Castillo (ChaTo)
1.5K views51 slides
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote) by
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Lora Aroyo
5.2K views82 slides
Pydata Taipei 2020 by
Pydata Taipei 2020Pydata Taipei 2020
Pydata Taipei 2020Tunghai University
66 views31 slides

What's hot(15)

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone by Lora Aroyo
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneMy ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
Lora Aroyo2.8K views
CrowdTruth: Machine-Human Computation for Harnessing Disagreement in Semantic... by Lora Aroyo
CrowdTruth: Machine-Human Computation for Harnessing Disagreement in Semantic...CrowdTruth: Machine-Human Computation for Harnessing Disagreement in Semantic...
CrowdTruth: Machine-Human Computation for Harnessing Disagreement in Semantic...
Lora Aroyo668 views
CCCT University of Amsterdam Seminars 2013: Crowdsourcing Session by Lora Aroyo
CCCT University of Amsterdam Seminars 2013: Crowdsourcing SessionCCCT University of Amsterdam Seminars 2013: Crowdsourcing Session
CCCT University of Amsterdam Seminars 2013: Crowdsourcing Session
Lora Aroyo752 views
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote) by Lora Aroyo
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Lora Aroyo5.2K views
Introduction to Bayesian Truth Serum by Fuming Shih
Introduction to Bayesian Truth SerumIntroduction to Bayesian Truth Serum
Introduction to Bayesian Truth Serum
Fuming Shih9K views
Understanding the world with NLP: interactions between society, behaviour and... by Diana Maynard
Understanding the world with NLP: interactions between society, behaviour and...Understanding the world with NLP: interactions between society, behaviour and...
Understanding the world with NLP: interactions between society, behaviour and...
Diana Maynard202 views
Groundhog Day: Near-Duplicate Detection on Twitter by Ke Tao
Groundhog Day: Near-Duplicate Detection on Twitter Groundhog Day: Near-Duplicate Detection on Twitter
Groundhog Day: Near-Duplicate Detection on Twitter
Ke Tao2.4K views
Mining Social Media with Linked Open Data, Entity Recognition, and Event Extr... by Leon Derczynski
Mining Social Media with Linked Open Data, Entity Recognition, and Event Extr...Mining Social Media with Linked Open Data, Entity Recognition, and Event Extr...
Mining Social Media with Linked Open Data, Entity Recognition, and Event Extr...
Leon Derczynski4.2K views
Fake News Detector by IrisYoon5
Fake News DetectorFake News Detector
Fake News Detector
IrisYoon5454 views
Rigourous evaluation of nlp models in real world deployment by Sandy Man
Rigourous evaluation of nlp models in real world deploymentRigourous evaluation of nlp models in real world deployment
Rigourous evaluation of nlp models in real world deployment
Sandy Man117 views
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ... by UXPA International
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...
Mechanical Turk Demystified: Best practices for sourcing and scaling quality ...
Challenges of social media analysis in the real world by Diana Maynard
Challenges of social media analysis in the real worldChallenges of social media analysis in the real world
Challenges of social media analysis in the real world
Diana Maynard2.3K views

Viewers also liked

Towards Better Media Understanding and Searchability by
Towards Better Media Understanding and SearchabilityTowards Better Media Understanding and Searchability
Towards Better Media Understanding and Searchabilityoanainel
5.3K views14 slides
Dive+ NL eScience symposium 2015 by
Dive+ NL eScience symposium 2015Dive+ NL eScience symposium 2015
Dive+ NL eScience symposium 2015CrowdTruth
10.5K views29 slides
Dive+@ICTOpen2017 by
Dive+@ICTOpen2017Dive+@ICTOpen2017
Dive+@ICTOpen2017oanainel
4.5K views11 slides
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora) by
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Lora Aroyo
4.9K views18 slides
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data by
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataVisualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataCrowdTruth
4.4K views18 slides
Gamification of crowdsourcing tasks: What motivates a medical expert? by
Gamification of crowdsourcing tasks: What motivates a medical expert?Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?CrowdTruth
4.6K views26 slides

Viewers also liked(20)

Towards Better Media Understanding and Searchability by oanainel
Towards Better Media Understanding and SearchabilityTowards Better Media Understanding and Searchability
Towards Better Media Understanding and Searchability
oanainel5.3K views
Dive+ NL eScience symposium 2015 by CrowdTruth
Dive+ NL eScience symposium 2015Dive+ NL eScience symposium 2015
Dive+ NL eScience symposium 2015
CrowdTruth10.5K views
Dive+@ICTOpen2017 by oanainel
Dive+@ICTOpen2017Dive+@ICTOpen2017
Dive+@ICTOpen2017
oanainel4.5K views
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora) by Lora Aroyo
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Lora Aroyo4.9K views
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data by CrowdTruth
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataVisualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
CrowdTruth4.4K views
Gamification of crowdsourcing tasks: What motivates a medical expert? by CrowdTruth
Gamification of crowdsourcing tasks: What motivates a medical expert?Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
CrowdTruth4.6K views
Crowdsourcing Disagreement on Open-Domain Questions by Benjamin Timmermans
Crowdsourcing Disagreement on Open-Domain QuestionsCrowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain Questions
Benjamin Timmermans3.6K views
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio... by CrowdTruth
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
CrowdTruth4.8K views
DIVE Semantic Web Challenge Presentation by Victor de Boer
DIVE Semantic Web Challenge Presentation DIVE Semantic Web Challenge Presentation
DIVE Semantic Web Challenge Presentation
Victor de Boer2.4K views
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search by Lora Aroyo
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
Lora Aroyo18.3K views
CrowdTruth Games @NLeSc eHumanities day 2015 by Lora Aroyo
CrowdTruth Games @NLeSc eHumanities day 2015CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015
Lora Aroyo2.9K views
Boosting Named Entity Extraction through Crowdsourcing by oanainel
Boosting Named Entity Extraction through CrowdsourcingBoosting Named Entity Extraction through Crowdsourcing
Boosting Named Entity Extraction through Crowdsourcing
oanainel2.2K views
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age by Lora Aroyo
Europeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital AgeEuropeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Lora Aroyo578 views
Genuine semantic publishing by Tobias Kuhn
Genuine semantic publishingGenuine semantic publishing
Genuine semantic publishing
Tobias Kuhn603 views
CEDAR: From Fragment to Fabric - Dutch Census Data in a Web of Global Cultura... by PRELIDA Project
CEDAR: From Fragment to Fabric - Dutch Census Data in a Web of Global Cultura...CEDAR: From Fragment to Fabric - Dutch Census Data in a Web of Global Cultura...
CEDAR: From Fragment to Fabric - Dutch Census Data in a Web of Global Cultura...
PRELIDA Project1.8K views
Harnessing the Power of Machines & Crowds for Event Extraction by oanainel
Harnessing the Power of Machines & Crowds for Event ExtractionHarnessing the Power of Machines & Crowds for Event Extraction
Harnessing the Power of Machines & Crowds for Event Extraction
oanainel2.5K views
Closing Event - Watson Innovation Course by Lora Aroyo
Closing Event - Watson Innovation CourseClosing Event - Watson Innovation Course
Closing Event - Watson Innovation Course
Lora Aroyo914 views
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+ by Lora Aroyo
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
Lora Aroyo654 views
Stitch by Stitch: Annotating Fashion at the Rijksmuseum by Lora Aroyo
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Lora Aroyo16.5K views

Similar to Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014

Soc 156 – Sociology of CommunicationReview Sheet – FinalShor.docx by
Soc 156 – Sociology of CommunicationReview Sheet – FinalShor.docxSoc 156 – Sociology of CommunicationReview Sheet – FinalShor.docx
Soc 156 – Sociology of CommunicationReview Sheet – FinalShor.docxwhitneyleman54422
3 views189 slides
anchoring-heuristic Decision Making by
anchoring-heuristic Decision Makinganchoring-heuristic Decision Making
anchoring-heuristic Decision MakingÖzkan Özer
4.5K views60 slides
Teaching lean startup capital enterprise by
Teaching lean startup   capital enterpriseTeaching lean startup   capital enterprise
Teaching lean startup capital enterpriseFounder-Centric
784 views103 slides
Econ 5315 Hw 10 by
Econ 5315 Hw 10Econ 5315 Hw 10
Econ 5315 Hw 10Joyce Williams
2 views77 slides
psychology in media by mostafa ewees by
psychology in media by mostafa eweespsychology in media by mostafa ewees
psychology in media by mostafa eweesMostafa Ewees
504 views50 slides
XAI-proposal2.pptx by
XAI-proposal2.pptxXAI-proposal2.pptx
XAI-proposal2.pptxvincenttong18
1 view20 slides

Similar to Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014(20)

Soc 156 – Sociology of CommunicationReview Sheet – FinalShor.docx by whitneyleman54422
Soc 156 – Sociology of CommunicationReview Sheet – FinalShor.docxSoc 156 – Sociology of CommunicationReview Sheet – FinalShor.docx
Soc 156 – Sociology of CommunicationReview Sheet – FinalShor.docx
anchoring-heuristic Decision Making by Özkan Özer
anchoring-heuristic Decision Makinganchoring-heuristic Decision Making
anchoring-heuristic Decision Making
Özkan Özer4.5K views
Teaching lean startup capital enterprise by Founder-Centric
Teaching lean startup   capital enterpriseTeaching lean startup   capital enterprise
Teaching lean startup capital enterprise
Founder-Centric784 views
psychology in media by mostafa ewees by Mostafa Ewees
psychology in media by mostafa eweespsychology in media by mostafa ewees
psychology in media by mostafa ewees
Mostafa Ewees504 views
Gender and language (linguistics, social network theory, Twitter!) by Tyler Schnoebelen
Gender and language (linguistics, social network theory, Twitter!)Gender and language (linguistics, social network theory, Twitter!)
Gender and language (linguistics, social network theory, Twitter!)
Tyler Schnoebelen3.3K views
Gender, language, and Twitter: Social theory and computational methods by Idibon1
Gender, language, and Twitter: Social theory and computational methodsGender, language, and Twitter: Social theory and computational methods
Gender, language, and Twitter: Social theory and computational methods
Idibon1572 views
Revising lftvd by TomEccles4
Revising lftvdRevising lftvd
Revising lftvd
TomEccles4130 views
Game Theory And Best Decision Essay by Jamie Miller
Game Theory And Best Decision EssayGame Theory And Best Decision Essay
Game Theory And Best Decision Essay
Jamie Miller2 views
Dialogue based Meaning Negotiation by Terry Payne
Dialogue based Meaning NegotiationDialogue based Meaning Negotiation
Dialogue based Meaning Negotiation
Terry Payne412 views
#CrowdTruth: Linked Data for Information Extraction @ISWC2015 by Lora Aroyo
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
Lora Aroyo856 views
Ois-Quiz Study For Chapter 8 And 9 by Rebecca Harris
Ois-Quiz Study For Chapter 8 And 9Ois-Quiz Study For Chapter 8 And 9
Ois-Quiz Study For Chapter 8 And 9
Rebecca Harris3 views
Connective Media Technologies - A Look Into Reddit's Star Dish by Frances Coronel
Connective Media Technologies - A Look Into Reddit's Star DishConnective Media Technologies - A Look Into Reddit's Star Dish
Connective Media Technologies - A Look Into Reddit's Star Dish
Frances Coronel277 views
II-SDV 2016 Srinivasan Parthiban - KOL Analytics from Biomedical Literature by Dr. Haxel Consult
II-SDV 2016 Srinivasan Parthiban - KOL Analytics from Biomedical LiteratureII-SDV 2016 Srinivasan Parthiban - KOL Analytics from Biomedical Literature
II-SDV 2016 Srinivasan Parthiban - KOL Analytics from Biomedical Literature
Dr. Haxel Consult1.4K views

More from Lora Aroyo

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning by
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningLora Aroyo
586 views20 slides
Harnessing Human Semantics at Scale (updated) by
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Lora Aroyo
124 views66 slides
Data excellence: Better data for better AI by
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AILora Aroyo
987 views35 slides
CHIP Demonstrator presentation @ CATCH Symposium by
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumLora Aroyo
263 views22 slides
Semantic Web Challenge: CHIP Demonstrator by
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorLora Aroyo
181 views17 slides
The Rijksmuseum Collection as Linked Data by
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataLora Aroyo
1.9K views26 slides

More from Lora Aroyo(20)

CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning by Lora Aroyo
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
Lora Aroyo586 views
Harnessing Human Semantics at Scale (updated) by Lora Aroyo
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)
Lora Aroyo124 views
Data excellence: Better data for better AI by Lora Aroyo
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
Lora Aroyo987 views
CHIP Demonstrator presentation @ CATCH Symposium by Lora Aroyo
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH Symposium
Lora Aroyo263 views
Semantic Web Challenge: CHIP Demonstrator by Lora Aroyo
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP Demonstrator
Lora Aroyo181 views
The Rijksmuseum Collection as Linked Data by Lora Aroyo
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked Data
Lora Aroyo1.9K views
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum by Lora Aroyo
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Lora Aroyo832 views
FAIRview: Responsible Video Summarization @NYCML'18 by Lora Aroyo
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18
Lora Aroyo699 views
Understanding bias in video news & news filtering algorithms by Lora Aroyo
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithms
Lora Aroyo356 views
StorySourcing: Telling Stories with Humans & Machines by Lora Aroyo
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & Machines
Lora Aroyo726 views
Data Science with Humans in the Loop by Lora Aroyo
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the Loop
Lora Aroyo3.7K views
Digital Humanities Benelux 2017: Keynote Lora Aroyo by Lora Aroyo
Digital Humanities Benelux 2017: Keynote Lora AroyoDigital Humanities Benelux 2017: Keynote Lora Aroyo
Digital Humanities Benelux 2017: Keynote Lora Aroyo
Lora Aroyo2.4K views
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev... by Lora Aroyo
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
Lora Aroyo4.5K views
Data Science with Human in the Loop @Faculty of Science #Leiden University by Lora Aroyo
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden University
Lora Aroyo1.1K views
"Video Killed the Radio Star": From MTV to Snapchat by Lora Aroyo
"Video Killed the Radio Star": From MTV to Snapchat"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat
Lora Aroyo622 views
UMAP 2016 Opening Ceremony by Lora Aroyo
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening Ceremony
Lora Aroyo475 views
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr... by Lora Aroyo
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Lora Aroyo1.1K views
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle... by Lora Aroyo
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Lora Aroyo1.2K views
Keynote @Final NWO CATCH Program Event by Lora Aroyo
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program Event
Lora Aroyo960 views
Semantic Digital Humanities Workshop 2015 @Oxford by Lora Aroyo
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @Oxford
Lora Aroyo1.8K views

Recently uploaded

【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院 by
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院IttrainingIttraining
69 views8 slides
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda... by
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...ShapeBlue
44 views13 slides
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue by
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueShapeBlue
62 views54 slides
Five Things You SHOULD Know About Postman by
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About PostmanPostman
38 views43 slides
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... by
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc
72 views29 slides
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De... by
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...Moses Kemibaro
27 views38 slides

Recently uploaded(20)

【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院 by IttrainingIttraining
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院
【USB韌體設計課程】精選講義節錄-USB的列舉過程_艾鍗學院
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda... by ShapeBlue
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
Hypervisor Agnostic DRS in CloudStack - Brief overview & demo - Vishesh Jinda...
ShapeBlue44 views
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue by ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlueVNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
VNF Integration and Support in CloudStack - Wei Zhou - ShapeBlue
ShapeBlue62 views
Five Things You SHOULD Know About Postman by Postman
Five Things You SHOULD Know About PostmanFive Things You SHOULD Know About Postman
Five Things You SHOULD Know About Postman
Postman38 views
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... by TrustArc
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc72 views
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De... by Moses Kemibaro
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Don’t Make A Human Do A Robot’s Job! : 6 Reasons Why AI Will Save Us & Not De...
Moses Kemibaro27 views
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ... by ShapeBlue
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
ShapeBlue46 views
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue by ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueCloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
ShapeBlue26 views
HTTP headers that make your website go faster - devs.gent November 2023 by Thijs Feryn
HTTP headers that make your website go faster - devs.gent November 2023HTTP headers that make your website go faster - devs.gent November 2023
HTTP headers that make your website go faster - devs.gent November 2023
Thijs Feryn26 views
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue75 views
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava... by ShapeBlue
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
ShapeBlue28 views
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker48 views
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R... by ShapeBlue
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
ShapeBlue37 views
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit... by ShapeBlue
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
ShapeBlue40 views
"Surviving highload with Node.js", Andrii Shumada by Fwdays
"Surviving highload with Node.js", Andrii Shumada "Surviving highload with Node.js", Andrii Shumada
"Surviving highload with Node.js", Andrii Shumada
Fwdays33 views
Why and How CloudStack at weSystems - Stephan Bienek - weSystems by ShapeBlue
Why and How CloudStack at weSystems - Stephan Bienek - weSystemsWhy and How CloudStack at weSystems - Stephan Bienek - weSystems
Why and How CloudStack at weSystems - Stephan Bienek - weSystems
ShapeBlue81 views
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by ShapeBlue
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
ShapeBlue55 views

Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014

  • 1. Truth is a Lie CrowdTruth: The 7 Myths of Human Annota9on Lora Aroyo
  • 6. Human annota9on of seman)c interpreta)on tasks as cri)cal part of cogni)ve systems engineering – standard prac)ce based on an9quated ideal of a single correct truth – 7 myths of human annota)on – new theory of truth: CrowdTruth Take Home Message Lora Aroyo
  • 7. I amar prestar aen... • amount of data & scale of computa9on available have increased by a previously inconceivable amount • CS & AI moved out of thought problems to empirical science • current methods pre-­‐date this fundamental shi? • the ideal of “one truth” is a lie • crowdsourcing & seman9cs together correct the fallacy and improve analy)c systems The world has changed: there is a need to form a new theory of truth -­‐ appropriate to cogni)ve systems Lora Aroyo
  • 8. Seman)c interpreta)on is needed in all sciences – Data abstracted into categories – PaIerns, correla9ons, associa9ons & implica9ons are extracted Seman9c Interpreta9on Cogni9ve Compu9ng: providing some way of scalable seman)c interpreta)on Lora Aroyo
  • 9. • Humans analyze examples: annota)ons for ground truth = the correct output for each example • Machines learn from the examples • Ground Truth Quality: – measured by inter-­‐annotator agreement – founded on ideal for single, universally constant truth – high agreement = high quality – disagreement must be eliminated Tradi9onal Human Annota9on Lora Aroyo Current gold standard acquisi9on & quality evalua9on are outdated
  • 10. • Cogni)ve Compu)ng increases the need for machines to handle the scale of data • Results in increasing need for new gold standards able to measure machine performance on tasks that require seman)c interpreta)on Need for Change Lora Aroyo The New Ground Truth is CrowdTruth
  • 11. • One truth: data collec)on efforts assume one correct interpreta)on for every example • All examples are created equal: ground truth treats all examples the same – either match the correct result or not • Detailed guidelines help: if examples cause disagreement -­‐ add instruc)ons to limit interpreta)ons • Disagreement is bad: increase quality of annota)on data by reducing disagreement among the annotators • One is enough: most of the annotated examples are evaluated by one person • Experts are beIer: annotators with domain knowledge provide beIer annota)ons • Once done, forever valid: annota)ons are not updated; new data not aligned with previous 7 Myths myths directly influence the prac)ce of collec)ng human annotated data; Need to be revisited in the context of new changing world & in the face of a new theory of truth (CrowdTruth) Lora Aroyo
  • 12. current ground truth collec)on efforts assume one correct interpreta)on for every example the ideal of truth is a fallacy for seman9c interpreta9on and needs to be changed 1. One Truth What if there are MORE? Lora Aroyo
  • 13. Which is the mood most appropriate Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Other passionate, rollicking, literate, humorous, silly, aggressive, fiery, does not fit into rousing, cheerful, fun, poignant, wis9ul, campy, quirky, tense, anxious, any of the 5 confident, sweet, amiable, bi>ersweet, whimsical, wi>y, intense, vola?le, clusters boisterous, good-­‐natured autumnal, wry visceral rowdy brooding Lora Aroyo Choose one: for each song? one truth? Results in: (Lee and Hu 2012)
  • 14. • typically annotators are asked whether a binary property holds for each example • o?en not given a chance to say that the property may par9ally hold, or holds but is not clearly expressed • mathema9cs of using ground truth treats every example the same – either match correct result or not • poor quality examples tend to generate high disagreement disagreement allows us to weight sentences = the ability to train & evaluate a machine more flexibly 2. All Examples Are Created Equal What if they are DIFFERENT? Lora Aroyo
  • 15. Is TREAT relation expressed between the highlighted terms? ANTIBIOTICS are the first line treatment for indications of TYPHUS. clearly With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. treats less clear treats equal training data? disagreement can indicate vagueness & ambiguity of sentences Lora Aroyo
  • 16. • Perfuming agreement scores by forcing annotators to make choices they may think are not valid • Low annotator agreement is addressed by detailed guidelines for annotators to consistently handle the cases that generate disagreement • Remove poten9al signal on examples that are ambiguous precise annota)on guidelines do eliminate disagreement but do not increase quality 3. Detailed Guidelines Help What if they HURT? Lora Aroyo
  • 17. Which mood cluster is most appropriate for a song? Instruc9ons Your task is to listen to the following 30 second music clips and select disagreement can indicate problems with the task the most appropriate mood cluster that represents the mood of the music. Try to think about the mood carried by the music and please try to ignore any lyrics. If you feel the music does not fit into any of the 5 clusters please select “Other”. The descrip)ons of the clusters are provided in the panel at the top of the page for your reference. Answer the ques)ons carefully. Your work will not be accepted if your answers are inconsistent and/or incomplete. restric2ng guidelines help? (Lee and Hu 2012) Lora Aroyo
  • 18. • rather than accep)ng disagreement as a natural property of seman)c interpreta)on • tradi)onally, disagreement is considered a measure of poor quality because: – task is poorly defined or – annotators lack training this makes the elimina9on of disagreement the GOAL 4. Disagreement is Bad What if it is GOOD? Lora Aroyo
  • 19. Does each sentence express the TREAT relation? ANTIBIOTICS are the first line treatment for indications of TYPHUS. à agreement 95% Patients with TYPHUS who were given ANTIBIOTICS exhibited side-effects. à agreement 80% With ANTIBIOTICS in short supply, DDT was used during WWII to control the insect vectors of TYPHUS. à agreement 50% disagreement bad? disagreement can reflect the degree of clarity in a sentence Lora Aroyo
  • 20. • over 90% of annotated examples – seen by 1-­‐2 annotators • small number overlap – to measure agreement five or six popular interpreta9ons can’t be captured by one or two people 5. One is Enough What if it is NOT ENOUGH? Lora Aroyo
  • 21. One Quality? accumulated results for each rela)on across all the sentences 20 workers/sentence (and higher) yields same rela9ve disagreement Lora Aroyo
  • 22. • conven9onal wisdom: human annotators with domain knowledge provide beIer annotated data, e.g – medical texts should be annotated by medical experts • but experts are expensive & don’t scale mul9ple perspec9ves on data can be useful, beyond what experts believe is salient or correct 6. Experts Are BeIer What if the CROWD IS BETTER? Lora Aroyo
  • 23. What is the (medical) relation between the highlighted (medical) terms? • 91% of expert annotations covered by the crowd • expert annotators reach agreement only in 30% • most popular crowd vote covers 95% of this expert annotation agreement experts beIer than crowd? Lora Aroyo
  • 24. • perspec9ves change over 9me – old training data might contain examples that are not valid or only par)ally valid later • con9nuous collec9on of training data over )me allows the adapta)on of gold standards to changing )mes – popularity of music – levels of educa)on 7. Once Done, Forever Valid What if VALIDITY CHANGES?
  • 25. Which are mentions of terrorists in this sentence? OSAMA BIN LADEN used money from his own construction company to support the MUHAJADEEN in Afghanistan against Soviet forces. forever valid? 1990: hero 2011: terrorist both types should be valid -­‐ two roles for same en9ty -­‐ adapta9on of gold standards to changing 9mes Lora Aroyo
  • 27. • annotator disagreement is signal, not noise. • it is indicative of the variation in human semantic interpretation of signs • it can indicate ambiguity, vagueness, similarity, over-generality, as well as quality crowdtruth.org
  • 30. The Team 2013 hIp://crowd-­‐watson.nl