1. http://lora-aroyo.org @laroyo
Lora Aroyo
DATA SCIENCE WITH HUMANS IN THE LOOP

2. http://lora-aroyo.org @laroyo
Bulgaria NYC
The Netherlands
2
ABOUT ME ... Personal Data Science
Sofia

3. http://lora-aroyo.org @laroyo 3
E-LEARNING & AI
To understand the value of semantic technologies for e-learning
we need to understand the people, specifically how they
interact and consume information

4. ⌂ http://lora-aroyo.org @laroyo
MY
RESEARCH
FAMILY
4
… and many many many many more

5. ⌂ http://lora-aroyo.org @laroyo
MY
RESEARCH
FAMILY
5
… and many many many many more

6. http://lora-aroyo.org @laroyo 6
CROWDTRUTH
TEAM

7. http://lora-aroyo.org @laroyo
EVOLUTION OF SEMANTIC WEB
7
Great moments from 1980s till now

8. http://lora-aroyo.org @laroyo
EMPIRE OF THE EXPERTS
8
80’s
Advances in hardware and SDEs
PCs, workstations, Symbolics, Sun
New architectures like Hypercube
LISP, Prolog, OPS
AI can now BUILD SYSTEMS
Primary focus on experts and rules
What is the knowledge of experts
Graphs, logic, rules, frames
How do experts reason?
Deduction, induction
Work on form & process academic
inside the system, to make the
reasoning inside the system proper
and as good as possible
Industry forged ahead with ad-hoc &
proprietary systems and actually tried
to build expert systems
Originals of uncertain KR
Fuzzy, probabilistic

9. http://lora-aroyo.org @laroyo
EMPIRE OF THE EXPERTS
9
80’s
Piero Bonissone and
the DELTA/CATS expert
system for locomotive repair
with David Smith, a
locomotive repair expert

10. http://lora-aroyo.org @laroyo
EMPIRE OF THE EXPERTS
10
80’s
Buchanan and Shortliff’s
MYCIN project at Stanford
built a huge rule base
for medical diagnosis
working with an extensive
team of medical experts.

11. http://lora-aroyo.org @laroyo
KNOWLEDGE
ACQUISITION FROM
EXPERTS
11
90’s
Common KADS
founded by Bob Wielinga as
a methodology for expert
knowledge acquisition. It
was deeply psychology
based - it was about people,
about their knowledge and
especially about their
expertise. How do people
know what they know, and
how can you acquire that
knowledge?

12. http://lora-aroyo.org @laroyo
STRUCTURED KNOWLEDGE ENGINEERING

13. http://lora-aroyo.org @laroyo
INTEROPERABILITY & STANDARDS ODYSSEY
13
00’s

14. http://lora-aroyo.org @laroyo
AI AWAKENS
14
10’s

15. http://lora-aroyo.org @laroyo 15
2011
IBM WATSON @JEOPARDY

16. http://lora-aroyo.org @laroyo
BIG DATA
16
10’s

17. http://lora-aroyo.org @laroyo 17
BIG CROWDS
10’s
Human Annotation Central in Machine Learning Training & Evaluation

18. http://lora-aroyo.org @laroyo
COMFORT ZONE
18
7 MYTHS ABOUT HUMAN ANNOTATION

19. http://lora-aroyo.org @laroyo
ONE TRUTH
19
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
7 MYTHS ABOUT HUMAN ANNOTATION
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

20. http://lora-aroyo.org @laroyo 20
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
7 MYTHS ABOUT HUMAN ANNOTATION
ALL EXAMPLES EQUAL
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

21. http://lora-aroyo.org @laroyo 21
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
7 MYTHS ABOUT HUMAN ANNOTATION
DISAGREEMENT BAD
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

22. http://lora-aroyo.org @laroyo 22
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
7 MYTHS ABOUT HUMAN ANNOTATION
EXPERTS RULE
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

23. http://lora-aroyo.org @laroyo 23
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
7 MYTHS ABOUT HUMAN ANNOTATION
ONE IS ENOUGH
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

24. http://lora-aroyo.org @laroyo 24
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old7 MYTHS ABOUT HUMAN ANNOTATION
BINARY WORLD
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

25. http://lora-aroyo.org @laroyo 25
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
DOES THIS SENTENCE EXPRESS TREATS RELATION?

26. http://lora-aroyo.org @laroyo 26
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
DOES THIS SENTENCE EXPRESS TREATS RELATION?

27. http://lora-aroyo.org @laroyo 27
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
DOES THIS SENTENCE EXPRESS TREATS RELATION?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.

28. http://lora-aroyo.org @laroyo 28
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DO EXPERTS SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
✓
✓
✘

29. http://lora-aroyo.org @laroyo 29
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DOES A LAY ANNOTATOR SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
✓
✓
✘

30. http://lora-aroyo.org @laroyo 30
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DOES ANOTHER LAY ANNOTATOR SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
✓
✘
✘

31. http://lora-aroyo.org @laroyo 31
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DOES A THIRD LAY ANNOTATOR SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
✓
✓
✓

32. http://lora-aroyo.org @laroyo 32
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
WHAT DOES THE CROWD SAY?
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
Intuition: This is better
95%
75%
50%

33. http://lora-aroyo.org @laroyo 33
For prevention of malaria, use only in individuals traveling
to malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
95%
75%
50%
There’s a difference between these two
This one isn’t utterly wrong
BETTER
WORSE
WHAT DOES THE CROWD SAY?

34. http://lora-aroyo.org @laroyo 34
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

35. http://lora-aroyo.org @laroyo 35
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

36. http://lora-aroyo.org @laroyo 36
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

37. http://lora-aroyo.org @laroyo 37
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

38. http://lora-aroyo.org @laroyo 38
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
COMFORT ZONE
Disrupted
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

39. http://lora-aroyo.org @laroyo 39
For prevention of malaria, use only in individuals traveling to
malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
ENCOURAGING DISAGREEMENT
Rheumatoid arthritis and MALARIA have been treated
with CHLOROQUINE for decades.
Treats: Chloroquine, Malaria
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
Intuition: This is better
95%
75%
50%

40. http://lora-aroyo.org @laroyo
CROWD TASK

41. http://lora-aroyo.org @laroyo
WORKER VECTOR FOR A SENTENCE
treats associated _with othersymptom
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.

42. http://lora-aroyo.org @laroyo
MANY WORKERS FOR THE SAME SENTENCE
treats otherassociated _withsymptom
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.

43. http://lora-aroyo.org @laroyo
ALL WORKER VECTORS
AGGREGATED IN A SENTENCE VECTOR
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
treats othernoneassociated _withsymptommanifestation
side
effect

44. http://lora-aroyo.org @laroyo
SENTENCE VECTORS FOR THE 3 SENTENCES
treats othernoneassociated _withsymptommanifestation
side
effect
treats othernoneassociated _withcontraindicatesmanifestation
treats other

45. http://lora-aroyo.org @laroyo 45
One truth: knowledge acquisition for
the semantic web assumes one correct
interpretation for every example
All examples are created equal: triples
are triples, one is not more important
than another, they are all either true or
false
Disagreement bad: when people
disagree, they don’t understand the
problem
Experts rule: knowledge is captured
from domain experts
One is enough: knowledge by a single
expert is sufficient
Detailed explanations help: if
examples cause disagreement - add
instructions
Once done, forever valid: knowledge is
not updated; new data not aligned with
old
TIME TO DISRUPT THE
COMFORT ZONE
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty

46. http://lora-aroyo.org @laroyo
EXCITING DISCOVERIESEXCITING DISCOVERIES

47. http://lora-aroyo.org @laroyo
UMLS RELATION EXTRACTION PROJECT
NLP
UMLS

48. http://lora-aroyo.org @laroyo
The final frontier
VECTOR SPACE

49. http://lora-aroyo.org @laroyo
KNOWLEDGE
ACQUISITIONSTRUCTURED KNOWLEDGE ENGINEERING

50. http://lora-aroyo.org @laroyo
HYPER-DIMENSIONAL SPACE

51. http://lora-aroyo.org @laroyo
HYPER-DIMENSIONAL SPACE
3-axis tensor

52. http://lora-aroyo.org @laroyo
HYPER-DIMENSIONAL SPACE
… matrix

53. http://lora-aroyo.org @laroyo
3-AXIS TENSOR
Workers axis

54. http://lora-aroyo.org @laroyo
Relations axis
3-AXIS TENSOR

55. http://lora-aroyo.org @laroyo
Sentences axis
3-AXIS TENSOR

56. http://lora-aroyo.org @laroyo
HYPER-DIMENSIONAL SPACE
Worker votes

57. http://lora-aroyo.org @laroyo
HYPER-DIMENSIONAL SPACE
R1
R2
R3
R4
R5
R6
R7
R8
R9
R10
R11
R12
Sentence plane into a sentence vector

58. http://lora-aroyo.org @laroyo
HYPER-DIMENSIONAL SPACE
Sentence Slice

59. http://lora-aroyo.org @laroyo
HYPER-DIMENSIONAL SPACE
3 Sentence Slices

60. http://lora-aroyo.org @laroyo
DISAGREEMENT IS SIGNAL
Variety of sources for disagreement

61. http://lora-aroyo.org @laroyo
Source 1: People’s bias & perspective
DISAGREEMENT IS SIGNAL

62. http://lora-aroyo.org @laroyo
DISAGREEMENT IS SIGNAL
Source 1: Worker systematically give same answer

63. http://lora-aroyo.org @laroyo
DISAGREEMENT IS SIGNAL
Source 1: Worker systematically give same answer

64. http://lora-aroyo.org @laroyo
DISAGREEMENT IS SIGNAL
Source 1: Worker systematically give same answer

65. http://lora-aroyo.org @laroyo
Source 2: Target semantics
DISAGREEMENT IS SIGNAL

66. http://lora-aroyo.org @laroyo
SentencesSource 3: Sentences
DISAGREEMENT IS SIGNAL

67. http://lora-aroyo.org @laroyo
TRIANGLE OF MEANING
Model of semantic interpretation (Ogden & Richards, 1936)

68. http://lora-aroyo.org @laroyo
TRIANGLE OF MEANING
Model of semantic interpretation

69. http://lora-aroyo.org @laroyo
treats other
CrowdTruth metrics for quality assessment
TRIANGLE OF MEANING

70. http://lora-aroyo.org @laroyo
treats other
Spam
QUALITY ASSESSMENT

71. http://lora-aroyo.org @laroyo
treats othernoneassociated _withsymptommanifestation
side
effect
Among 56 subjects reporting to a clinic with symptoms
of MALARIA 53 (95%) had ordinarily effective levels
of CHLOROQUINE in blood.
Sentence ambiguity
QUALITY ASSESSMENT

72. http://lora-aroyo.org @laroyo
TREATS RELATION
Yes or No?
treats othernoneassociated _withcontraindicatesmanifestation
treats othernoneassociated _withcontraindicatesmanifestation
treats other

73. http://lora-aroyo.org @laroyo
THE WORLD IS SMOOTH AND NOT BINARY

74. http://lora-aroyo.org @laroyo
Agreement as percentage
CROWDTRUTH METRICS
For prevention of malaria, use only in individuals traveling to
malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
25% 25% 75% 12% 12% 50%
treats othernoneassociated _withcontraindicatesmanifestation

75. http://lora-aroyo.org @laroyo
For prevention of malaria, use only in individuals traveling to
malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
CROWDTRUTH METRICS
Applying all sides of the triangle
treats other

76. http://lora-aroyo.org @laroyo
CROWDTRUTH METRICS
For prevention of malaria, use only in individuals traveling to
malarious areas where CHLOROQUINE resistant P.
falciparum MALARIA has not been reported.
treats other
Applying all sides of the triangle
99%

77. http://lora-aroyo.org @laroyo
CROWDTRUTH METRICS
Applying all sides of the triangle

78. http://lora-aroyo.org @laroyo
CROWDTRUTH
KNOWLEDGE TENSOR

79. http://lora-aroyo.org @laroyo
CROWDTRUTH VS. EXPERTS
crowd as good or better than from experts

80. http://lora-aroyo.org @laroyo
AMBIGUITY IMPACTS ACCURACY
more ambiguous sentences were harder to classify

81. http://lora-aroyo.org @laroyo
CROWDTRUTH METRICS
Quality assessment

82. http://lora-aroyo.org @laroyo
CROWDTRUTH.ORG
a spatial representation of meaning that harnesses disagreement

83. http://lora-aroyo.org @laroyo
On the role of user-generated metadata in audio visual collections (2011).
R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011
VIDEO METADATA ENRICHMENT
The Netherlands Institute for Sound and Vision
1

84. http://lora-aroyo.org @laroyo
DIVE+
Explorative Search
2
DIVE into the event-based browsing of linked historical media (2015)
V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics:

85. http://lora-aroyo.org @laroyo
DEEP QA IN CULTURAL HERITAGE
Mauritshuis use case
3

86. http://lora-aroyo.org @laroyo
CROWDTRUTH IN DEPLOYMENT
Google Maps
questions
Google
Maps
reviewers
4

87. http://lora-aroyo.org @laroyo
CROWDTRUTH IN DEPLOYMENT
Google Maps
emotions
mTURK
crowd
5

88. http://lora-aroyo.org @laroyo
WHAT DOES THE FUTURE HOLD

89. http://lora-aroyo.org @laroyo
USER-CENTRIC DATA SCIENCE
Formerly the Web & Media group

90. http://lora-aroyo.org @laroyo
H2020 ReTV
Trans-Vector Platform (TVP)
Lora Aroyo, (coordinator) VU
Amsterdam, Computer Science
Lyndon Nixon, MODUL, AT
Vasileios Mezaris, CERTH, GR
Arno Scharl, Weblyzard, AT
Bea Knecht, Zattoo, DE
Johan Oomen, Sound and Vision, NL
Nicolas Patz, Rundfunk
Berlin-brandenburg, DE

91. http://lora-aroyo.org @laroyo
CAPTURING BIAS
Startimpuls for the Dutch National Science Agenda
Lora Aroyo, (coordinator) VU
Amsterdam, Computer Science
Alessandro Bozzon, TU Delft CS &
Delft Data Science
Alec Badenoch, Utrecht University,
Media & Culture Studies
Antoaneta Dimitrova, Leiden
University, Institute of Public
Administration
Johan Oomen, Netherlands
Institute for Sound and Vision

92. http://lora-aroyo.org @laroyo
CROWDTRUTH ROCKS!
Disagreement is signal
CrowdTruth is a spatial representation
of meaning that harnesses disagreement
Crowds bring natural diversity
CrowdTruth defines hyper-dimensional
space to represent ambiguity
Crowds help gathering
real human semantics

93. http://lora-aroyo.org @laroyo
The world is full
of shades of grey
Experts and crowds
are complimentary
Capturing and understanding opinions,
perspectives and contexts in the center
of understanding people
TIME TO BREAK FREE
CrowdTruth defines multi-dimensional
space to measure quality

94. http://lora-aroyo.org @laroyo
Lora Aroyo
DATA SCIENCE WITH HUMANS IN THE LOOP