SlideShare a Scribd company logo
1 of 84
Download to read offline
http://lora-aroyo.org @laroyo
Disrupting the Semantic
Lora Aroyo
Web & Media Group
Web & Media Group
http://lora-aroyo.org @laroyo
Bulgaria
The Netherlands
Sofia
NYC
Personal
Semantics
Web & Media Group
http://lora-aroyo.org @laroyo
Riva del Garda, Italy, 2014
Semantic
Social Life
Web & Media Group
http://lora-aroyo.org @laroyo
4
To understand the value of
Semantic Web for e-learning
you have to understand people,
e.g. how they learn, interact &
consume information
Web & Media Group
http://lora-aroyo.org @laroyo
5
To understand the value of
Semantic Web for e-learning
you have to understand people,
e.g. how they interact &
consume information
Web & Media Group
http://lora-aroyo.org @laroyo
6
To understand the value of Semantic Web
for cultural heritage
you have to understand people, e.g.
how they interact & consume information
Web & Media Group
http://lora-aroyo.org @laroyo
7
To understand the value of Semantic Web
for cultural heritage
you have to understand people, e.g.
how they interact & consume information
Web & Media Group
http://lora-aroyo.org @laroyo
To understand the value of Semantic Web
for digital humanities, you have to
understand people, e.g.
how they interact & consume information
Web & Media Group
http://lora-aroyo.org @laroyo
people are in the center of everything
people & their semantics, i.e. their real-world behavior,
online interactions, information needs, information
consumption habits, personal preferences ...
Web & Media Group
http://lora-aroyo.org @laroyo
CrowdTruth team
http://lora-aroyo.org @laroyo
Web & Media Group
the evolution of the semantic web:
great moments from the 1980s to ESWC 2017
http://lora-aroyo.org @laroyo
50’AI more or less begins
......
80’expert systems
90’knowledge acquisition from experts
00’standards & interoperability
10’big data & large crowds
A long time ago
in a galaxy far, far away …
http://lora-aroyo.org @laroyo
80’s - empire of the experts
http://lora-aroyo.org @laroyo
Advances in hardware and SDEs
PCs, workstations, Symbolics, Sun
New architectures like the Hypercube
LISP, Prolog, OPS
AI can now BUILD SYSTEMS
Primary focus on experts and rules
What is the knowledge of experts
What is the form of this knowledge?
Graphs, logic, rules, frames
How do experts reason?
Deduction, induction
80’s - empire of the experts
Work on form & process remained
academic
what happened inside the system, to
make the reasoning inside the system
proper and as good as possible
industry forged ahead with ad-hoc
& proprietary systems and actually
tried to build expert systems
Originals of uncertain KR
Fuzzy, probabilistic
http://lora-aroyo.org @laroyo
Piero Bonissone and the
DELTA/CATS expert system for
locomotive repair with David Smith, a
locomotive repair expert
Buchanan and Shortliff’s MYCIN project at
Stanford built an huge rule base for medicat
diagnosis working with an extensive team of
medical experts.
http://lora-aroyo.org @laroyo
90’s - knowledge acquisition from experts
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
90’s - knowledge acquisition from experts
The 90’s brought [attention for] knowledge acquisition.
Knowing that expert systems by then can functionally work, the focus [in
practice as well as scientific research and technology development] shifted
to the then-bigger challenge of how to acquire knowledge in real-world
scenarios.
It seems natural that after the look inside the systems, then one needed
to pay attention to how actually get the knowledge from the world outside
and frame it into the proper structured knowledge for inside the system.
Dream of the 90’s
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
00’s - interoperability & standards odyssey
http://lora-aroyo.org @laroyo
10’s - AI Awakens
• Machine Learning
• Neural networks
• Solving basic perceptual problems instead of high-expertise ones
• Ambiguity tolerant reasoning
• Non-taxonomic ordering → non-taxonomic reasoning
• folksonomies, clustering, diversity of perspectives, embeddings
Web & Media Group
http://lora-aroyo.org @laroyo
2011
http://lora-aroyo.org @laroyo
10’s – Big Data
Web & Media Group
http://lora-aroyo.org @laroyo
Human Annotation
Central in Machine Learning
Training & Evaluation
10’s – Crowds
http://lora-aroyo.org @laroyo
Web & Media Group
Team BellKor wins Netflix Prize
20071998 2006 2009
Web & Media Group
http://lora-aroyo.org @laroyo
Web & Media Group
http://lora-aroyo.org @laroyo
the semantic
comfort
zone
Web & Media Group
http://lora-aroyo.org @laroyo
One truth: knowledge acquisition for the semantic web
assumes one correct interpretation for every example
All examples are created equal: triples are triples, one is not
more important than another, they are all either true or false
Disagreement bad: when people disagree, they don’t
understand the problem
Experts rule: knowledge is captured from domain experts
One is enough: knowledge by a single expert is sufficient
Detailed explanations help: if examples cause disagreement
- add instructions
Once done, forever valid: knowledge is not updated; new
data not aligned with old
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
video archive
enrichment
Search Behavior of Media Professionals at an Audiovisual Archive:
A Transaction Log Analysis (2009).
B. Huurnink, L. Hollink, W. van den Heuvel, M. de Rijke.
Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
video archive
enrichment
Goal:
make the
multimedia content of
Dutch National Video Archive
accessible to large audiences
Comfort Zone Solution:
media professionals watch & annotate videos. Of course!
Web & Media Group
http://lora-aroyo.org @laroyo
but ...
Expensive
Doesn’t scale
time-consuming
5 times the video duration
professional vocabulary
experts use a specific vocabulary
that is unknown to general audiences
Web & Media Group
http://lora-aroyo.org @laroyo
… and
people search for fragments
experts annotate full videos
not finding
35% of search queries result in not found
Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
real world QA
for Watson
Crowdsourcing ground truth for Question Answering using CrowdTruth (2015).
B Timmermans, L Aroyo, C Welty
Web & Media Group
http://lora-aroyo.org @laroyo
Goal:
gather questions
that real people ask
for training & evaluating Watson
Data:
30K Questions + Candidate Answers.
from Yahoo! Answers
Comfort Zone Solution:
ask people if the passage answers the question (Y/N). Simple!
Use Case:
real world QA
for Watson
Web & Media Group
http://lora-aroyo.org @laroyo
Contradicting evidence
Is Coral a plant?
• “Coral almost could be considered half-plant [..]”
• “[..] organism, such as a coral, resembling a stony plant.”
Unanswerable questions
• Can I take a pill if you don't have a child yet?
• Is the spelling for being drunk right?
• Is napster black?
Unclear answer type
Is paper animal plant or man made?
Multiple right answers to a question
What is the best university in NY? (subjective)
YES or NO?
Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
medical relation
extraction
for Watson
Crowdsourcing Ground Truth for Medical Relation Extraction (2017).
A Dumitrache, L Aroyo, C Welty
Web & Media Group
http://lora-aroyo.org @laroyo
Goal:
gather data to train
Watson to read
medical text & automatically
extract a medical relations KB
Comfort Zone Solution:
having medical experts read & annotate examples
Use Case:
medical relation
extraction
for Watson
Web & Media Group
http://lora-aroyo.org @laroyo
ANTIBIOTICS are the first line treatment for
indications of TYPHUS.
treats(ANTIBIOTICS, TYPHUS)? Expert: yes
Patients with TYPHUS who were given ANTIBIOTICS
exhibited side-effects.
treats(ANTIBIOTICS, TYPHUS)? Expert: yes
With ANTIBIOTICS in short supply, DDT was used
during WWII to control the insect vectors of
TYPHUS.
treats(ANTIBIOTICS, TYPHUS)? Expert: yes.
Are these three really all the same???
Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
map music to moods
Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
map music to moods
Goal:
annotate songs with emotional tags
Comfort Zone Solution:
people assign the prevalent mood of a song
Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Other
passionate, rollicking, literate, humorous, silly, aggressive, fiery, does not fit into
rousing, cheerful, fun, poignant, wistful, campy, quirky, tense, anxious, any of the 5
confident, sweet, amiable, bittersweet, whimsical, witty, intense, volatile, clusters
boisterous, good-natured autumnal, wry visceral
rowdy brooding
Choose one:
Which is the mood most appropriate
for each song?
Goal:
(Lee and Hu 2012)
1 song - 1 mood???
Web & Media Group
http://lora-aroyo.org @laroyo
One truth: knowledge acquisition for the semantic web
assumes one correct interpretation for every example
All examples are created equal: triples are triples, one is not
more important than another, they are all either true or false
Disagreement bad: when people disagree, they don’t
understand the problem
Experts rule: knowledge is captured from domain experts
One is enough: knowledge by a single expert is sufficient
Detailed explanations help: if examples cause disagreement
- add instructions
Once done, forever valid: knowledge is not updated; new
data not aligned with old
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
Web & Media Group
http://lora-aroyo.org @laroyo
One truth: knowledge acquisition for the semantic web
assumes one correct interpretation for every example
All examples are created equal: triples are triples, one is not
more important than another, they are all either true or false
Disagreement bad: when people disagree, they don’t
understand the problem
Experts rule: knowledge is captured from domain experts
One is enough: knowledge by a single expert is sufficient
Detailed explanations help: if examples cause disagreement
- add instructions
Once done, forever valid: knowledge is not updated; new
data not aligned with old
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
Semantic
Comfort Zone
Web & Media Group
http://lora-aroyo.org @laroyo
One truth: knowledge acquisition for the semantic web
assumes one correct interpretation for every example
All examples are created equal: triples are triples, one is not
more important than another, they are all either true or false
Disagreement bad: when people disagree, they don’t
understand the problem
Experts rule: knowledge is captured from domain experts
One is enough: knowledge by a single expert is sufficient
Detailed explanations help: if examples cause disagreement
- add instructions
Once done, forever valid: knowledge is not updated; new
data not aligned with old
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
Semantic
Comfort Zone
disrupted
Web & Media Group
http://lora-aroyo.org @laroyo
Web & Media Group
http://lora-aroyo.org @laroyo
interestingly …
Web & Media Group
http://lora-aroyo.org @laroyo
• collective decisions of large groups
of people
• a group of error-prone
decision-makers can be surprisingly
good at picking the best choice
• when thumbs up or thumbs down - the
chance of picking the right answer
needs to be > 50%
• the odds that a most of them will pick
the right answer is greater than any of
them will pick it on their own
• performance gets better as size grows
1785
Marquis de Condorcet
“wisdom of crowds”
Web & Media Group
http://lora-aroyo.org @laroyo
•asked 787 people to
guess the weight of
an ox
•none got the right
answer
•their collective guess
was almost perfect
1906
Sir Francis Galton
“wisdom of crowds”
Web & Media Group
http://lora-aroyo.org @laroyo
WWII Math Rosies
1942: Ballistics calculations and flight trajectories
Web & Media Group
http://lora-aroyo.org @laroyo
NASA’s Computer Room
transcribe raw flight data from celluloid film & oscillograph paper
Web & Media Group
http://lora-aroyo.org @laroyo
can we harness it?
http://lora-aroyo.org @laroyo
Web & Media Group
CrowdTruth
http://crowdtruth.org/
http://lora-aroyo.org @laroyo
Web & Media Group
CrowdTruth
Three basic causes of disagreement: workers,
examples, target semantics
Disagreement is signal, not noise.
It is indicative of the variation in human semantic
interpretation
It can indicate ambiguity, vagueness, similarity,
over-generality, etc, as well as quality
Crowdtruth: Machine-human computation framework for harnessing disagreement
in gathering annotated data (2014)
O Inel, A Dumitrache, l.Aroyo, C. Welty
Web & Media Group
http://lora-aroyo.org @laroyo
one truth: multiple truths
all examples are created equal:
each example is unique
disagreement bad: disagreement is good
experts rule: crowd rules
one is enough: the more the better
detailed explanations help:
keep it simple stupid
once done, forever valid:
maintenance is necessary
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
Web & Media Group
http://lora-aroyo.org @laroyo
changes needed
video archive
enrichment
improve support
for fragment search
time-based annotations
bridging vocabulary gap between
searcher & cataloguer
Web & Media Group
http://lora-aroyo.org @laroyo
crowdsourcing
video tagging
two
video tagging pilots
Web & Media Group
http://lora-aroyo.org @laroyo
@waisda
http://waisda.nl
engage
crowds
through
continuous
gaming
http://lora-aroyo.org @laroyo
Web & Media Group
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
http://lora-aroyo.org @laroyo
Web & Media Group
time-based
bernhard
just “tags”
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
http://lora-aroyo.org @laroyo
Web & Media Group
objects (57%)
westminster abbey
abbey
priester
geestelijken
hek
paarden
tocht
aankomst
koets
kroning
mensenmassa
parade
kroon
regen
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
http://lora-aroyo.org @laroyo
Web & Media Group
persons (31%)
bernhard
juliana
objects (57%)
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
http://lora-aroyo.org @laroyo
Web & Media Group
user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
locations (7%)
engeland
locations (7%)
persons (31%)
objects (57%)
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
http://lora-aroyo.org @laroyo
Web & Media Group
user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
locations (7%)
describe mainly short segments
often not very specific
don’t describe programmes as a whole
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
Web & Media Group
http://lora-aroyo.org @laroyo
crowdsourcing
medical relation
extraction
diversity of opinions
independent perspectives
multitude of contexts
we exposed a richer set of possibilities
that help in identifying, processing
& understanding context
Web & Media Group
http://lora-aroyo.org @laroyo
Does this sentence express
TREATS(Antibiotics, Typhus)?
Patients with TYPHUS who were given
ANTIBIOTICS exhibited several side-effects.
With ANTIBIOTICS in short supply, DDT was
used during World War II to control the insect
vectors of TYPHUS.
ANTIBIOTICS are the first line treatment for
indications of TYPHUS. 95%
75%
50%
The crowd results captures the natural ambiguity
http://lora-aroyo.org @laroyo
Web & Media Group
What is the relation between the highlighted terms?
He was the first physician to identify the relationship
between HEMOPHILIA and HEMOPHILIC ARTHROPATHY.
Experts Hallucinate
Crowd reads text literally - provide better examples to machine
experts: cause
crowd: no relation
http://lora-aroyo.org @laroyo
Web & Media Group
Unclear relationship between the two arguments reflected
in the disagreement
Medical Relation Extraction
http://lora-aroyo.org @laroyo
Web & Media Group
Clearly expressed relation between the two arguments reflected in
the agreement
Medical Relation Extraction
http://lora-aroyo.org @laroyo
Web & Media Group
Unclear relationship between the two arguments reflected
in the disagreement
Medical Relation Extraction
http://lora-aroyo.org @laroyo
Web & Media Group
http://lora-aroyo.org @laroyo
Web & Media Group
Learning Curves
(crowd with pos./neg. threshold at 0.5)
above 400 sent.: crowd consistently over baseline & single
above 600 sent.: crowd out-performs experts
http://lora-aroyo.org @laroyo
Web & Media Group
Learning Curves Extended
(crowd with pos./neg. threshold at 0.5)
crowd consistently performs better than baseline
http://lora-aroyo.org @laroyo
Web & Media Group
# of Workers: Impact on Sentence-Relation Score
Web & Media Group
http://lora-aroyo.org @laroyo
Training a Relation Extraction Classifier
F1
Cost per
sentence
CrowdTruth 0.642 $0.66
Expert Annotator 0.638 $2.00
Single Annotator 0.492 $0.08
“wisdom of the crowd”
provides training data that is at least as good
if not better than experts
only with proper analytic framework for
harnessing disagreement from the crowd
http://lora-aroyo.org @laroyo
Web & Media Group
map music to moods
Goal:
tag songs with emotional clusters
Comfort Zone Solution:
people assign the prevalent mood of a song
Web & Media Group
http://lora-aroyo.org @laroyo
Web & Media Group
http://lora-aroyo.org @laroyo
Is this song ….
?Passionate
Rousing
Confident
Boisterous
Rowdy
Literate
Poignant
Wistful
Bittersweet
Autumnal
Brooding
Rollicking
Cheerful
Fun
Sweet
Amiable
Good-natured
Humorous
Silly
Campy
Whimsical
Witty
Wry
Aggressive
Fiery
Tense
Anxious
Intense
Volatile
Web & Media Group
http://lora-aroyo.org @laroyo
If “One Truth” & “No Disagreement”
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5
W1 1
W2 1
W3 1
W4 1
W5 1
W6 1
W7
W8
W9 1
W10 1
Totals 1 3 1 2 1
Web & Media Group
http://lora-aroyo.org @laroyo
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W1 1 1 1
W2 1 1 1
W3 1 1 1
W4 1 1
W5 1 1
W6 1 1 1
W7 1 1 1
W8 1 1 1
W9 1 1
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
If “Many Truths” & “Disagreement”
Web & Media Group
http://lora-aroyo.org @laroyo
can indicate
alternative interpretations
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
Disagreement as Signal
can indicate
ambiguity in the
categorisation
can indicate
low quality workers
http://lora-aroyo.org @laroyo
so …
http://lora-aroyo.org @laroyo
getting
comfortable
again
http://lora-aroyo.org @laroyo
Take Home Message
People first, experts second
True and False is not enough,
There is diversity in human interpretation
CrowdTruth introduces a spatial representation
of meaning that harnesses disagreement
With CrowdTruth untrained workers can be just as
reliable as highly trained experts
http://lora-aroyo.org @laroyo
http://data.crowdtruth.org/

More Related Content

What's hot

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchLora Aroyo
 
Social Web 2014: Final Presentations (Part I)
Social Web 2014: Final Presentations (Part I)Social Web 2014: Final Presentations (Part I)
Social Web 2014: Final Presentations (Part I)Lora Aroyo
 
Dispute finder
Dispute finderDispute finder
Dispute finderrobennals
 
"Why the Semantic Web will Never Work" (note the quotes)
"Why the Semantic Web will Never Work"  (note the quotes)"Why the Semantic Web will Never Work"  (note the quotes)
"Why the Semantic Web will Never Work" (note the quotes)James Hendler
 
Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Lora Aroyo
 
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 Lora Aroyo
 
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...CrowdTruth
 
State of RecSys: Recap of RecSys 2012
State of RecSys: Recap of RecSys 2012State of RecSys: Recap of RecSys 2012
State of RecSys: Recap of RecSys 2012Alan Said
 
"What is Data Science?"
"What is Data Science?""What is Data Science?"
"What is Data Science?"Renee Teate
 
On Beyond OWL: challenges for ontologies on the Web
On Beyond OWL: challenges for ontologies on the WebOn Beyond OWL: challenges for ontologies on the Web
On Beyond OWL: challenges for ontologies on the WebJames Hendler
 
WordPress in Higher Education
WordPress in Higher EducationWordPress in Higher Education
WordPress in Higher EducationShane Pearlman
 
Can Open Data Save The Public Realm
Can Open Data Save The Public RealmCan Open Data Save The Public Realm
Can Open Data Save The Public RealmChris Taggart
 

What's hot (13)

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
 
Social Web 2014: Final Presentations (Part I)
Social Web 2014: Final Presentations (Part I)Social Web 2014: Final Presentations (Part I)
Social Web 2014: Final Presentations (Part I)
 
Dispute finder
Dispute finderDispute finder
Dispute finder
 
"Why the Semantic Web will Never Work" (note the quotes)
"Why the Semantic Web will Never Work"  (note the quotes)"Why the Semantic Web will Never Work"  (note the quotes)
"Why the Semantic Web will Never Work" (note the quotes)
 
Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)
 
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
 
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
 
State of RecSys: Recap of RecSys 2012
State of RecSys: Recap of RecSys 2012State of RecSys: Recap of RecSys 2012
State of RecSys: Recap of RecSys 2012
 
Diversity (in Media)
Diversity (in Media)Diversity (in Media)
Diversity (in Media)
 
"What is Data Science?"
"What is Data Science?""What is Data Science?"
"What is Data Science?"
 
On Beyond OWL: challenges for ontologies on the Web
On Beyond OWL: challenges for ontologies on the WebOn Beyond OWL: challenges for ontologies on the Web
On Beyond OWL: challenges for ontologies on the Web
 
WordPress in Higher Education
WordPress in Higher EducationWordPress in Higher Education
WordPress in Higher Education
 
Can Open Data Save The Public Realm
Can Open Data Save The Public RealmCan Open Data Save The Public Realm
Can Open Data Save The Public Realm
 

Similar to My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

What Do Future Technology and Trends Mean for You?
What Do Future Technology and Trends Mean for You?   				What Do Future Technology and Trends Mean for You?
What Do Future Technology and Trends Mean for You? Anne Adrian
 
Adape Social Marketing Overview
Adape   Social Marketing OverviewAdape   Social Marketing Overview
Adape Social Marketing OverviewClive Lam
 
Florida librarydirectors
Florida librarydirectorsFlorida librarydirectors
Florida librarydirectorsStephen Abram
 
CrowdTruth @VU Faculty Colloquium (June 2015)
CrowdTruth @VU Faculty Colloquium (June 2015)CrowdTruth @VU Faculty Colloquium (June 2015)
CrowdTruth @VU Faculty Colloquium (June 2015)Lora Aroyo
 
Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AILora Aroyo
 
Hypothesis quick overview 2011-10-19
Hypothesis  quick overview 2011-10-19Hypothesis  quick overview 2011-10-19
Hypothesis quick overview 2011-10-19dwhly
 
Copy of lit project
Copy of lit projectCopy of lit project
Copy of lit projectjsnart
 
Intro to Social Media for DSFRS - 22nd July2010
Intro to Social Media for DSFRS - 22nd July2010Intro to Social Media for DSFRS - 22nd July2010
Intro to Social Media for DSFRS - 22nd July2010Carl Haggerty
 
The Human Intranet
The Human IntranetThe Human Intranet
The Human IntranetAndy Gibson
 
Top Three Challenges to Building an Organization Dedicated to Social Learning
Top Three Challenges to Building an Organization Dedicated to Social LearningTop Three Challenges to Building an Organization Dedicated to Social Learning
Top Three Challenges to Building an Organization Dedicated to Social LearningintroNetworks.com
 
PSH Mobile Voice 2016 Personal Virtual Assistants 
Are Not Enough?
PSH Mobile Voice 2016 Personal Virtual Assistants 
Are Not Enough?PSH Mobile Voice 2016 Personal Virtual Assistants 
Are Not Enough?
PSH Mobile Voice 2016 Personal Virtual Assistants 
Are Not Enough?Paul Heirendt
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityLora Aroyo
 
Crowdsourcing 101 for GLAMs
Crowdsourcing 101 for GLAMsCrowdsourcing 101 for GLAMs
Crowdsourcing 101 for GLAMsOlaf Janssen
 
Practical Machine Ethics @ SXSW2019
Practical Machine Ethics @ SXSW2019Practical Machine Ethics @ SXSW2019
Practical Machine Ethics @ SXSW2019Jesus Ramos
 
How Not to Get Fired Using Social Media at Work - EEO, Diversity and Social M...
How Not to Get Fired Using Social Media at Work - EEO, Diversity and Social M...How Not to Get Fired Using Social Media at Work - EEO, Diversity and Social M...
How Not to Get Fired Using Social Media at Work - EEO, Diversity and Social M...GovLoop
 
The Architecture of Understanding
The Architecture of UnderstandingThe Architecture of Understanding
The Architecture of UnderstandingPeter Morville
 
Dyslexia Essay Introduction. Online assignment writing service.
Dyslexia Essay Introduction. Online assignment writing service.Dyslexia Essay Introduction. Online assignment writing service.
Dyslexia Essay Introduction. Online assignment writing service.Brenda Gutierrez
 
The Vortex: Experiments in Online Collaboration
The Vortex: Experiments in Online CollaborationThe Vortex: Experiments in Online Collaboration
The Vortex: Experiments in Online CollaborationNancy Wright White
 

Similar to My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone (20)

What Do Future Technology and Trends Mean for You?
What Do Future Technology and Trends Mean for You?   				What Do Future Technology and Trends Mean for You?
What Do Future Technology and Trends Mean for You?
 
Adape Social Marketing Overview
Adape   Social Marketing OverviewAdape   Social Marketing Overview
Adape Social Marketing Overview
 
Sjsul web2.011
Sjsul web2.011Sjsul web2.011
Sjsul web2.011
 
Florida librarydirectors
Florida librarydirectorsFlorida librarydirectors
Florida librarydirectors
 
CrowdTruth @VU Faculty Colloquium (June 2015)
CrowdTruth @VU Faculty Colloquium (June 2015)CrowdTruth @VU Faculty Colloquium (June 2015)
CrowdTruth @VU Faculty Colloquium (June 2015)
 
Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
 
Hypothesis quick overview 2011-10-19
Hypothesis  quick overview 2011-10-19Hypothesis  quick overview 2011-10-19
Hypothesis quick overview 2011-10-19
 
Copy of lit project
Copy of lit projectCopy of lit project
Copy of lit project
 
Intro to Social Media for DSFRS - 22nd July2010
Intro to Social Media for DSFRS - 22nd July2010Intro to Social Media for DSFRS - 22nd July2010
Intro to Social Media for DSFRS - 22nd July2010
 
The Human Intranet
The Human IntranetThe Human Intranet
The Human Intranet
 
Top Three Challenges to Building an Organization Dedicated to Social Learning
Top Three Challenges to Building an Organization Dedicated to Social LearningTop Three Challenges to Building an Organization Dedicated to Social Learning
Top Three Challenges to Building an Organization Dedicated to Social Learning
 
PSH Mobile Voice 2016 Personal Virtual Assistants 
Are Not Enough?
PSH Mobile Voice 2016 Personal Virtual Assistants 
Are Not Enough?PSH Mobile Voice 2016 Personal Virtual Assistants 
Are Not Enough?
PSH Mobile Voice 2016 Personal Virtual Assistants 
Are Not Enough?
 
Wisconsin la2011
Wisconsin la2011Wisconsin la2011
Wisconsin la2011
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden University
 
Crowdsourcing 101 for GLAMs
Crowdsourcing 101 for GLAMsCrowdsourcing 101 for GLAMs
Crowdsourcing 101 for GLAMs
 
Practical Machine Ethics @ SXSW2019
Practical Machine Ethics @ SXSW2019Practical Machine Ethics @ SXSW2019
Practical Machine Ethics @ SXSW2019
 
How Not to Get Fired Using Social Media at Work - EEO, Diversity and Social M...
How Not to Get Fired Using Social Media at Work - EEO, Diversity and Social M...How Not to Get Fired Using Social Media at Work - EEO, Diversity and Social M...
How Not to Get Fired Using Social Media at Work - EEO, Diversity and Social M...
 
The Architecture of Understanding
The Architecture of UnderstandingThe Architecture of Understanding
The Architecture of Understanding
 
Dyslexia Essay Introduction. Online assignment writing service.
Dyslexia Essay Introduction. Online assignment writing service.Dyslexia Essay Introduction. Online assignment writing service.
Dyslexia Essay Introduction. Online assignment writing service.
 
The Vortex: Experiments in Online Collaboration
The Vortex: Experiments in Online CollaborationThe Vortex: Experiments in Online Collaboration
The Vortex: Experiments in Online Collaboration
 

More from Lora Aroyo

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfLora Aroyo
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningLora Aroyo
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumLora Aroyo
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorLora Aroyo
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataLora Aroyo
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumLora Aroyo
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18Lora Aroyo
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsLora Aroyo
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesLora Aroyo
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the LoopLora Aroyo
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...Lora Aroyo
 
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital AgeEuropeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital AgeLora Aroyo
 
"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to SnapchatLora Aroyo
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyLora Aroyo
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...Lora Aroyo
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumLora Aroyo
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Lora Aroyo
 
Keynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventLora Aroyo
 
Closing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseClosing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseLora Aroyo
 
CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015Lora Aroyo
 

More from Lora Aroyo (20)

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH Symposium
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP Demonstrator
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked Data
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithms
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & Machines
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the Loop
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
 
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital AgeEuropeana GA 2016: Harnessing Crowds, Niches & Professionals  in the Digital Age
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
 
"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat"Video Killed the Radio Star": From MTV to Snapchat
"Video Killed the Radio Star": From MTV to Snapchat
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening Ceremony
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
 
Keynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program Event
 
Closing Event - Watson Innovation Course
Closing Event - Watson Innovation CourseClosing Event - Watson Innovation Course
Closing Event - Watson Innovation Course
 
CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015CrowdTruth Games @NLeSc eHumanities day 2015
CrowdTruth Games @NLeSc eHumanities day 2015
 

Recently uploaded

TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsRizwan Syed
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationSlibray Presentation
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxLoriGlavin3
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 

Recently uploaded (20)

TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Scanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL CertsScanning the Internet for External Cloud Exposures via SSL Certs
Scanning the Internet for External Cloud Exposures via SSL Certs
 
Connect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck PresentationConnect Wave/ connectwave Pitch Deck Presentation
Connect Wave/ connectwave Pitch Deck Presentation
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
DMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special EditionDMCC Future of Trade Web3 - Special Edition
DMCC Future of Trade Web3 - Special Edition
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data PrivacyTrustArc Webinar - How to Build Consumer Trust Through Data Privacy
TrustArc Webinar - How to Build Consumer Trust Through Data Privacy
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptxPasskey Providers and Enabling Portability: FIDO Paris Seminar.pptx
Passkey Providers and Enabling Portability: FIDO Paris Seminar.pptx
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 

My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone

  • 1. http://lora-aroyo.org @laroyo Disrupting the Semantic Lora Aroyo Web & Media Group
  • 2. Web & Media Group http://lora-aroyo.org @laroyo Bulgaria The Netherlands Sofia NYC Personal Semantics
  • 3. Web & Media Group http://lora-aroyo.org @laroyo Riva del Garda, Italy, 2014 Semantic Social Life
  • 4. Web & Media Group http://lora-aroyo.org @laroyo 4 To understand the value of Semantic Web for e-learning you have to understand people, e.g. how they learn, interact & consume information
  • 5. Web & Media Group http://lora-aroyo.org @laroyo 5 To understand the value of Semantic Web for e-learning you have to understand people, e.g. how they interact & consume information
  • 6. Web & Media Group http://lora-aroyo.org @laroyo 6 To understand the value of Semantic Web for cultural heritage you have to understand people, e.g. how they interact & consume information
  • 7. Web & Media Group http://lora-aroyo.org @laroyo 7 To understand the value of Semantic Web for cultural heritage you have to understand people, e.g. how they interact & consume information
  • 8. Web & Media Group http://lora-aroyo.org @laroyo To understand the value of Semantic Web for digital humanities, you have to understand people, e.g. how they interact & consume information
  • 9. Web & Media Group http://lora-aroyo.org @laroyo people are in the center of everything people & their semantics, i.e. their real-world behavior, online interactions, information needs, information consumption habits, personal preferences ...
  • 10. Web & Media Group http://lora-aroyo.org @laroyo CrowdTruth team
  • 11. http://lora-aroyo.org @laroyo Web & Media Group the evolution of the semantic web: great moments from the 1980s to ESWC 2017
  • 12. http://lora-aroyo.org @laroyo 50’AI more or less begins ...... 80’expert systems 90’knowledge acquisition from experts 00’standards & interoperability 10’big data & large crowds A long time ago in a galaxy far, far away …
  • 13. http://lora-aroyo.org @laroyo 80’s - empire of the experts
  • 14. http://lora-aroyo.org @laroyo Advances in hardware and SDEs PCs, workstations, Symbolics, Sun New architectures like the Hypercube LISP, Prolog, OPS AI can now BUILD SYSTEMS Primary focus on experts and rules What is the knowledge of experts What is the form of this knowledge? Graphs, logic, rules, frames How do experts reason? Deduction, induction 80’s - empire of the experts Work on form & process remained academic what happened inside the system, to make the reasoning inside the system proper and as good as possible industry forged ahead with ad-hoc & proprietary systems and actually tried to build expert systems Originals of uncertain KR Fuzzy, probabilistic
  • 15. http://lora-aroyo.org @laroyo Piero Bonissone and the DELTA/CATS expert system for locomotive repair with David Smith, a locomotive repair expert Buchanan and Shortliff’s MYCIN project at Stanford built an huge rule base for medicat diagnosis working with an extensive team of medical experts.
  • 16. http://lora-aroyo.org @laroyo 90’s - knowledge acquisition from experts
  • 18. http://lora-aroyo.org @laroyo 90’s - knowledge acquisition from experts The 90’s brought [attention for] knowledge acquisition. Knowing that expert systems by then can functionally work, the focus [in practice as well as scientific research and technology development] shifted to the then-bigger challenge of how to acquire knowledge in real-world scenarios. It seems natural that after the look inside the systems, then one needed to pay attention to how actually get the knowledge from the world outside and frame it into the proper structured knowledge for inside the system. Dream of the 90’s
  • 20. http://lora-aroyo.org @laroyo 00’s - interoperability & standards odyssey
  • 21. http://lora-aroyo.org @laroyo 10’s - AI Awakens • Machine Learning • Neural networks • Solving basic perceptual problems instead of high-expertise ones • Ambiguity tolerant reasoning • Non-taxonomic ordering → non-taxonomic reasoning • folksonomies, clustering, diversity of perspectives, embeddings
  • 22. Web & Media Group http://lora-aroyo.org @laroyo 2011
  • 24. Web & Media Group http://lora-aroyo.org @laroyo Human Annotation Central in Machine Learning Training & Evaluation 10’s – Crowds
  • 25. http://lora-aroyo.org @laroyo Web & Media Group Team BellKor wins Netflix Prize 20071998 2006 2009
  • 26. Web & Media Group http://lora-aroyo.org @laroyo
  • 27. Web & Media Group http://lora-aroyo.org @laroyo the semantic comfort zone
  • 28. Web & Media Group http://lora-aroyo.org @laroyo One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 29. Web & Media Group http://lora-aroyo.org @laroyo Use Case: video archive enrichment Search Behavior of Media Professionals at an Audiovisual Archive: A Transaction Log Analysis (2009). B. Huurnink, L. Hollink, W. van den Heuvel, M. de Rijke.
  • 30. Web & Media Group http://lora-aroyo.org @laroyo Use Case: video archive enrichment Goal: make the multimedia content of Dutch National Video Archive accessible to large audiences Comfort Zone Solution: media professionals watch & annotate videos. Of course!
  • 31. Web & Media Group http://lora-aroyo.org @laroyo but ... Expensive Doesn’t scale time-consuming 5 times the video duration professional vocabulary experts use a specific vocabulary that is unknown to general audiences
  • 32. Web & Media Group http://lora-aroyo.org @laroyo … and people search for fragments experts annotate full videos not finding 35% of search queries result in not found
  • 33. Web & Media Group http://lora-aroyo.org @laroyo Use Case: real world QA for Watson Crowdsourcing ground truth for Question Answering using CrowdTruth (2015). B Timmermans, L Aroyo, C Welty
  • 34. Web & Media Group http://lora-aroyo.org @laroyo Goal: gather questions that real people ask for training & evaluating Watson Data: 30K Questions + Candidate Answers. from Yahoo! Answers Comfort Zone Solution: ask people if the passage answers the question (Y/N). Simple! Use Case: real world QA for Watson
  • 35. Web & Media Group http://lora-aroyo.org @laroyo Contradicting evidence Is Coral a plant? • “Coral almost could be considered half-plant [..]” • “[..] organism, such as a coral, resembling a stony plant.” Unanswerable questions • Can I take a pill if you don't have a child yet? • Is the spelling for being drunk right? • Is napster black? Unclear answer type Is paper animal plant or man made? Multiple right answers to a question What is the best university in NY? (subjective) YES or NO?
  • 36. Web & Media Group http://lora-aroyo.org @laroyo Use Case: medical relation extraction for Watson Crowdsourcing Ground Truth for Medical Relation Extraction (2017). A Dumitrache, L Aroyo, C Welty
  • 37. Web & Media Group http://lora-aroyo.org @laroyo Goal: gather data to train Watson to read medical text & automatically extract a medical relations KB Comfort Zone Solution: having medical experts read & annotate examples Use Case: medical relation extraction for Watson
  • 38. Web & Media Group http://lora-aroyo.org @laroyo ANTIBIOTICS are the first line treatment for indications of TYPHUS. treats(ANTIBIOTICS, TYPHUS)? Expert: yes Patients with TYPHUS who were given ANTIBIOTICS exhibited side-effects. treats(ANTIBIOTICS, TYPHUS)? Expert: yes With ANTIBIOTICS in short supply, DDT was used during WWII to control the insect vectors of TYPHUS. treats(ANTIBIOTICS, TYPHUS)? Expert: yes. Are these three really all the same???
  • 39. Web & Media Group http://lora-aroyo.org @laroyo Use Case: map music to moods
  • 40. Web & Media Group http://lora-aroyo.org @laroyo Use Case: map music to moods Goal: annotate songs with emotional tags Comfort Zone Solution: people assign the prevalent mood of a song
  • 41. Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Other passionate, rollicking, literate, humorous, silly, aggressive, fiery, does not fit into rousing, cheerful, fun, poignant, wistful, campy, quirky, tense, anxious, any of the 5 confident, sweet, amiable, bittersweet, whimsical, witty, intense, volatile, clusters boisterous, good-natured autumnal, wry visceral rowdy brooding Choose one: Which is the mood most appropriate for each song? Goal: (Lee and Hu 2012) 1 song - 1 mood???
  • 42. Web & Media Group http://lora-aroyo.org @laroyo One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 43. Web & Media Group http://lora-aroyo.org @laroyo One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty Semantic Comfort Zone
  • 44. Web & Media Group http://lora-aroyo.org @laroyo One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty Semantic Comfort Zone disrupted
  • 45. Web & Media Group http://lora-aroyo.org @laroyo
  • 46. Web & Media Group http://lora-aroyo.org @laroyo interestingly …
  • 47. Web & Media Group http://lora-aroyo.org @laroyo • collective decisions of large groups of people • a group of error-prone decision-makers can be surprisingly good at picking the best choice • when thumbs up or thumbs down - the chance of picking the right answer needs to be > 50% • the odds that a most of them will pick the right answer is greater than any of them will pick it on their own • performance gets better as size grows 1785 Marquis de Condorcet “wisdom of crowds”
  • 48. Web & Media Group http://lora-aroyo.org @laroyo •asked 787 people to guess the weight of an ox •none got the right answer •their collective guess was almost perfect 1906 Sir Francis Galton “wisdom of crowds”
  • 49. Web & Media Group http://lora-aroyo.org @laroyo WWII Math Rosies 1942: Ballistics calculations and flight trajectories
  • 50. Web & Media Group http://lora-aroyo.org @laroyo NASA’s Computer Room transcribe raw flight data from celluloid film & oscillograph paper
  • 51. Web & Media Group http://lora-aroyo.org @laroyo can we harness it?
  • 52. http://lora-aroyo.org @laroyo Web & Media Group CrowdTruth http://crowdtruth.org/
  • 53. http://lora-aroyo.org @laroyo Web & Media Group CrowdTruth Three basic causes of disagreement: workers, examples, target semantics Disagreement is signal, not noise. It is indicative of the variation in human semantic interpretation It can indicate ambiguity, vagueness, similarity, over-generality, etc, as well as quality Crowdtruth: Machine-human computation framework for harnessing disagreement in gathering annotated data (2014) O Inel, A Dumitrache, l.Aroyo, C. Welty
  • 54. Web & Media Group http://lora-aroyo.org @laroyo one truth: multiple truths all examples are created equal: each example is unique disagreement bad: disagreement is good experts rule: crowd rules one is enough: the more the better detailed explanations help: keep it simple stupid once done, forever valid: maintenance is necessary “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 55. Web & Media Group http://lora-aroyo.org @laroyo changes needed video archive enrichment improve support for fragment search time-based annotations bridging vocabulary gap between searcher & cataloguer
  • 56. Web & Media Group http://lora-aroyo.org @laroyo crowdsourcing video tagging two video tagging pilots
  • 57. Web & Media Group http://lora-aroyo.org @laroyo @waisda http://waisda.nl engage crowds through continuous gaming
  • 58. http://lora-aroyo.org @laroyo Web & Media Group “On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
  • 59. http://lora-aroyo.org @laroyo Web & Media Group time-based bernhard just “tags” “On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
  • 60. http://lora-aroyo.org @laroyo Web & Media Group objects (57%) westminster abbey abbey priester geestelijken hek paarden tocht aankomst koets kroning mensenmassa parade kroon regen “On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
  • 61. http://lora-aroyo.org @laroyo Web & Media Group persons (31%) bernhard juliana objects (57%) “On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
  • 62. http://lora-aroyo.org @laroyo Web & Media Group user vocabulary 8% in professional vocabulary 23% in Dutch lexicon 89% found on Google locations (7%) engeland locations (7%) persons (31%) objects (57%) “On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
  • 63. http://lora-aroyo.org @laroyo Web & Media Group user vocabulary 8% in professional vocabulary 23% in Dutch lexicon 89% found on Google locations (7%) describe mainly short segments often not very specific don’t describe programmes as a whole “On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011 user vocabulary 8% in professional vocabulary 23% in Dutch lexicon 89% found on Google
  • 64. Web & Media Group http://lora-aroyo.org @laroyo crowdsourcing medical relation extraction diversity of opinions independent perspectives multitude of contexts we exposed a richer set of possibilities that help in identifying, processing & understanding context
  • 65. Web & Media Group http://lora-aroyo.org @laroyo Does this sentence express TREATS(Antibiotics, Typhus)? Patients with TYPHUS who were given ANTIBIOTICS exhibited several side-effects. With ANTIBIOTICS in short supply, DDT was used during World War II to control the insect vectors of TYPHUS. ANTIBIOTICS are the first line treatment for indications of TYPHUS. 95% 75% 50% The crowd results captures the natural ambiguity
  • 66. http://lora-aroyo.org @laroyo Web & Media Group What is the relation between the highlighted terms? He was the first physician to identify the relationship between HEMOPHILIA and HEMOPHILIC ARTHROPATHY. Experts Hallucinate Crowd reads text literally - provide better examples to machine experts: cause crowd: no relation
  • 67. http://lora-aroyo.org @laroyo Web & Media Group Unclear relationship between the two arguments reflected in the disagreement Medical Relation Extraction
  • 68. http://lora-aroyo.org @laroyo Web & Media Group Clearly expressed relation between the two arguments reflected in the agreement Medical Relation Extraction
  • 69. http://lora-aroyo.org @laroyo Web & Media Group Unclear relationship between the two arguments reflected in the disagreement Medical Relation Extraction
  • 71. http://lora-aroyo.org @laroyo Web & Media Group Learning Curves (crowd with pos./neg. threshold at 0.5) above 400 sent.: crowd consistently over baseline & single above 600 sent.: crowd out-performs experts
  • 72. http://lora-aroyo.org @laroyo Web & Media Group Learning Curves Extended (crowd with pos./neg. threshold at 0.5) crowd consistently performs better than baseline
  • 73. http://lora-aroyo.org @laroyo Web & Media Group # of Workers: Impact on Sentence-Relation Score
  • 74. Web & Media Group http://lora-aroyo.org @laroyo Training a Relation Extraction Classifier F1 Cost per sentence CrowdTruth 0.642 $0.66 Expert Annotator 0.638 $2.00 Single Annotator 0.492 $0.08 “wisdom of the crowd” provides training data that is at least as good if not better than experts only with proper analytic framework for harnessing disagreement from the crowd
  • 75. http://lora-aroyo.org @laroyo Web & Media Group map music to moods Goal: tag songs with emotional clusters Comfort Zone Solution: people assign the prevalent mood of a song
  • 76. Web & Media Group http://lora-aroyo.org @laroyo
  • 77. Web & Media Group http://lora-aroyo.org @laroyo Is this song …. ?Passionate Rousing Confident Boisterous Rowdy Literate Poignant Wistful Bittersweet Autumnal Brooding Rollicking Cheerful Fun Sweet Amiable Good-natured Humorous Silly Campy Whimsical Witty Wry Aggressive Fiery Tense Anxious Intense Volatile
  • 78. Web & Media Group http://lora-aroyo.org @laroyo If “One Truth” & “No Disagreement” Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 W1 1 W2 1 W3 1 W4 1 W5 1 W6 1 W7 W8 W9 1 W10 1 Totals 1 3 1 2 1
  • 79. Web & Media Group http://lora-aroyo.org @laroyo Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other W1 1 1 1 W2 1 1 1 W3 1 1 1 W4 1 1 W5 1 1 W6 1 1 1 W7 1 1 1 W8 1 1 1 W9 1 1 W10 1 1 1 1 1 Totals 3 5 6 5 2 8 If “Many Truths” & “Disagreement”
  • 80. Web & Media Group http://lora-aroyo.org @laroyo can indicate alternative interpretations Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other W10 1 1 1 1 1 Totals 3 5 6 5 2 8 Disagreement as Signal can indicate ambiguity in the categorisation can indicate low quality workers
  • 83. http://lora-aroyo.org @laroyo Take Home Message People first, experts second True and False is not enough, There is diversity in human interpretation CrowdTruth introduces a spatial representation of meaning that harnesses disagreement With CrowdTruth untrained workers can be just as reliable as highly trained experts