Lora Aroyo, profile picture
Uploaded byLora Aroyo
PDF, PPTX3,714 views

Data Science with Humans in the Loop

The document discusses the evolution of semantic technologies in e-learning and highlights the importance of human annotation in machine learning. It addresses misconceptions in knowledge acquisition, emphasizing the need for diverse perspectives and the role of expert and lay annotators. The document also explores the role of disagreement as a valuable signal in understanding semantic relationships.

Embed presentation

Download as PDF, PPTX
http://lora-aroyo.org @laroyo Lora Aroyo DATA SCIENCE WITH HUMANS IN THE LOOP
http://lora-aroyo.org @laroyo Bulgaria NYC The Netherlands 2 ABOUT ME ... Personal Data Science Sofia
http://lora-aroyo.org @laroyo 3 E-LEARNING & AI To understand the value of semantic technologies for e-learning we need to understand the people, specifically how they interact and consume information
⌂ http://lora-aroyo.org @laroyo MY RESEARCH FAMILY 4 … and many many many many more
⌂ http://lora-aroyo.org @laroyo MY RESEARCH FAMILY 5 … and many many many many more
http://lora-aroyo.org @laroyo 6 CROWDTRUTH TEAM
http://lora-aroyo.org @laroyo EVOLUTION OF SEMANTIC WEB 7 Great moments from 1980s till now
http://lora-aroyo.org @laroyo EMPIRE OF THE EXPERTS 8 80’s Advances in hardware and SDEs PCs, workstations, Symbolics, Sun New architectures like Hypercube LISP, Prolog, OPS AI can now BUILD SYSTEMS Primary focus on experts and rules What is the knowledge of experts Graphs, logic, rules, frames How do experts reason? Deduction, induction Work on form & process academic inside the system, to make the reasoning inside the system proper and as good as possible Industry forged ahead with ad-hoc & proprietary systems and actually tried to build expert systems Originals of uncertain KR Fuzzy, probabilistic
http://lora-aroyo.org @laroyo EMPIRE OF THE EXPERTS 9 80’s Piero Bonissone and the DELTA/CATS expert system for locomotive repair with David Smith, a locomotive repair expert
http://lora-aroyo.org @laroyo EMPIRE OF THE EXPERTS 10 80’s Buchanan and Shortliff’s MYCIN project at Stanford built a huge rule base for medical diagnosis working with an extensive team of medical experts.
http://lora-aroyo.org @laroyo KNOWLEDGE ACQUISITION FROM EXPERTS 11 90’s Common KADS founded by Bob Wielinga as a methodology for expert knowledge acquisition. It was deeply psychology based - it was about people, about their knowledge and especially about their expertise. How do people know what they know, and how can you acquire that knowledge?
http://lora-aroyo.org @laroyo STRUCTURED KNOWLEDGE ENGINEERING
http://lora-aroyo.org @laroyo INTEROPERABILITY & STANDARDS ODYSSEY 13 00’s
http://lora-aroyo.org @laroyo AI AWAKENS 14 10’s
http://lora-aroyo.org @laroyo 15 2011 IBM WATSON @JEOPARDY
http://lora-aroyo.org @laroyo BIG DATA 16 10’s
http://lora-aroyo.org @laroyo 17 BIG CROWDS 10’s Human Annotation Central in Machine Learning Training & Evaluation
http://lora-aroyo.org @laroyo COMFORT ZONE 18 7 MYTHS ABOUT HUMAN ANNOTATION
http://lora-aroyo.org @laroyo ONE TRUTH 19 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example 7 MYTHS ABOUT HUMAN ANNOTATION “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 20 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false 7 MYTHS ABOUT HUMAN ANNOTATION ALL EXAMPLES EQUAL “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 21 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem 7 MYTHS ABOUT HUMAN ANNOTATION DISAGREEMENT BAD “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 22 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts 7 MYTHS ABOUT HUMAN ANNOTATION EXPERTS RULE “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 23 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient 7 MYTHS ABOUT HUMAN ANNOTATION ONE IS ENOUGH “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 24 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old7 MYTHS ABOUT HUMAN ANNOTATION BINARY WORLD “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 25 Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria DOES THIS SENTENCE EXPRESS TREATS RELATION?
http://lora-aroyo.org @laroyo 26 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria DOES THIS SENTENCE EXPRESS TREATS RELATION?
http://lora-aroyo.org @laroyo 27 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. DOES THIS SENTENCE EXPRESS TREATS RELATION? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
http://lora-aroyo.org @laroyo 28 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DO EXPERTS SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✘
http://lora-aroyo.org @laroyo 29 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES A LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✘
http://lora-aroyo.org @laroyo 30 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES ANOTHER LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✘ ✘
http://lora-aroyo.org @laroyo 31 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES A THIRD LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✓
http://lora-aroyo.org @laroyo 32 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES THE CROWD SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Intuition: This is better 95% 75% 50%
http://lora-aroyo.org @laroyo 33 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. 95% 75% 50% There’s a difference between these two This one isn’t utterly wrong BETTER WORSE WHAT DOES THE CROWD SAY?
http://lora-aroyo.org @laroyo 34 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 35 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 36 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 37 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 38 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo 39 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. ENCOURAGING DISAGREEMENT Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Intuition: This is better 95% 75% 50%
http://lora-aroyo.org @laroyo CROWD TASK
http://lora-aroyo.org @laroyo WORKER VECTOR FOR A SENTENCE treats associated _with othersymptom Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
http://lora-aroyo.org @laroyo MANY WORKERS FOR THE SAME SENTENCE treats otherassociated _withsymptom Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
http://lora-aroyo.org @laroyo ALL WORKER VECTORS AGGREGATED IN A SENTENCE VECTOR Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. treats othernoneassociated _withsymptommanifestation side effect
http://lora-aroyo.org @laroyo SENTENCE VECTORS FOR THE 3 SENTENCES treats othernoneassociated _withsymptommanifestation side effect treats othernoneassociated _withcontraindicatesmanifestation treats other
http://lora-aroyo.org @laroyo 45 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old TIME TO DISRUPT THE COMFORT ZONE “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
http://lora-aroyo.org @laroyo EXCITING DISCOVERIESEXCITING DISCOVERIES
http://lora-aroyo.org @laroyo UMLS RELATION EXTRACTION PROJECT NLP UMLS
http://lora-aroyo.org @laroyo The final frontier VECTOR SPACE
http://lora-aroyo.org @laroyo KNOWLEDGE ACQUISITIONSTRUCTURED KNOWLEDGE ENGINEERING
http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE
http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE 3-axis tensor
http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE … matrix
http://lora-aroyo.org @laroyo 3-AXIS TENSOR Workers axis
http://lora-aroyo.org @laroyo Relations axis 3-AXIS TENSOR
http://lora-aroyo.org @laroyo Sentences axis 3-AXIS TENSOR
http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE Worker votes
http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Sentence plane into a sentence vector
http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE Sentence Slice
http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE 3 Sentence Slices
http://lora-aroyo.org @laroyo DISAGREEMENT IS SIGNAL Variety of sources for disagreement
http://lora-aroyo.org @laroyo Source 1: People’s bias & perspective DISAGREEMENT IS SIGNAL
http://lora-aroyo.org @laroyo DISAGREEMENT IS SIGNAL Source 1: Worker systematically give same answer
http://lora-aroyo.org @laroyo DISAGREEMENT IS SIGNAL Source 1: Worker systematically give same answer
http://lora-aroyo.org @laroyo DISAGREEMENT IS SIGNAL Source 1: Worker systematically give same answer
http://lora-aroyo.org @laroyo Source 2: Target semantics DISAGREEMENT IS SIGNAL
http://lora-aroyo.org @laroyo SentencesSource 3: Sentences DISAGREEMENT IS SIGNAL
http://lora-aroyo.org @laroyo TRIANGLE OF MEANING Model of semantic interpretation (Ogden & Richards, 1936)
http://lora-aroyo.org @laroyo TRIANGLE OF MEANING Model of semantic interpretation
http://lora-aroyo.org @laroyo treats other CrowdTruth metrics for quality assessment TRIANGLE OF MEANING
http://lora-aroyo.org @laroyo treats other Spam QUALITY ASSESSMENT
http://lora-aroyo.org @laroyo treats othernoneassociated _withsymptommanifestation side effect Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Sentence ambiguity QUALITY ASSESSMENT
http://lora-aroyo.org @laroyo TREATS RELATION Yes or No? treats othernoneassociated _withcontraindicatesmanifestation treats othernoneassociated _withcontraindicatesmanifestation treats other
http://lora-aroyo.org @laroyo THE WORLD IS SMOOTH AND NOT BINARY
http://lora-aroyo.org @laroyo Agreement as percentage CROWDTRUTH METRICS For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. 25% 25% 75% 12% 12% 50% treats othernoneassociated _withcontraindicatesmanifestation
http://lora-aroyo.org @laroyo For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. CROWDTRUTH METRICS Applying all sides of the triangle treats other
http://lora-aroyo.org @laroyo CROWDTRUTH METRICS For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. treats other Applying all sides of the triangle 99%
http://lora-aroyo.org @laroyo CROWDTRUTH METRICS Applying all sides of the triangle
http://lora-aroyo.org @laroyo CROWDTRUTH KNOWLEDGE TENSOR
http://lora-aroyo.org @laroyo CROWDTRUTH VS. EXPERTS crowd as good or better than from experts
http://lora-aroyo.org @laroyo AMBIGUITY IMPACTS ACCURACY more ambiguous sentences were harder to classify
http://lora-aroyo.org @laroyo CROWDTRUTH METRICS Quality assessment
http://lora-aroyo.org @laroyo CROWDTRUTH.ORG a spatial representation of meaning that harnesses disagreement
http://lora-aroyo.org @laroyo On the role of user-generated metadata in audio visual collections (2011). R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011 VIDEO METADATA ENRICHMENT The Netherlands Institute for Sound and Vision 1
http://lora-aroyo.org @laroyo DIVE+ Explorative Search 2 DIVE into the event-based browsing of linked historical media (2015) V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics:
http://lora-aroyo.org @laroyo DEEP QA IN CULTURAL HERITAGE Mauritshuis use case 3
http://lora-aroyo.org @laroyo CROWDTRUTH IN DEPLOYMENT Google Maps questions Google Maps reviewers 4
http://lora-aroyo.org @laroyo CROWDTRUTH IN DEPLOYMENT Google Maps emotions mTURK crowd 5
http://lora-aroyo.org @laroyo WHAT DOES THE FUTURE HOLD
http://lora-aroyo.org @laroyo USER-CENTRIC DATA SCIENCE Formerly the Web & Media group
http://lora-aroyo.org @laroyo H2020 ReTV Trans-Vector Platform (TVP) Lora Aroyo, (coordinator) VU Amsterdam, Computer Science Lyndon Nixon, MODUL, AT Vasileios Mezaris, CERTH, GR Arno Scharl, Weblyzard, AT Bea Knecht, Zattoo, DE Johan Oomen, Sound and Vision, NL Nicolas Patz, Rundfunk Berlin-brandenburg, DE
http://lora-aroyo.org @laroyo CAPTURING BIAS Startimpuls for the Dutch National Science Agenda Lora Aroyo, (coordinator) VU Amsterdam, Computer Science Alessandro Bozzon, TU Delft CS & Delft Data Science Alec Badenoch, Utrecht University, Media & Culture Studies Antoaneta Dimitrova, Leiden University, Institute of Public Administration Johan Oomen, Netherlands Institute for Sound and Vision
http://lora-aroyo.org @laroyo CROWDTRUTH ROCKS! Disagreement is signal CrowdTruth is a spatial representation of meaning that harnesses disagreement Crowds bring natural diversity CrowdTruth defines hyper-dimensional space to represent ambiguity Crowds help gathering real human semantics
http://lora-aroyo.org @laroyo The world is full of shades of grey Experts and crowds are complimentary Capturing and understanding opinions, perspectives and contexts in the center of understanding people TIME TO BREAK FREE CrowdTruth defines multi-dimensional space to measure quality
http://lora-aroyo.org @laroyo Lora Aroyo DATA SCIENCE WITH HUMANS IN THE LOOP

More Related Content

PDF
Data Science with Human in the Loop @Faculty of Science #Leiden University
byLora Aroyo
 
PDF
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
byLora Aroyo
 
PDF
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
byLora Aroyo
 
PDF
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
byLora Aroyo
 
PPT
Weaponized Web Archives: Provenance Laundering of Short Order Evidence
byMichael Nelson
 
PPTX
If I Googled You, What Would I Find? Managing your digital footprint - Nicola...
byEDINA, University of Edinburgh
 
PDF
@WebSciDL PhD Student Project Reviews August 5&6, 2015
byMichael Nelson
 
PPT
Weaponized Web Archives: Provenance Laundering of Short Order Evidence
byMichael Nelson
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
byLora Aroyo
 
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
byLora Aroyo
 
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
byLora Aroyo
 
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
byLora Aroyo
 
Weaponized Web Archives: Provenance Laundering of Short Order Evidence
byMichael Nelson
 
If I Googled You, What Would I Find? Managing your digital footprint - Nicola...
byEDINA, University of Edinburgh
 
@WebSciDL PhD Student Project Reviews August 5&6, 2015
byMichael Nelson
 
Weaponized Web Archives: Provenance Laundering of Short Order Evidence
byMichael Nelson
 

What's hot

PPTX
HKU Data Curation MLIM7350 Class 10
byScott Edmunds
 
PDF
Machine Intelligence - Wie Systeme lernen und unseren Alltag verändern
byMark Cieliebak
 
PPT
What's New and Exciting in Scholarly Publishing
byTeeTee
 
PPT
The Vortex: Experiments in Online Collaboration
byNancy Wright White
 
PPT
The Future of Libraries
byPeter Bromberg
 
PPT
Keeping up: strategic use of online social networks for librarian current awa...
bysuelibrarian
 
PPT
Online information 2010_track_two_final_corrected
byBasset Hervé
 
HKU Data Curation MLIM7350 Class 10
byScott Edmunds
 
Machine Intelligence - Wie Systeme lernen und unseren Alltag verändern
byMark Cieliebak
 
What's New and Exciting in Scholarly Publishing
byTeeTee
 
The Vortex: Experiments in Online Collaboration
byNancy Wright White
 
The Future of Libraries
byPeter Bromberg
 
Keeping up: strategic use of online social networks for librarian current awa...
bysuelibrarian
 
Online information 2010_track_two_final_corrected
byBasset Hervé
 

Viewers also liked

PDF
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
byBayes Nets meetup London
 
PDF
Crowdsourcing for Information Retrieval: From Statistics to Ethics
byMatthew Lease
 
PDF
Crowdsourcing for NLP Ground Truth Data
byLora Aroyo
 
PPTX
2016 03-02 EyesOnALZ All-hands meeting_mohammad haft_p1
byEyesOnALZ
 
PPTX
EUDAT Webinar "Organise, retrieve and aggregate data using annotations with B...
byEUDAT
 
PDF
Translating research data into Gene Ontology annotations
byPascale Gaudet
 
PPTX
From Sequence to Knowledge (Phage Genomics Workshop Intro at the 22nd Biennia...
byRamy K. Aziz
 
PDF
Editing Functionality - Apollo Workshop
byMonica Munoz-Torres
 
PPTX
3-step Parallel Corpus Cleaning using Monolingual Crowd Workers
byToshiaki Nakazawa
 
PPTX
10 facts about jobs in the future
byPew Research Center's Internet & American Life Project
 
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
byBayes Nets meetup London
 
Crowdsourcing for Information Retrieval: From Statistics to Ethics
byMatthew Lease
 
Crowdsourcing for NLP Ground Truth Data
byLora Aroyo
 
2016 03-02 EyesOnALZ All-hands meeting_mohammad haft_p1
byEyesOnALZ
 
EUDAT Webinar "Organise, retrieve and aggregate data using annotations with B...
byEUDAT
 
Translating research data into Gene Ontology annotations
byPascale Gaudet
 
From Sequence to Knowledge (Phage Genomics Workshop Intro at the 22nd Biennia...
byRamy K. Aziz
 
Editing Functionality - Apollo Workshop
byMonica Munoz-Torres
 
3-step Parallel Corpus Cleaning using Monolingual Crowd Workers
byToshiaki Nakazawa
 
10 facts about jobs in the future
byPew Research Center's Internet & American Life Project
 

More from Lora Aroyo

PDF
Multilingual, Multi-cultural Multi-modal Benchmark - AILuminate Benchmark Suite
byLora Aroyo
 
PDF
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
byLora Aroyo
 
PDF
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
byLora Aroyo
 
PDF
Harnessing Human Semantics at Scale (updated)
byLora Aroyo
 
PDF
Data excellence: Better data for better AI
byLora Aroyo
 
PDF
CHIP Demonstrator presentation @ CATCH Symposium
byLora Aroyo
 
PDF
Semantic Web Challenge: CHIP Demonstrator
byLora Aroyo
 
PDF
The Rijksmuseum Collection as Linked Data
byLora Aroyo
 
PDF
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
byLora Aroyo
 
PDF
FAIRview: Responsible Video Summarization @NYCML'18
byLora Aroyo
 
PDF
Understanding bias in video news & news filtering algorithms
byLora Aroyo
 
PDF
StorySourcing: Telling Stories with Humans & Machines
byLora Aroyo
 
PPTX
Digital Humanities Benelux 2017: Keynote Lora Aroyo
byLora Aroyo
 
PDF
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
byLora Aroyo
 
PDF
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
byLora Aroyo
 
PDF
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
byLora Aroyo
 
PDF
"Video Killed the Radio Star": From MTV to Snapchat
byLora Aroyo
 
PPTX
UMAP 2016 Opening Ceremony
byLora Aroyo
 
PDF
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
byLora Aroyo
 
PDF
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
byLora Aroyo
 
Multilingual, Multi-cultural Multi-modal Benchmark - AILuminate Benchmark Suite
byLora Aroyo
 
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
byLora Aroyo
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
byLora Aroyo
 
Harnessing Human Semantics at Scale (updated)
byLora Aroyo
 
Data excellence: Better data for better AI
byLora Aroyo
 
CHIP Demonstrator presentation @ CATCH Symposium
byLora Aroyo
 
Semantic Web Challenge: CHIP Demonstrator
byLora Aroyo
 
The Rijksmuseum Collection as Linked Data
byLora Aroyo
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
byLora Aroyo
 
FAIRview: Responsible Video Summarization @NYCML'18
byLora Aroyo
 
Understanding bias in video news & news filtering algorithms
byLora Aroyo
 
StorySourcing: Telling Stories with Humans & Machines
byLora Aroyo
 
Digital Humanities Benelux 2017: Keynote Lora Aroyo
byLora Aroyo
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
byLora Aroyo
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
byLora Aroyo
 
Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age
byLora Aroyo
 
"Video Killed the Radio Star": From MTV to Snapchat
byLora Aroyo
 
UMAP 2016 Opening Ceremony
byLora Aroyo
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
byLora Aroyo
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
byLora Aroyo
 

Recently uploaded

PDF
TravelTech Paris 2025 | The EU Digital Identity Wallet and it’s impact on Tra...
byapidays
 
PDF
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
PDF
Safer’s Picks: The 5 FME Transformers You Didn’t Know You Needed
bySafe Software
 
PDF
How does MES(Manufacturing Execution System) work?
byakipii ogaoga
 
PPTX
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
PPTX
Computer architecture, Computer architecture ,Computer architecture
bywaheedqazi123
 
PDF
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
PDF
Poročilo odbora CIS (CH08873) za leto 2025 na letni skupščini IEEE Slovenija ...
byUniversity of Maribor
 
PDF
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
 
PDF
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
PDF
The new book “Marketing in the Metaverse” spotlights Scott M. Graffius’ research
byScott M. Graffius
 
PDF
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
PDF
How to Prepare for the RWA Tokenization Trend
byrose mason
 
PPTX
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
PDF
Transcript: Escape from the Forbidden Zone: Smuggling green and inclusive tec...
byBookNet Canada
 
PPTX
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
PPTX
PPTX game guess the logo with a twistppt
byAdrianAnig
 
PDF
Security-Fails-in-the-Gaps-Between-Systems.pptx.pdf
byOhhproJunction
 
PDF
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
PDF
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 
TravelTech Paris 2025 | The EU Digital Identity Wallet and it’s impact on Tra...
byapidays
 
final~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.pdf
byomarbishtawi04
 
Safer’s Picks: The 5 FME Transformers You Didn’t Know You Needed
bySafe Software
 
How does MES(Manufacturing Execution System) work?
byakipii ogaoga
 
Introducing VisualSim 2610 The Next Leap in System Level Modeling
byDeepak Shankar
 
Computer architecture, Computer architecture ,Computer architecture
bywaheedqazi123
 
The Emergence of Empathic XR: AWE Asia 2026 keynote
byMark Billinghurst
 
Poročilo odbora CIS (CH08873) za leto 2025 na letni skupščini IEEE Slovenija ...
byUniversity of Maribor
 
Traditional-Security-Models-No-Longer-Work.pptx (1).pdf
byOhhproJunction
 
NTG -Company Profile_Software_Vendor_Company_in_MENA
byMustafa Kuğu
 
The new book “Marketing in the Metaverse” spotlights Scott M. Graffius’ research
byScott M. Graffius
 
Writing GPU-Ready AI Models in Pure Java with Babylon
byAna-Maria Mihalceanu
 
How to Prepare for the RWA Tokenization Trend
byrose mason
 
Automation Without Apprentices: How AI Challenges the Open Source Way
byIcinga
 
Transcript: Escape from the Forbidden Zone: Smuggling green and inclusive tec...
byBookNet Canada
 
Beyond the Algorithm: Designing Human-Centric Public Service with AI
byAlan Dix
 
PPTX game guess the logo with a twistppt
byAdrianAnig
 
Security-Fails-in-the-Gaps-Between-Systems.pptx.pdf
byOhhproJunction
 
Automated Governance for FME Flow: Smarter Admin at Scale
bySafe Software
 
Why Many Smart Device Platforms Fail to Scale?
byPromeraki Developments
 

Data Science with Humans in the Loop

  • 1.
    http://lora-aroyo.org @laroyo Lora Aroyo DATASCIENCE WITH HUMANS IN THE LOOP
  • 2.
    http://lora-aroyo.org @laroyo Bulgaria NYC TheNetherlands 2 ABOUT ME ... Personal Data Science Sofia
  • 3.
    http://lora-aroyo.org @laroyo 3 E-LEARNING& AI To understand the value of semantic technologies for e-learning we need to understand the people, specifically how they interact and consume information
  • 4.
  • 5.
  • 6.
  • 7.
    http://lora-aroyo.org @laroyo EVOLUTION OFSEMANTIC WEB 7 Great moments from 1980s till now
  • 8.
    http://lora-aroyo.org @laroyo EMPIRE OFTHE EXPERTS 8 80’s Advances in hardware and SDEs PCs, workstations, Symbolics, Sun New architectures like Hypercube LISP, Prolog, OPS AI can now BUILD SYSTEMS Primary focus on experts and rules What is the knowledge of experts Graphs, logic, rules, frames How do experts reason? Deduction, induction Work on form & process academic inside the system, to make the reasoning inside the system proper and as good as possible Industry forged ahead with ad-hoc & proprietary systems and actually tried to build expert systems Originals of uncertain KR Fuzzy, probabilistic
  • 9.
    http://lora-aroyo.org @laroyo EMPIRE OFTHE EXPERTS 9 80’s Piero Bonissone and the DELTA/CATS expert system for locomotive repair with David Smith, a locomotive repair expert
  • 10.
    http://lora-aroyo.org @laroyo EMPIRE OFTHE EXPERTS 10 80’s Buchanan and Shortliff’s MYCIN project at Stanford built a huge rule base for medical diagnosis working with an extensive team of medical experts.
  • 11.
    http://lora-aroyo.org @laroyo KNOWLEDGE ACQUISITION FROM EXPERTS 11 90’s CommonKADS founded by Bob Wielinga as a methodology for expert knowledge acquisition. It was deeply psychology based - it was about people, about their knowledge and especially about their expertise. How do people know what they know, and how can you acquire that knowledge?
  • 12.
  • 13.
  • 14.
  • 15.
  • 16.
  • 17.
    http://lora-aroyo.org @laroyo 17 BIGCROWDS 10’s Human Annotation Central in Machine Learning Training & Evaluation
  • 18.
  • 19.
    http://lora-aroyo.org @laroyo ONE TRUTH 19 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example 7 MYTHS ABOUT HUMAN ANNOTATION “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 20.
    http://lora-aroyo.org @laroyo 20 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false 7 MYTHS ABOUT HUMAN ANNOTATION ALL EXAMPLES EQUAL “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 21.
    http://lora-aroyo.org @laroyo 21 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem 7 MYTHS ABOUT HUMAN ANNOTATION DISAGREEMENT BAD “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 22.
    http://lora-aroyo.org @laroyo 22 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts 7 MYTHS ABOUT HUMAN ANNOTATION EXPERTS RULE “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 23.
    http://lora-aroyo.org @laroyo 23 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient 7 MYTHS ABOUT HUMAN ANNOTATION ONE IS ENOUGH “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 24.
    http://lora-aroyo.org @laroyo 24 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old7 MYTHS ABOUT HUMAN ANNOTATION BINARY WORLD “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 25.
    http://lora-aroyo.org @laroyo 25 Rheumatoidarthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria DOES THIS SENTENCE EXPRESS TREATS RELATION?
  • 26.
    http://lora-aroyo.org @laroyo 26 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria DOES THIS SENTENCE EXPRESS TREATS RELATION?
  • 27.
    http://lora-aroyo.org @laroyo 27 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. DOES THIS SENTENCE EXPRESS TREATS RELATION? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
  • 28.
    http://lora-aroyo.org @laroyo 28 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DO EXPERTS SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✘
  • 29.
    http://lora-aroyo.org @laroyo 29 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES A LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✘
  • 30.
    http://lora-aroyo.org @laroyo 30 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES ANOTHER LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✘ ✘
  • 31.
    http://lora-aroyo.org @laroyo 31 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES A THIRD LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✓
  • 32.
    http://lora-aroyo.org @laroyo 32 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES THE CROWD SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Intuition: This is better 95% 75% 50%
  • 33.
    http://lora-aroyo.org @laroyo 33 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. 95% 75% 50% There’s a difference between these two This one isn’t utterly wrong BETTER WORSE WHAT DOES THE CROWD SAY?
  • 34.
    http://lora-aroyo.org @laroyo 34 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 35.
    http://lora-aroyo.org @laroyo 35 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 36.
    http://lora-aroyo.org @laroyo 36 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 37.
    http://lora-aroyo.org @laroyo 37 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 38.
    http://lora-aroyo.org @laroyo 38 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 39.
    http://lora-aroyo.org @laroyo 39 Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. ENCOURAGING DISAGREEMENT Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Intuition: This is better 95% 75% 50%
  • 40.
  • 41.
    http://lora-aroyo.org @laroyo WORKER VECTORFOR A SENTENCE treats associated _with othersymptom Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
  • 42.
    http://lora-aroyo.org @laroyo MANY WORKERSFOR THE SAME SENTENCE treats otherassociated _withsymptom Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
  • 43.
    http://lora-aroyo.org @laroyo ALL WORKERVECTORS AGGREGATED IN A SENTENCE VECTOR Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. treats othernoneassociated _withsymptommanifestation side effect
  • 44.
    http://lora-aroyo.org @laroyo SENTENCE VECTORSFOR THE 3 SENTENCES treats othernoneassociated _withsymptommanifestation side effect treats othernoneassociated _withcontraindicatesmanifestation treats other
  • 45.
    http://lora-aroyo.org @laroyo 45 Onetruth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old TIME TO DISRUPT THE COMFORT ZONE “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  • 46.
  • 47.
  • 48.
  • 49.
  • 50.
  • 51.
  • 52.
  • 53.
  • 54.
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60.
    http://lora-aroyo.org @laroyo DISAGREEMENT ISSIGNAL Variety of sources for disagreement
  • 61.
    http://lora-aroyo.org @laroyo Source 1:People’s bias & perspective DISAGREEMENT IS SIGNAL
  • 62.
    http://lora-aroyo.org @laroyo DISAGREEMENT ISSIGNAL Source 1: Worker systematically give same answer
  • 63.
    http://lora-aroyo.org @laroyo DISAGREEMENT ISSIGNAL Source 1: Worker systematically give same answer
  • 64.
    http://lora-aroyo.org @laroyo DISAGREEMENT ISSIGNAL Source 1: Worker systematically give same answer
  • 65.
    http://lora-aroyo.org @laroyo Source 2:Target semantics DISAGREEMENT IS SIGNAL
  • 66.
    http://lora-aroyo.org @laroyo SentencesSource 3:Sentences DISAGREEMENT IS SIGNAL
  • 67.
    http://lora-aroyo.org @laroyo TRIANGLE OFMEANING Model of semantic interpretation (Ogden & Richards, 1936)
  • 68.
    http://lora-aroyo.org @laroyo TRIANGLE OFMEANING Model of semantic interpretation
  • 69.
    http://lora-aroyo.org @laroyo treats other CrowdTruthmetrics for quality assessment TRIANGLE OF MEANING
  • 70.
  • 71.
    http://lora-aroyo.org @laroyo treats othernoneassociated_withsymptommanifestation side effect Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Sentence ambiguity QUALITY ASSESSMENT
  • 72.
    http://lora-aroyo.org @laroyo TREATS RELATION Yesor No? treats othernoneassociated _withcontraindicatesmanifestation treats othernoneassociated _withcontraindicatesmanifestation treats other
  • 73.
  • 74.
    http://lora-aroyo.org @laroyo Agreement aspercentage CROWDTRUTH METRICS For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. 25% 25% 75% 12% 12% 50% treats othernoneassociated _withcontraindicatesmanifestation
  • 75.
    http://lora-aroyo.org @laroyo For preventionof malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. CROWDTRUTH METRICS Applying all sides of the triangle treats other
  • 76.
    http://lora-aroyo.org @laroyo CROWDTRUTH METRICS Forprevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. treats other Applying all sides of the triangle 99%
  • 77.
  • 78.
  • 79.
    http://lora-aroyo.org @laroyo CROWDTRUTH VS.EXPERTS crowd as good or better than from experts
  • 80.
    http://lora-aroyo.org @laroyo AMBIGUITY IMPACTSACCURACY more ambiguous sentences were harder to classify
  • 81.
  • 82.
    http://lora-aroyo.org @laroyo CROWDTRUTH.ORG a spatialrepresentation of meaning that harnesses disagreement
  • 83.
    http://lora-aroyo.org @laroyo On therole of user-generated metadata in audio visual collections (2011). R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011 VIDEO METADATA ENRICHMENT The Netherlands Institute for Sound and Vision 1
  • 84.
    http://lora-aroyo.org @laroyo DIVE+ Explorative Search 2 DIVEinto the event-based browsing of linked historical media (2015) V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics:
  • 85.
    http://lora-aroyo.org @laroyo DEEP QAIN CULTURAL HERITAGE Mauritshuis use case 3
  • 86.
    http://lora-aroyo.org @laroyo CROWDTRUTH INDEPLOYMENT Google Maps questions Google Maps reviewers 4
  • 87.
    http://lora-aroyo.org @laroyo CROWDTRUTH INDEPLOYMENT Google Maps emotions mTURK crowd 5
  • 88.
  • 89.
    http://lora-aroyo.org @laroyo USER-CENTRIC DATASCIENCE Formerly the Web & Media group
  • 90.
    http://lora-aroyo.org @laroyo H2020 ReTV Trans-VectorPlatform (TVP) Lora Aroyo, (coordinator) VU Amsterdam, Computer Science Lyndon Nixon, MODUL, AT Vasileios Mezaris, CERTH, GR Arno Scharl, Weblyzard, AT Bea Knecht, Zattoo, DE Johan Oomen, Sound and Vision, NL Nicolas Patz, Rundfunk Berlin-brandenburg, DE
  • 91.
    http://lora-aroyo.org @laroyo CAPTURING BIAS Startimpulsfor the Dutch National Science Agenda Lora Aroyo, (coordinator) VU Amsterdam, Computer Science Alessandro Bozzon, TU Delft CS & Delft Data Science Alec Badenoch, Utrecht University, Media & Culture Studies Antoaneta Dimitrova, Leiden University, Institute of Public Administration Johan Oomen, Netherlands Institute for Sound and Vision
  • 92.
    http://lora-aroyo.org @laroyo CROWDTRUTH ROCKS! Disagreementis signal CrowdTruth is a spatial representation of meaning that harnesses disagreement Crowds bring natural diversity CrowdTruth defines hyper-dimensional space to represent ambiguity Crowds help gathering real human semantics
  • 93.
    http://lora-aroyo.org @laroyo The worldis full of shades of grey Experts and crowds are complimentary Capturing and understanding opinions, perspectives and contexts in the center of understanding people TIME TO BREAK FREE CrowdTruth defines multi-dimensional space to measure quality
  • 94.
    http://lora-aroyo.org @laroyo Lora Aroyo DATASCIENCE WITH HUMANS IN THE LOOP