Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Science with Humans in the Loop

1,041 views

Published on

Inaugural speech by Lora Aroyo, Vrije Universiteit Amsterdam
Human-Computer Interaction chair
Video: https://youtu.be/9jlCJULSrhc
Video with slides: https://av-media.vu.nl/VUMedia/Play/5745f2482d3f4fe7a547458393af322a1d

Published in: Technology
  • Be the first to comment

Data Science with Humans in the Loop

  1. 1. http://lora-aroyo.org @laroyo Lora Aroyo DATA SCIENCE WITH HUMANS IN THE LOOP
  2. 2. http://lora-aroyo.org @laroyo Bulgaria NYC The Netherlands 2 ABOUT ME ... Personal Data Science Sofia
  3. 3. http://lora-aroyo.org @laroyo 3 E-LEARNING & AI To understand the value of semantic technologies for e-learning we need to understand the people, specifically how they interact and consume information
  4. 4. ⌂ http://lora-aroyo.org @laroyo MY RESEARCH FAMILY 4 … and many many many many more
  5. 5. ⌂ http://lora-aroyo.org @laroyo MY RESEARCH FAMILY 5 … and many many many many more
  6. 6. http://lora-aroyo.org @laroyo 6 CROWDTRUTH TEAM
  7. 7. http://lora-aroyo.org @laroyo EVOLUTION OF SEMANTIC WEB 7 Great moments from 1980s till now
  8. 8. http://lora-aroyo.org @laroyo EMPIRE OF THE EXPERTS 8 80’s Advances in hardware and SDEs PCs, workstations, Symbolics, Sun New architectures like Hypercube LISP, Prolog, OPS AI can now BUILD SYSTEMS Primary focus on experts and rules What is the knowledge of experts Graphs, logic, rules, frames How do experts reason? Deduction, induction Work on form & process academic inside the system, to make the reasoning inside the system proper and as good as possible Industry forged ahead with ad-hoc & proprietary systems and actually tried to build expert systems Originals of uncertain KR Fuzzy, probabilistic
  9. 9. http://lora-aroyo.org @laroyo EMPIRE OF THE EXPERTS 9 80’s Piero Bonissone and the DELTA/CATS expert system for locomotive repair with David Smith, a locomotive repair expert
  10. 10. http://lora-aroyo.org @laroyo EMPIRE OF THE EXPERTS 10 80’s Buchanan and Shortliff’s MYCIN project at Stanford built a huge rule base for medical diagnosis working with an extensive team of medical experts.
  11. 11. http://lora-aroyo.org @laroyo KNOWLEDGE ACQUISITION FROM EXPERTS 11 90’s Common KADS founded by Bob Wielinga as a methodology for expert knowledge acquisition. It was deeply psychology based - it was about people, about their knowledge and especially about their expertise. How do people know what they know, and how can you acquire that knowledge?
  12. 12. http://lora-aroyo.org @laroyo STRUCTURED KNOWLEDGE ENGINEERING
  13. 13. http://lora-aroyo.org @laroyo INTEROPERABILITY & STANDARDS ODYSSEY 13 00’s
  14. 14. http://lora-aroyo.org @laroyo AI AWAKENS 14 10’s
  15. 15. http://lora-aroyo.org @laroyo 15 2011 IBM WATSON @JEOPARDY
  16. 16. http://lora-aroyo.org @laroyo BIG DATA 16 10’s
  17. 17. http://lora-aroyo.org @laroyo 17 BIG CROWDS 10’s Human Annotation Central in Machine Learning Training & Evaluation
  18. 18. http://lora-aroyo.org @laroyo COMFORT ZONE 18 7 MYTHS ABOUT HUMAN ANNOTATION
  19. 19. http://lora-aroyo.org @laroyo ONE TRUTH 19 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example 7 MYTHS ABOUT HUMAN ANNOTATION “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  20. 20. http://lora-aroyo.org @laroyo 20 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false 7 MYTHS ABOUT HUMAN ANNOTATION ALL EXAMPLES EQUAL “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  21. 21. http://lora-aroyo.org @laroyo 21 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem 7 MYTHS ABOUT HUMAN ANNOTATION DISAGREEMENT BAD “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  22. 22. http://lora-aroyo.org @laroyo 22 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts 7 MYTHS ABOUT HUMAN ANNOTATION EXPERTS RULE “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  23. 23. http://lora-aroyo.org @laroyo 23 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient 7 MYTHS ABOUT HUMAN ANNOTATION ONE IS ENOUGH “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  24. 24. http://lora-aroyo.org @laroyo 24 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old7 MYTHS ABOUT HUMAN ANNOTATION BINARY WORLD “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  25. 25. http://lora-aroyo.org @laroyo 25 Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria DOES THIS SENTENCE EXPRESS TREATS RELATION?
  26. 26. http://lora-aroyo.org @laroyo 26 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria DOES THIS SENTENCE EXPRESS TREATS RELATION?
  27. 27. http://lora-aroyo.org @laroyo 27 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. DOES THIS SENTENCE EXPRESS TREATS RELATION? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
  28. 28. http://lora-aroyo.org @laroyo 28 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DO EXPERTS SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✘
  29. 29. http://lora-aroyo.org @laroyo 29 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES A LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✘
  30. 30. http://lora-aroyo.org @laroyo 30 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES ANOTHER LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✘ ✘
  31. 31. http://lora-aroyo.org @laroyo 31 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES A THIRD LAY ANNOTATOR SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. ✓ ✓ ✓
  32. 32. http://lora-aroyo.org @laroyo 32 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. WHAT DOES THE CROWD SAY? Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Intuition: This is better 95% 75% 50%
  33. 33. http://lora-aroyo.org @laroyo 33 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. 95% 75% 50% There’s a difference between these two This one isn’t utterly wrong BETTER WORSE WHAT DOES THE CROWD SAY?
  34. 34. http://lora-aroyo.org @laroyo 34 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  35. 35. http://lora-aroyo.org @laroyo 35 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  36. 36. http://lora-aroyo.org @laroyo 36 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  37. 37. http://lora-aroyo.org @laroyo 37 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  38. 38. http://lora-aroyo.org @laroyo 38 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old COMFORT ZONE Disrupted “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  39. 39. http://lora-aroyo.org @laroyo 39 For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. ENCOURAGING DISAGREEMENT Rheumatoid arthritis and MALARIA have been treated with CHLOROQUINE for decades. Treats: Chloroquine, Malaria Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Intuition: This is better 95% 75% 50%
  40. 40. http://lora-aroyo.org @laroyo CROWD TASK
  41. 41. http://lora-aroyo.org @laroyo WORKER VECTOR FOR A SENTENCE treats associated _with othersymptom Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
  42. 42. http://lora-aroyo.org @laroyo MANY WORKERS FOR THE SAME SENTENCE treats otherassociated _withsymptom Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood.
  43. 43. http://lora-aroyo.org @laroyo ALL WORKER VECTORS AGGREGATED IN A SENTENCE VECTOR Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. treats othernoneassociated _withsymptommanifestation side effect
  44. 44. http://lora-aroyo.org @laroyo SENTENCE VECTORS FOR THE 3 SENTENCES treats othernoneassociated _withsymptommanifestation side effect treats othernoneassociated _withcontraindicatesmanifestation treats other
  45. 45. http://lora-aroyo.org @laroyo 45 One truth: knowledge acquisition for the semantic web assumes one correct interpretation for every example All examples are created equal: triples are triples, one is not more important than another, they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if examples cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old TIME TO DISRUPT THE COMFORT ZONE “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
  46. 46. http://lora-aroyo.org @laroyo EXCITING DISCOVERIESEXCITING DISCOVERIES
  47. 47. http://lora-aroyo.org @laroyo UMLS RELATION EXTRACTION PROJECT NLP UMLS
  48. 48. http://lora-aroyo.org @laroyo The final frontier VECTOR SPACE
  49. 49. http://lora-aroyo.org @laroyo KNOWLEDGE ACQUISITIONSTRUCTURED KNOWLEDGE ENGINEERING
  50. 50. http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE
  51. 51. http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE 3-axis tensor
  52. 52. http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE … matrix
  53. 53. http://lora-aroyo.org @laroyo 3-AXIS TENSOR Workers axis
  54. 54. http://lora-aroyo.org @laroyo Relations axis 3-AXIS TENSOR
  55. 55. http://lora-aroyo.org @laroyo Sentences axis 3-AXIS TENSOR
  56. 56. http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE Worker votes
  57. 57. http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11 R12 Sentence plane into a sentence vector
  58. 58. http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE Sentence Slice
  59. 59. http://lora-aroyo.org @laroyo HYPER-DIMENSIONAL SPACE 3 Sentence Slices
  60. 60. http://lora-aroyo.org @laroyo DISAGREEMENT IS SIGNAL Variety of sources for disagreement
  61. 61. http://lora-aroyo.org @laroyo Source 1: People’s bias & perspective DISAGREEMENT IS SIGNAL
  62. 62. http://lora-aroyo.org @laroyo DISAGREEMENT IS SIGNAL Source 1: Worker systematically give same answer
  63. 63. http://lora-aroyo.org @laroyo DISAGREEMENT IS SIGNAL Source 1: Worker systematically give same answer
  64. 64. http://lora-aroyo.org @laroyo DISAGREEMENT IS SIGNAL Source 1: Worker systematically give same answer
  65. 65. http://lora-aroyo.org @laroyo Source 2: Target semantics DISAGREEMENT IS SIGNAL
  66. 66. http://lora-aroyo.org @laroyo SentencesSource 3: Sentences DISAGREEMENT IS SIGNAL
  67. 67. http://lora-aroyo.org @laroyo TRIANGLE OF MEANING Model of semantic interpretation (Ogden & Richards, 1936)
  68. 68. http://lora-aroyo.org @laroyo TRIANGLE OF MEANING Model of semantic interpretation
  69. 69. http://lora-aroyo.org @laroyo treats other CrowdTruth metrics for quality assessment TRIANGLE OF MEANING
  70. 70. http://lora-aroyo.org @laroyo treats other Spam QUALITY ASSESSMENT
  71. 71. http://lora-aroyo.org @laroyo treats othernoneassociated _withsymptommanifestation side effect Among 56 subjects reporting to a clinic with symptoms of MALARIA 53 (95%) had ordinarily effective levels of CHLOROQUINE in blood. Sentence ambiguity QUALITY ASSESSMENT
  72. 72. http://lora-aroyo.org @laroyo TREATS RELATION Yes or No? treats othernoneassociated _withcontraindicatesmanifestation treats othernoneassociated _withcontraindicatesmanifestation treats other
  73. 73. http://lora-aroyo.org @laroyo THE WORLD IS SMOOTH AND NOT BINARY
  74. 74. http://lora-aroyo.org @laroyo Agreement as percentage CROWDTRUTH METRICS For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. 25% 25% 75% 12% 12% 50% treats othernoneassociated _withcontraindicatesmanifestation
  75. 75. http://lora-aroyo.org @laroyo For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. CROWDTRUTH METRICS Applying all sides of the triangle treats other
  76. 76. http://lora-aroyo.org @laroyo CROWDTRUTH METRICS For prevention of malaria, use only in individuals traveling to malarious areas where CHLOROQUINE resistant P. falciparum MALARIA has not been reported. treats other Applying all sides of the triangle 99%
  77. 77. http://lora-aroyo.org @laroyo CROWDTRUTH METRICS Applying all sides of the triangle
  78. 78. http://lora-aroyo.org @laroyo CROWDTRUTH KNOWLEDGE TENSOR
  79. 79. http://lora-aroyo.org @laroyo CROWDTRUTH VS. EXPERTS crowd as good or better than from experts
  80. 80. http://lora-aroyo.org @laroyo AMBIGUITY IMPACTS ACCURACY more ambiguous sentences were harder to classify
  81. 81. http://lora-aroyo.org @laroyo CROWDTRUTH METRICS Quality assessment
  82. 82. http://lora-aroyo.org @laroyo CROWDTRUTH.ORG a spatial representation of meaning that harnesses disagreement
  83. 83. http://lora-aroyo.org @laroyo On the role of user-generated metadata in audio visual collections (2011). R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011 VIDEO METADATA ENRICHMENT The Netherlands Institute for Sound and Vision 1
  84. 84. http://lora-aroyo.org @laroyo DIVE+ Explorative Search 2 DIVE into the event-based browsing of linked historical media (2015) V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics:
  85. 85. http://lora-aroyo.org @laroyo DEEP QA IN CULTURAL HERITAGE Mauritshuis use case 3
  86. 86. http://lora-aroyo.org @laroyo CROWDTRUTH IN DEPLOYMENT Google Maps questions Google Maps reviewers 4
  87. 87. http://lora-aroyo.org @laroyo CROWDTRUTH IN DEPLOYMENT Google Maps emotions mTURK crowd 5
  88. 88. http://lora-aroyo.org @laroyo WHAT DOES THE FUTURE HOLD
  89. 89. http://lora-aroyo.org @laroyo USER-CENTRIC DATA SCIENCE Formerly the Web & Media group
  90. 90. http://lora-aroyo.org @laroyo H2020 ReTV Trans-Vector Platform (TVP) Lora Aroyo, (coordinator) VU Amsterdam, Computer Science Lyndon Nixon, MODUL, AT Vasileios Mezaris, CERTH, GR Arno Scharl, Weblyzard, AT Bea Knecht, Zattoo, DE Johan Oomen, Sound and Vision, NL Nicolas Patz, Rundfunk Berlin-brandenburg, DE
  91. 91. http://lora-aroyo.org @laroyo CAPTURING BIAS Startimpuls for the Dutch National Science Agenda Lora Aroyo, (coordinator) VU Amsterdam, Computer Science Alessandro Bozzon, TU Delft CS & Delft Data Science Alec Badenoch, Utrecht University, Media & Culture Studies Antoaneta Dimitrova, Leiden University, Institute of Public Administration Johan Oomen, Netherlands Institute for Sound and Vision
  92. 92. http://lora-aroyo.org @laroyo CROWDTRUTH ROCKS! Disagreement is signal CrowdTruth is a spatial representation of meaning that harnesses disagreement Crowds bring natural diversity CrowdTruth defines hyper-dimensional space to represent ambiguity Crowds help gathering real human semantics
  93. 93. http://lora-aroyo.org @laroyo The world is full of shades of grey Experts and crowds are complimentary Capturing and understanding opinions, perspectives and contexts in the center of understanding people TIME TO BREAK FREE CrowdTruth defines multi-dimensional space to measure quality
  94. 94. http://lora-aroyo.org @laroyo Lora Aroyo DATA SCIENCE WITH HUMANS IN THE LOOP

×