Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

StorySourcing: Telling Stories with Humans & Machines

203 views

Published on

Keynote at Narrative Matters 2018
Lora Aroyo

Published in: Technology
  • Be the first to comment

StorySourcing: Telling Stories with Humans & Machines

  1. 1. http://lora-aroyo.org @laroyo Lora Aroyo StorySourcing: TELLING STORIES WITH HUMANS & MACHINES User Centric Data Science Group
  2. 2. http://lora-aroyo.org @laroyo Information Heritage Organizations as Inventories of the World André Malraux, The Imaginary Museum of World Sculpture, 1953
  3. 3. http://lora-aroyo.org @laroyo Interpretation Heritage Organizations as a Place to Engage with the World André Malraux, The Imaginary Museum of World Sculpture, 1953
  4. 4. http://lora-aroyo.org @laroyo CULTURAL HERITAGE 4 Before the Digital Age Lots of manual effort Focus on internal collection management Focus on art historical significance Access targeted to researchers & professionals Small curated selection online for general audiences onsite
  5. 5. http://lora-aroyo.org @laroyo DIGITAL HERITAGE 5 Bringing collections online Focus on massive digitization of heritage collections Getting large collections online Still need significant art historical understanding to get access Metadata not sufficient for the online presence
  6. 6. http://lora-aroyo.org @laroyo Knowledge Representation, Taxonomies, Thesauri METADATA ENRICHMENT Shared structured knowledge
  7. 7. http://lora-aroyo.org @laroyo Linked Data, Semantic Web, Interoperability, Standards METADATA ENRICHMENT Shift from metadata for internal use to metadata for online access
  8. 8. http://lora-aroyo.org @laroyo Linked Data, Semantic Web, Interoperability, Standards METADATA ENRICHMENT Building community for shared knowledge creation, use & maintenance http://www.getty.edu/research/tools/vocabularies/lod/index.html
  9. 9. http://lora-aroyo.org @laroyo Rijksmuseum Using Linked Data to Diversify Search Results a Case Study in Cultural Heritage Chris Dijkshoorn, Lora Aroyo, Guus Schreiber, Jan Wielemaker, and Lizzy Jongma
  10. 10. http://lora-aroyo.org @laroyo 2005 - 2007 http://multimedian.project.cwi.nl/
  11. 11. http://lora-aroyo.org @laroyo 2005 - 2007 http://multimedian.project.cwi.nl/
  12. 12. http://lora-aroyo.org @laroyo 2005 - 2007 http://multimedian.project.cwi.nl/
  13. 13. http://lora-aroyo.org @laroyo 2005 - 2007 http://multimedian.project.cwi.nl/
  14. 14. http://lora-aroyo.org @laroyo http://multimedian.project.cwi.nl/ 2005 - 2007
  15. 15. http://lora-aroyo.org @laroyo 2005 - 2007 http://multimedian.project.cwi.nl/
  16. 16. http://lora-aroyo.org @laroyo 2005 - 2007 http://multimedian.project.cwi.nl/
  17. 17. http://lora-aroyo.org @laroyo 2005 - 2007 http://multimedian.project.cwi.nl/
  18. 18. http://lora-aroyo.org @laroyo BIG DATA Shift from single institutions to connected heritage https://www.europeana.eu/portal/en
  19. 19. http://lora-aroyo.org @laroyo Europeana.eu sharing cultural heritage for enjoyment, education and research In 2008 launched with 4.5 mil digitised items & 1,000 contributing organisations In 2018 it collaborates with thousands of European archives, libraries & museums > 50 mil digitised items: ● Books ● Music ● Artworks Thematic collections on: ● Art ● Fashion ● Music ● Photography ● World War I https://www.europeana.eu/portal/en
  20. 20. http://lora-aroyo.org @laroyo ADDRESSED THE WEB ACCESS & SCALE ISSUES ... through using automated methods to enrich & curate metadata André Malraux, The Imaginary Museum of World Sculpture, 1953
  21. 21. http://lora-aroyo.org @laroyo André Malraux, The Imaginary Museum of World Sculpture, 1953 BUT THAT WASN’T ENOUGH FOR TRUE ENGAGEMENT Still there is much more focus on information support rather than interpretation support for online collections
  22. 22. http://lora-aroyo.org @laroyo Gravity (2013) LOST IN CULTURAL SPACE MORE THAN EVER The sense of disconnect was now bigger as there has never been so much online information and so difficult to find ...
  23. 23. http://lora-aroyo.org @laroyo 23 … BECAUSE THERE WAS NO CONTEXT Entities were not sufficient to endure engagement with online collections
  24. 24. http://lora-aroyo.org @laroyo 24 “THE GALLERY OF CORNELIS VAN DER GEEST” Willem van Haecht, 1628
  25. 25. http://lora-aroyo.org @laroyo 25 WHAT HAPPENS IN THIS PAINTING? Hunting, dogs, outdoors
  26. 26. http://lora-aroyo.org @laroyo 26 WHAT HAPPENS IN THIS PAINTING? Religious, Madonna, Madonna and Child, Quentin Metsys
  27. 27. http://lora-aroyo.org @laroyo 27 WHAT HAPPENS IN THIS PAINTING? Archduke Albert and Archduchess Isabella, Cornelis van der Geest
  28. 28. http://lora-aroyo.org @laroyo 28 WHAT HAPPENS IN THIS PAINTING? Battle scenes, warriors, soldiers
  29. 29. http://lora-aroyo.org @laroyo 29 SO MANY STORIES THAT CAN BE TOLD ...
  30. 30. http://lora-aroyo.org @laroyo 30 SO MANY INTERPRETATIONS ...
  31. 31. http://lora-aroyo.org @laroyo 31 theory of interpretation of information bringing people and technology together to: ● model information ● offer engaging interaction ● support interpretation DIGITAL HERMENEUTICS Chiel van den Akker, Susan Legêne, Marieke van Erp, Lora Aroyo, Roxane Segers, Lourens van der Meij, Jacco van Ossenbruggen, Guus Schreiber, Bob Wielinga, Johan Oomen, and Geertje Jacobs (2011). Digital hermeneutics: Agora and the online understanding of cultural heritage. In Proceedings of the 3rd International Web Science Conference (WebSci '11). ACM, New York, NY, USA
  32. 32. http://lora-aroyo.org @laroyo LINKING OBJECTS THROUGH EVENTS & ENTITIES Erp, M. van; Oomen, J.; Segers, R.; Akker, C. van de; Aroyo, L.; Jacobs, G.; Legêne, S; Meij, L. van der; O ssenbruggen, J.R. van; Schreiber, G. Automatic Heritage Metadata Enrichment with Historic Events Museums and the Web 2011 http://diveproject.beeldengeluid.nl/
  33. 33. http://lora-aroyo.org @laroyo Erp, M. van; Oomen, J.; Segers, R.; Akker, C. van de; Aroyo, L.; Jacobs, G.; Legêne, S; Meij, L. van der; O ssenbruggen, J.R. van; Schreiber, G. Automatic Heritage Metadata Enrichment with Historic Events Museums and the Web 2011 http://diveproject.beeldengeluid.nl/ ENGAGING USERS THROUGH EVENT NARRATIVES
  34. 34. http://lora-aroyo.org @laroyo AGORA PROJECT Modeling Historical Events Segers, R., Erp, M.V., Meij, L.V., Aroyo, L., Schreiber, G., Wielinga, B.F., Ossenbruggen, J.V., Oomen, J., & Jacobs, G. (2011). Hacking History : Automatic Historical Event Extraction for Enriching Cultural Heritage Multimedia Collections. In Proc. of the 6th International Conference on Knowledge Capture (K-CAP’11)
  35. 35. http://lora-aroyo.org @laroyo AGORA PROJECT Modeling Historical Events Segers, R., Erp, M.V., Meij, L.V., Aroyo, L., Schreiber, G., Wielinga, B.F., Ossenbruggen, J.V., Oomen, J., & Jacobs, G. (2011). Hacking History : Automatic Historical Event Extraction for Enriching Cultural Heritage Multimedia Collections. In Proc. of the 6th International Conference on Knowledge Capture (K-CAP’11)
  36. 36. http://lora-aroyo.org @laroyo AGORA PROJECT Event Properties & Relations Segers, R., Erp, M.V., Meij, L.V., Aroyo, L., Schreiber, G., Wielinga, B.F., Ossenbruggen, J.V., Oomen, J., & Jacobs, G. (2011). Hacking History : Automatic Historical Event Extraction for Enriching Cultural Heritage Multimedia Collections. In Proc. of the 6th International Conference on Knowledge Capture (K-CAP’11)
  37. 37. http://lora-aroyo.org @laroyo AGORA PROJECT Proto-narratives with Events Segers, R., Erp, M.V., Meij, L.V., Aroyo, L., Schreiber, G., Wielinga, B.F., Ossenbruggen, J.V., Oomen, J., & Jacobs, G. (2011). Hacking History : Automatic Historical Event Extraction for Enriching Cultural Heritage Multimedia Collections. In Proc. of the 6th International Conference on Knowledge Capture (K-CAP’11)
  38. 38. http://lora-aroyo.org @laroyo DIVE+ Event-centric Explorative Search DIVE into the event-based browsing of linked historical media (2015) V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics http://diveproject.beeldengeluid.nl/
  39. 39. http://lora-aroyo.org @laroyo DIVE+ Explorative Search DIVE into the event-based browsing of linked historical media (2015) V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics: http://diveproject.beeldengeluid.nl/
  40. 40. http://lora-aroyo.org @laroyo DIVE+ Filters for Events DIVE into the event-based browsing of linked historical media (2015) V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics: http://diveproject.beeldengeluid.nl/ filter on events
  41. 41. http://lora-aroyo.org @laroyo DIVE+ Building Exploration Narratives DIVE into the event-based browsing of linked historical media (2015) V De Boer, J Oomen, O Inel, L Aroyo, E Van Staveren, in Journal of Web Semantics: http://diveproject.beeldengeluid.nl/ narrative
  42. 42. http://lora-aroyo.org @laroyo DIVE+ MEDIA SUITE Explorative Search for Media Collections de Boer V., Melgar L., Inel O., Ortiz C.M., Aroyo L., Oomen J. (2017) Enriching Media Collections for Event-Based Exploration. In Proceedings of Metadata and Semantic Research (MTSR 2017), Communications in Computer and Information Science, vol 755. Springer http://mediasuite.clariah.nl/
  43. 43. http://lora-aroyo.org @laroyo DIVE+ MEDIA SUITE Explorative Search for Media Collections de Boer V., Melgar L., Inel O., Ortiz C.M., Aroyo L., Oomen J. (2017) Enriching Media Collections for Event-Based Exploration. In Proceedings of Metadata and Semantic Research (MTSR 2017), Communications in Computer and Information Science, vol 755. Springer http://mediasuite.clariah.nl/
  44. 44. http://lora-aroyo.org @laroyo Narratives in animated GIFs Remixing Archival Stories with millenials Inel O., Sauer, S., Aroyo L. (2018) A Study of Narrative Creation by Means of Crowds and Niches http://diveproject.beeldengeluid.nl/
  45. 45. http://lora-aroyo.org @laroyo CrowDDriven Engaging Audiences with Tagging & Curating Diego Rens, Marco Schreurs, Egemen Uzunali and Youssef Azriouil. (Master Thesis) Supervised by Lora Aroyo http://diveproject.beeldengeluid.nl/
  46. 46. http://lora-aroyo.org @laroyo Tagasauris, Inc. DIVE Event-based Browser for TV Media Exploration http://tagasauris.com
  47. 47. http://lora-aroyo.org @laroyo M.C. Escher, Day and Night , 1938 BUT THIS ONLY WORKS IF THERE ARE EVENTS ... Event vocabularies are difficult: too many, not structured, not shared, not standardized, lots of variations, perspectives, no agreement across communities
  48. 48. http://lora-aroyo.org @laroyo CROWDTRUTH.ORG a spatial representation of meaning that harnesses disagreement http://crowdtruth.orghttp://data.crowdtruth.org
  49. 49. http://lora-aroyo.org @laroyo CROWDTRUTH.ORG a spatial representation of meaning that harnesses disagreement http://crowdtruth.orghttp://data.crowdtruth.org a human computation (crowdsourcing) approach to: ● gather diversity of perspectives & opinions from crowds & niches ● expand expert vocabularies with these ● gather new type of gold standard for machines
  50. 50. http://lora-aroyo.org @laroyo COMFORT ZONE 50 Defending the single truth, the institutional quality validation http://crowdtruth.org
  51. 51. http://lora-aroyo.org @laroyo 51 One truth: knowledge acquisition and curation assume one correct interpretation for every object All cases are created equal: they are all either true or false Disagreement bad: when people disagree, they don’t understand the problem Experts rule: knowledge is always captured from domain experts One is enough: knowledge by a single expert is sufficient Detailed explanations help: if cases cause disagreement - add instructions Once done, forever valid: knowledge is not updated; new data not aligned with old a set of assumptions and rules that we rarely question “Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty http://crowdtruth.org COMFORT ZONE DISRUPTED
  52. 52. http://lora-aroyo.org @laroyo 52 COMFORT ZONE DISRUPTED Everything is relative, and life is full of perspectives and opinions M.C. Escher, Relativity , 1953
  53. 53. http://lora-aroyo.org @laroyo On the role of user-generated metadata in audio visual collections (2011). R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011 VIDEO METADATA ENRICHMENT The Netherlands Institute for Sound and Vision http://waisda.nl
  54. 54. http://lora-aroyo.org @laroyo On the role of user-generated metadata in audio visual collections (2011). R. Gligorov, M. Hildebrand, J. van Ossenbruggen, G. Schreiber, L. Aroyo K-CAP2011 VIDEO METADATA ENRICHMENT The Netherlands Institute for Sound and Vision http://spotvogel.vroegevogels.vara.nl/
  55. 55. http://lora-aroyo.org @laroyo L. Aroyo, C. Welty: CrowdTruth: Harnessing disagreement in crowdsourcing relex gold standard. ACM WebSci 2013. L. Aroyo, C. Welty. The Three Sides of CrowdTruth, Journal of Human Computation, 2014 VIDEO ENRICHMENT CrowdTruth with Amazon Mechanical Turk & Figure Eight http://crowdtruth.org
  56. 56. http://lora-aroyo.org @laroyo L. Aroyo, C. Welty: CrowdTruth: Harnessing disagreement in crowdsourcing relex gold standard. ACM WebSci 2013. L. Aroyo, C. Welty. The Three Sides of CrowdTruth, Journal of Human Computation, 2014 IMAGE ENRICHMENT CrowdTruth with Amazon Mechanical Turk & Figure Eight http://crowdtruth.org
  57. 57. http://lora-aroyo.org @laroyo L. Aroyo, C. Welty: CrowdTruth: Harnessing disagreement in crowdsourcing relex gold standard. ACM WebSci 2013. L. Aroyo, C. Welty. The Three Sides of CrowdTruth, Journal of Human Computation, 2014 IMAGE ENRICHMENT CrowdTruth with Amazon Mechanical Turk & Figure Eight http://crowdtruth.org
  58. 58. http://lora-aroyo.org @laroyo Nikita Galinkin, Zoltán Szlávik, Lora Aroyo and Benjamin Timmermans (2017). Catch Them If You Can: A Simulation Study on Malicious Behavior in a Cultural Heritage Question Answering System. The 29th Benelux Conference on Artificial Intelligence (BNAIC 2017). IMAGE ENRICHMENT CrowdTruth with Mauritshuis http://crowdtruth.org
  59. 59. http://lora-aroyo.org @laroyo Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage NICHESOURCING: FINDING NICHES IN THE CROWD Accurator tool: SealincMedia Project http://sealincmedia.wordpress.com
  60. 60. http://lora-aroyo.org @laroyo Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage NICHESOURCING IN THE CULTURAL HERITAGE Accurator tool http://annotate.accurator.nl
  61. 61. http://lora-aroyo.org @laroyo Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage NICHESOURCING IN THE CULTURAL HERITAGE Accurator tool http://annotate.accurator.nl
  62. 62. http://lora-aroyo.org @laroyo Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage NICHESOURCING IN THE CULTURAL HERITAGE Accurator tool http://annotate.accurator.nl
  63. 63. http://lora-aroyo.org @laroyo Chris Dijkshoorn, Victor De Boer, Lora Aroyo, Guus Schreiber (2014). Accurator: Nichesourcing for Cultural Heritage CREATING EXPERTS WITH GAMES Accurator tool http://annotate.accurator.nl
  64. 64. http://lora-aroyo.org @laroyo DigiBird: on the fly collection integration supported by the crowd (2017) Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo NICHESOURCING EVENTS Part of the SealincMedia Project http://annotate.accurator.nl
  65. 65. http://lora-aroyo.org @laroyo DigiBird: on the fly collection integration supported by the crowd (2017) Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo NICHESOURCING EVENTS DigiBird Project http://annotate.accurator.nl
  66. 66. http://lora-aroyo.org @laroyo DigiBird: on the fly collection integration supported by the crowd (2017) Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo NICHESOURCING EVENTS DigiBird Project http://annotate.accurator.nl
  67. 67. http://lora-aroyo.org @laroyo DigiBird: on the fly collection integration supported by the crowd (2017) Chris Dijkshoorn, Christina-Lulia Bucur, Maarten Brinkerink, Sander Pieterse and Lora Aroyo NICHESOURCING EVENTS DigiBird Project http://annotate.accurator.nl
  68. 68. http://lora-aroyo.org @laroyo SUCCESS STORIES: NIOD Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/
  69. 69. http://lora-aroyo.org @laroyo ADDING EVENTS TO THE NOB THESAURUS Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/
  70. 70. http://lora-aroyo.org @laroyo EVENTS THESAURUS Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/
  71. 71. http://lora-aroyo.org @laroyo PERSONAL EVENTS Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/
  72. 72. http://lora-aroyo.org @laroyo PEOPLE PORTAL Linked Data & Crowdsourcing for historical & personal events https://www.oorlogsbronnen.nl/
  73. 73. http://lora-aroyo.org @laroyo 632.953 artworks - 411.745 Rijksstudios SUCCESS STORIES: RIJKSMUSEUM Crowdsourcing with Rijksstudio https://www.rijksmuseum.nl/en/rijksstudio
  74. 74. http://lora-aroyo.org @laroyo SUCCESS STORIES: RIJKSMUSEUM Rijksmuseum API https://www.rijksmuseum.nl/en/api
  75. 75. http://lora-aroyo.org @laroyo SUCCESS STORIES: RIJKSMUSEUM Creativity with Open Data
  76. 76. http://lora-aroyo.org @laroyo SUCCESS STORIES: RIJKSMUSEUM Creativity with Open Data
  77. 77. http://lora-aroyo.org @laroyo SUCCESS STORIES: RIJKSMUSEUM Creativity with Open Data
  78. 78. http://lora-aroyo.org @laroyo SUCCESS STORIES: RIJKSMUSEUM Creativity with Open Data
  79. 79. http://lora-aroyo.org @laroyo SUCCESS STORIES: RIJKSMUSEUM Creativity with Open Data
  80. 80. http://lora-aroyo.org @laroyo LESSONS LEARNED ... Crowds are large and contribute at scale Crowds bring natural diversity Crowds help gathering real human semantics There are niches of experts in the crowds Experts and crowds are complimentary together they encompass a multitude of opinions and perspectives Experts and crowds have different semantics Experts and crowds are interested in different stories Experts and crowds use different vocabularies Crowds are enthusiasts, motivated, driven by altruism
  81. 81. http://lora-aroyo.org @laroyo The world is full of shades of grey Capturing and understanding opinions, perspectives & contexts is in the center of understanding people LESSONS LEARNED ... CrowdTruth defines multi-dimensional space to measure quality CrowdTruth defines hyper-dimensional space to represent ambiguity Nichesourcing helps expanding expertise beyond the walls of organizations Nichesourcing needs active engagement online and with onsite campaigns
  82. 82. http://lora-aroyo.org @laroyo CROWDTRUTH.ORG Not just a framework for crowdsourcing, it is a state of mind ... http://crowdtruth.orghttp://data.crowdtruth.org
  83. 83. http://lora-aroyo.org @laroyo Lora Aroyo StorySourcing: TELLING STORIES WITH HUMANS & MACHINES User Centric Data Science Group http://lora-aroyo.org @laroyo

×