SlideShare a Scribd company logo
1 of 59
Download to read offline
http://lora-aroyo.org @laroyo
Harnessing Human Semantics at Scale
Measurable, Reproducible, Engaging, Sustainable
Crowdsourcing & Nichesourcing
Lora Aroyo
http://lora-aroyo.org @laroyo
20071998 2006 2009
from DVDs to data science
http://lora-aroyo.org @laroyo
20071998 2006 2009
Team BellKor wins Netflix Prize
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
from books to data science
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
from books to data science
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
from books to data science
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
from books to data science
http://lora-aroyo.org @laroyo
20061994 2003 2016 2017
from books to data science
http://lora-aroyo.org @laroyo
data is at the centre of every process
http://lora-aroyo.org @laroyo
data is essential to evolve with users
http://lora-aroyo.org @laroyo
Ceci n'est pas … la mona lisa
http://lora-aroyo.org @laroyo
Ceci n'est pas … la mona lisa
Louvre’s Mona Lisa
is only #14
http://lora-aroyo.org @laroyo
the battle of two worlds
9,3 million
Louvre
visitors 2014
14 million
website visitors
2,3 million
social media
http://lora-aroyo.org @laroyo
in the (very near) future
most visitors will be digital-born
not bound by time or location
native to new forms of co-makership
native to new media
Siebe Weide, Max Meijer and Marieke Krabshuis (2012).
Agenda 2026: Study on the Future of the Dutch Museum Sector
http://lora-aroyo.org @laroyo
variety of meanings
multitude of perspectives
abundance of sources
endless contexts
know your data
http://lora-aroyo.org @laroyo
crowdsourcing to know your data at scale
http://lora-aroyo.org @laroyo
variety of types
multitude of platforms
abundance of interactions
endless characteristics
know your crowds
http://lora-aroyo.org @laroyo
https://www.rijksmuseum.nl/en/rijksstudio
Engage with Co-creation
http://lora-aroyo.org @laroyo
Engage with Co-creativity
http://lora-aroyo.org @laroyo
Engage with Co-curation
http://lora-aroyo.org @laroyo
Engage the Expert Niche
http://annotate.accurator.nl
http://lora-aroyo.org @laroyo
expertise of Rijksmuseum professionals is
in annotating their collection
with art-historical information, e.g. when they
were created, by whom, etc.
http://lora-aroyo.org @laroyo
detailed domain-specific information
about depicted objects, e.g. which species the
animal or plant belongs to,
is in most cases not available
http://lora-aroyo.org @laroyo
use nichesourcing, i.e. niches of people with
the right expertise, to add more specific
information
http://lora-aroyo.org @laroyo
Keep Reproducing
http://annotate.accurator.nl
http://lora-aroyo.org @laroyo
Engage with Games
training the general crowd to be a niche:
game in which players can carry out an expert
annotation tasks with some assistance
http://lora-aroyo.org @laroyo
http://waisda.nl
Engage with Games
http://lora-aroyo.org @laroyo
http://waisda.nl
Engage with Games
http://lora-aroyo.org @laroyo
http://spotvogel.vroegevogels.vara.nl
Keep Reproducing
http://lora-aroyo.org @laroyo
CrowdTruth.org
Experiment with Paid Crowds
http://lora-aroyo.org @laroyo
CrowdTruth.org
Experiment with Paid Crowds
http://lora-aroyo.org @laroyo
CrowdTruth.org
Experiment with Paid Crowds
http://lora-aroyo.org @laroyo
http://crowdtruth.org/
http://lora-aroyo.org @laroyo
http://data.crowdtruth.org/
http://lora-aroyo.org @laroyo
Challenges
http://lora-aroyo.org @laroyo
Low reproducibility rates
Difficult to estimate & control the time to complete
Difficult to assess & compare quality
Demands continuous promotional effort
Active learning (human-in-the-loop) needs different expertise
Difficult to incorporate results into existing content infrastructure
Challenges
Crowdsourcing typically undertaken in isolation
http://lora-aroyo.org @laroyo
Assess Impact of Task Design
http://lora-aroyo.org @laroyo
Instructions
Layout
Sequence
Crowds
Payment
Campaign
Assess Impact of Task Design
experiment with different designs
http://lora-aroyo.org @laroyo
for example
mapping music to mood
http://lora-aroyo.org @laroyo
Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Other
passionate, rollicking, literate, humorous, silly, aggressive, fiery, does not fit into
rousing, cheerful, fun, poignant, wistful, campy, quirky, tense, anxious, any of the 5
confident, sweet, amiable, bittersweet, whimsical, witty, intense, volatile, clusters
boisterous, good-natured autumnal, wry visceral
rowdy brooding
Choose one:
Which is the mood most appropriate
for each song?
Goal:
(Lee and Hu 2012)
1 song - 1 mood???
http://lora-aroyo.org @laroyo
If “One Truth” & “No Disagreement”
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5
W1 1
W2 1
W3 1
W4 1
W5 1
W6 1
W7
W8
W9 1
W10 1
Totals 1 3 1 2 1
http://lora-aroyo.org @laroyo
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W1 1 1 1
W2 1 1 1
W3 1 1 1
W4 1 1
W5 1 1
W6 1 1 1
W7 1 1 1
W8 1 1 1
W9 1 1
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
If “Many Truths” & “Disagreement”
Web & Media Group
http://lora-aroyo.org @laroyo
simplification of context
this all results in
Web & Media Group
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
● Identify Crowdsourcing Goals through user log analysis
○ # queries, #unique queries, #queries of specific type
○ ranked by popularity
○ ranked by popularity and with error, e.g.
■ # queries entered over 50 times with 0 results
■ # queries of specific type with 0 results
○ which will have biggest impact
○ which has biggest urgency
● … or through other user analysis
Assess Impact of Results
http://lora-aroyo.org @laroyo
for example
in video search
http://lora-aroyo.org @laroyo
people search for fragments
experts annotate full videos
35% of search queries result in not found
people search for fragments
experts annotate full videos
35% of search queries result in not found
for example
in video search
http://lora-aroyo.org @laroyo
Measure Quality
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
http://lora-aroyo.org @laroyo
Measure Quality
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
time-based annotation
bernhard
88% of the tags useful
for specific genres
describe short segments
often not very specific
don’t describe program as a whole
http://lora-aroyo.org @laroyo
for example
in video search
video annotation is time-consuming
5 times the video duration
experts use a specific vocabulary
that is unknown to general audiences
video annotation is time-consuming
5 times the video duration
experts use a specific vocabulary
that is unknown to general audiences
http://lora-aroyo.org @laroyo
user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
locations (7%)
engeland
persons (31%)
objects (57%)
Measure Quality
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
Web & Media Group
http://lora-aroyo.org @laroyo
human subjectivity, ambiguity & uncertainty of expression
natural part of human semantics
http://lora-aroyo.org @laroyo
measure quality
quality is not just about spam
quality is typically multi-dimensional
understand the diversity in crowd answers
do not ignore multitude of interpretations
understand the variety of contexts
identify cases with high ambiguity, similarity, …
experiment with explicit metrics
experiment with different designs
http://lora-aroyo.org @laroyo
Measure Progress
6 months 2 years
340,551 tags 36,981 tags
137.421 matches
602 items 1.782 items
555 registered players 2,017 users (taggers)
thousands of anonymous players
12,279 visits (3+ min online)
44,362 pageviews
Riste Gligorov, Michiel Hildebrand, Jacco van Ossenbruggen, Guus Schreiber, Lora Aroyo
(2011). On the role of user-generated metadata in audio visual collections. International
conference on Knowledge capture K-CAP '11, Pages 145-152
http://lora-aroyo.org @laroyo
campaign, campaign, campaign
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
http://lora-aroyo.org @laroyo
Measurable quality
Reproducible results
Sustainable settings
Engaging interaction
Goals

More Related Content

What's hot

Semantic SEO in the post Hummingbird Era and WordLift
Semantic SEO in the post Hummingbird Era and WordLiftSemantic SEO in the post Hummingbird Era and WordLift
Semantic SEO in the post Hummingbird Era and WordLiftAndrea Volpini
 
Weaponized Web Archives: Provenance Laundering of Short Order Evidence
Weaponized Web Archives: Provenance Laundering of Short Order Evidence Weaponized Web Archives: Provenance Laundering of Short Order Evidence
Weaponized Web Archives: Provenance Laundering of Short Order Evidence Michael Nelson
 
CrowdTruth for User-Centric Relevance
CrowdTruth for User-Centric RelevanceCrowdTruth for User-Centric Relevance
CrowdTruth for User-Centric RelevanceLora Aroyo
 
The Potential of Web 3.0
The Potential of Web 3.0The Potential of Web 3.0
The Potential of Web 3.0Carsten Ullrich
 
The Internet (The Problem With...)
The Internet (The Problem With...)The Internet (The Problem With...)
The Internet (The Problem With...)Helior Colorado
 
Research Information Network 07 May 2009
Research Information Network 07 May 2009Research Information Network 07 May 2009
Research Information Network 07 May 2009rpg7ss
 
From Content Strategy to Drupal Site Building - Connecting the dots
From Content Strategy to Drupal Site Building - Connecting the dotsFrom Content Strategy to Drupal Site Building - Connecting the dots
From Content Strategy to Drupal Site Building - Connecting the dotsRonald Ashri
 
CS260
CS260CS260
CS260mor
 
Data Modelling at Scale
Data Modelling at ScaleData Modelling at Scale
Data Modelling at ScaleDavid Simons
 
Data and Creativity: The Perfect Marriage
Data and Creativity: The Perfect MarriageData and Creativity: The Perfect Marriage
Data and Creativity: The Perfect MarriageRafael Lebrón Febles
 
How The Open Data Community Died - A Warning From The Future
How The Open Data Community Died - A Warning From The FutureHow The Open Data Community Died - A Warning From The Future
How The Open Data Community Died - A Warning From The FutureChris Taggart
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningLora Aroyo
 
Web User Experience in 2021
Web User Experience in 2021Web User Experience in 2021
Web User Experience in 2021Drew Gorton
 
From Data Points to Data Lakes
From Data Points to Data LakesFrom Data Points to Data Lakes
From Data Points to Data LakesJesus Rogel
 

What's hot (15)

Semantic SEO in the post Hummingbird Era and WordLift
Semantic SEO in the post Hummingbird Era and WordLiftSemantic SEO in the post Hummingbird Era and WordLift
Semantic SEO in the post Hummingbird Era and WordLift
 
Weaponized Web Archives: Provenance Laundering of Short Order Evidence
Weaponized Web Archives: Provenance Laundering of Short Order Evidence Weaponized Web Archives: Provenance Laundering of Short Order Evidence
Weaponized Web Archives: Provenance Laundering of Short Order Evidence
 
CrowdTruth for User-Centric Relevance
CrowdTruth for User-Centric RelevanceCrowdTruth for User-Centric Relevance
CrowdTruth for User-Centric Relevance
 
The Potential of Web 3.0
The Potential of Web 3.0The Potential of Web 3.0
The Potential of Web 3.0
 
The Internet (The Problem With...)
The Internet (The Problem With...)The Internet (The Problem With...)
The Internet (The Problem With...)
 
Research Information Network 07 May 2009
Research Information Network 07 May 2009Research Information Network 07 May 2009
Research Information Network 07 May 2009
 
From Content Strategy to Drupal Site Building - Connecting the dots
From Content Strategy to Drupal Site Building - Connecting the dotsFrom Content Strategy to Drupal Site Building - Connecting the dots
From Content Strategy to Drupal Site Building - Connecting the dots
 
CS260
CS260CS260
CS260
 
Data Modelling at Scale
Data Modelling at ScaleData Modelling at Scale
Data Modelling at Scale
 
Data and Creativity: The Perfect Marriage
Data and Creativity: The Perfect MarriageData and Creativity: The Perfect Marriage
Data and Creativity: The Perfect Marriage
 
How The Open Data Community Died - A Warning From The Future
How The Open Data Community Died - A Warning From The FutureHow The Open Data Community Died - A Warning From The Future
How The Open Data Community Died - A Warning From The Future
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
 
Web User Experience in 2021
Web User Experience in 2021Web User Experience in 2021
Web User Experience in 2021
 
ChatOps FTW
ChatOps FTWChatOps FTW
ChatOps FTW
 
From Data Points to Data Lakes
From Data Points to Data LakesFrom Data Points to Data Lakes
From Data Points to Data Lakes
 

Similar to Digital Humanities Benelux 2017: Keynote Lora Aroyo

Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AILora Aroyo
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordLora Aroyo
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumLora Aroyo
 
Linked Data and Search: Thomas Steiner (Google Inc, Germany)
Linked Data and Search:  Thomas Steiner (Google Inc, Germany)Linked Data and Search:  Thomas Steiner (Google Inc, Germany)
Linked Data and Search: Thomas Steiner (Google Inc, Germany)FIA2010
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18Lora Aroyo
 
Modelling Users’ Profiles and Interests based on Cross-Folksonomy Analysis ...
Modelling Users’ Profiles and  Interests based on  Cross-Folksonomy Analysis ...Modelling Users’ Profiles and  Interests based on  Cross-Folksonomy Analysis ...
Modelling Users’ Profiles and Interests based on Cross-Folksonomy Analysis ...Martin Szomszor
 
Optimizing for Voice Search #SMXL18
Optimizing for Voice Search #SMXL18Optimizing for Voice Search #SMXL18
Optimizing for Voice Search #SMXL18Aleyda Solís
 
Selfish Accessibility: Government Digital Service
Selfish Accessibility: Government Digital ServiceSelfish Accessibility: Government Digital Service
Selfish Accessibility: Government Digital ServiceAdrian Roselli
 
Selfish Accessibility — WordCamp Europe 2017
Selfish Accessibility — WordCamp Europe 2017Selfish Accessibility — WordCamp Europe 2017
Selfish Accessibility — WordCamp Europe 2017Adrian Roselli
 
Gaming with Open Source Frameworks
Gaming with Open Source FrameworksGaming with Open Source Frameworks
Gaming with Open Source FrameworksAlmog Koren
 
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...Rory Hope
 
International SEO: The Weird Technical Parts - Pubcon Vegas 2019 Patrick Stox
International SEO: The Weird Technical Parts - Pubcon Vegas 2019 Patrick StoxInternational SEO: The Weird Technical Parts - Pubcon Vegas 2019 Patrick Stox
International SEO: The Weird Technical Parts - Pubcon Vegas 2019 Patrick Stoxpatrickstox
 
Content and seo for Tulos by Gianluca Fiorelli
Content and seo for Tulos by Gianluca FiorelliContent and seo for Tulos by Gianluca Fiorelli
Content and seo for Tulos by Gianluca FiorelliTulos Helsinki Oy
 
Selfish Accessibility: WordCamp London 2017
Selfish Accessibility: WordCamp London 2017Selfish Accessibility: WordCamp London 2017
Selfish Accessibility: WordCamp London 2017Adrian Roselli
 
Community 2.0: Beyond Using Software Livre
Community 2.0: Beyond Using Software LivreCommunity 2.0: Beyond Using Software Livre
Community 2.0: Beyond Using Software LivreLeslie Hawthorn
 
Openlaw - A Project of Alphalaw
Openlaw - A Project of AlphalawOpenlaw - A Project of Alphalaw
Openlaw - A Project of AlphalawSunghyun Kim
 
Catalyze Webcast - Five Myths Of RIA With Laurie Gray - 031808
Catalyze Webcast - Five Myths Of RIA With Laurie Gray - 031808Catalyze Webcast - Five Myths Of RIA With Laurie Gray - 031808
Catalyze Webcast - Five Myths Of RIA With Laurie Gray - 031808Tom Humbarger
 
Indie Game Market Research Presentation
Indie Game Market Research PresentationIndie Game Market Research Presentation
Indie Game Market Research PresentationLogan Williams
 
Social Semantic Web on Facebook Open Graph protocol and Twitter Annotations
Social Semantic Web on Facebook Open Graph protocol and Twitter AnnotationsSocial Semantic Web on Facebook Open Graph protocol and Twitter Annotations
Social Semantic Web on Facebook Open Graph protocol and Twitter AnnotationsMyungjin Lee
 

Similar to Digital Humanities Benelux 2017: Keynote Lora Aroyo (20)

Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @Oxford
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
 
Linked Data and Search: Thomas Steiner (Google Inc, Germany)
Linked Data and Search:  Thomas Steiner (Google Inc, Germany)Linked Data and Search:  Thomas Steiner (Google Inc, Germany)
Linked Data and Search: Thomas Steiner (Google Inc, Germany)
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18
 
Modelling Users’ Profiles and Interests based on Cross-Folksonomy Analysis ...
Modelling Users’ Profiles and  Interests based on  Cross-Folksonomy Analysis ...Modelling Users’ Profiles and  Interests based on  Cross-Folksonomy Analysis ...
Modelling Users’ Profiles and Interests based on Cross-Folksonomy Analysis ...
 
Optimizing for Voice Search #SMXL18
Optimizing for Voice Search #SMXL18Optimizing for Voice Search #SMXL18
Optimizing for Voice Search #SMXL18
 
Yahoo for the Masses
Yahoo for the MassesYahoo for the Masses
Yahoo for the Masses
 
Selfish Accessibility: Government Digital Service
Selfish Accessibility: Government Digital ServiceSelfish Accessibility: Government Digital Service
Selfish Accessibility: Government Digital Service
 
Selfish Accessibility — WordCamp Europe 2017
Selfish Accessibility — WordCamp Europe 2017Selfish Accessibility — WordCamp Europe 2017
Selfish Accessibility — WordCamp Europe 2017
 
Gaming with Open Source Frameworks
Gaming with Open Source FrameworksGaming with Open Source Frameworks
Gaming with Open Source Frameworks
 
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...
 
International SEO: The Weird Technical Parts - Pubcon Vegas 2019 Patrick Stox
International SEO: The Weird Technical Parts - Pubcon Vegas 2019 Patrick StoxInternational SEO: The Weird Technical Parts - Pubcon Vegas 2019 Patrick Stox
International SEO: The Weird Technical Parts - Pubcon Vegas 2019 Patrick Stox
 
Content and seo for Tulos by Gianluca Fiorelli
Content and seo for Tulos by Gianluca FiorelliContent and seo for Tulos by Gianluca Fiorelli
Content and seo for Tulos by Gianluca Fiorelli
 
Selfish Accessibility: WordCamp London 2017
Selfish Accessibility: WordCamp London 2017Selfish Accessibility: WordCamp London 2017
Selfish Accessibility: WordCamp London 2017
 
Community 2.0: Beyond Using Software Livre
Community 2.0: Beyond Using Software LivreCommunity 2.0: Beyond Using Software Livre
Community 2.0: Beyond Using Software Livre
 
Openlaw - A Project of Alphalaw
Openlaw - A Project of AlphalawOpenlaw - A Project of Alphalaw
Openlaw - A Project of Alphalaw
 
Catalyze Webcast - Five Myths Of RIA With Laurie Gray - 031808
Catalyze Webcast - Five Myths Of RIA With Laurie Gray - 031808Catalyze Webcast - Five Myths Of RIA With Laurie Gray - 031808
Catalyze Webcast - Five Myths Of RIA With Laurie Gray - 031808
 
Indie Game Market Research Presentation
Indie Game Market Research PresentationIndie Game Market Research Presentation
Indie Game Market Research Presentation
 
Social Semantic Web on Facebook Open Graph protocol and Twitter Annotations
Social Semantic Web on Facebook Open Graph protocol and Twitter AnnotationsSocial Semantic Web on Facebook Open Graph protocol and Twitter Annotations
Social Semantic Web on Facebook Open Graph protocol and Twitter Annotations
 

More from Lora Aroyo

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfLora Aroyo
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumLora Aroyo
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorLora Aroyo
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataLora Aroyo
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsLora Aroyo
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesLora Aroyo
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the LoopLora Aroyo
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...Lora Aroyo
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Lora Aroyo
 
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchLora Aroyo
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyLora Aroyo
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...Lora Aroyo
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumLora Aroyo
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Lora Aroyo
 
Keynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventLora Aroyo
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+Lora Aroyo
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...Lora Aroyo
 
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015Lora Aroyo
 

More from Lora Aroyo (18)

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH Symposium
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP Demonstrator
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked Data
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithms
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & Machines
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the Loop
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
 
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening Ceremony
 
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...
Crowdsourcing & Nichesourcing: Enriching Cultural Heritage with Experts & Cr...
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
 
Keynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program EventKeynote @Final NWO CATCH Program Event
Keynote @Final NWO CATCH Program Event
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
 
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
 

Recently uploaded

Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactivestartupro
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfROWELL MARQUINA
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesManik S Magar
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Hiroshi SHIBATA
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Memoori
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementDianaGray10
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?Rustici Software
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentPim van der Noll
 
Introduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxIntroduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxmprakaash5
 
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfHCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfROWELL MARQUINA
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 

Recently uploaded (20)

Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactive
 
QMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdfQMMS Lesson 2 - Using MS Excel Formula.pdf
QMMS Lesson 2 - Using MS Excel Formula.pdf
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotesMuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
MuleSoft Online Meetup Group - B2B Crash Course: Release SparkNotes
 
Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024Long journey of Ruby standard library at RubyConf AU 2024
Long journey of Ruby standard library at RubyConf AU 2024
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions management
 
Which standard is best for your content?
Which standard is best for your content?Which standard is best for your content?
Which standard is best for your content?
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native developmentEmixa Mendix Meetup 11 April 2024 about Mendix Native development
Emixa Mendix Meetup 11 April 2024 about Mendix Native development
 
Introduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxIntroduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptx
 
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdfHCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
HCI Lesson 1 - Introduction to Human-Computer Interaction.pdf
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 

Digital Humanities Benelux 2017: Keynote Lora Aroyo