SlideShare a Scribd company logo
Lora Aroyo @laroyo http://lora-aroyo.org
Turning Sci-Fi into Reality
Lora Aroyo
Lora Aroyo @laroyo http://lora-aroyo.org
Sci-Fi in 60’s, 70’s & 80’s
Lora Aroyo @laroyo http://lora-aroyo.org
Sci-Fi in 2002
Lora Aroyo @laroyo http://lora-aroyo.org
Sci-Fi in 2013
Lora Aroyo @laroyo http://lora-aroyo.org
Sci-Fi in 2015
Lora Aroyo @laroyo http://lora-aroyo.org
Turning Sci-Fi into Reality
Lora Aroyo @laroyo http://lora-aroyo.org
scientific challenge in 1997
Lora Aroyo @laroyo http://lora-aroyo.org
chess
finite, mathematically well-defined search space
limited number of moves & states
grounded in explicit, unambiguous math rules
Lora Aroyo @laroyo http://lora-aroyo.org
IBM Confidential
scientific challenge in 2011
Lora Aroyo @laroyo http://lora-aroyo.org
IBM Confidential
natural language
ambiguous, contextual & implicit
grounded only in human cognition
infinite number of ways to express same meaning
Lora Aroyo @laroyo http://lora-aroyo.org
http://lora-aroyo.org @laroyo 11
keeps expanding to new domains
Lora Aroyo @laroyo http://lora-aroyo.org
Lora Aroyo @laroyo http://lora-aroyo.org
http://lora-aroyo.org @laroyo 13
Lora Aroyo @laroyo http://lora-aroyo.org
1. Expand
“EXPANDs human
cognition - makes the jobs
we do easier, like a
cognitive prosthesis.

Especially when dealing
with processing massive
data, or data that requires
human interpretation”
Lora Aroyo @laroyo http://lora-aroyo.org
“LEARNs as you use it –
most machine errors are
easy for a human to
detect.

We can instrument usage
of systems to better
understand the system
and the problem it solves”
2. Learn
Lora Aroyo @laroyo http://lora-aroyo.org
“INTERACTs naturally -
bringing machines closer to
their users by letting them
understand natural language,
spoken or written, be able to
process images & videos. 

These simple human problems
are extremely complex for
machines, but are hallmarks
of a new computing era.”
3. Interact
Lora Aroyo @laroyo http://lora-aroyo.org
▪  A new software paradigm has emerged
–  Increasingly, computational tasks require inexact solutions that
combine multiple methods in unpredictable ways
▪  Knowledge is not the destination
–  Watson does not answer a question by translating natural language
input into formally represented knowledge and simply running queries
against this knowledge
▪  Machine intelligence is not human intelligence
–  The difference is most notable in the mistakes they make
When Building Cognitive Computing Systems …
http://www.yovisto.com/video/19674
Chris Welty’s keynote at WWW2012
Lora Aroyo @laroyo http://lora-aroyo.org
Cognitive
Systems
can fail
and still
be useful
A dramatic shift for software systems
Focus on improvement, not perfection
We must be able to answer, “Is it better?”
Cognitive Computing: It’s OK to be wrong!
Lora Aroyo @laroyo http://lora-aroyo.org
Cognitive Systems Engineering Pillars
MEASURE
MEASURE
MEASURE
DATA
DATA
DATA
TRUTH
TRUTH
TRUTH
Lora Aroyo @laroyo http://lora-aroyo.org
Cognitive Systems Engineering Pillars
MEASURE
MEASURE
MEASURE
DATA
DATA
DATA
TRUTH
TRUTH
TRUTH
They define the system.
If you change one of them,
the system should change
Lora Aroyo @laroyo http://lora-aroyo.org
Winning Human
Performance
2007 QA Computer System
Grand Champion
Human Performance
Top human
players are
remarkably
good.
Each dot – actual historical human Jeopardy! games
More Confident Less Confident
What it Takes to compete against Top Human Jeopardy! Players
Our Analysis Reveals the Winner’s Cloud
Lora Aroyo @laroyo http://lora-aroyo.org
2007 QA Computer System
In 2007, we committed to
making a Huge Leap!
Computers?
Not So Good.
Winning Human
Performance
Grand Champion
Human Performance
What it Takes to compete against Top Human Jeopardy! Players
Our Analysis Reveals the Winner’s Cloud
Each dot – actual historical human Jeopardy! games
More Confident Less Confident
Lora Aroyo @laroyo http://lora-aroyo.org
Precision
% of Answered
Deep QA: Incremental Progress
6/2007-11/2010
Now Playing in the
Winners Cloud
11/2010
04/2010
10/2009
5/2009
12/2008
8/2008
5/2008
12/2007
Baseline
Lora Aroyo @laroyo http://lora-aroyo.org
… and WATSON won
Lora Aroyo @laroyo http://lora-aroyo.org
Watch…
Day 2 (15 Feb 2011)
Day 1 (14 Feb 2011)
Also on YouTube:
https://www.youtube.com/watch?v=i-vMW_Ce51w
Lora Aroyo @laroyo http://lora-aroyo.org
Crowd-Watson team 2013
http://lora-aroyo.org @laroyo 26
Lora Aroyo @laroyo http://lora-aroyo.org
CrowdTruth team growing, 2014
http://lora-aroyo.org @laroyo 27
Lora Aroyo @laroyo http://lora-aroyo.org
http://lora-aroyo.org @laroyo 28
Lora Aroyo @laroyo http://lora-aroyo.org
http://lora-aroyo.org @laroyo 29
Lora Aroyo @laroyo http://lora-aroyo.org
Lora Aroyo @laroyo http://lora-aroyo.org
Crowd-Watson 2015
Lora Aroyo @laroyo http://lora-aroyo.org
http://crowdtruth.org

More Related Content

Viewers also liked

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
Lora Aroyo
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
Lora Aroyo
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Lora Aroyo
 
CrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsCrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsLora Aroyo
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Lora Aroyo
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening Ceremony
Lora Aroyo
 
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
CrowdTruth
 
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Lora Aroyo
 
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
Lora Aroyo
 
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataVisualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
CrowdTruth
 
Crowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain QuestionsCrowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain Questions
Benjamin Timmermans
 
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Lora Aroyo
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @Oxford
Lora Aroyo
 
ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016
oanainel
 
Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...
Benjamin Timmermans
 
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
CrowdTruth
 
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Lora Aroyo
 
Towards Better Media Understanding and Searchability
Towards Better Media Understanding and SearchabilityTowards Better Media Understanding and Searchability
Towards Better Media Understanding and Searchability
oanainel
 
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Lora Aroyo
 

Viewers also liked (19)

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
 
CrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsCrowdTruth for Digital Hermeneutics
CrowdTruth for Digital Hermeneutics
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening Ceremony
 
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
 
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
 
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
 
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataVisualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
 
Crowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain QuestionsCrowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain Questions
 
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @Oxford
 
ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016
 
Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...
 
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
 
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
 
Towards Better Media Understanding and Searchability
Towards Better Media Understanding and SearchabilityTowards Better Media Understanding and Searchability
Towards Better Media Understanding and Searchability
 
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
 

Similar to Closing Event - Watson Innovation Course

Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
Lora Aroyo
 
Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)
Lora Aroyo
 
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
Distilled
 
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Codemotion
 
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
Distilled
 
Pietro leo - Active Intelligence
Pietro leo - Active IntelligencePietro leo - Active Intelligence
Pietro leo - Active Intelligence
Pietro Leo
 
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
MnSearch, The Minnesota Search Engine Marketing Association
 
2005 BT Technology Timeline
2005 BT Technology Timeline2005 BT Technology Timeline
2005 BT Technology Timelinebutest
 
Mobile Games Market Trends Q1 2018 from Priori Data
Mobile Games Market Trends Q1 2018 from Priori DataMobile Games Market Trends Q1 2018 from Priori Data
Mobile Games Market Trends Q1 2018 from Priori Data
Benjamin Nolan
 
Artificial intelligence for humans… #AIDC2018 keynote
Artificial intelligence for humans… #AIDC2018 keynoteArtificial intelligence for humans… #AIDC2018 keynote
Artificial intelligence for humans… #AIDC2018 keynote
Christian Heilmann
 
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Patrizio Pelliccione
 
Harkable Day of Innovation Oct 2013 - Hark in the Park
Harkable Day of Innovation Oct 2013 - Hark in the ParkHarkable Day of Innovation Oct 2013 - Hark in the Park
Harkable Day of Innovation Oct 2013 - Hark in the Park
Harkable
 
AI, Innovation & Ethics in Marketing by PR Smith, founder of SOSTAC® Plans ...
AI, Innovation & Ethics in Marketing  by PR Smith, founder of SOSTAC® Plans  ...AI, Innovation & Ethics in Marketing  by PR Smith, founder of SOSTAC® Plans  ...
AI, Innovation & Ethics in Marketing by PR Smith, founder of SOSTAC® Plans ...
Institutul de Marketing
 

Similar to Closing Event - Watson Innovation Course (13)

Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
 
Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)
 
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
 
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
 
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
 
Pietro leo - Active Intelligence
Pietro leo - Active IntelligencePietro leo - Active Intelligence
Pietro leo - Active Intelligence
 
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
 
2005 BT Technology Timeline
2005 BT Technology Timeline2005 BT Technology Timeline
2005 BT Technology Timeline
 
Mobile Games Market Trends Q1 2018 from Priori Data
Mobile Games Market Trends Q1 2018 from Priori DataMobile Games Market Trends Q1 2018 from Priori Data
Mobile Games Market Trends Q1 2018 from Priori Data
 
Artificial intelligence for humans… #AIDC2018 keynote
Artificial intelligence for humans… #AIDC2018 keynoteArtificial intelligence for humans… #AIDC2018 keynote
Artificial intelligence for humans… #AIDC2018 keynote
 
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
 
Harkable Day of Innovation Oct 2013 - Hark in the Park
Harkable Day of Innovation Oct 2013 - Hark in the ParkHarkable Day of Innovation Oct 2013 - Hark in the Park
Harkable Day of Innovation Oct 2013 - Hark in the Park
 
AI, Innovation & Ethics in Marketing by PR Smith, founder of SOSTAC® Plans ...
AI, Innovation & Ethics in Marketing  by PR Smith, founder of SOSTAC® Plans  ...AI, Innovation & Ethics in Marketing  by PR Smith, founder of SOSTAC® Plans  ...
AI, Innovation & Ethics in Marketing by PR Smith, founder of SOSTAC® Plans ...
 

More from Lora Aroyo

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
Lora Aroyo
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
Lora Aroyo
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH Symposium
Lora Aroyo
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP Demonstrator
Lora Aroyo
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked Data
Lora Aroyo
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Lora Aroyo
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18
Lora Aroyo
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithms
Lora Aroyo
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & Machines
Lora Aroyo
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the Loop
Lora Aroyo
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
Lora Aroyo
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Lora Aroyo
 
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneMy ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
Lora Aroyo
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden University
Lora Aroyo
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
Lora Aroyo
 
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
Lora Aroyo
 

More from Lora Aroyo (16)

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH Symposium
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP Demonstrator
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked Data
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithms
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & Machines
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the Loop
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
 
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneMy ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden University
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
 
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
 

Recently uploaded

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Tobias Schneck
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Jeffrey Haguewood
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
ThousandEyes
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
Ana-Maria Mihalceanu
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
Guy Korland
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
DianaGray10
 

Recently uploaded (20)

Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Assuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyesAssuring Contact Center Experiences for Your Customers With ThousandEyes
Assuring Contact Center Experiences for Your Customers With ThousandEyes
 
Monitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR EventsMonitoring Java Application Security with JDK Tools and JFR Events
Monitoring Java Application Security with JDK Tools and JFR Events
 
GraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge GraphGraphRAG is All You need? LLM & Knowledge Graph
GraphRAG is All You need? LLM & Knowledge Graph
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
FIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdfFIDO Alliance Osaka Seminar: Overview.pdf
FIDO Alliance Osaka Seminar: Overview.pdf
 
UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3UiPath Test Automation using UiPath Test Suite series, part 3
UiPath Test Automation using UiPath Test Suite series, part 3
 

Closing Event - Watson Innovation Course

  • 1. Lora Aroyo @laroyo http://lora-aroyo.org Turning Sci-Fi into Reality Lora Aroyo
  • 2. Lora Aroyo @laroyo http://lora-aroyo.org Sci-Fi in 60’s, 70’s & 80’s
  • 3. Lora Aroyo @laroyo http://lora-aroyo.org Sci-Fi in 2002
  • 4. Lora Aroyo @laroyo http://lora-aroyo.org Sci-Fi in 2013
  • 5. Lora Aroyo @laroyo http://lora-aroyo.org Sci-Fi in 2015
  • 6. Lora Aroyo @laroyo http://lora-aroyo.org Turning Sci-Fi into Reality
  • 7. Lora Aroyo @laroyo http://lora-aroyo.org scientific challenge in 1997
  • 8. Lora Aroyo @laroyo http://lora-aroyo.org chess finite, mathematically well-defined search space limited number of moves & states grounded in explicit, unambiguous math rules
  • 9. Lora Aroyo @laroyo http://lora-aroyo.org IBM Confidential scientific challenge in 2011
  • 10. Lora Aroyo @laroyo http://lora-aroyo.org IBM Confidential natural language ambiguous, contextual & implicit grounded only in human cognition infinite number of ways to express same meaning
  • 11. Lora Aroyo @laroyo http://lora-aroyo.org http://lora-aroyo.org @laroyo 11 keeps expanding to new domains
  • 12. Lora Aroyo @laroyo http://lora-aroyo.org
  • 13. Lora Aroyo @laroyo http://lora-aroyo.org http://lora-aroyo.org @laroyo 13
  • 14. Lora Aroyo @laroyo http://lora-aroyo.org 1. Expand “EXPANDs human cognition - makes the jobs we do easier, like a cognitive prosthesis. Especially when dealing with processing massive data, or data that requires human interpretation”
  • 15. Lora Aroyo @laroyo http://lora-aroyo.org “LEARNs as you use it – most machine errors are easy for a human to detect. We can instrument usage of systems to better understand the system and the problem it solves” 2. Learn
  • 16. Lora Aroyo @laroyo http://lora-aroyo.org “INTERACTs naturally - bringing machines closer to their users by letting them understand natural language, spoken or written, be able to process images & videos. These simple human problems are extremely complex for machines, but are hallmarks of a new computing era.” 3. Interact
  • 17. Lora Aroyo @laroyo http://lora-aroyo.org ▪  A new software paradigm has emerged –  Increasingly, computational tasks require inexact solutions that combine multiple methods in unpredictable ways ▪  Knowledge is not the destination –  Watson does not answer a question by translating natural language input into formally represented knowledge and simply running queries against this knowledge ▪  Machine intelligence is not human intelligence –  The difference is most notable in the mistakes they make When Building Cognitive Computing Systems … http://www.yovisto.com/video/19674 Chris Welty’s keynote at WWW2012
  • 18. Lora Aroyo @laroyo http://lora-aroyo.org Cognitive Systems can fail and still be useful A dramatic shift for software systems Focus on improvement, not perfection We must be able to answer, “Is it better?” Cognitive Computing: It’s OK to be wrong!
  • 19. Lora Aroyo @laroyo http://lora-aroyo.org Cognitive Systems Engineering Pillars MEASURE MEASURE MEASURE DATA DATA DATA TRUTH TRUTH TRUTH
  • 20. Lora Aroyo @laroyo http://lora-aroyo.org Cognitive Systems Engineering Pillars MEASURE MEASURE MEASURE DATA DATA DATA TRUTH TRUTH TRUTH They define the system. If you change one of them, the system should change
  • 21. Lora Aroyo @laroyo http://lora-aroyo.org Winning Human Performance 2007 QA Computer System Grand Champion Human Performance Top human players are remarkably good. Each dot – actual historical human Jeopardy! games More Confident Less Confident What it Takes to compete against Top Human Jeopardy! Players Our Analysis Reveals the Winner’s Cloud
  • 22. Lora Aroyo @laroyo http://lora-aroyo.org 2007 QA Computer System In 2007, we committed to making a Huge Leap! Computers? Not So Good. Winning Human Performance Grand Champion Human Performance What it Takes to compete against Top Human Jeopardy! Players Our Analysis Reveals the Winner’s Cloud Each dot – actual historical human Jeopardy! games More Confident Less Confident
  • 23. Lora Aroyo @laroyo http://lora-aroyo.org Precision % of Answered Deep QA: Incremental Progress 6/2007-11/2010 Now Playing in the Winners Cloud 11/2010 04/2010 10/2009 5/2009 12/2008 8/2008 5/2008 12/2007 Baseline
  • 24. Lora Aroyo @laroyo http://lora-aroyo.org … and WATSON won
  • 25. Lora Aroyo @laroyo http://lora-aroyo.org Watch… Day 2 (15 Feb 2011) Day 1 (14 Feb 2011) Also on YouTube: https://www.youtube.com/watch?v=i-vMW_Ce51w
  • 26. Lora Aroyo @laroyo http://lora-aroyo.org Crowd-Watson team 2013 http://lora-aroyo.org @laroyo 26
  • 27. Lora Aroyo @laroyo http://lora-aroyo.org CrowdTruth team growing, 2014 http://lora-aroyo.org @laroyo 27
  • 28. Lora Aroyo @laroyo http://lora-aroyo.org http://lora-aroyo.org @laroyo 28
  • 29. Lora Aroyo @laroyo http://lora-aroyo.org http://lora-aroyo.org @laroyo 29
  • 30. Lora Aroyo @laroyo http://lora-aroyo.org
  • 31. Lora Aroyo @laroyo http://lora-aroyo.org Crowd-Watson 2015
  • 32. Lora Aroyo @laroyo http://lora-aroyo.org http://crowdtruth.org