SlideShare a Scribd company logo
1 of 32
Download to read offline
Lora Aroyo @laroyo http://lora-aroyo.org
Turning Sci-Fi into Reality
Lora Aroyo
Lora Aroyo @laroyo http://lora-aroyo.org
Sci-Fi in 60’s, 70’s & 80’s
Lora Aroyo @laroyo http://lora-aroyo.org
Sci-Fi in 2002
Lora Aroyo @laroyo http://lora-aroyo.org
Sci-Fi in 2013
Lora Aroyo @laroyo http://lora-aroyo.org
Sci-Fi in 2015
Lora Aroyo @laroyo http://lora-aroyo.org
Turning Sci-Fi into Reality
Lora Aroyo @laroyo http://lora-aroyo.org
scientific challenge in 1997
Lora Aroyo @laroyo http://lora-aroyo.org
chess
finite, mathematically well-defined search space
limited number of moves & states
grounded in explicit, unambiguous math rules
Lora Aroyo @laroyo http://lora-aroyo.org
IBM Confidential
scientific challenge in 2011
Lora Aroyo @laroyo http://lora-aroyo.org
IBM Confidential
natural language
ambiguous, contextual & implicit
grounded only in human cognition
infinite number of ways to express same meaning
Lora Aroyo @laroyo http://lora-aroyo.org
http://lora-aroyo.org @laroyo 11
keeps expanding to new domains
Lora Aroyo @laroyo http://lora-aroyo.org
Lora Aroyo @laroyo http://lora-aroyo.org
http://lora-aroyo.org @laroyo 13
Lora Aroyo @laroyo http://lora-aroyo.org
1. Expand
“EXPANDs human
cognition - makes the jobs
we do easier, like a
cognitive prosthesis.

Especially when dealing
with processing massive
data, or data that requires
human interpretation”
Lora Aroyo @laroyo http://lora-aroyo.org
“LEARNs as you use it –
most machine errors are
easy for a human to
detect.

We can instrument usage
of systems to better
understand the system
and the problem it solves”
2. Learn
Lora Aroyo @laroyo http://lora-aroyo.org
“INTERACTs naturally -
bringing machines closer to
their users by letting them
understand natural language,
spoken or written, be able to
process images & videos. 

These simple human problems
are extremely complex for
machines, but are hallmarks
of a new computing era.”
3. Interact
Lora Aroyo @laroyo http://lora-aroyo.org
▪  A new software paradigm has emerged
–  Increasingly, computational tasks require inexact solutions that
combine multiple methods in unpredictable ways
▪  Knowledge is not the destination
–  Watson does not answer a question by translating natural language
input into formally represented knowledge and simply running queries
against this knowledge
▪  Machine intelligence is not human intelligence
–  The difference is most notable in the mistakes they make
When Building Cognitive Computing Systems …
http://www.yovisto.com/video/19674
Chris Welty’s keynote at WWW2012
Lora Aroyo @laroyo http://lora-aroyo.org
Cognitive
Systems
can fail
and still
be useful
A dramatic shift for software systems
Focus on improvement, not perfection
We must be able to answer, “Is it better?”
Cognitive Computing: It’s OK to be wrong!
Lora Aroyo @laroyo http://lora-aroyo.org
Cognitive Systems Engineering Pillars
MEASURE
MEASURE
MEASURE
DATA
DATA
DATA
TRUTH
TRUTH
TRUTH
Lora Aroyo @laroyo http://lora-aroyo.org
Cognitive Systems Engineering Pillars
MEASURE
MEASURE
MEASURE
DATA
DATA
DATA
TRUTH
TRUTH
TRUTH
They define the system.
If you change one of them,
the system should change
Lora Aroyo @laroyo http://lora-aroyo.org
Winning Human
Performance
2007 QA Computer System
Grand Champion
Human Performance
Top human
players are
remarkably
good.
Each dot – actual historical human Jeopardy! games
More Confident Less Confident
What it Takes to compete against Top Human Jeopardy! Players
Our Analysis Reveals the Winner’s Cloud
Lora Aroyo @laroyo http://lora-aroyo.org
2007 QA Computer System
In 2007, we committed to
making a Huge Leap!
Computers?
Not So Good.
Winning Human
Performance
Grand Champion
Human Performance
What it Takes to compete against Top Human Jeopardy! Players
Our Analysis Reveals the Winner’s Cloud
Each dot – actual historical human Jeopardy! games
More Confident Less Confident
Lora Aroyo @laroyo http://lora-aroyo.org
Precision
% of Answered
Deep QA: Incremental Progress
6/2007-11/2010
Now Playing in the
Winners Cloud
11/2010
04/2010
10/2009
5/2009
12/2008
8/2008
5/2008
12/2007
Baseline
Lora Aroyo @laroyo http://lora-aroyo.org
… and WATSON won
Lora Aroyo @laroyo http://lora-aroyo.org
Watch…
Day 2 (15 Feb 2011)
Day 1 (14 Feb 2011)
Also on YouTube:
https://www.youtube.com/watch?v=i-vMW_Ce51w
Lora Aroyo @laroyo http://lora-aroyo.org
Crowd-Watson team 2013
http://lora-aroyo.org @laroyo 26
Lora Aroyo @laroyo http://lora-aroyo.org
CrowdTruth team growing, 2014
http://lora-aroyo.org @laroyo 27
Lora Aroyo @laroyo http://lora-aroyo.org
http://lora-aroyo.org @laroyo 28
Lora Aroyo @laroyo http://lora-aroyo.org
http://lora-aroyo.org @laroyo 29
Lora Aroyo @laroyo http://lora-aroyo.org
Lora Aroyo @laroyo http://lora-aroyo.org
Crowd-Watson 2015
Lora Aroyo @laroyo http://lora-aroyo.org
http://crowdtruth.org

More Related Content

Viewers also liked

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchLora Aroyo
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+Lora Aroyo
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumLora Aroyo
 
CrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsCrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsLora Aroyo
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Lora Aroyo
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyLora Aroyo
 
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?CrowdTruth
 
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Lora Aroyo
 
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 Lora Aroyo
 
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataVisualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataCrowdTruth
 
Crowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain QuestionsCrowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain QuestionsBenjamin Timmermans
 
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Lora Aroyo
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordLora Aroyo
 
ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016oanainel
 
Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Benjamin Timmermans
 
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...CrowdTruth
 
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Lora Aroyo
 
Towards Better Media Understanding and Searchability
Towards Better Media Understanding and SearchabilityTowards Better Media Understanding and Searchability
Towards Better Media Understanding and Searchabilityoanainel
 
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Lora Aroyo
 

Viewers also liked (19)

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New SearchSXSW2017 @NewDutchMedia Talk: Exploration is the New Search
SXSW2017 @NewDutchMedia Talk: Exploration is the New Search
 
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons  with DIVE+
DIVE+ @ NLeSymposium 2015: Towards New Cultural Commons with DIVE+
 
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the RijksmuseumStitch by Stitch: Annotating Fashion at the Rijksmuseum
Stitch by Stitch: Annotating Fashion at the Rijksmuseum
 
CrowdTruth for Digital Hermeneutics
CrowdTruth for Digital HermeneuticsCrowdTruth for Digital Hermeneutics
CrowdTruth for Digital Hermeneutics
 
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...
 
UMAP 2016 Opening Ceremony
UMAP 2016 Opening CeremonyUMAP 2016 Opening Ceremony
UMAP 2016 Opening Ceremony
 
Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?Gamification of crowdsourcing tasks: What motivates a medical expert?
Gamification of crowdsourcing tasks: What motivates a medical expert?
 
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)
 
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014
 
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing DataVisualization of Disagreement-based Quality Metrics of Crowdsourcing Data
Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data
 
Crowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain QuestionsCrowdsourcing Disagreement on Open-Domain Questions
Crowdsourcing Disagreement on Open-Domain Questions
 
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)
 
Semantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @OxfordSemantic Digital Humanities Workshop 2015 @Oxford
Semantic Digital Humanities Workshop 2015 @Oxford
 
ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016ESWC - PhD Symposium 2016
ESWC - PhD Symposium 2016
 
Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...Exploiting disagreement through open ended tasks for capturing interpretation...
Exploiting disagreement through open ended tasks for capturing interpretation...
 
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...
 
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...
 
Towards Better Media Understanding and Searchability
Towards Better Media Understanding and SearchabilityTowards Better Media Understanding and Searchability
Towards Better Media Understanding and Searchability
 
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014
 

Similar to Closing Event - Watson Innovation Course

Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AILora Aroyo
 
Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Lora Aroyo
 
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...Distilled
 
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...Codemotion
 
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...Distilled
 
Pietro leo - Active Intelligence
Pietro leo - Active IntelligencePietro leo - Active Intelligence
Pietro leo - Active IntelligencePietro Leo
 
2005 BT Technology Timeline
2005 BT Technology Timeline2005 BT Technology Timeline
2005 BT Technology Timelinebutest
 
Mobile Games Market Trends Q1 2018 from Priori Data
Mobile Games Market Trends Q1 2018 from Priori DataMobile Games Market Trends Q1 2018 from Priori Data
Mobile Games Market Trends Q1 2018 from Priori DataBenjamin Nolan
 
Artificial intelligence for humans… #AIDC2018 keynote
Artificial intelligence for humans… #AIDC2018 keynoteArtificial intelligence for humans… #AIDC2018 keynote
Artificial intelligence for humans… #AIDC2018 keynoteChristian Heilmann
 
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019Patrizio Pelliccione
 
Harkable Day of Innovation Oct 2013 - Hark in the Park
Harkable Day of Innovation Oct 2013 - Hark in the ParkHarkable Day of Innovation Oct 2013 - Hark in the Park
Harkable Day of Innovation Oct 2013 - Hark in the ParkHarkable
 
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneMy ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneLora Aroyo
 
AI, Innovation & Ethics in Marketing by PR Smith, founder of SOSTAC® Plans ...
AI, Innovation & Ethics in Marketing  by PR Smith, founder of SOSTAC® Plans  ...AI, Innovation & Ethics in Marketing  by PR Smith, founder of SOSTAC® Plans  ...
AI, Innovation & Ethics in Marketing by PR Smith, founder of SOSTAC® Plans ...Institutul de Marketing
 

Similar to Closing Event - Watson Innovation Course (14)

Data excellence: Better data for better AI
Data excellence: Better data for better AIData excellence: Better data for better AI
Data excellence: Better data for better AI
 
Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)Harnessing Human Semantics at Scale (updated)
Harnessing Human Semantics at Scale (updated)
 
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
SearchLove London 2017 | Rob Bucci | Reverse-Engineering Google's Research on...
 
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
Christian Heilmann - Building human interfaces powered by AI - Codemotion Ber...
 
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
SearchLove San Diego 2017 | Rob Bucci | Snooping Into Google's Insights on Se...
 
Pietro leo - Active Intelligence
Pietro leo - Active IntelligencePietro leo - Active Intelligence
Pietro leo - Active Intelligence
 
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
Snooping into Google’s Insights on Searcher Intent: How to Target the Right C...
 
2005 BT Technology Timeline
2005 BT Technology Timeline2005 BT Technology Timeline
2005 BT Technology Timeline
 
Mobile Games Market Trends Q1 2018 from Priori Data
Mobile Games Market Trends Q1 2018 from Priori DataMobile Games Market Trends Q1 2018 from Priori Data
Mobile Games Market Trends Q1 2018 from Priori Data
 
Artificial intelligence for humans… #AIDC2018 keynote
Artificial intelligence for humans… #AIDC2018 keynoteArtificial intelligence for humans… #AIDC2018 keynote
Artificial intelligence for humans… #AIDC2018 keynote
 
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
Software Engineering for ML/AI, keynote at FAS*/ICAC/SASO 2019
 
Harkable Day of Innovation Oct 2013 - Hark in the Park
Harkable Day of Innovation Oct 2013 - Hark in the ParkHarkable Day of Innovation Oct 2013 - Hark in the Park
Harkable Day of Innovation Oct 2013 - Hark in the Park
 
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort ZoneMy ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
My ESWC 2017 keynote: Disrupting the Semantic Comfort Zone
 
AI, Innovation & Ethics in Marketing by PR Smith, founder of SOSTAC® Plans ...
AI, Innovation & Ethics in Marketing  by PR Smith, founder of SOSTAC® Plans  ...AI, Innovation & Ethics in Marketing  by PR Smith, founder of SOSTAC® Plans  ...
AI, Innovation & Ethics in Marketing by PR Smith, founder of SOSTAC® Plans ...
 

More from Lora Aroyo

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfLora Aroyo
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningLora Aroyo
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumLora Aroyo
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorLora Aroyo
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataLora Aroyo
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumLora Aroyo
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18Lora Aroyo
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsLora Aroyo
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesLora Aroyo
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the LoopLora Aroyo
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...Lora Aroyo
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Lora Aroyo
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityLora Aroyo
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...Lora Aroyo
 
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015Lora Aroyo
 

More from Lora Aroyo (15)

NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdfNeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
NeurIPS2023 Keynote: The Many Faces of Responsible AI.pdf
 
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine LearningCATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
CATS4ML Data Challenge: Crowdsourcing Adverse Test Sets for Machine Learning
 
CHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH SymposiumCHIP Demonstrator presentation @ CATCH Symposium
CHIP Demonstrator presentation @ CATCH Symposium
 
Semantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP DemonstratorSemantic Web Challenge: CHIP Demonstrator
Semantic Web Challenge: CHIP Demonstrator
 
The Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked DataThe Rijksmuseum Collection as Linked Data
The Rijksmuseum Collection as Linked Data
 
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @RijksmuseumKeynote at International Conference of Art Libraries 2018 @Rijksmuseum
Keynote at International Conference of Art Libraries 2018 @Rijksmuseum
 
FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18FAIRview: Responsible Video Summarization @NYCML'18
FAIRview: Responsible Video Summarization @NYCML'18
 
Understanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithmsUnderstanding bias in video news & news filtering algorithms
Understanding bias in video news & news filtering algorithms
 
StorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & MachinesStorySourcing: Telling Stories with Humans & Machines
StorySourcing: Telling Stories with Humans & Machines
 
Data Science with Humans in the Loop
Data Science with Humans in the LoopData Science with Humans in the Loop
Data Science with Humans in the Loop
 
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...
 
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017
 
Data Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityData Science with Human in the Loop @Faculty of Science #Leiden University
Data Science with Human in the Loop @Faculty of Science #Leiden University
 
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
#CrowdTruth: Biomedical Data Mining, Modeling & Semantic Integration (BDM2I 2...
 
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015#CrowdTruth: Linked Data for Information Extraction @ISWC2015
#CrowdTruth: Linked Data for Information Extraction @ISWC2015
 

Recently uploaded

Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)wesley chun
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherRemote DBA Services
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...Neo4j
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...Martijn de Jong
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptxHampshireHUG
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfsudhanshuwaghmare1
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slidevu2urc
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘RTylerCroy
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerThousandEyes
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationRadu Cotescu
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesBoston Institute of Analytics
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024The Digital Insurer
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilV3cube
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Scriptwesley chun
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)Gabriella Davis
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobeapidays
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsMaria Levchenko
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024The Digital Insurer
 

Recently uploaded (20)

Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)Powerful Google developer tools for immediate impact! (2023-24 C)
Powerful Google developer tools for immediate impact! (2023-24 C)
 
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...2024: Domino Containers - The Next Step. News from the Domino Container commu...
2024: Domino Containers - The Next Step. News from the Domino Container commu...
 
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
04-2024-HHUG-Sales-and-Marketing-Alignment.pptx
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Boost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdfBoost Fertility New Invention Ups Success Rates.pdf
Boost Fertility New Invention Ups Success Rates.pdf
 
Histor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slideHistor y of HAM Radio presentation slide
Histor y of HAM Radio presentation slide
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected WorkerHow to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
 
Scaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organizationScaling API-first – The story of a global engineering organization
Scaling API-first – The story of a global engineering organization
 
HTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation StrategiesHTML Injection Attacks: Impact and Mitigation Strategies
HTML Injection Attacks: Impact and Mitigation Strategies
 
Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024Partners Life - Insurer Innovation Award 2024
Partners Life - Insurer Innovation Award 2024
 
Developing An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of BrazilDeveloping An App To Navigate The Roads of Brazil
Developing An App To Navigate The Roads of Brazil
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)A Domino Admins Adventures (Engage 2024)
A Domino Admins Adventures (Engage 2024)
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
Handwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed textsHandwritten Text Recognition for manuscripts and early printed texts
Handwritten Text Recognition for manuscripts and early printed texts
 
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024
 

Closing Event - Watson Innovation Course

  • 1. Lora Aroyo @laroyo http://lora-aroyo.org Turning Sci-Fi into Reality Lora Aroyo
  • 2. Lora Aroyo @laroyo http://lora-aroyo.org Sci-Fi in 60’s, 70’s & 80’s
  • 3. Lora Aroyo @laroyo http://lora-aroyo.org Sci-Fi in 2002
  • 4. Lora Aroyo @laroyo http://lora-aroyo.org Sci-Fi in 2013
  • 5. Lora Aroyo @laroyo http://lora-aroyo.org Sci-Fi in 2015
  • 6. Lora Aroyo @laroyo http://lora-aroyo.org Turning Sci-Fi into Reality
  • 7. Lora Aroyo @laroyo http://lora-aroyo.org scientific challenge in 1997
  • 8. Lora Aroyo @laroyo http://lora-aroyo.org chess finite, mathematically well-defined search space limited number of moves & states grounded in explicit, unambiguous math rules
  • 9. Lora Aroyo @laroyo http://lora-aroyo.org IBM Confidential scientific challenge in 2011
  • 10. Lora Aroyo @laroyo http://lora-aroyo.org IBM Confidential natural language ambiguous, contextual & implicit grounded only in human cognition infinite number of ways to express same meaning
  • 11. Lora Aroyo @laroyo http://lora-aroyo.org http://lora-aroyo.org @laroyo 11 keeps expanding to new domains
  • 12. Lora Aroyo @laroyo http://lora-aroyo.org
  • 13. Lora Aroyo @laroyo http://lora-aroyo.org http://lora-aroyo.org @laroyo 13
  • 14. Lora Aroyo @laroyo http://lora-aroyo.org 1. Expand “EXPANDs human cognition - makes the jobs we do easier, like a cognitive prosthesis. Especially when dealing with processing massive data, or data that requires human interpretation”
  • 15. Lora Aroyo @laroyo http://lora-aroyo.org “LEARNs as you use it – most machine errors are easy for a human to detect. We can instrument usage of systems to better understand the system and the problem it solves” 2. Learn
  • 16. Lora Aroyo @laroyo http://lora-aroyo.org “INTERACTs naturally - bringing machines closer to their users by letting them understand natural language, spoken or written, be able to process images & videos. These simple human problems are extremely complex for machines, but are hallmarks of a new computing era.” 3. Interact
  • 17. Lora Aroyo @laroyo http://lora-aroyo.org ▪  A new software paradigm has emerged –  Increasingly, computational tasks require inexact solutions that combine multiple methods in unpredictable ways ▪  Knowledge is not the destination –  Watson does not answer a question by translating natural language input into formally represented knowledge and simply running queries against this knowledge ▪  Machine intelligence is not human intelligence –  The difference is most notable in the mistakes they make When Building Cognitive Computing Systems … http://www.yovisto.com/video/19674 Chris Welty’s keynote at WWW2012
  • 18. Lora Aroyo @laroyo http://lora-aroyo.org Cognitive Systems can fail and still be useful A dramatic shift for software systems Focus on improvement, not perfection We must be able to answer, “Is it better?” Cognitive Computing: It’s OK to be wrong!
  • 19. Lora Aroyo @laroyo http://lora-aroyo.org Cognitive Systems Engineering Pillars MEASURE MEASURE MEASURE DATA DATA DATA TRUTH TRUTH TRUTH
  • 20. Lora Aroyo @laroyo http://lora-aroyo.org Cognitive Systems Engineering Pillars MEASURE MEASURE MEASURE DATA DATA DATA TRUTH TRUTH TRUTH They define the system. If you change one of them, the system should change
  • 21. Lora Aroyo @laroyo http://lora-aroyo.org Winning Human Performance 2007 QA Computer System Grand Champion Human Performance Top human players are remarkably good. Each dot – actual historical human Jeopardy! games More Confident Less Confident What it Takes to compete against Top Human Jeopardy! Players Our Analysis Reveals the Winner’s Cloud
  • 22. Lora Aroyo @laroyo http://lora-aroyo.org 2007 QA Computer System In 2007, we committed to making a Huge Leap! Computers? Not So Good. Winning Human Performance Grand Champion Human Performance What it Takes to compete against Top Human Jeopardy! Players Our Analysis Reveals the Winner’s Cloud Each dot – actual historical human Jeopardy! games More Confident Less Confident
  • 23. Lora Aroyo @laroyo http://lora-aroyo.org Precision % of Answered Deep QA: Incremental Progress 6/2007-11/2010 Now Playing in the Winners Cloud 11/2010 04/2010 10/2009 5/2009 12/2008 8/2008 5/2008 12/2007 Baseline
  • 24. Lora Aroyo @laroyo http://lora-aroyo.org … and WATSON won
  • 25. Lora Aroyo @laroyo http://lora-aroyo.org Watch… Day 2 (15 Feb 2011) Day 1 (14 Feb 2011) Also on YouTube: https://www.youtube.com/watch?v=i-vMW_Ce51w
  • 26. Lora Aroyo @laroyo http://lora-aroyo.org Crowd-Watson team 2013 http://lora-aroyo.org @laroyo 26
  • 27. Lora Aroyo @laroyo http://lora-aroyo.org CrowdTruth team growing, 2014 http://lora-aroyo.org @laroyo 27
  • 28. Lora Aroyo @laroyo http://lora-aroyo.org http://lora-aroyo.org @laroyo 28
  • 29. Lora Aroyo @laroyo http://lora-aroyo.org http://lora-aroyo.org @laroyo 29
  • 30. Lora Aroyo @laroyo http://lora-aroyo.org
  • 31. Lora Aroyo @laroyo http://lora-aroyo.org Crowd-Watson 2015
  • 32. Lora Aroyo @laroyo http://lora-aroyo.org http://crowdtruth.org