SlideShare a Scribd company logo
RESPONSIBLEINTELLIGENCE
1
RESPONSIBLE INTELLIGENCE
ARI KAMLANI
akamlani@gmail.com @akamlani
Data Scientist | Technology Advisor
RESPONSIBLEINTELLIGENCE
2
IS YOUR MODEL USEFUL?
MODEL FIDELITY
“ All Models are Wrong,
but some are Useful...”
- George E.P. Box, 1976
If a Model is not Interpretable,
how useful is it?
Some Models are roughly
accurate, but perform terrible
on live data…
ari kamlani ©2018
RESPONSIBLEINTELLIGENCE
3
PIPELINE
Deploy MonitorEvaluationModeling
Pre-Process
more than metrics…
Does the model provide valuable insights?
ari kamlani ©2018
RESPONSIBLEINTELLIGENCE
4
DATA QUALITY
• Data Collection
• Is Data Representative and Unbiased – No Proxies
• Are Ground Truth (GT) Annotations Accurate
• Collecting Rare Event Data can be Expensive
• Examples @JPMC
• Targeted Campaign Segment Populations
• Customer Preference Distributions Change over Time
ari kamlani ©2018
RESPONSIBLEINTELLIGENCE
5
IS YOUR MODEL PERFORMING AS INTENDED?
MODEL FAIRNESS
• Fairness vs Pattern Detection
• Are Predictions Sensible?
• Did the Model learn the right Features?
Would you Trade Accuracy for Explanations?
Simple(Linear) vs Complex/Black-Box(Non-Linear)
Plausible Justification for behavior
ari kamlani ©2018
RESPONSIBLEINTELLIGENCE
6
ARE THE PREDICTIONS REASONABLE?
TRUSTING A MODEL
• Model Governance/Audit
• Highly Regularized Industries need to qualify Risk
• Different Risks in FOMO Opportunities vs Safety as in SDC
• Performance
• Achievable Upper limit as denoted by Domain SME
• Model Robustness should account for Dynamic User Behavior
ari kamlani ©2018
RESPONSIBLEINTELLIGENCE
7
BRINGING DEBUGGABILITY INTO MACHINE INTELLIGENCE
INTELLIGENT EXPLANATIONS
ari kamlani ©2018
RESPONSIBLEINTELLIGENCE
8
• Semantic Concept Mapping
• Describe a Scenario Modeled as a set of Events and Constraints
• Producing a Contextual pattern representing underlying Behavior
CAN A HUMAN UNDERSTAND THE UNDERLYING SEMANTIC CONCEPT?
ONTOLOGIES
ari kamlani ©2018
RESPONSIBLEINTELLIGENCE
9
MONITORING MODELS
• Contrast Offline vs Online ~ Digital Twin (in Industrial IoT)
• Continuous Benchmark w/live data
• Data Provenance ~ Track your Experiments & Data Sources
Have Distributions changed? When to Retire a Model?
ari kamlani ©2018

More Related Content

Similar to Responsible intelligence

AI in the Real World: Challenges, and Risks and how to handle them?
AI in the Real World: Challenges, and Risks and how to handle them?AI in the Real World: Challenges, and Risks and how to handle them?
AI in the Real World: Challenges, and Risks and how to handle them?
Srinath Perera
 
MLSD18. Supervised Summary
MLSD18. Supervised SummaryMLSD18. Supervised Summary
MLSD18. Supervised Summary
BigML, Inc
 
Customer Intelligence & Analytics - Part I
Customer Intelligence & Analytics - Part ICustomer Intelligence & Analytics - Part I
Customer Intelligence & Analytics - Part I
Vivastream
 
Agile Analytics: Delivering on Promises by Atif Abdul Rahman
Agile Analytics: Delivering on Promises by Atif Abdul RahmanAgile Analytics: Delivering on Promises by Atif Abdul Rahman
Agile Analytics: Delivering on Promises by Atif Abdul Rahman
Agile ME
 
Agile Analytics
Agile AnalyticsAgile Analytics
Agile Analytics
Atif Shaikh
 
Evolving Your Email Targeting for Increased Revenue
Evolving Your Email Targeting for Increased RevenueEvolving Your Email Targeting for Increased Revenue
Evolving Your Email Targeting for Increased Revenue
Yes Lifecycle Marketing
 
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
StampedeCon
 
Lecture notes on being Data-Driven and doing Data Science
Lecture notes on being Data-Driven and doing Data Science Lecture notes on being Data-Driven and doing Data Science
Lecture notes on being Data-Driven and doing Data Science
Johan Himberg
 
Why is your model stuck in the lab? How to move your model from the lab to pr...
Why is your model stuck in the lab? How to move your model from the lab to pr...Why is your model stuck in the lab? How to move your model from the lab to pr...
Why is your model stuck in the lab? How to move your model from the lab to pr...
Data Con LA
 
Big Data Day Baku 2016 - Imran Baghirov
Big Data Day Baku 2016  - Imran BaghirovBig Data Day Baku 2016  - Imran Baghirov
Big Data Day Baku 2016 - Imran Baghirov
Imran Baghirov
 
GoDataDriven Giovanni Lanzani
GoDataDriven Giovanni LanzaniGoDataDriven Giovanni Lanzani
GoDataDriven Giovanni Lanzani
TalentEvent
 
Closing the Programmatic Loop
Closing the Programmatic LoopClosing the Programmatic Loop
Closing the Programmatic Loop
MediaPost
 
Giovanni Lanzani GoDataDriven
Giovanni Lanzani GoDataDrivenGiovanni Lanzani GoDataDriven
Giovanni Lanzani GoDataDriven
BigDataExpo
 
Apply AI Finance Webinar.pdf
Apply AI Finance Webinar.pdfApply AI Finance Webinar.pdf
Apply AI Finance Webinar.pdf
tevinek
 
How Will Your ML Project Fail
How Will Your ML Project FailHow Will Your ML Project Fail
How Will Your ML Project Fail
Elena Samuylova
 
BigML Education - OptiML
BigML Education - OptiMLBigML Education - OptiML
BigML Education - OptiML
BigML, Inc
 
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Matt Stubbs
 
ProductTank: Nairobi - 'Turning Data into Insights' talk by Tristan Aubert, D...
ProductTank: Nairobi - 'Turning Data into Insights' talk by Tristan Aubert, D...ProductTank: Nairobi - 'Turning Data into Insights' talk by Tristan Aubert, D...
ProductTank: Nairobi - 'Turning Data into Insights' talk by Tristan Aubert, D...
Amos Njoroge
 
What is agile coaching?
What is agile coaching?What is agile coaching?
What is agile coaching?
Pierre E. NEIS
 
Community-Assisted Software Engineering Decision Making
Community-Assisted Software Engineering Decision MakingCommunity-Assisted Software Engineering Decision Making
Community-Assisted Software Engineering Decision Making
gregoryg
 

Similar to Responsible intelligence (20)

AI in the Real World: Challenges, and Risks and how to handle them?
AI in the Real World: Challenges, and Risks and how to handle them?AI in the Real World: Challenges, and Risks and how to handle them?
AI in the Real World: Challenges, and Risks and how to handle them?
 
MLSD18. Supervised Summary
MLSD18. Supervised SummaryMLSD18. Supervised Summary
MLSD18. Supervised Summary
 
Customer Intelligence & Analytics - Part I
Customer Intelligence & Analytics - Part ICustomer Intelligence & Analytics - Part I
Customer Intelligence & Analytics - Part I
 
Agile Analytics: Delivering on Promises by Atif Abdul Rahman
Agile Analytics: Delivering on Promises by Atif Abdul RahmanAgile Analytics: Delivering on Promises by Atif Abdul Rahman
Agile Analytics: Delivering on Promises by Atif Abdul Rahman
 
Agile Analytics
Agile AnalyticsAgile Analytics
Agile Analytics
 
Evolving Your Email Targeting for Increased Revenue
Evolving Your Email Targeting for Increased RevenueEvolving Your Email Targeting for Increased Revenue
Evolving Your Email Targeting for Increased Revenue
 
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
How to Talk about AI to Non-analaysts - Stampedecon AI Summit 2017
 
Lecture notes on being Data-Driven and doing Data Science
Lecture notes on being Data-Driven and doing Data Science Lecture notes on being Data-Driven and doing Data Science
Lecture notes on being Data-Driven and doing Data Science
 
Why is your model stuck in the lab? How to move your model from the lab to pr...
Why is your model stuck in the lab? How to move your model from the lab to pr...Why is your model stuck in the lab? How to move your model from the lab to pr...
Why is your model stuck in the lab? How to move your model from the lab to pr...
 
Big Data Day Baku 2016 - Imran Baghirov
Big Data Day Baku 2016  - Imran BaghirovBig Data Day Baku 2016  - Imran Baghirov
Big Data Day Baku 2016 - Imran Baghirov
 
GoDataDriven Giovanni Lanzani
GoDataDriven Giovanni LanzaniGoDataDriven Giovanni Lanzani
GoDataDriven Giovanni Lanzani
 
Closing the Programmatic Loop
Closing the Programmatic LoopClosing the Programmatic Loop
Closing the Programmatic Loop
 
Giovanni Lanzani GoDataDriven
Giovanni Lanzani GoDataDrivenGiovanni Lanzani GoDataDriven
Giovanni Lanzani GoDataDriven
 
Apply AI Finance Webinar.pdf
Apply AI Finance Webinar.pdfApply AI Finance Webinar.pdf
Apply AI Finance Webinar.pdf
 
How Will Your ML Project Fail
How Will Your ML Project FailHow Will Your ML Project Fail
How Will Your ML Project Fail
 
BigML Education - OptiML
BigML Education - OptiMLBigML Education - OptiML
BigML Education - OptiML
 
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
Big Data LDN 2018: HOW AUTOMATION CAN ACCELERATE THE DELIVERY OF MACHINE LEAR...
 
ProductTank: Nairobi - 'Turning Data into Insights' talk by Tristan Aubert, D...
ProductTank: Nairobi - 'Turning Data into Insights' talk by Tristan Aubert, D...ProductTank: Nairobi - 'Turning Data into Insights' talk by Tristan Aubert, D...
ProductTank: Nairobi - 'Turning Data into Insights' talk by Tristan Aubert, D...
 
What is agile coaching?
What is agile coaching?What is agile coaching?
What is agile coaching?
 
Community-Assisted Software Engineering Decision Making
Community-Assisted Software Engineering Decision MakingCommunity-Assisted Software Engineering Decision Making
Community-Assisted Software Engineering Decision Making
 

Recently uploaded

University of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma TranscriptUniversity of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma Transcript
soxrziqu
 
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
74nqk8xf
 
Influence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business PlanInfluence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business Plan
jerlynmaetalle
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Kiwi Creative
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
rwarrenll
 
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
zsjl4mimo
 
State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023
kuntobimo2016
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
Lars Albertsson
 
Everything you wanted to know about LIHTC
Everything you wanted to know about LIHTCEverything you wanted to know about LIHTC
Everything you wanted to know about LIHTC
Roger Valdez
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
74nqk8xf
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
Social Samosa
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
manishkhaire30
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
AlessioFois2
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
Sm321
 
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
g4dpvqap0
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
Timothy Spann
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
Timothy Spann
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
nyfuhyz
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
aqzctr7x
 

Recently uploaded (20)

University of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma TranscriptUniversity of New South Wales degree offer diploma Transcript
University of New South Wales degree offer diploma Transcript
 
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
一比一原版(牛布毕业证书)牛津布鲁克斯大学毕业证如何办理
 
Influence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business PlanInfluence of Marketing Strategy and Market Competition on Business Plan
Influence of Marketing Strategy and Market Competition on Business Plan
 
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging DataPredictably Improve Your B2B Tech Company's Performance by Leveraging Data
Predictably Improve Your B2B Tech Company's Performance by Leveraging Data
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.My burning issue is homelessness K.C.M.O.
My burning issue is homelessness K.C.M.O.
 
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
一比一原版(Harvard毕业证书)哈佛大学毕业证如何办理
 
State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023State of Artificial intelligence Report 2023
State of Artificial intelligence Report 2023
 
End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024End-to-end pipeline agility - Berlin Buzzwords 2024
End-to-end pipeline agility - Berlin Buzzwords 2024
 
Everything you wanted to know about LIHTC
Everything you wanted to know about LIHTCEverything you wanted to know about LIHTC
Everything you wanted to know about LIHTC
 
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
一比一原版(Coventry毕业证书)考文垂大学毕业证如何办理
 
The Ipsos - AI - Monitor 2024 Report.pdf
The  Ipsos - AI - Monitor 2024 Report.pdfThe  Ipsos - AI - Monitor 2024 Report.pdf
The Ipsos - AI - Monitor 2024 Report.pdf
 
Learn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queriesLearn SQL from basic queries to Advance queries
Learn SQL from basic queries to Advance queries
 
A presentation that explain the Power BI Licensing
A presentation that explain the Power BI LicensingA presentation that explain the Power BI Licensing
A presentation that explain the Power BI Licensing
 
Challenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more importantChallenges of Nation Building-1.pptx with more important
Challenges of Nation Building-1.pptx with more important
 
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
一比一原版(爱大毕业证书)爱丁堡大学毕业证如何办理
 
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Dat...
 
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
06-12-2024-BudapestDataForum-BuildingReal-timePipelineswithFLaNK AIM
 
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
一比一原版(UMN文凭证书)明尼苏达大学毕业证如何办理
 
一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理一比一原版(UO毕业证)渥太华大学毕业证如何办理
一比一原版(UO毕业证)渥太华大学毕业证如何办理
 

Responsible intelligence