SlideShare a Scribd company logo
1 of 33
Download to read offline
ML Interpretability: Beyond
Feature Importance
Javier Antorán (ja666@cam.ac.uk)

PhD Student, University of Cambridge

@JaviAC7
Recent Trends in ML
Research and what they
mean for Interpretability
Javier Antorán (ja666@cam.ac.uk)

PhD Student, University of Cambridge

@JaviAC7
Talk Outline
• Existing Approaches to Interpretability

• “Counterfactual” Interpretability

• Uncertainty in ML

• Transparency in ML Systems that Express Uncertainty
with CLUE

• Questions / Feedback
Interpretable Data Driven
Decision Making
• Generalised Linear Models:
μ = w0 + w1x1 + w2x2 + w3x3 . . .
• Decision Trees:
• Importance of is 

• Polarity of is
xi |wi |
xi sign(wi)
Not Very Interpretable Data
Driven Decision Making
• Capture non-linear functions (NNs are universal function approximators)
• Scale to high dimensional problems
• Scale to massive datasets
• Simulate complex systems
• Etc
Feature Importance: LIME
• We approximate non-linear model Locally with Linear Model
μ = fNN(x)
μapprox = w0 + w1x1 + w2x2 + w3x3 . . .
Here explanation is more reliable in thanx1 x0
Lime Explanation is w1, w2, w3 . . .
Feature Importance on
Images
LIME [Ribeiro et. al., 2016] SHAP [Lundberg and Lee, 2017]
• Per class weight vectors forms a “template images” of
positive and negative contributions
• Can become meaningless for strongly non-linear functions
Counterfactual
Explanations to the Rescue!
• Counterfactuals capture the notion what would have
happened if something had been different
• We can ask a similar question: “What features would I need to remove
such that my model’s confidence decreases?”

• Or: “What features would I need to remove such that my model’s
prediction changes?”
• This gives a model-agnostic question we can answer to
provide insights to users. — Explanation has Clear Meaning
Counterfactual Explanations
for Image Classification
Chang et. al., 2018
Uncertainty in ML
People saying AI will
take over the world:
Meanwhile, my
Deep Neural Network:
0.978
Sources of Uncertainty
Have we observed enough data to make confident predictions?
— Model (Epistemic) Uncertainty
°5 0 5
°5
0
5
°10 0 10
°10
0
10
°10 0 10
°10
0
10
0.0
0.7
H(nats)
Is there class overlap in our data? — Noise (Aleatoric) Uncertainty
Bayesian Linear Regression Example:
What is Model Uncertainty (1/3)
Likelihood Prior
++ +
What is Model Uncertainty (2/3)
Neural Net Non-Convex
Optimisation Landscape:
Weights
Loss
What is Model Uncertainty (3/3)
Weights go from single values to probability distributions!
Approximations
Weights
Loss Approximate Distribution
Realistic Loss Landscape
Our Uncertainty Estimates are Almost Always Biased by Our
Approximations
Quantifying Uncertainty
• Classification

• Regression
Predictive
Variance
Low Entropy High Entropy
Uncertainty in Practise
• Robustness in Critical Applications (Driving,
Medical Diagnosis, etc) — Reject Option

• Dataset Building, Safety, Fairness (Identifying
disparities in representation of subgroups in data,
etc )

• Active Learning (Sparse Labels, Drug Discovery)

• Try out Bayesian Deep Learning with our Public Repo
Aleatoric Epistemic
Kendall and Gal, 2017
github.com/JavierAntoran/Bayesian-Neural-Networks
Are Uncertainty Aware
Systems Interpretable?
• Thankfully, Yes!*
• They are as interpretable as regular ML models**
• Uncertainty can help users understand prediction in some cases
Wickstrøm, et. al., 2019
Polyp segmentation example:
**But what about when our ML
System Doesn’t Know the Answer?
x*
Accept a Certain
Prediction
Reject an Uncertain Prediction
?
Get Explanation
ML User / Practitioner Workflow:
• LIME [Ribeiro et al., 2017]
• SHAP [Lundberg et al., 2017]
• Integrated Gradients
[Sundararajan et al., 2017]
• FIDO [Chang et al., 2017]
Explaining Uncertainty
Estimates
• We would like to highlight the evidence for each class.
• What if there is conflicting evidence (noise) or a lack of evidence for
predefined classes (model uncertainty)?
• Recall that Uncertainty Estimates are Non-Linear even for simplest models.
• Problem is well posed again when using Counterfactuals
°5 0 5
°5
0
5
°10 0 10
°10
0
10
°10 0 10
°10
0
10
0.0
0.7
H(nats)
How can we Ensure that
Counterfactuals are Relevant?
• Adversarial examples
for uncertainty
• Adversarial examples
Lets Look at Data Driven
Drug Discovery
• We can restrict hypothesis space to manifold captured
by generative model: this ensures relevant proposals
Gómez-Bombarelli et. al., 2018
Lets do the same for
Counterfactuals
CLUE: Counterfactual Latent
Uncertainty Explanations
“What is the smallest change we need to make to an
input, while staying in-distribution, such that our
model produces more certain predictions?”
The CLUE Algorithm
Iterative Optimisation:
Displaying CLUEs to Users
Comparing CLUE to Feature
Importance (LIME / SHAP)
• In high uncertainty scenarios, it is difficult to build an
explanation in terms of the provided information (features)

• CLUE’s counterfactual nature allows it to add new information
User Study: Setup (1/2)
Human Simulability: Users are shown context examples and
are tasked with predicting model behaviour on new datapoint.
User Study: Setup (2/2)
Tasks:
• COMPAS (Criminal Recidivism Prediction, 7 dim)
• LAST (Academic Performance Prediction, 4 dim)
Users:
• University Students with ML experience
• 10 Users per approach, 10 Questions per Dataset
User Study: Results
Method N. participants Accuracy (%)
Random 10 61.67
Sensitivity 10 52.78
Human 10 62.22
CLUE 10 82.22
CLUE’s improvement over all other approaches is statistically significant
(Using Nemenyi test for average ranks across test questions)
Now with Images:
We modify the MNIST train set to introduce Out Of
Distribution (model) uncertainty.
Method N. participants Accuracy
Unc. 5 0.67
CLUE 5 0.88
Thanks to my Collaborators!
Javier Antorán
ja666@cam.ac.uk
Umang Bhatt
usb20@cam.ac.uk
José Miguel
Hernández-Lobato
jmh233@cam.ac.uk
Adrian Weller
aw665@cam.ac.uk
Tameem Adel
tah47@cam.ac.uk
Questions?
Read The Full Paper at:
arxiv.org/abs/2006.06848
See More of my Research (+slides):
javierantoran.github.io/about/
Contact Me:
ja666@cam.ac.uk

More Related Content

What's hot

Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
Gradient-Based Meta-Learning with Learned Layerwise Metric and SubspaceGradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
Gradient-Based Meta-Learning with Learned Layerwise Metric and SubspaceYoonho Lee
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine LearningShimi Bandiel
 
Classification case study + intro to cnn
Classification case study + intro to cnnClassification case study + intro to cnn
Classification case study + intro to cnnVincent Tatan
 
Interpretable machine learning : Methods for understanding complex models
Interpretable machine learning : Methods for understanding complex modelsInterpretable machine learning : Methods for understanding complex models
Interpretable machine learning : Methods for understanding complex modelsManojit Nandi
 
[Revised] Intro to CNN
[Revised] Intro to CNN[Revised] Intro to CNN
[Revised] Intro to CNNVincent Tatan
 
Meta Learning Shared Hierarchies
Meta Learning Shared HierarchiesMeta Learning Shared Hierarchies
Meta Learning Shared HierarchiesYoonho Lee
 
Brian Mac Namee - Predict Webinar 3 - Short Intro to Deep Learing
Brian Mac Namee - Predict Webinar 3 - Short Intro to Deep LearingBrian Mac Namee - Predict Webinar 3 - Short Intro to Deep Learing
Brian Mac Namee - Predict Webinar 3 - Short Intro to Deep LearingPredict Conference
 
Introduction to ml ops in daily apps
Introduction to ml ops in daily appsIntroduction to ml ops in daily apps
Introduction to ml ops in daily appsVincent Tatan
 
Introduction to Few shot learning
Introduction to Few shot learningIntroduction to Few shot learning
Introduction to Few shot learningRidge-i, Inc.
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsbutest
 
Approximated and User Steerable tSNE for Progressive Visual Analytics
Approximated and User Steerable tSNE for Progressive Visual AnalyticsApproximated and User Steerable tSNE for Progressive Visual Analytics
Approximated and User Steerable tSNE for Progressive Visual AnalyticsNicola Pezzotti
 
Machine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative ModelsMachine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative Modelsbutest
 
MachineLlearning introduction
MachineLlearning introductionMachineLlearning introduction
MachineLlearning introductionThe IOT Academy
 
Machine learning basics
Machine learning basics Machine learning basics
Machine learning basics Akanksha Bali
 
Explainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretableExplainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretableAditya Bhattacharya
 

What's hot (17)

Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
Gradient-Based Meta-Learning with Learned Layerwise Metric and SubspaceGradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
Gradient-Based Meta-Learning with Learned Layerwise Metric and Subspace
 
Introduction to Machine Learning
Introduction to Machine LearningIntroduction to Machine Learning
Introduction to Machine Learning
 
Classification case study + intro to cnn
Classification case study + intro to cnnClassification case study + intro to cnn
Classification case study + intro to cnn
 
Interpretable machine learning : Methods for understanding complex models
Interpretable machine learning : Methods for understanding complex modelsInterpretable machine learning : Methods for understanding complex models
Interpretable machine learning : Methods for understanding complex models
 
[Revised] Intro to CNN
[Revised] Intro to CNN[Revised] Intro to CNN
[Revised] Intro to CNN
 
Meta Learning Shared Hierarchies
Meta Learning Shared HierarchiesMeta Learning Shared Hierarchies
Meta Learning Shared Hierarchies
 
Brian Mac Namee - Predict Webinar 3 - Short Intro to Deep Learing
Brian Mac Namee - Predict Webinar 3 - Short Intro to Deep LearingBrian Mac Namee - Predict Webinar 3 - Short Intro to Deep Learing
Brian Mac Namee - Predict Webinar 3 - Short Intro to Deep Learing
 
Introduction to ml ops in daily apps
Introduction to ml ops in daily appsIntroduction to ml ops in daily apps
Introduction to ml ops in daily apps
 
Introduction to Few shot learning
Introduction to Few shot learningIntroduction to Few shot learning
Introduction to Few shot learning
 
Icml2017 overview
Icml2017 overviewIcml2017 overview
Icml2017 overview
 
Cristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical modelsCristopher M. Bishop's tutorial on graphical models
Cristopher M. Bishop's tutorial on graphical models
 
Approximated and User Steerable tSNE for Progressive Visual Analytics
Approximated and User Steerable tSNE for Progressive Visual AnalyticsApproximated and User Steerable tSNE for Progressive Visual Analytics
Approximated and User Steerable tSNE for Progressive Visual Analytics
 
Machine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative ModelsMachine Learning: Generative and Discriminative Models
Machine Learning: Generative and Discriminative Models
 
Introduction to ml
Introduction to mlIntroduction to ml
Introduction to ml
 
MachineLlearning introduction
MachineLlearning introductionMachineLlearning introduction
MachineLlearning introduction
 
Machine learning basics
Machine learning basics Machine learning basics
Machine learning basics
 
Explainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretableExplainable AI - making ML and DL models more interpretable
Explainable AI - making ML and DL models more interpretable
 

Similar to Rsqrd AI - ML Interpretability: Beyond Feature Importance

Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Ian Morgan
 
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Bayes Nets meetup London
 
Model Evaluation in the land of Deep Learning
Model Evaluation in the land of Deep LearningModel Evaluation in the land of Deep Learning
Model Evaluation in the land of Deep LearningPramit Choudhary
 
[Explained] "Partial Success in Closing the Gap between Human and Machine Vis...
[Explained] "Partial Success in Closing the Gap between Human and Machine Vis...[Explained] "Partial Success in Closing the Gap between Human and Machine Vis...
[Explained] "Partial Success in Closing the Gap between Human and Machine Vis...Sou Yoshihara
 
Discussion of Persi Diaconis' lecture at ISBA 2016
Discussion of Persi Diaconis' lecture at ISBA 2016Discussion of Persi Diaconis' lecture at ISBA 2016
Discussion of Persi Diaconis' lecture at ISBA 2016Christian Robert
 
​​Explainability in AI and Recommender systems: let’s make it interactive!
​​Explainability in AI and Recommender systems: let’s make it interactive!​​Explainability in AI and Recommender systems: let’s make it interactive!
​​Explainability in AI and Recommender systems: let’s make it interactive!Eindhoven University of Technology / JADS
 
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdfVincenzo Lomonaco
 
Get hands-on with Explainable AI at Machine Learning Interpretability(MLI) Gym!
Get hands-on with Explainable AI at Machine Learning Interpretability(MLI) Gym!Get hands-on with Explainable AI at Machine Learning Interpretability(MLI) Gym!
Get hands-on with Explainable AI at Machine Learning Interpretability(MLI) Gym!Sri Ambati
 
Hacking Predictive Modeling - RoadSec 2018
Hacking Predictive Modeling - RoadSec 2018Hacking Predictive Modeling - RoadSec 2018
Hacking Predictive Modeling - RoadSec 2018HJ van Veen
 
Module 3: Linear Regression
Module 3:  Linear RegressionModule 3:  Linear Regression
Module 3: Linear RegressionSara Hooker
 
Lecture 2 - Introduction to Machine Learning, a lecture in subject module Sta...
Lecture 2 - Introduction to Machine Learning, a lecture in subject module Sta...Lecture 2 - Introduction to Machine Learning, a lecture in subject module Sta...
Lecture 2 - Introduction to Machine Learning, a lecture in subject module Sta...Maninda Edirisooriya
 
17- Kernels and Clustering.pptx
17- Kernels and Clustering.pptx17- Kernels and Clustering.pptx
17- Kernels and Clustering.pptxssuser2023c6
 
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Daniel Katz
 
Deep learning: Cutting through the Myths and Hype
Deep learning: Cutting through the Myths and HypeDeep learning: Cutting through the Myths and Hype
Deep learning: Cutting through the Myths and HypeSiby Jose Plathottam
 
Linear models for data science
Linear models for data scienceLinear models for data science
Linear models for data scienceBrad Klingenberg
 
Spsshelp 100608163328-phpapp01
Spsshelp 100608163328-phpapp01Spsshelp 100608163328-phpapp01
Spsshelp 100608163328-phpapp01Henock Beyene
 
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...GeekPwn Keen
 
Statistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignoreStatistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignoreTuri, Inc.
 

Similar to Rsqrd AI - ML Interpretability: Beyond Feature Importance (20)

Declarative data analysis
Declarative data analysisDeclarative data analysis
Declarative data analysis
 
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
 
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
Professor Steve Roberts; The Bayesian Crowd: scalable information combinati...
 
Model Evaluation in the land of Deep Learning
Model Evaluation in the land of Deep LearningModel Evaluation in the land of Deep Learning
Model Evaluation in the land of Deep Learning
 
[Explained] "Partial Success in Closing the Gap between Human and Machine Vis...
[Explained] "Partial Success in Closing the Gap between Human and Machine Vis...[Explained] "Partial Success in Closing the Gap between Human and Machine Vis...
[Explained] "Partial Success in Closing the Gap between Human and Machine Vis...
 
Discussion of Persi Diaconis' lecture at ISBA 2016
Discussion of Persi Diaconis' lecture at ISBA 2016Discussion of Persi Diaconis' lecture at ISBA 2016
Discussion of Persi Diaconis' lecture at ISBA 2016
 
​​Explainability in AI and Recommender systems: let’s make it interactive!
​​Explainability in AI and Recommender systems: let’s make it interactive!​​Explainability in AI and Recommender systems: let’s make it interactive!
​​Explainability in AI and Recommender systems: let’s make it interactive!
 
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf
2023-08-22 CoLLAs Tutorial - Beyond CIL.pdf
 
Get hands-on with Explainable AI at Machine Learning Interpretability(MLI) Gym!
Get hands-on with Explainable AI at Machine Learning Interpretability(MLI) Gym!Get hands-on with Explainable AI at Machine Learning Interpretability(MLI) Gym!
Get hands-on with Explainable AI at Machine Learning Interpretability(MLI) Gym!
 
Hacking Predictive Modeling - RoadSec 2018
Hacking Predictive Modeling - RoadSec 2018Hacking Predictive Modeling - RoadSec 2018
Hacking Predictive Modeling - RoadSec 2018
 
Module 3: Linear Regression
Module 3:  Linear RegressionModule 3:  Linear Regression
Module 3: Linear Regression
 
Lecture 2 - Introduction to Machine Learning, a lecture in subject module Sta...
Lecture 2 - Introduction to Machine Learning, a lecture in subject module Sta...Lecture 2 - Introduction to Machine Learning, a lecture in subject module Sta...
Lecture 2 - Introduction to Machine Learning, a lecture in subject module Sta...
 
17- Kernels and Clustering.pptx
17- Kernels and Clustering.pptx17- Kernels and Clustering.pptx
17- Kernels and Clustering.pptx
 
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
Legal Analytics Course - Class 6 - Overfitting, Underfitting, & Cross-Validat...
 
Deep learning: Cutting through the Myths and Hype
Deep learning: Cutting through the Myths and HypeDeep learning: Cutting through the Myths and Hype
Deep learning: Cutting through the Myths and Hype
 
Linear models for data science
Linear models for data scienceLinear models for data science
Linear models for data science
 
Spsshelp 100608163328-phpapp01
Spsshelp 100608163328-phpapp01Spsshelp 100608163328-phpapp01
Spsshelp 100608163328-phpapp01
 
LR2. Summary Day 2
LR2. Summary Day 2LR2. Summary Day 2
LR2. Summary Day 2
 
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
RECENT PROGRESS IN ADVERSARIAL DEEP LEARNING ATTACK AND DEFENSE - Wenbo Guo a...
 
Statistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignoreStatistics in the age of data science, issues you can not ignore
Statistics in the age of data science, issues you can not ignore
 

Recently uploaded

DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.Curtis Poe
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxBkGupta21
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLScyllaDB
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????blackmambaettijean
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxLoriGlavin3
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 

Recently uploaded (20)

DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.How AI, OpenAI, and ChatGPT impact business and software.
How AI, OpenAI, and ChatGPT impact business and software.
 
unit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptxunit 4 immunoblotting technique complete.pptx
unit 4 immunoblotting technique complete.pptx
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Developer Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQLDeveloper Data Modeling Mistakes: From Postgres to NoSQL
Developer Data Modeling Mistakes: From Postgres to NoSQL
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
What is Artificial Intelligence?????????
What is Artificial Intelligence?????????What is Artificial Intelligence?????????
What is Artificial Intelligence?????????
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptxA Deep Dive on Passkeys: FIDO Paris Seminar.pptx
A Deep Dive on Passkeys: FIDO Paris Seminar.pptx
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 

Rsqrd AI - ML Interpretability: Beyond Feature Importance

  • 1. ML Interpretability: Beyond Feature Importance Javier Antorán (ja666@cam.ac.uk) PhD Student, University of Cambridge @JaviAC7
  • 2. Recent Trends in ML Research and what they mean for Interpretability Javier Antorán (ja666@cam.ac.uk) PhD Student, University of Cambridge @JaviAC7
  • 3. Talk Outline • Existing Approaches to Interpretability • “Counterfactual” Interpretability • Uncertainty in ML • Transparency in ML Systems that Express Uncertainty with CLUE • Questions / Feedback
  • 4. Interpretable Data Driven Decision Making • Generalised Linear Models: μ = w0 + w1x1 + w2x2 + w3x3 . . . • Decision Trees: • Importance of is • Polarity of is xi |wi | xi sign(wi)
  • 5. Not Very Interpretable Data Driven Decision Making • Capture non-linear functions (NNs are universal function approximators) • Scale to high dimensional problems • Scale to massive datasets • Simulate complex systems • Etc
  • 6. Feature Importance: LIME • We approximate non-linear model Locally with Linear Model μ = fNN(x) μapprox = w0 + w1x1 + w2x2 + w3x3 . . . Here explanation is more reliable in thanx1 x0 Lime Explanation is w1, w2, w3 . . .
  • 7. Feature Importance on Images LIME [Ribeiro et. al., 2016] SHAP [Lundberg and Lee, 2017] • Per class weight vectors forms a “template images” of positive and negative contributions • Can become meaningless for strongly non-linear functions
  • 8. Counterfactual Explanations to the Rescue! • Counterfactuals capture the notion what would have happened if something had been different • We can ask a similar question: “What features would I need to remove such that my model’s confidence decreases?” • Or: “What features would I need to remove such that my model’s prediction changes?” • This gives a model-agnostic question we can answer to provide insights to users. — Explanation has Clear Meaning
  • 9. Counterfactual Explanations for Image Classification Chang et. al., 2018
  • 10. Uncertainty in ML People saying AI will take over the world: Meanwhile, my Deep Neural Network: 0.978
  • 11. Sources of Uncertainty Have we observed enough data to make confident predictions? — Model (Epistemic) Uncertainty °5 0 5 °5 0 5 °10 0 10 °10 0 10 °10 0 10 °10 0 10 0.0 0.7 H(nats) Is there class overlap in our data? — Noise (Aleatoric) Uncertainty Bayesian Linear Regression Example:
  • 12. What is Model Uncertainty (1/3) Likelihood Prior
  • 13. ++ + What is Model Uncertainty (2/3) Neural Net Non-Convex Optimisation Landscape: Weights Loss
  • 14. What is Model Uncertainty (3/3) Weights go from single values to probability distributions!
  • 15. Approximations Weights Loss Approximate Distribution Realistic Loss Landscape Our Uncertainty Estimates are Almost Always Biased by Our Approximations
  • 16. Quantifying Uncertainty • Classification • Regression Predictive Variance Low Entropy High Entropy
  • 17. Uncertainty in Practise • Robustness in Critical Applications (Driving, Medical Diagnosis, etc) — Reject Option • Dataset Building, Safety, Fairness (Identifying disparities in representation of subgroups in data, etc ) • Active Learning (Sparse Labels, Drug Discovery) • Try out Bayesian Deep Learning with our Public Repo Aleatoric Epistemic Kendall and Gal, 2017 github.com/JavierAntoran/Bayesian-Neural-Networks
  • 18. Are Uncertainty Aware Systems Interpretable? • Thankfully, Yes!* • They are as interpretable as regular ML models** • Uncertainty can help users understand prediction in some cases Wickstrøm, et. al., 2019 Polyp segmentation example:
  • 19. **But what about when our ML System Doesn’t Know the Answer? x* Accept a Certain Prediction Reject an Uncertain Prediction ? Get Explanation ML User / Practitioner Workflow: • LIME [Ribeiro et al., 2017] • SHAP [Lundberg et al., 2017] • Integrated Gradients [Sundararajan et al., 2017] • FIDO [Chang et al., 2017]
  • 20. Explaining Uncertainty Estimates • We would like to highlight the evidence for each class. • What if there is conflicting evidence (noise) or a lack of evidence for predefined classes (model uncertainty)? • Recall that Uncertainty Estimates are Non-Linear even for simplest models. • Problem is well posed again when using Counterfactuals °5 0 5 °5 0 5 °10 0 10 °10 0 10 °10 0 10 °10 0 10 0.0 0.7 H(nats)
  • 21. How can we Ensure that Counterfactuals are Relevant? • Adversarial examples for uncertainty • Adversarial examples
  • 22. Lets Look at Data Driven Drug Discovery • We can restrict hypothesis space to manifold captured by generative model: this ensures relevant proposals Gómez-Bombarelli et. al., 2018
  • 23. Lets do the same for Counterfactuals
  • 24. CLUE: Counterfactual Latent Uncertainty Explanations “What is the smallest change we need to make to an input, while staying in-distribution, such that our model produces more certain predictions?”
  • 27. Comparing CLUE to Feature Importance (LIME / SHAP) • In high uncertainty scenarios, it is difficult to build an explanation in terms of the provided information (features) • CLUE’s counterfactual nature allows it to add new information
  • 28. User Study: Setup (1/2) Human Simulability: Users are shown context examples and are tasked with predicting model behaviour on new datapoint.
  • 29. User Study: Setup (2/2) Tasks: • COMPAS (Criminal Recidivism Prediction, 7 dim) • LAST (Academic Performance Prediction, 4 dim) Users: • University Students with ML experience • 10 Users per approach, 10 Questions per Dataset
  • 30. User Study: Results Method N. participants Accuracy (%) Random 10 61.67 Sensitivity 10 52.78 Human 10 62.22 CLUE 10 82.22 CLUE’s improvement over all other approaches is statistically significant (Using Nemenyi test for average ranks across test questions)
  • 31. Now with Images: We modify the MNIST train set to introduce Out Of Distribution (model) uncertainty. Method N. participants Accuracy Unc. 5 0.67 CLUE 5 0.88
  • 32. Thanks to my Collaborators! Javier Antorán ja666@cam.ac.uk Umang Bhatt usb20@cam.ac.uk José Miguel Hernández-Lobato jmh233@cam.ac.uk Adrian Weller aw665@cam.ac.uk Tameem Adel tah47@cam.ac.uk
  • 33. Questions? Read The Full Paper at: arxiv.org/abs/2006.06848 See More of my Research (+slides): javierantoran.github.io/about/ Contact Me: ja666@cam.ac.uk