Explainable AI

Explainable AI

• 1.
  Explainable AI Wagston Staehler& Luciano Alves March, 2020
• 2.
  The Presenters Wagston Staehler ComputerEngineer MSc Computer Science Researcher at HP Luciano Alves BSc Computer Science MSc Computer Science ML Engineer at HP
• 3.
  Machine Learning Dataset Train Model Fine tuning Optimization …  Accuracy  Precision/Recall  Error  Deploy!  Real-world!  Data has meaning!
• 4.
  Interpretability and Explainability •Interpretability: to establish a cause and effect relationship • Explainability: to explain the internal mechanics of ML or DL in human terms • Most of the time, the terms are interchangeable • Interpretability, explainability, explainable AI, XAI
• 5.
  Agenda 1. Awkward real-worldsituations… 2. Why interpretability? 3. Some techniques and examples 4. Final considerations
• 6.
  Awkward real-world situations... 1
• 7.
  Some awkward situations… https://www.nature.com/articles /d41586-018-05707-8 https://www.technologyreview.c om/f/612502/ai-has-a-culturally- biased-worldview-that-google- has-a-plan-to-change/
• 8.
  Some awkward situations… https://ai.googleblog.com/2018/09/introducing-inclusive-images-competition.html
• 9.
  Some awkward situations… S.Kaufman, S. Rosset, and C. Perlich. Leakage in data mining: Formulation, detection, and avoidance. In Knowledge Discovery and Data Mining (KDD), 2011 Cancer detection from mammography data is highly correlated with…PATIENT ID! Symptoms # Days in Hospital Mammography Feat. 1 Mammography Feat. 2 Cancer Diagnosis abc True def True … True False False False Patient ID 1 2 ... 499 500 501 502 No source obfuscation ⇒ Data leakage! if (patient ID < 500) then cancer=True
• 10.
  One more awkwardsituation… • Cost-effective Health Care (CEHC) built models to predict probability of death for patients [Cooper et al. 97] - Interpretable Machine Learning, Been Kim from Google Brain, Tutorial ICML2017 - Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission, Caruana et al. ’15 Has asthma? ⇒ Lower risk for pneumonia Doctors think he/she is high risk Aggressive treatment
• 11.
  Why interpretability? 2
• 12.
  Motivation • Building trust •Informing human decision-making • “Insights can be more valuable than predictions” • Legal/Ethics • Prove that the system is legally compliant and that it does not discriminate any particular group • General Data Protection Regulation (GDPR) requires explanations on automated decision-making and profiling (https://gdpr-info.eu) • Brazilian LGPD, Lei Geral de Proteção de Dados Pessoais
• 13.
  Motivation • Safety • Makesure the system is taking safe decisions • Debugging • Understand why a system doesn’t work • Model has good accuracy in validation set but it is not performing well in the wild • Science • Comprehend how the system took that conclusion and learn with it
• 14.
  Motivation • Informing featureengineering • Create new features using transformations of your raw data or other features to improve model accuracy • Directing future data collection • Understand the value of the features that you already have and, therefore, decide what new values will be more helpful
• 15.
  Interpretability • Helps alot with • Privacy • Accountability • Trust • Causality • May not be required when • No significant consequences • Well known field
• 16.
  Types of InterpretableMethods • Pre-modelling explainability • Understand/describe data used to develop models • Explainable modelling • Develop inherently more explainable models • Post-modelling explainability • Extract explanations to describe pre-developed models https://towardsdatascience.com/the-how-of-explainable-ai-post-modelling-explainability-8b4cbc7adf5f
• 17.
  Some practical techniques andexamples 3
• 18.
  Data Visualization • Byunderstanding well the data, we know what to expect from the trained model
• 19.
  Feature Importance pickup_longitude pickup_latitudedropoff_longitude dropoff_latitude passenger_count fare_amount -73.982738 40.761270 -73.991242 40.750562 2 5.7 -73.987130 40.733143 -73.991567 40.758092 1 7.7 -73.968095 40.768008 -73.956655 40.783762 1 5.3 -73.980002 40.751662 -73.973802 40.764842 1 7.5 -73.951300 40.774138 -73.990095 40.751048 1 16.5 https://www.kaggle.com/dansbecker/permutation-importance (from eli5.sklearn import PermutationImportance)
• 20.
  Partial Dependence Plots •Partial dependence plots show how a feature affects predictions • Useful to answer questions like: • How would similarly sized houses be priced in different areas? • Are predicted health differences between two groups due to differences in their diets, or due to some other factor?
• 21.
  Partial Dependence Plots(Example) https://christophm.github.io/interpretable-ml-book/pdp.html from matplotlib import pyplot as plt from pdpbox import pdp partial_plot = pdp.pdp_isolate(model, dataset, model_features, feature) pdp.pdp_plot(partial_plot, 'Partial Plot') plt.show()
• 22.
  LIME (Local InterpretableModel-Agnostic Explanation) • Do you trust this algorithm to work well in the real word? • Classification: Wolf or Husky? https://arxiv.org/abs/1602.04938 "Why Should I Trust You?”, Marco Tulio Ribeiro, Sameer Singh, Carlos Guestrin
• 23.
  LIME (Local InterpretableModel-Agnostic Explanation) • Do you trust this algorithm to work well in the real word? • How do you think the algorithm is able to distinguish between these photos of wolves and huskies?
• 24.
  • Do youtrust this algorithm to work well in the real word? • How do you think the algorithm is able to distinguish between these photos of wolves and huskies? LIME (Local Interpretable Model-Agnostic Explanation) Snow detector!!!
• 25.
  LIME 1. Select instanceof interest. 2. Perturb a dataset with small variations of instance. 3. Predict these new dataset with the original model. 4. Sampled instances are weighted by proximity to the instance of interest 5. Train a weighted, interpretable model on the dataset with the variations. 6. Explain the prediction by interpreting the local model.
• 26.
  LIME Example: TabularData grey blue A A. Random forest predictions given features x1 and x2
• 27.
  LIME Example: TabularData grey blue A B A. Random forest predictions given features x1 and x2 B. Instance of interest (yellow dot) and data sampled from a normal distribution (black dots)
• 28.
  LIME Example: TabularData grey blue A B C A. Random forest predictions given features x1 and x2 B. Instance of interest (yellow dot) and data sampled from a normal distribution (black dots) C. Assign higher weight to points near the instance of interest
• 29.
  LIME Example: TabularData A. Random forest predictions given features x1 and x2 B. Instance of interest (yellow dot) and data sampled from a normal distribution (black dots) C. Assign higher weight to points near the instance of interest D. Colors and signs of the grid show the classifications of the locally learned model from the weighted samples - grey + blue A B C D
• 30.
  LIME Example: Text •Text differs from tabular data • Model Spam classifier • Randomly removed words ID CONTENT CLASS 267 PSY is a good guy 0 173 For Christmas Song visit my channel! ;) 1 For Christmas Song visit my channel ;) prob weight 1 0 1 1 0 0 1 0.09 0.57 0 1 1 1 1 0 1 0.09 0.71 1 0 0 1 1 1 1 0.99 0.71 1 0 1 1 1 1 1 0.99 0.86 0 1 1 1 0 0 1 0.09 0.57 Instance of Interest Sampled Data Instances
• 31.
  LIME Example: Text •Text differs from tabular data • Model Spam classifier • Randomly removed words ID CONTENT CLASS 267 PSY is a good guy 0 173 For Christmas Song visit my channel! ;) 1 For Christmas Song visit my channel ;) prob weight 1 0 1 1 0 0 1 0.09 0.57 0 1 1 1 1 0 1 0.09 0.71 1 0 0 1 1 1 1 0.99 0.71 1 0 1 1 1 1 1 0.99 0.86 0 1 1 1 0 0 1 0.09 0.57 Instance of Interest Sampled Data Instances 1 - 𝟓 𝟕 = 0.71
• 32.
  LIME Example: Text CaseLabel Prob Feature Feature Weight 1 0.0872151 good 0.000000 1 0.0872151 a 0.000000 1 0.0872151 PSY 0.000000 2 0.9939759 channel! 6.908755 2 0.9939759 visit 0.000000 2 0.9939759 Song 0.000000 • The word “channel” indicates a high probability of spam • Non-spam comment with zero weight estimated
• 33.
  LIME Example: Image •Image differs from Text and Tabular Data • Image segmentation • Randomly selects one pixel and turns ON or OFF by color similarity Original Image Interpretable Components
• 34.
  LIME Example: Image OriginalImage P(tree frog) = 0.54 Perturbed Instances P(tree frog) 0.85 0.00001 0.52 Explanation Locally Weighted regression
• 35.
  Shapley Values • 1953- A group of differently skilled participants are all cooperating with each other for a collective reward. How should the reward be fairly divided amongst the group? • Definition: A prediction can be explained by assuming that each feature of the instance is a player in a game where the prediction is the payout. The Shapley value – a method from a coalitional game theory – tells us how to fairly distribute the payout among the features. Shapley, Lloyd S. “A value for n-person games.” Contributions to the Theory of Games 2.28 (1953): 307-317.
• 36.
  Shapley Values • Alice,Bob, Celine write papers as members of a research group 80, if c = {A} 56, if c = {B} 70, if c = {C} 80, if c = {A, B} 85, if c = {A, C} 72, if c = {B, C} 90, if c = {A, B, C} v(c) = 𝜙𝑖 𝐺 = 1 𝑛! 𝜋∈∏ 𝑛 △ 𝜋 𝐺 (𝑖)
• 37.
  Shapley Values 80, ifc = {A} 56, if c = {B} 70, if c = {C} 80, if c = {A, B} 85, if c = {A, C} 72, if c = {B, C} 90, if c = {A, B, C} v(c) = 𝝅 𝛿 𝜋 𝐺 (A, B, C) (80, 0, 10) 𝜙𝑖 𝐺 = 1 𝑛! 𝜋∈∏ 𝑛 △ 𝜋 𝐺 (𝑖)
• 38.
  Shapley Values 80, ifc = {A} 56, if c = {B} 70, if c = {C} 80, if c = {A, B} 85, if c = {A, C} 72, if c = {B, C} 90, if c = {A, B, C} v(c) = 𝝅 𝛿 𝜋 𝐺 (A, B, C) (80, 0, 10) (A, C, B) (80, 5, 5) 𝜙𝑖 𝐺 = 1 𝑛! 𝜋∈∏ 𝑛 △ 𝜋 𝐺 (𝑖)
• 39.
  Shapley Values 80, ifc = {A} 56, if c = {B} 70, if c = {C} 80, if c = {A, B} 85, if c = {A, C} 72, if c = {B, C} 90, if c = {A, B, C} v(c) = 𝝅 𝛿 𝜋 𝐺 (A, B, C) (80, 0, 10) (A, C, B) (80, 5, 5) (B, A, C) (24, 56, 10) 𝜙𝑖 𝐺 = 1 𝑛! 𝜋∈∏ 𝑛 △ 𝜋 𝐺 (𝑖)
• 40.
  Shapley Values 80, ifc = {A} 56, if c = {B} 70, if c = {C} 80, if c = {A, B} 85, if c = {A, C} 72, if c = {B, C} 90, if c = {A, B, C} v(c) = 𝝅 𝛿 𝜋 𝐺 (A, B, C) (80, 0, 10) (A, C, B) (80, 5, 5) (B, A, C) (24, 56, 10) (B, C, A) (18, 56, 16) 𝜙𝑖 𝐺 = 1 𝑛! 𝜋∈∏ 𝑛 △ 𝜋 𝐺 (𝑖)
• 41.
  Shapley Values 80, ifc = {A} 56, if c = {B} 70, if c = {C} 80, if c = {A, B} 85, if c = {A, C} 72, if c = {B, C} 90, if c = {A, B, C} v(c) = 𝝅 𝛿 𝜋 𝐺 (A, B, C) (80, 0, 10) (A, C, B) (80, 5, 5) (B, A, C) (24, 56, 10) (B, C, A) (18, 56, 16) (C, A, B) (15, 5, 70) 𝜙𝑖 𝐺 = 1 𝑛! 𝜋∈∏ 𝑛 △ 𝜋 𝐺 (𝑖)
• 42.
  Shapley Values 80, ifc = {A} 56, if c = {B} 70, if c = {C} 80, if c = {A, B} 85, if c = {A, C} 72, if c = {B, C} 90, if c = {A, B, C} v(c) = 𝝅 𝛿 𝜋 𝐺 (A, B, C) (80, 0, 10) (A, C, B) (80, 5, 5) (B, A, C) (24, 56, 10) (B, C, A) (18, 56, 16) (C, A, B) (15, 5, 70) (C, B, A) (18, 2, 70) 𝜙𝑖 𝐺 = 1 𝑛! 𝜋∈∏ 𝑛 △ 𝜋 𝐺 (𝑖)
• 43.
  Shapley Values 80, ifc = {A} 56, if c = {B} 70, if c = {C} 80, if c = {A, B} 85, if c = {A, C} 72, if c = {B, C} 90, if c = {A, B, C} v(c) = 𝝅 𝛿 𝜋 𝐺 (A, B, C) (80, 0, 10) (A, C, B) (80, 5, 5) (B, A, C) (24, 56, 10) (B, C, A) (18, 56, 16) (C, A, B) (15, 5, 70) (C, B, A) (18, 2, 70) ∅ (39.17, 20.67, 30.17) 𝜙𝑖 𝐺 = 1 𝑛! 𝜋∈∏ 𝑛 △ 𝜋 𝐺 (𝑖)
• 44.
  Other Techniques • DeepLIFT •Layerwise Relevance Propagation • QII, Quantitative Input Influence • Integrated Gradients • Grad-CAM
• 45.
  Final Considerations 4
• 46.
  Final Thoughts Sharayu Rane(2018)
• 47.
  Final Thoughts • Insaneincreasing of model’s complexity! • That requires tools to help interpret it • Interpretability can assure models’ quality • High level of generalization, complaint to law and ethics • Increase models’ acceptance • Without trust, we may lose ML benefits!
• 48.
  References • Kaggle MachineLearning Explainability • https://www.kaggle.com/learn/machine-learning-explainability • Interpretable Machine Learning, Been Kim • https://people.csail.mit.edu/beenkim/papers/BeenK_FinaleDV_ICML2017_tutorial.pdf • How to improve interpretability of machine learning systems • https://hub.packtpub.com/improve-interpretability-machine-learning-systems • Why should I trust you? (LIME paper) • https://arxiv.org/abs/1602.04938 • Distill Feature Visualization • https://distill.pub/2017/feature-visualization • Book: A Guide for Making Black Box Models Explainable, Christoph Molnar • https://christophm.github.io/interpretable-ml-book
• 49.
  References • GitHubs • https://github.com/slundberg/shap •https://github.com/marcotcr/lime
• 50.
  Thank you!!
• 51.
  Interpretability and AdversarialExamples Protecting Voice Controlled Systems Using Sound Source Identification Based on Acoustic Cues, Yuan Gong, Christian Poellabauer
• 52.
  Interpretability and AdversarialExamples Does Interpretability of Neural Networks Imply Adversarial Robustness? Adam Noack, Isaac Ahern, Dejing Dou, Boyang Li https://arxiv.org/abs/1912.03430 • This study indicates that improving interpretability also helps with adversarial robustness
• 53.
  Automated ML Historical Data TestSet Training Set Model Training/ Building Featuring Importance Test Model Predictions Explaining Predictions Deployed Model Predict Creation Production Feature Extraction Streaming Data Feature Extraction Evaluate Results Good Interpretability Interpretability Good Bad