Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Manipulating and measuring model interpretability

194 views

Published on

Forough Poursabzi, Researcher, Microsoft Research

Published in: Technology
  • Be the first to comment

Manipulating and measuring model interpretability

  1. 1. Manipulating and Measuring Model Interpretability Microsoft Research NYC Forough Poursabzi- Sangdeh Dan Goldstein Jake Hofman Jenn Wortman Vaughan Hanna Wallach
  2. 2. u = k(x, u) INTERPRETABLE MACHINE LEARNING
  3. 3. u = k(x, u) e.g., generalized additive models Lou et al. 2012 and 2013 Simple models INTERPRETABLE MACHINE LEARNING
  4. 4. u = k(x, u) e.g., LIME Ribiero et al. 2016 Post-hoc explanations e.g., generalized additive models Lou et al. 2012 and 2013 Simple models INTERPRETABLE MACHINE LEARNING
  5. 5. INTERPRETABILITY? u = k(x, u) • What makes a model or explanation interpretable?
  6. 6. DIFFERENT SCENARIOS, DIFFERENT PEOPLE, DIFFERENT NEEDS u = k(x, u) Explain a prediction Understand model Make better decisions Debug model De-bias model Inspire trust CEOs Approach A Data scientists Approach C Laypeople Regulators Approach B
  7. 7. Interpretability INTERPRETABILITY AS A LATENT PROPERTY
  8. 8. Interpretability INTERPRETABILITY AS A LATENT PROPERTY number of features linearity black-box vs. clear visualizations types of features …
  9. 9. Interpretability INTERPRETABILITY AS A LATENT PROPERTY number of features linearity black-box vs. clear visualizations types of features … … trust ability to debug ability to simulate ability to explain ability to detect mistakes
  10. 10. Interpretability INTERPRETABILITY AS A LATENT PROPERTY number of features linearity black-box vs. clear visualizations types of features … properties of model and system design … trust ability to debug ability to simulate ability to explain ability to detect mistakes
  11. 11. Interpretability INTERPRETABILITY AS A LATENT PROPERTY number of features linearity black-box vs. clear visualizations types of features … properties of human behavior properties of model and system design … trust ability to debug ability to simulate ability to explain ability to detect mistakes
  12. 12. Interpretability INTERPRETABILITY AS A LATENT PROPERTY number of features linearity black-box vs. clear visualizations types of features … properties of human behavior We need interdisciplinary approaches properties of model and system design … trust ability to debug ability to simulate ability to explain ability to detect mistakes
  13. 13. Interpretability FOCUS ON LAYPEOPLE number of features linearity black-box vs. clear visualizations types of features … properties of human behavior Randomized human-subject experiments properties of model and system design … trust ability to debug ability to simulate ability to explain ability to detect mistakes
  14. 14. USER EXPERIMENT, PREDICTIVE TASK u = k(x, u) • Predict the price of apartments in NYC with the help of a model
  15. 15. EXPERIMENTAL CONDITIONS
  16. 16. EXPERIMENTAL CONDITIONS
  17. 17. EXPERIMENTAL CONDITIONS
  18. 18. EXPERIMENTAL CONDITIONS
  19. 19. EXPERIMENTAL CONDITIONS CLEAR-2 feature BB-2 feature CLEAR-8 feature BB-8 feature
  20. 20. TIGHTLY CONTROLLED EXPERIMENTS CLEAR-2 feature BB-2 feature CLEAR-8 feature BB-8 feature
  21. 21. TIGHTLY CONTROLLED EXPERIMENTS CLEAR-2 feature BB-2 feature CLEAR-8 feature BB-8 feature
  22. 22. TIGHTLY CONTROLLED EXPERIMENTS CLEAR-2 feature BB-2 feature CLEAR-8 feature BB-8 feature
  23. 23. USER INTERFACE AND INTERACTIONS u = k(x, u) • Training phase: participants get familiar with the model • Testing phase step 1: simulate the model’s prediction Simulate the model
  24. 24. USER INTERFACE AND INTERACTIONS u = k(x, u) • Testing phase step 2: observe the model’s prediction and guess the price Predict actual selling price
  25. 25. PRE-REGISTERED HYPOTHESES u = k(x, u) • CLEAR-2 feature will be easiest for participants to simulate • Participants will trust CLEAR-2 feature more than BB-8 feature • Participants’ behaviors will vary when they see unusual examples where the model makes inaccurate predictions https://aspredicted.org/xy5s6.pdf
  26. 26. SIMULATION ERROR u = k(x, u) CLEAR-2 feature will be easiest for participants to simulate
  27. 27. SIMULATION ERROR u = k(x, u) CLEAR-2 feature will be easiest for participants to simulate m $um
  28. 28. SIMULATION ERROR u = k(x, u) CLEAR-2 feature will be easiest for participants to simulate Simulation error CLEAR−2 CLEAR−8 BB−2 BB−8 $0k $100k $200k Meansimulationerror m $um
  29. 29. SIMULATION ERROR u = k(x, u) CLEAR-2 feature will be easiest for participants to simulate Simulation error CLEAR−2 CLEAR−8 BB−2 BB−8 $0k $100k $200k Meansimulationerror m $um
  30. 30. TRUST (DEVIATION FROM THE MODEL) Participants will trust CLEAR-2 feature more than BB-8 feature
  31. 31. TRUST (DEVIATION FROM THE MODEL) Participants will trust CLEAR-2 feature more than BB-8 feature m $ua
  32. 32. Deviation CLEAR−2 CLEAR−8 BB−2 BB−8 $0k $50k $100k $150k Meandeviationfromthemodel TRUST (DEVIATION FROM THE MODEL) Participants will trust CLEAR-2 feature more than BB-8 feature m $ua
  33. 33. Deviation CLEAR−2 CLEAR−8 BB−2 BB−8 $0k $50k $100k $150k Meandeviationfromthemodel TRUST (DEVIATION FROM THE MODEL) Participants will trust CLEAR-2 feature more than BB-8 feature m $ua
  34. 34. WEIRD APARTMENT u = k(x, u)
  35. 35. DETECTION OF MISTAKES Participants’ behaviors will vary when they see unusual examples where the model makes inaccurate predictions
  36. 36. DETECTION OF MISTAKES Participants’ behaviors will vary when they see unusual examples where the model makes inaccurate predictions m $ua
  37. 37. DETECTION OF MISTAKES Participants’ behaviors will vary when they see unusual examples where the model makes inaccurate predictions Apartment 12: 1 bed, 3 bath CLEAR−2 CLEAR−8 BB−2 BB−8 $0k $50k $100k $150k $200k $250k $300k Meandeviationfromthemodel forapartment12 m $ua
  38. 38. DETECTION OF MISTAKES Participants’ behaviors will vary when they see unusual examples where the model makes inaccurate predictions Apartment 12: 1 bed, 3 bath CLEAR−2 CLEAR−8 BB−2 BB−8 $0k $50k $100k $150k $200k $250k $300k Meandeviationfromthemodel forapartment12 m $ua When participants see unusual examples, they are less likely to correct inaccurate predictions made by clear models than black-box models
  39. 39. WHAT IS UP WITH THIS?
  40. 40. CONJECTURE: VISUAL OVERLOAD
  41. 41. CONJECTURE: VISUAL OVERLOAD
  42. 42. CONJECTURE: ANCHORING EFFECT
  43. 43. CONJECTURE: ANCHORING EFFECT User’s simulation of the model’s prediction
  44. 44. EXPLICIT ATTENTION CHECK
  45. 45. USER INTERFACE AND INTERACTIONS u = k(x, u) • We remove potential anchors
  46. 46. PRE-REGISTERED HYPOTHESES u = k(x, u) • Explicit attention checks on unusual inputs will affect participants’ abilities in detecting model’s mistakes • Model transparency affects participants’ abilities in detecting model’s mistakes, both with and without attention checks https://aspredicted.org/5xy8y.pdf
  47. 47. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  48. 48. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES • No attention checks: clear models lower users’ ability to correct model’s mistakes Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  49. 49. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES • No attention checks: clear models lower users’ ability to correct model’s mistakes Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  50. 50. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES • No attention checks: clear models lower users’ ability to correct model’s mistakes Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  51. 51. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES • Attention checks improve users’ ability to correct model’s mistakes • No attention checks: clear models lower users’ ability to correct model’s mistakes Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  52. 52. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES • Attention checks improve users’ ability to correct model’s mistakes • No attention checks: clear models lower users’ ability to correct model’s mistakes Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  53. 53. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES • Attention checks improve users’ ability to correct model’s mistakes • No attention checks: clear models lower users’ ability to correct model’s mistakes Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  54. 54. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES • Attention checks improve users’ ability to correct model’s mistakes • No attention checks: clear models lower users’ ability to correct model’s mistakes • With attention checks, there is no difference between clear and black-box Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  55. 55. Apartment 6: 1 bed, 3 bath, 726 sq ft Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantpredictio Model's prediction CLEAR BB DETECTION OF MISTAKES • Attention checks improve users’ ability to correct model’s mistakes • No attention checks: clear models lower users’ ability to correct model’s mistakes • With attention checks, there is no difference between clear and black-box Apartment 6: 1 bed, 3 bath Apartment 8: 1 bed, 3 bath, 350 sq ft No attention check With attention check No attention check With attention check $0M $0.5M $1M $1.5M Meanparticipantprediction Model's prediction CLEAR BB
  56. 56. SUMMARY OF RESULTS u = k(x, u) • A clear model with a small number of features is easier for participants to simulate - People have a better understanding of simple and transparent models • No significant difference in participants’ trust in the model - Contrary to intuition, people do not necessarily trust simple and transparent models more • Participants were less able to correct inaccurate predictions of a clear model than a black- box model - Too much transparency can be harmful - Design implications (e.g., highlighting unusual inputs, display model internals on demand)
  57. 57. • Interpretability is not a purely computational problem - We need interdisciplinary research to understand interpretability • Our surprising results underscore that interpretability research is much more complicated - We need more empirical studies - Other scenarios, domains, models, factors, outcomes TAKEAWAYS
  58. 58. u = k(x, u) https://csel.cs.colorado.edu/~fopo5620/ forough.poursabzi@microsoft.com Thanks!

×