SlideShare a Scribd company logo

Rsqrd AI - Challenges in Deploying Explainable Machine Learning

In this talk, Umang Bhatt presents his work on understanding how explainability is used in the industry, research done in collaboration with Partnership on AI. Umang Bhatt is a Research Fellow at the Partnership on AI and a Ph.D. student in the Machine Learning Group at the University of Cambridge. His research interests lie in statistical machine learning, explainable artificial intelligence, and human-machine collaboration.

1 of 45
Download to read offline
Challenges in Deploying
Explainable Machine Learning
Umang Bhatt
PhD Student, University of Cambridge
Research Fellow, Partnership on AI
Student Fellow, Leverhulme Center for the Future of Intelligence
@umangsbhatt
This Talk
1. How are existing approaches to explainability
used in practice?

2. Can existing explainability tools be used to ensure
model unfairness?

3. How can we create explainability tools for external
stakeholders?
This Talk
1. How are existing approaches to explainability
used in practice?

2. Can existing explainability tools be used to ensure
model unfairness?

3. How can we create explainability tools for external
stakeholders?
Explainable Machine Learning
in Deployment
Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly,
Yuhuan Jia, Joydeep Ghosh, Ruchir Puri, José Moura, and Peter Eckersley
Appeared at the ACM Conference on Fairness, Accountability, and Transparency 2020

https://arxiv.org/abs/1909.06342
Growth of Transparency Literature
Many algorithms proposed to “explain” machine learning
model output

We study how organizations use these algorithms, if at all
Our Approach
30 minute to 2 hour semi-structured interviews

50 individuals from 30 organizations interviewed
Ad

Recommended

IRJET- NEEV: An Education Informational Chatbot
IRJET-  	  NEEV: An Education Informational ChatbotIRJET-  	  NEEV: An Education Informational Chatbot
IRJET- NEEV: An Education Informational ChatbotIRJET Journal
 
IRJET- Artificial Intelligence Based Chat-Bot
IRJET-  	  Artificial Intelligence Based Chat-BotIRJET-  	  Artificial Intelligence Based Chat-Bot
IRJET- Artificial Intelligence Based Chat-BotIRJET Journal
 
Communication Skills Improving Assistance
Communication Skills Improving AssistanceCommunication Skills Improving Assistance
Communication Skills Improving Assistanceijtsrd
 
IRJET - YouTube Spam Comments Detection
IRJET - YouTube Spam Comments DetectionIRJET - YouTube Spam Comments Detection
IRJET - YouTube Spam Comments DetectionIRJET Journal
 
The 't' in tel software development for tel research problems, pitfalls, and ...
The 't' in tel software development for tel research problems, pitfalls, and ...The 't' in tel software development for tel research problems, pitfalls, and ...
The 't' in tel software development for tel research problems, pitfalls, and ...Roland Klemke
 
Exams are online!
Exams are online!Exams are online!
Exams are online!Derek Moore
 
Dissertation Review Essay
Dissertation Review EssayDissertation Review Essay
Dissertation Review EssayLiz Bundren
 
Teaching, Organizational Leadership, And Health Promotion
Teaching, Organizational Leadership, And Health PromotionTeaching, Organizational Leadership, And Health Promotion
Teaching, Organizational Leadership, And Health PromotionAngela Baxter
 

More Related Content

Similar to Rsqrd AI - Challenges in Deploying Explainable Machine Learning

'Humans still needed' - research project reveals impact of artificial intelli...
'Humans still needed' - research project reveals impact of artificial intelli...'Humans still needed' - research project reveals impact of artificial intelli...
'Humans still needed' - research project reveals impact of artificial intelli...Chartered Institute of Public Relations
 
The Producer Consumer Problem Considered Harmful
The Producer Consumer Problem Considered HarmfulThe Producer Consumer Problem Considered Harmful
The Producer Consumer Problem Considered HarmfulSusan White
 
Product Analyst Advisor
Product Analyst AdvisorProduct Analyst Advisor
Product Analyst AdvisorIRJET Journal
 
Digital Technology And Its Impact On The Future Of A...
Digital Technology And Its Impact On The Future Of A...Digital Technology And Its Impact On The Future Of A...
Digital Technology And Its Impact On The Future Of A...Mia Hart
 
Graduate Level Writing Analysis And Development Plan
Graduate Level Writing Analysis And Development PlanGraduate Level Writing Analysis And Development Plan
Graduate Level Writing Analysis And Development PlanSheila Phillips
 
Oleksander Krakovetskyi "Explaining a Machine Learning blackbox"
Oleksander Krakovetskyi "Explaining a Machine Learning blackbox"Oleksander Krakovetskyi "Explaining a Machine Learning blackbox"
Oleksander Krakovetskyi "Explaining a Machine Learning blackbox"Fwdays
 
RE2021 tutorial human values in requirements engineering
RE2021 tutorial   human values in requirements engineeringRE2021 tutorial   human values in requirements engineering
RE2021 tutorial human values in requirements engineeringJon Whittle
 
20240104 HICSS Panel on AI and Legal Ethical 20240103 v7.pptx
20240104 HICSS  Panel on AI and Legal Ethical 20240103 v7.pptx20240104 HICSS  Panel on AI and Legal Ethical 20240103 v7.pptx
20240104 HICSS Panel on AI and Legal Ethical 20240103 v7.pptxISSIP
 
Usability Testing: Making it fast, good, and cheap
Usability Testing: Making it fast, good, and cheapUsability Testing: Making it fast, good, and cheap
Usability Testing: Making it fast, good, and cheapWhitney Quesenbery
 
Should Fda Promote More Process Analytical Technology And...
Should Fda Promote More Process Analytical Technology And...Should Fda Promote More Process Analytical Technology And...
Should Fda Promote More Process Analytical Technology And...Stephanie Williams
 
Should Fda Promote More Process Analytical Technology And...
Should Fda Promote More Process Analytical Technology And...Should Fda Promote More Process Analytical Technology And...
Should Fda Promote More Process Analytical Technology And...Megan Foster
 
Elder Abuse Research
Elder Abuse ResearchElder Abuse Research
Elder Abuse ResearchLaura Torres
 
Balanced Evaluation Paper
Balanced Evaluation PaperBalanced Evaluation Paper
Balanced Evaluation PaperRachel Phillips
 
Requirements Engineering for the Humanities
Requirements Engineering for the HumanitiesRequirements Engineering for the Humanities
Requirements Engineering for the HumanitiesShawn Day
 
Exploration & Promotion: Implementation Strategies of Corporate Social Software
Exploration & Promotion: Implementation Strategies of Corporate Social SoftwareExploration & Promotion: Implementation Strategies of Corporate Social Software
Exploration & Promotion: Implementation Strategies of Corporate Social SoftwareAlexander Stocker
 
ifib Lunchbag: CHI2018 Highlights - Algorithms in (Social) Practice and more
ifib Lunchbag: CHI2018 Highlights - Algorithms in (Social) Practice and moreifib Lunchbag: CHI2018 Highlights - Algorithms in (Social) Practice and more
ifib Lunchbag: CHI2018 Highlights - Algorithms in (Social) Practice and morehen_drik
 
BA and Beyond 19 Sponsor spotlight - Namahn - Beating complexity with complexity
BA and Beyond 19 Sponsor spotlight - Namahn - Beating complexity with complexityBA and Beyond 19 Sponsor spotlight - Namahn - Beating complexity with complexity
BA and Beyond 19 Sponsor spotlight - Namahn - Beating complexity with complexityBA and Beyond
 
The Roles of Social Networks In Knowledge Management
The Roles of Social Networks In Knowledge ManagementThe Roles of Social Networks In Knowledge Management
The Roles of Social Networks In Knowledge ManagementMonzer Osama Alchikh WARAK
 
IEEE augmented reality learning experience model (ARLEM)
IEEE augmented reality learning experience model (ARLEM)IEEE augmented reality learning experience model (ARLEM)
IEEE augmented reality learning experience model (ARLEM)fridolin.wild
 

Similar to Rsqrd AI - Challenges in Deploying Explainable Machine Learning (20)

'Humans still needed' - research project reveals impact of artificial intelli...
'Humans still needed' - research project reveals impact of artificial intelli...'Humans still needed' - research project reveals impact of artificial intelli...
'Humans still needed' - research project reveals impact of artificial intelli...
 
The Producer Consumer Problem Considered Harmful
The Producer Consumer Problem Considered HarmfulThe Producer Consumer Problem Considered Harmful
The Producer Consumer Problem Considered Harmful
 
Product Analyst Advisor
Product Analyst AdvisorProduct Analyst Advisor
Product Analyst Advisor
 
Digital Technology And Its Impact On The Future Of A...
Digital Technology And Its Impact On The Future Of A...Digital Technology And Its Impact On The Future Of A...
Digital Technology And Its Impact On The Future Of A...
 
Graduate Level Writing Analysis And Development Plan
Graduate Level Writing Analysis And Development PlanGraduate Level Writing Analysis And Development Plan
Graduate Level Writing Analysis And Development Plan
 
Oleksander Krakovetskyi "Explaining a Machine Learning blackbox"
Oleksander Krakovetskyi "Explaining a Machine Learning blackbox"Oleksander Krakovetskyi "Explaining a Machine Learning blackbox"
Oleksander Krakovetskyi "Explaining a Machine Learning blackbox"
 
Usability Testing Report
Usability Testing ReportUsability Testing Report
Usability Testing Report
 
RE2021 tutorial human values in requirements engineering
RE2021 tutorial   human values in requirements engineeringRE2021 tutorial   human values in requirements engineering
RE2021 tutorial human values in requirements engineering
 
20240104 HICSS Panel on AI and Legal Ethical 20240103 v7.pptx
20240104 HICSS  Panel on AI and Legal Ethical 20240103 v7.pptx20240104 HICSS  Panel on AI and Legal Ethical 20240103 v7.pptx
20240104 HICSS Panel on AI and Legal Ethical 20240103 v7.pptx
 
Usability Testing: Making it fast, good, and cheap
Usability Testing: Making it fast, good, and cheapUsability Testing: Making it fast, good, and cheap
Usability Testing: Making it fast, good, and cheap
 
Should Fda Promote More Process Analytical Technology And...
Should Fda Promote More Process Analytical Technology And...Should Fda Promote More Process Analytical Technology And...
Should Fda Promote More Process Analytical Technology And...
 
Should Fda Promote More Process Analytical Technology And...
Should Fda Promote More Process Analytical Technology And...Should Fda Promote More Process Analytical Technology And...
Should Fda Promote More Process Analytical Technology And...
 
Elder Abuse Research
Elder Abuse ResearchElder Abuse Research
Elder Abuse Research
 
Balanced Evaluation Paper
Balanced Evaluation PaperBalanced Evaluation Paper
Balanced Evaluation Paper
 
Requirements Engineering for the Humanities
Requirements Engineering for the HumanitiesRequirements Engineering for the Humanities
Requirements Engineering for the Humanities
 
Exploration & Promotion: Implementation Strategies of Corporate Social Software
Exploration & Promotion: Implementation Strategies of Corporate Social SoftwareExploration & Promotion: Implementation Strategies of Corporate Social Software
Exploration & Promotion: Implementation Strategies of Corporate Social Software
 
ifib Lunchbag: CHI2018 Highlights - Algorithms in (Social) Practice and more
ifib Lunchbag: CHI2018 Highlights - Algorithms in (Social) Practice and moreifib Lunchbag: CHI2018 Highlights - Algorithms in (Social) Practice and more
ifib Lunchbag: CHI2018 Highlights - Algorithms in (Social) Practice and more
 
BA and Beyond 19 Sponsor spotlight - Namahn - Beating complexity with complexity
BA and Beyond 19 Sponsor spotlight - Namahn - Beating complexity with complexityBA and Beyond 19 Sponsor spotlight - Namahn - Beating complexity with complexity
BA and Beyond 19 Sponsor spotlight - Namahn - Beating complexity with complexity
 
The Roles of Social Networks In Knowledge Management
The Roles of Social Networks In Knowledge ManagementThe Roles of Social Networks In Knowledge Management
The Roles of Social Networks In Knowledge Management
 
IEEE augmented reality learning experience model (ARLEM)
IEEE augmented reality learning experience model (ARLEM)IEEE augmented reality learning experience model (ARLEM)
IEEE augmented reality learning experience model (ARLEM)
 

Recently uploaded

How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxInfosec
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanDatabarracks
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVARobert McDermott
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaISPMAIndia
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17Ana-Maria Mihalceanu
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxMaarten Balliauw
 
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxLeveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxVotarikari Shravan
 
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Adrian Sanabria
 
AI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementAI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementMimmo Squillace
 
"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor FesenkoFwdays
 
"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura RochniakFwdays
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stackSummit
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Umar Saif
 
H3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxH3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxMemory Fabric Forum
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Product School
 
"DevOps Practisting Platform on EKS with Karpenter autoscaling", Dmytro Kozhevin
"DevOps Practisting Platform on EKS with Karpenter autoscaling", Dmytro Kozhevin"DevOps Practisting Platform on EKS with Karpenter autoscaling", Dmytro Kozhevin
"DevOps Practisting Platform on EKS with Karpenter autoscaling", Dmytro KozhevinFwdays
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfkatalinjordans1
 
Curtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfCurtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfDomotica daVinci
 
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...Product School
 

Recently uploaded (20)

How AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptxHow AI and ChatGPT are changing cybersecurity forever.pptx
How AI and ChatGPT are changing cybersecurity forever.pptx
 
How to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response PlanHow to write an effective Cyber Incident Response Plan
How to write an effective Cyber Incident Response Plan
 
Introduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVAIntroduction to Multimodal LLMs with LLaVA
Introduction to Multimodal LLMs with LLaVA
 
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish GuptaBuilding Products That Think- Bhaskaran Srinivasan & Ashish Gupta
Building Products That Think- Bhaskaran Srinivasan & Ashish Gupta
 
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17Enhancing Productivity and Insight  A Tour of JDK Tools Progress Beyond Java 17
Enhancing Productivity and Insight A Tour of JDK Tools Progress Beyond Java 17
 
Bringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptxBringing nullability into existing code - dammit is not the answer.pptx
Bringing nullability into existing code - dammit is not the answer.pptx
 
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docxLeveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
Leveraging SLF4j for Effective Logging in IBM App Connect Enterprise.docx
 
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
Early Tech Adoption: Foolish or Pragmatic? - 17th ISACA South Florida WOW Con...
 
AI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvementAI Act & Standardization: UNINFO involvement
AI Act & Standardization: UNINFO involvement
 
"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko"Platform Engineering with Development Containers", Igor Fesenko
"Platform Engineering with Development Containers", Igor Fesenko
 
"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak"Testing of Helm Charts or There and Back Again", Yura Rochniak
"Testing of Helm Charts or There and Back Again", Yura Rochniak
 
How we think about an advisor tech stack
How we think about an advisor tech stackHow we think about an advisor tech stack
How we think about an advisor tech stack
 
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
Progress Report: Ministry of IT under Dr. Umar Saif Aug 23-Feb'24
 
H3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptxH3 Platform CXL Solution_Memory Fabric Forum.pptx
H3 Platform CXL Solution_Memory Fabric Forum.pptx
 
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...Relationship Counselling: From Disjointed Features to Product-First Thinking ...
Relationship Counselling: From Disjointed Features to Product-First Thinking ...
 
"DevOps Practisting Platform on EKS with Karpenter autoscaling", Dmytro Kozhevin
"DevOps Practisting Platform on EKS with Karpenter autoscaling", Dmytro Kozhevin"DevOps Practisting Platform on EKS with Karpenter autoscaling", Dmytro Kozhevin
"DevOps Practisting Platform on EKS with Karpenter autoscaling", Dmytro Kozhevin
 
Power of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdfPower of 2024 - WITforce Odyssey.pptx.pdf
Power of 2024 - WITforce Odyssey.pptx.pdf
 
Curtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdfCurtain Module Manual Zigbee Neo CS01-1C.pdf
Curtain Module Manual Zigbee Neo CS01-1C.pdf
 
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
Synergy in Leadership and Product Excellence: A Blueprint for Growth by CPO, ...
 
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...In sharing we trust. Taking advantage of a diverse consortium to build a tran...
In sharing we trust. Taking advantage of a diverse consortium to build a tran...
 

Rsqrd AI - Challenges in Deploying Explainable Machine Learning

  • 1. Challenges in Deploying Explainable Machine Learning Umang Bhatt PhD Student, University of Cambridge Research Fellow, Partnership on AI Student Fellow, Leverhulme Center for the Future of Intelligence @umangsbhatt
  • 2. This Talk 1. How are existing approaches to explainability used in practice? 2. Can existing explainability tools be used to ensure model unfairness? 3. How can we create explainability tools for external stakeholders?
  • 3. This Talk 1. How are existing approaches to explainability used in practice? 2. Can existing explainability tools be used to ensure model unfairness? 3. How can we create explainability tools for external stakeholders?
  • 4. Explainable Machine Learning in Deployment Umang Bhatt, Alice Xiang, Shubham Sharma, Adrian Weller, Ankur Taly, Yuhuan Jia, Joydeep Ghosh, Ruchir Puri, José Moura, and Peter Eckersley Appeared at the ACM Conference on Fairness, Accountability, and Transparency 2020 https://arxiv.org/abs/1909.06342
  • 5. Growth of Transparency Literature Many algorithms proposed to “explain” machine learning model output We study how organizations use these algorithms, if at all
  • 6. Our Approach 30 minute to 2 hour semi-structured interviews 50 individuals from 30 organizations interviewed
  • 7. Shared Language • Transparency: Providing stakeholders with relevant information about how the model works: this includes documentation of the training procedure, analysis of training data distribution, code releases, feature- level explanations, etc. • Explainability: Providing insights into a model’s behavior for specific datapoint(s)
  • 8. Our Questions • What type of explanations have you used (e.g., feature-based, sample- based, counterfactual, or natural language)? • Who is the audience for the model explanation (e.g., research scientists, product managers, domain experts, or users)? • In what context have you deployed the explanations (e.g., informing the development process, informing human decision makers about the model, or informing the end user on how actions were taken based on the model’s output)?
  • 9. Types of Explanations Feature Importance Sample Importance Counterfactuals
  • 11. Findings 1.Explainability is used for debugging internally 2.Goals of explainability are not clearly defined within organizations 3.Technical limitations make explainability hard to deploy in real-time
  • 12. Findings 1.Explainability is used for debugging internally 2.Goals of explainability are not clearly defined within organizations 3.Technical limitations make explainability hard to deploy in real-time
  • 14. Findings 1.Explainability is used for debugging internally 2.Goals of explainability are not clearly defined within organizations 3.Technical limitations make explainability hard to deploy in real-time
  • 16. Findings 1.Explainability is used for debugging internally 2.Goals of explainability are not clearly defined within organizations 3.Technical limitations make explainability hard to deploy in real-time
  • 17. Limitations • Spurious correlations exposed by feature level explanations • No causal underpinnings to the models themselves
  • 18. Limitations (cont.) • Sample importance is computationally infeasible to deploy at scale • Privacy concerns of model inversion exist
  • 19. Findings 1.Explainability is used for debugging internally 2.Goals of explainability are not clearly defined within organizations 3.Technical limitations make explainability hard to deploy in real-time
  • 20. This Talk 1. How are existing approaches to explainability used in practice? 2. Can existing explainability tools be used to ensure model unfairness? 3. How can we create explainability tools for external stakeholders?
  • 21. This Talk 1. How are existing approaches to explainability used in practice? Only used by developers 2. Can existing explainability tools be used to ensure model unfairness? 3. How can we create explainability tools for external stakeholders?
  • 22. This Talk 1. How are existing approaches to explainability used in practice? Only used by developers 2. Can existing explainability tools be used to ensure model unfairness? 3. How can we create explainability tools for external stakeholders?
  • 23. You Shouldn’t Trust Me: Learning Models Which Conceal Unfairness From Multiple Explanation Methods Botty Dimanov, Umang Bhatt, Mateja Jamnik, and Adrian Weller To appear at the European Conference on Artificial Intelligence 2020 http://ecai2020.eu/papers/72_paper.pdf
  • 24. Takeaway Feature importance reveals nothing reliable about model fairness.
  • 25. Why do we care? FeatureImportance Feature FeatureImportance Feature Model A Model B
  • 26. Why do we care? FeatureImportance Feature FeatureImportance Feature Model A Model B Unfair Fair
  • 27. Can we manipulate explanations? Modified explanations via adversarial perturbations of inputs • Ghorbani, Abid, and Zou. AAAI 2019 • Dombrowski et al. NeurIPS 2019 • Slack et al. AIES 2019 Control visual explanations via adversarial perturbations of parameters • Heo, Joo, and Moon. NeurIPS 2019 Downgrade explanations via adversarial perturbations of parameters to hide unfairness
  • 28. fθ fθ+δ f : X ↦ Y Explanation g(f, x)j Our Setup Classifier Our Goal ∀i, fθ+δ(x(i) ) ≈ fθ(x(i) ) Desirable Properties Model Similarity ∀i, |g(fθ+δ, x(i) )j | ≪ |g(fθ, x(i) )j |Low Target Feature Attribution
  • 29. Our Method argminδ L′ = L(fθ+δ, x, y) + α n ∇X:,j L(fθ+δ, x, y) p Adversarial Explanation Attack
  • 31. Results (Importance Ranking) fθ fθ+δ Our adversarial explanation attack: 1.Significantly decreases relative importance. 2.Generalizes to test points. 3.Transfers across explanation methods.
  • 32. Findings • Little change in accuracy, but difference in outputs is detectable • Low attribution achieved with respect to multiple explanation methods • High unfairness across multiple fairness metrics (compared to holding feature constant) fθ fθ+δ
  • 33. This Talk 1. How are existing approaches to explainability used in practice? Only used by developers 2. Can existing explainability tools be used to ensure model unfairness? 3. How can we create explainability tools for external stakeholders?
  • 34. This Talk 1. How are existing approaches to explainability used in practice? Only used by developers 2. Can existing explainability tools be used to ensure model unfairness? Not in their current form 3. How can we create explainability tools for external stakeholders?
  • 35. This Talk 1. How are existing approaches to explainability used in practice? Only used by developers 2. Can existing explainability tools be used to ensure model unfairness? Not in their current form 3. How can we create explainability tools for external stakeholders?
  • 36. Machine Learning Explainability for External Stakeholders Umang Bhatt, McKane Andrus, Adrian Weller, and Alice Xiang Convening hosted by CFI, PAI, and IBM Appeared at the ICML 2020 Workshop on Extending Explainable AI: Beyond Deep Models and Classifiers https://arxiv.org/abs/2007.05408
  • 37. Overview • 33 participants from 5 countries • 15 ML experts, 3 designers, 6 legal experts, 9 policymakers • Domain expertise: Finance, Healthcare, Media, and Social Services • Goal: facilitate an inter-stakeholder conversation around explainable machine learning
  • 38. Two Takeaways 1. There is a need for community engagement in the development of explainable machine learning. 2. There are nuances in the deployment of explainable machine learning.
  • 39. Community Engagement 1. In which context will this explanation be used? Does the context change the properties of the explanations we expose? 2. How should the explanation be evaluated? Both quantitatively and qualitatively… 3. Can we prevent data misuse and preferential treatment by involving affected groups in the development process? 4. Can we educate external stakeholders (and data scientists) regarding the functionalities and limitations of explainable machine learning?
  • 40. Deploying Explainability 1. How does uncertainty in the model and introduced by the (potentially approximate) explanation technique affect the resulting explanations? 2. How can stakeholders interact with the resulting explanations? Can explanations be a conduit for interacting with the model? 3. How, if at all, will stakeholder behavior change as a result of the explanation shown? 4. Over time, how will the explanation technique adapt to changes in stakeholder behavior?
  • 41. Two Takeaways 1. There is a need for community engagement in the development of explainable machine learning. 2. There are nuances in the deployment of explainable machine learning.
  • 42. This Talk 1. How are existing approaches to explainability used in practice? Only used by developers 2. Can existing explainability tools be used to ensure model unfairness? Not in their current form 3. How can we create explainability tools for external stakeholders?
  • 43. This Talk 1. How are existing approaches to explainability used in practice? Only used by developers 2. Can existing explainability tools be used to ensure model unfairness? Not in their current form 3. How can we create explainability tools for external stakeholders? Community engagement and thoughtful deployment
  • 44. This Talk 1. How are existing approaches to explainability used in practice? Only used by developers 2. Can existing explainability tools be used to ensure model unfairness? Not in their current form 3. How can we create explainability tools for external stakeholders? Community engagement and thoughtful deployment
  • 45. Challenges in Deploying Explainable Machine Learning Umang Bhatt usb20@cam.ac.uk @umangsbhatt Thanks for listening!