Ethical AI: Establish an AI/ML Governance framework addressing Reproducibility, Explainability, Bias & Accountability for Enterprise AI use-cases. Presentation on “Open Source Enterprise AI/ML Governance” at Linux Foundation’s Open Compliance Summit, Dec 2020 (https://events.linuxfoundation.org/open-compliance-summit/) Full article: https://towardsdatascience.com/ethical-ai-its-implications-for-enterprise-ai-use-cases-and-governance-81602078f5db

1. Open-Source Enterprise
AI/ML Governance
Debmalya Biswas
@debmalyabiswas

2. Agenda
• Enterprise AI
• Ethical AI
– Regulations and Guidelines
• Explainability
• Bias and Fairness
• Accountability

3. Artificial Intellignence
• AI is the quest to build software
running on machines that can
'think' and act like humans.
• Machine Learning (ML) is a
subset of AI focused on the
design of learning algorithms, as
well as scaling existing
algorithms, to work with large
data sets.
• Deep learning (DL) is a branch
of ML that attempts to mimic the
working of the brain in the form of
multilayered neural networks. Source: Nvidia AI

4. (Supervised) Machine Learning
• Machine (Deep) Learning at its
core is the practice of using
algorithms to parse data, learn
from it, and then make an
inference or prediction about
something in the world.
• (So rather than hand-coding
software routines with a specific
set of instructions to accomplish
a particular task) the machine is
“trained” using large amounts of
data on an algorithm that gives
it the ability to learn how to
perform the task.
(Labeled)
Data
Train ML
Model
Predictions

5. Enterprise AI
• Enterprise AI use-cases
are pervasive

6. Open Source in AI
• and they get
implemented
and deployed
via a diverse
mix of
approaches.
Open-Source Innovation in
AI, ML, DL and Data (link)

7. Ethical AI
“Ethical AI, also known as responsible AI, is the practice of using AI with
good intention to empower employees and businesses, and fairly impact
customers and society. Ethical AI enables companies to engender trust and
scale AI with confidence.” [1]
Failing to operationalize Ethical AI can not only expose enterprises to
reputational, regulatory, and legal risks; but also lead to wasted resources,
inefficiencies in product development, and even an inability to use data to
train AI models. [2]
[1] R. Porter. Beyond the promise: implementing Ethical AI, 2020 (link)
[2] R. Blackman. A Practical Guide to Building Ethical AI, 2020 (link)

8. Regulations & Guidelines
• Good news: is that there has been a
recent trend towards ensuring that AI
applications are responsibly trained and
deployed, in line with the enterprise
strategy and policies.
• Bad news: Efforts have been
complicated by different governmental
organizations and regulatory bodies
releasing their own guidelines and
policies; with little to no standardization
on the definition of terms,
– “there are 20+ definitions of ‘fairness’” [1]
– “no single ethical principle was common
to all of the 84 documents on ethical
AI we reviewed” [2]
[1] S. Verma, J. Rubin. Fairness definitions explained, 2020 (link)
[2] A. Jobin, M. Ienca, E. Vayena. The global landscape of AI Ethics Guidelines, 2019 (link)
EU Ethics guidelines for
trustworthy AI (link)
UK Guidance on the AI
auditing framework (link)
Singapore Model AI
Governance Framework
(link)

9. Regulations & Guidelines (2)
• Software companies (e.g.
Google, Microsoft, IBM) and the
large consultancies (e.g.
Accenture, Deloitte) have also
jumped on the bandwagon,
publishing their own AI Code of
Ethics cookbooks.
• At this stage, they all seem like
public relations exercises, with
very little details on how to apply
those high-level principles,
across AI use-cases at scale.
AI at Google: our principles
(link)
Microsoft AI Principals
(link)
IBM Trusted AI for Business
(link)
Accenture Responsible AI: A
Framework for Building Trust in
Your AI Solutions (link)

10. Ethical AI - Principles
• Explainability
• Bias & Fairness
• Accountability
• (Reproducibility)
• (Data Privacy)
*Full article on Medium:
D. Biswas. Ethical AI: its implications for Enterprise
AI Use-cases and Governance, 2020 (link)

11. Explainable AI
• Explainable AI is an umbrella term
for a range of tools, algorithms and
methods; which accompany AI
model predictions with
explanations.
• Explainability of AI models ranks
high among the list of ‘non-
functional’ AI features to be
considered by enterprises.
– For example, this implies having to
explain why an ML model profiled a
user to be in a specific segment —
which led him/her to receiving an
advertisement.
(Labeled)
Data
Train ML
Model
Predictions
Explanation
Model
Explainable
Predictions

12. Explainable AI - Regulations
• Limits to decision making based solely
on automated processing and
profiling (Art. 22)
• Right to be provided with meaningful
information about the logic involved
in the decision (Art. 13, 15)
EU GDPR – Right to Explainability
• GDPR does not mandate the ‘Right to
Explainability’, rather it mandates the ‘Right
to Information’.
• GDPR does allow the possibility of
completely automated decision making as
long as personal data is not involved, and the
goal is not to evaluate the personality of a
user — Human intervention is needed in
such scenarios.

13. Explainable AI - Feasibility
• Machine (Deep) Learning algorithms
vary in the level of accuracy and
explainability that they can provide -
the two are often inversely
proportional.
• Explainability starts becoming more
difficult as as we move to Random
Forests, which are basically an
ensemble of Decision Trees. At the end
of the spectrum are Neural Networks
(Deep Learning), which have shown
human-level accuracy.
Explainability
Accuracy
Logistic Regression
Decision Trees
Random Forest
(Ensemble of
Decision Trees)
Deep Learning
(Neural Networks)

14. Explainable AI - Abstraction
AI Developer
Goal: ensure/improve
performance
Regulatory Bodies
Goal: Ensure compliance with legislation,
protect interests of constituents
End Users
Goal: Understanding of
decision, trust model output
“important thing is to explain the right thing to the right person in the right way at the right time”*
Singapore AI Governance framework: “technical explainability may not always be enlightening, esp. to the man
in the street… providing an individual with counterfactuals (such as “you would have been approved if your
average debt was 15% lower” or “these are users with similar profiles to yours that received a different
decision”) can be a powerful type of explanation”
*N. Xie, et. al. Explainable Deep Learning: A
Field Guide for the Uninitiated, 2020 (link)

15. Explainable AI - Frameworks
• Local Interpretable
Model-Agnostic
Explanations (LIME*)
provides easy to
understand explanations
of a prediction by
training an explainability
model based on samples
around a prediction.
• The approximate nature
of the explainability
model might limit its
usage for compliance
needs.
*M. T. Ribeiro, S. Singh, C. Guestrin. “Why Should I Trust You?” Explaining
the Predictions of Any Classifier, 2016 (link)
LIME output showing the important features, positively
and negatively impacting the model’s prediction.

16. Explainable AI - SOTA
Facebook’s ‘Why you’re seeing this Post’,
which is an extension of their ‘Why am I seeing this Ad’*
*CNBC. Facebook has a new tool that
explains why you’re seeing certain
posts on your News Feed, 2019 (link)

17. Bias
• Bias is a phenomenon that occurs
when an algorithm produces results
that are systemically prejudiced due
to erroneous assumptions in the
machine learning process*.
• AI models should behave in all
fairness towards everyone, without
any bias. However, defining
‘fairness’ is easier said than done.
– Does fairness mean, e.g., that the
same proportion of male and female
applicants get high risk assessment
scores?
– Or that the same level of risk result
in the same score regardless of
gender?
– (Impossible to fulfill both)
* SearchEnterprise AI. Machine Learning bias (AI bias) (link)
Google Photo labeling pictures of a black
Haitian-American programmer as “gorilla”
“White Barack Obama”
images (link)
A computer program used for bail and
sentencing decisions was labeled biased
against blacks. (link)

18. Types of Bias
• Bias creeps into AI models, primarily
due to the inherent bias already
present in the training data. So the
‘data’ part of AI model development is
key to addressing bias.
– Historical Bias: arises due to historical
inequality of human decisions
captured in the training data
– Representation Bias: arises due to
training data that is not representative
of the actual population
• Ensure that training data is
representative and uniformly
distributed over the target population
- with respect to the selected
features.
Source: H. Suresh, J. V. Guttag. A Framework for Understanding
Unintended Consequences of Machine Learning, 2020 (link)

19. Bias (Explainable AI) - Tools
Google’s Explainable AI
Service (link)
Azure Responsible ML
Service (link)
IBM AI Explainability 360
Toolkit (link)
PwC’s Responsible AI
Toolkit (link)Source: TensorFlow Fairness Indicators (link)
• TensorFlow Fairness
Indicators is a library that
enables easy computation
of commonly-identified
fairness metrics.
• We need to plan for bias
detection to be performed
on a continuous basis. As
new data comes in
(feedback loops), a model
that is unbiased today can
become biased tomorrow.

20. Accountability
• Similar to the debate on self-driving
cars with respect to “who is
responsible” if an accident
happens?
• The same debate applies in the
case of AI models as well — who is
accountable if something goes
wrong?, e.g. as explained above in
the case of a biased AI model
deployment.
• Accountability is esp. important if
the AI model is developed and
maintained by an outsourced
partner/vendor.
* S. Greenman. The risks of AI Outsourcing — how to
successfully work with AI Startups, 2019 (link)
Risks of AI
Outsourcing

21. Accountability - Checklist
What contractual promises should
we negotiate (e.g. warranties, SLAs)?
What measures to we need to
implement if something goes wrong
(e.g. contingency planning)?
Questions to consider/clarify before signing the
contract with your preferred partner
• Liability Given that we are engaging with a 3rd
party, to what extent are they liable? This is
tricky to negotiate and depends on the extent to
which the AI system can operate independently.
– For example, in the case of a Chatbot, if the bot
is allowed to provide only a limited output (e.g.
respond to a consumer with only limited number
of pre-approved responses), then the risk is likely
to be a lot lower as compared to an open-ended
bot* that is free to respond.

22. Accountability – Checklist (2)
If the vendor is generating the
training data, basically bearing the
cost for annotation; do we still
want to own the training data?
• Data ownership: Data is critical to AI systems,
as such negotiation of ownership issues around
not only training data, but input data, output
data, and other generated data is critical.
• For example, in the context of a Chatbot
communicating with our consumers:
– Input data could be the questions asked by
consumers whilst interacting with the bot.
– Output data could be the bot’s responses, i.e. the
answers given to the consumers by the bot.
– Other generated data include the insights
gathered as a result of our consumers use of the
AI, e.g. the number of questions asked, types of
questions asked, etc. * D. Biswas. Privacy Preserving Chatbot
Conversations, NeurIPS-PPML, 2020 (link)

23. Accountability – Checklist (3)
Who owns the rights of the underlying
algorithm? Is it proprietary to a 3rd party? If
yes, have we negotiated appropriate license
rights, such that we can we use the AI
system in the manner that we want?
When we engage with vendors to develop
an AI system, is patent protection possible,
and if so, who has the right to file for the
patent?
• Confidentiality and IP/Non-
Compete clauses: In addition to
(training) data confidentiality, do we
want to prevent the vendor
from providing our competitors with
access to the trained model, or at
least any improvements to it —
particularly if it is giving us a
competitive advantage?
• With respect to IP, we are primarily
interested in the IP of the source code
- at an algorithmic level.

24. Conclusion
As with everything in life, esp. in IT, there is no clear black
and white and a blanket AI policy mandating the usage of
only explainable AI models is not optimal — you will miss
out a lot on what non-explainable algorithms can provide.
• In terms of bias and explainability as well, we have the full
spectrum from ‘fully explainable’ to ‘partially explainable, but
auditable’ to ‘fully opaque, but with very high accuracy’.
• Depending on the use-case and geographic regulations, there is
always scope for negotiation.
• The regulations related to different use-cases (e.g. profiling,
automated decision making), are different in different
geographies.
AI Ethics
Committee