This document discusses approaches for adding trust, transparency and accountability to AI models deployed with KFServing. It proposes integrating open-source explainability, fairness and adversarial robustness tools like AIX360, AIF360 and ART to analyze model payloads and provide explanations. The tools would calculate metrics from logged predictions to detect bias or anomalies. Designs are presented for capturing events from KFServing in brokers like Kafka for offline processing. This would allow auditing models over time to ensure trusted performance.

1. KFServing Payload Logging
Animesh Singh, Clive Cox

2. Trust and Transparency
into AI..
Does it really matter?

3. It does…more so
in these
unprecedented
times….

4. “If we fail to make ethical
and inclusive artificial
intelligence we risk losing
gains made in civil rights
and gender equity under
the guise of machine
neutrality.”
Joy Buolamwini
Gender Shades
MIT Media Lab
Joy Buolamwini – MIT Media Lab <https://www.media.mit.edu/people/joyab/overview/> (CC BY 4.0).

5. Is it fair? Is it easy to
understand?
Is it accountable?
Did anyone
tamper with it?
FAIRNESS EXPLAINABILITY
ROBUSTNESS
LINEAGE
Vision for Trusted AI

6. Bias in AI
Example: Criminal Justice System
● Since 2008, nearly
every arrestee in
Broward County,
Florida has been
assigned a risk score
using Northpointe’s
COMPAS algorithm.
● Defendants with low
risk scores are
released on bail.
● It falsely flagged
black defendants as
future criminals,
wrongly labeling
them this way at
almost twice the rate
as white defendants
Machine Bias— ProPublica<https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing>.
.

7. AI needs to explain its decision!
And there are different ways to explain
One explanation does not fit all
Different stakeholders require explanations for different purposes and with
different objectives, and explanations will have to be tailored to their needs.
End users/customers (trust)
Doctors: Why did you recommend this treatment?
Customers: Why was my loan denied?
Teachers: Why was my teaching evaluated in this way?
Gov’t/regulators (compliance, safety)
Prove to me that you didn't discriminate.
Developers (quality, “debuggability”)
Is our system performing well?
How can we improve it?

8. https://arxiv.org/pdf/1707.08945.pdf
8
Adversarial threats to AI

9. Trust and Transparency
into AI..
How Open Source is addressing this…..

10. FAIRNESS EXPLAINABILITY
ROBUSTNESS LINEAGE
Trusted AI Lifecycle through Open Source..
Adversarial Robustness
360
↳ (ART)
AI Fairness
360
↳ (AIF360)
AI Explainability 360
↳ (AIX360)
github.com/trusted-
ai/adversarial-
robustness-toolbox
art-demo.mybluemix.net
github.com/ trusted-ai
/AIF360
aif360.mybluemix.net
• github.com/ trusted-ai
/AIX360
aix360.mybluemix.net
https://aifs360.myblue
mix.net/
Is it fair?
Is it easy to
understand?
Is it accountable?
Did anyone
tamper with it?
AI Factsheets
360

11. …..In Open Governance
Adversarial Robustness
360
↳ (ART)
AI Fairness
360
↳ (AIF360)
AI Explainability 360
↳ (AIX360)
github.com/trusted-
ai/adversarial-
robustness-toolbox
art-demo.mybluemix.net
github.com/ trusted-ai
/AIF360
aif360.mybluemix.net
• github.com/ trusted-ai
/AIX360
aix360.mybluemix.net
https://aifs360.myblue
mix.net/
AI Factsheets
360

12. ..And integrated with KFServing
https://ai-fairness-360.org/
https://ai-explainability-360.org/
https://adversarial-robustness-toolbox.org/

13. Alibi and Alibi Detect integrated with KFServing

14. AI Explainability and Adversarial Robustness
in KFServing
InferenceService
YAML
Kubernetes
API
ETCD
InferenceService
Controller
Webhook Configmap
(initial create)
Explainer
[Alibi, AIX360, ART]
Transformer
Predictor
Model Server
Model Agent
Queue proxy

15. AI Explainability 360 KFServing Spec

16. AIX 360 Explainer Example
AI Explainability 360 using LIME
AI Explainability 360-KFServing Integration

17. Adversarial Robustness 360 KFServing Spec

18. ART KFServing Example
Adversarial Robustness Toolbox (ART) using Square Attack
Adversarial Robustness Toolbox-KFServing Integration

19. What about more advanced metrics?
Bias, Drift, Outlier, Anomaly Detection….
Is single prediction enough?

20. AI Fairness 360
↳ (AIF360)
AIF360
20
Ø AIF360 toolkit is an open-source library to help detect
and remove bias in machine learning models. AIF360
translates algorithmic research from the lab into practice.
Applicable domains include finance, human capital
management, healthcare, and education.
Ø The AI Fairness 360 Python package includes a
comprehensive set of metrics for datasets and models to
test for biases, explanations for these metrics, and
algorithms to mitigate bias in datasets and models.
Ø Fairness metrics (70+)
Fairness metric explanations
Bias mitigation algorithms (10+)
Ø http://aif360.mybluemix.net/

21. AIF360 in KFServing
AIF calculates metrics from the payload logs of a model’s previous
predictions
Model Persistent Storage
Prediction
request Store logs
Analyze
prediction logs
AIF360
Explainer
Bias Metrics

22. Payload Logging for analysis and offline processing
Why:
● Capture payloads for analysis and
future retraining of the model
● Perform offline processing of the
requests and responses
KFServing Implementation (alpha):
● Add to any InferenceService Endpoint:
Predictor, Explainer, Transformer
● Log Requests, Responses or Both from
the Endpoint
● Simple specify a URL to send the
payloads
● URL will receive CloudEvents
POST /event HTTP/1.0
Host: example.com
Content-Type: application/json
ce-specversion: 1.0
ce-type: repo.newItem
ce-source: http://bigco.com/repo
ce-id: 610b6dd4-c85d-417b-b58f-3771e532
<payload>

23. Parameters for AIF360 in KFServing

24. Current Samples in KFServing
The current demos uses the logger to send events to the
message dumper for displaying the events as container logs.
Then use a local Jupyter notebook to parse those logs and
display the requests/response payload

25. What we are working towards….

26. Extending Payload Logging Design

27. The Goal of the initial design is to minimize the amount of custom code and try to use out of the box components as much as
possible. The AIF360 bias detector with self queuing approach
Flow:
Ø Logger sends events to the eventing broker ingress. If the broker ingress is behind an eventing channel, the events will be
persistent with the channel config such as Kakfa.
Ø Trigger watch all the broker events and filter them with an attribute. Then sends the matched events to the attribute subscribers.
Ø AIF360 is the subscriber of this trigger. The AIF360 server needs to have some queuing method to stall all the events. Once it
stalls enough events, it will process them as a batch and calculate the bias metrics based on that.
Ø The AIF360 Server can push all the AIF360 metrics and processed events to Prometheus or to a message dumper for displaying
the results
Custom containers:
● AIF360 bias detector with self queuing.
Extending Payload Logging Design

28. Ideal Design (Single User)
SK-Learn model
with logger
(kafka_connect
schema)
Eventing Broker
Ingress
Cloud event v1
Eventing
Channel
Channel pods
(In-mem, Kakfa, NATS, GCP pub/sub)
Kafka cluster
AIF360
bias detector
(without self-queuing) Visualizer UI
Results
Persistent DB
Pull Batch data and
push back fairness results
Kafka connect with DB driver
(consumer)
1 2
3
4
5
Prometheus
Apply Channel and
Kafka settings (Topics)
Kafka API
Metrics transformer
6
Pull db tables
Push time series metrics
topic

29. Ideal Design (Single User)
The Goal of the ideal design is to have more flexibility on where we can capture and store events. It also decouple several components to
make it more scalable. Please be aware that Kafka and sink is not necessary in this design. Kafka is here to showcase how easy to plugin a
new streaming services for capturing the event flow at different levels.
Flow:
Ø Logger sends Cloud Events with Kafka connect schema to the eventing broker ingress that’s behind an eventing channel.
Ø The Cloud Event will be translated and be persisted with the channel config such as Kafka using the Kafka API.
Ø The Kafka connect (standalone) is processing the streaming queue. This Kafka connect server transforms all the events and persist them
to a relational db.
Ø The AIF360 Server need to take batched events based on the time period and requests from the database. Then it calculate the bias
metrics store the results back to the relational db.
Ø The Metrics transformer takes all the explainer events/metrics such as ART, AIF, AIX and transform them into a set of time series metrics
that Prometheus can understand.
Ø A visualizer UI is needed for displaying the Prometheus data. It can be a Grafana UI with our custom Grafana graphs.
Custom containers:
● AIF360 bias detector (no custom queuing)
● Event processing container
● Visualizer UI

30. Multi-User (single event flow)
SK-Learn model
with logger
(kafka_connect
schema)
Eventing Broker
Ingress
Cloud event v1
Channel pods
(In-mem, Kakfa, NATS, GCP pub/sub)
Kafka cluster
AIF360
bias detector
(without self-queuing) Visualizer UI
Results
Persistent db
Pull Batch data and
push back fairness results
Kafka connect with db driver
(consumer A)
1 2
3
4
5
Prometheus
Apply Channel and
Kafka settings (Topics, auth)
Kafka API with SASL auth
Metrics transformer
6
Pull db tables
Push time series metrics
ACL: topic A
Eventing
Channel

31. Multi-User (user management)
Eventing
channel A CRD*
(with topic A)
Kafka Channel controller**
Kafka cluster
consumer A
2
3
Kafka API with
SASL auth for the user topic
Topic A
User management
and Authorization
(ACL)
Namespace A Namespace B
Eventing
channel B CRD*
(with topic B)
consumer B
Topic B
3
Kafka Broker**
*All the CRD workloads are
handled in the controller in the
Knative control plane namespace.
Knative Webhook**
Broker CRD* Broker CRD*
Apply Channel and
Kafka topic settings (Topics, auth)
Knative Eventing Control plane
(0.17+ with V1 CRD only)
Zookeeper
SK-Learn model
with logger
Cloud event v1
1
Namespace Kafka
**The channel controller and broker
has a dispatcher for offloading the
requests. It's omitted in the diagram
for simplification.
*The total number of pods in the Knative
Control plane is 8 (single replica) + 3
default TM channel pods that are
manually removable.

32. Multi-User
For Multi-users, each namespace can only have one channel and that channel will bind with an authentication method such
as SASL. On the Kafka level, it will use the same SASL to verify the Kafka API calls and authorize with zookeeper.
Zookeeper then handle the ACL to manage topic permissions for each user.
Flows:
1. KFServing logger sends a cloud event with SASL authentications (TBD) to the Knative broker.
2. KNative broker will verify the authentication and send the request to Kafka. Zookeeper then authorize the request and
determine the permissions. If allows, store the events into the specified topic.
3. Consumers can listen to the topic stream if they have the ACL. Here, authentications are done with the Kafka API.
4. If multiple users are sharing the same database and AIF server, then those components also need to implement ACL
and possibly some kind of authorization.
Challenges:
1. KFServing doesn’t have the concept of users, so anyone who login into Kubeflow have access to all KFServing model
endpoints. This also means KFServing doesn’t know what user token it should use when sending logger events to the
authenticate eventing channel.

33. Data Schema
Column Name Data Type Description
ts timestamp Measurement timestamp
mid str Uuid for the transaction
measurement str Description of what the measurement
value str Json string of the measurement
binding_id string ID of a service binding
subscription_id string ID of a subscription
deployment_id string ID of a deployment
asset_revision string ID of an asset revision (version)
process string Name of the producer process
KFServing payload logging using
Kafka
insert_json = {
"ts": time.time(),
"mid": str(uuid.uuid4()),
"measurement": "fairness",
"value": json.dumps(metrics_json),
"binding_id": None,
"subscription_id": None,
"deployment_id": model_name,
"asset_revision": None,
"process": "aif-cronjob"
}
insert_stmt = "INSERT INTO " + table + " ("
+ ", ".join(insert_json.keys()).rstrip(", ") + ") "
values_stmt = "VALUES (" + ("%s, " *
len(insert_json.keys())).rstrip(", ") + ")"

34. Iter8 Integration with KFServing
• Effort to standardize
around capturing
KFServing metrics and
taking actions based on
metrics
• Use an iter8 experiment
to safely expose
competing versions of a
model to traffic, gather
in-depth insights about
key performance and
metrics for your model
versions, and
intelligently rollout the
best version of your
models

35. LFAI Trusted AI
Committee
https://wiki.lfai.foundation/display/DL/Trusted+AI+Committee
Bring Trust, Transparency and Responsibility into AI
TRUSTED AI
ü Principles Working Group
ü Technical Working Group
Chairs Region Company
Animesh Singh North America IBM
Souad Ouali Europe Orange
Jeff Cao Asia Tencent

36. Links
Github:
https://github.com/kubeflow/kfserving/tree/master/docs/samples/explanation/a
ix/mnist
https://github.com/drewbutlerbb4/kfserving/tree/trusted-ai-
robustness/docs/samples/explanation/art/mnist
Design Document:
https://docs.google.com/document/d/1avFbKVcaV9zFfEGjvQCGYjaOG8Q_Cz2rk
TSVZOfme00/edit?usp=sharing