AI/ML Infra Meetup | ML explainability in Michelangelo

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

ML Explainability in
Michelangelo
Eric Wang
Michelangelo - Uber ML Platform Team

01 Challenges and Needs
02 Importance of ML explainability
03 Explainers
04 Architecture
05 User workﬂow and case studies
...

Challenges and Needs
Needs
- Understand and interpret how models make decisions.
- Provide transparency and understanding ...

Challenges and Needs
Model performance
Feature null rate monitoring
Michelangelo provided visual interfaces for:

Importance of Model Explainability
model_1
score
model_2
score
… ...
better
model?
better score?
??
Summary stats (AUC, MA...

User’s Request
Making models more transparent and
interpretable
Needs to implement Explainable AI (XAI) for their Keras mo...

Importance of Model Explainability
Model debugging in Michelangelo: to make the 80% effort more efficient and effective.
Expla...

Explanation methods

TreeShap
Interactive tree ensemble model visualizer on frontend
Data source: any serialized tree model (Spark GBDT, XGBoos...

KernelShap
The good
Model Agnostic
Local explanation support
Captures Feature Interactions
Comprehensive Explanations
The ...

Integrated gradients
Why?
● Gradient based with baselines comparison
● a popular interpretability technique to any
diﬀeren...

Integrated gradients
Beneﬁts
● Completeness
● Interpretability
● Feature dependency
agnostic
● Eﬃciency
Feature values
Pre...

Integrated gradients
Notes
● Flatten Input features
● Choose the right layers (especially with categorical features)
● Use...

Using integration gradients
Model, model and model
Explainer
Model packaging for serving
Basis Feature
set
Feature joins
P...

Using integration gradients
Save DL model separately
Explainer
Train Serving model
Raw model
Deploy to
endpoint
[keras.mod...

Using integration gradients
Flatten input features
- Entity - an Uber business entity
such as city, rider, and driver (ex:...

Importance
level
Using integration gradients
Flatten input features
Entity
Feature
Group 1
Feature
Group 2
Feature
Group 3...

Importance
level
Using integration gradients
Flatten input features
Feature 1 Feature 2 Feature 3
Input to
model
Vectorize...

Using integration gradients
Choose the right layers
● Support both pyTorch
and Keras
● Support gradients on
input or outpu...

Explainer
Using integration gradients
Use model wrapper to aggregate all outputs if possible
Model prediction pipelines
Ba...

Using integration gradients
Use model wrapper to aggregate all outputs if possible
Model prediction pipelines
Basis
Featur...

ML explainer in Michelangelo
Notebooks
1. Model debugging
2. Feature importance comparing
3. Visualization
Enabled for use...

● Backed in intuitive notions of what a good explanation
● Allows for both local and global reasoning, and it is
● Model a...

ML explainer in Michelangelo
Generate feature importance in training pipeline
Model training pipelines
Basis Feature
set
F...

ML explainer in Michelangelo
Monitoring Pipelines
1. Generate features importance during training time
2. Diﬀerent thresho...

Case Studies
Case 1. Identifying Useful Features
suburb Non-suburb
compare
Scenario
- A team at Uber is evaluating the ord...

Scenario
- A team at Uber is evaluating the order
conversion rate is very different
between suburb and non-suburb areas.
-...

Scenario
A team want to see what photos the model
predicted incorrectly. Since our action involves
a cost associated with ...

Architecture
XAI framework

Architecture
Components:
1. Data processing
a. Converting from pySpark to numpy
b. Feature ﬂatten for calculating gradient...

● Support LLM explanation in prompting engineer
● Feature selection assistant
● Interactive visualization tools
Future opp...

Q&A

AI/ML Infra Meetup | ML explainability in Michelangelo

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1059379 views
Storytelling For The Web: Integrate... by Chiara Aliotta 976667 views
Artificial Intelligence, Data and C... by OECD Directorate ... 914599 views
How to Leverage AI to Boost Employe... by SocialHRCamp 387831 views
2024 State of Marketing Report – by... by Marius Sescu 219508 views
Everything You Need To Know About C... by Expeed Software 236350 views

View on Slideshare

1 of 33 Ad

View on Slideshare

1 of 33 Ad