Tsvi Lev. Practical Explainability for AI - with examples

Tsvi Lev,
Corporate VP and Head of NEC Labs Israel
tsvi.lev@necam.com
XAI Beyond the Hype
#MAXA

2 © NEC Corporation 2015 NEC Group Internal Use Only2 © NEC Corporation 2019
NEC – we too were once a startup..

120 Years of Research Innovation

Applied AI is quite different from Kaggle competitions or papers
NEC Labs do Applied AI for vision, healthcare, cyber, edge. Our Customers:
▌don’t care about SOTA or AUC
▌lack training data
▌want AI to behave ‘like us’
ACM CCS Best Paper Edge AI for Vision Cyber Research AI for Healthcare

▌ The recent AI craze started with Digital Marketing success ($$$) and
Academic breakthroughs in Image and Speech Processing
▌Other application areas have different requirements
The truth about recent AI success
More $$$!
Better than Human

▌ Recommendation Engines replaced rule based systems
▌ Measured only by results (clicks, sales)
▌ Recently – issues of Bias and Accountability
Use Case: digital marketing
Accuracy
Stability
Compliance
Explainablity
Accountability
Bias Free

▌Replace non-expert Humans
▌Acceptance threshold prevents ‘just better than previous’ criteria
▌Stability and Compliance are key from day 0
▌Hence – slower than promised and hyped
Use Case: Mobility
Accuracy
Stability
Compliance
Explainablity
Accountability
Bias Free

▌ Assist, not replace Expert Human Judgement.
▌ Xplain-ability and Stability are >> than accuracy
▌ Hence – slower, but progress is stable
Use Case : Medical Diagnostics
Accuracy
Stability
Compliance
Explainablity
Accountability
Bias Free

The Future will be dominated by XAI
“Confidence in AI only comes with explainablity and
transparency” – Well Known B2B AI vendor

When do we most need XAI the most?
Recommendation Engines Man Machine Interfaces
Online Advertising Gaming
Single Decision=> Lives, much $$$ Many Micro Decision Aggregated

What may happen without XAI – AUC/ROC deterioration
Populations, physicians, imaging machines change – how to ensure AI is constant?
Benign Malignant! Benign

Users attach different gravity to certain error cases
▌ Errors in “simple” cases: blood vessel vs nodules
▌ Distribution of errors: no false alarms, then suddenly a ton for no reason
▌ “Butterfly in Asia” – tiny changes in input yield large changes in the result

Which Models are a best fit for XAI?

Model type does not guarantee explainability
MNIST decision tree (Hinton et al)

Explainable can mean different things
Causality Analysis
Reliability/Abstention
Model Improvement
Pattern Discovery
Post-hoc: methods to be
applied to other models.
IBM Research

Causality Analysis
Local Interpretable Model-agnostic Explanations (LIME)

Reliability/Abstention Criteria
▌Threshold – estimate the reliability by the output magnitude
▌Distance to labeled examples –
⚫Let V be the output vector of the network prior to the final SoftMax layer
⚫Per sample, we compute the distance between V(sample) and V(train) for the training set
⚫Check how consistent the network is across the closest training examples
Labeled Dataset FC outputs

Abstention Criteria
Compare activations along
the feedforward path

Model Improvement
Find similar labeled cases
- Requires a fine ‘matching’ measure
- Works well only if labelled set is ‘super-dense’
- Hence – apply continuous deformable transformations from
the new sample to the closest training set cases

Explainablity and its relation to Augmentation
▌Given a set of legitimate transformations {Ei(X)}, where X is a labeled
input vector, an explanation for the classification of another vector Y is a
series of infinitesimal transformations which starts with the labeled
example X and ends in Y
▌The ‘shorter’ the series, the ‘better’ the explanation
▌Legitimate transformation: class of {Ei(X)}=class of X (or Y).
VS
IBM Research

Some Visualizations to the rescue

The path is not a trivial function..
▌For example, simple averaging is not OK
▌But applying some deformable image transformation is OK:

Luckily, Medical Imaging Research has the same challenge
▌Human organs change in time, and automatic registration is required

▌Using Lenet-5, for a new sample, we find a labeled candidate ‘nearest
neighbor’ using the before-last FC layer activations.
▌We apply a deformable DNN to ‘reach’ sample from training patch
▌One can estimate ‘reliability’ using number of activation flips along path
between labeled and new sample
Sandbox case – MNIST...
New Sample Labeled Sample

▌If new sample is mislabeled by the network, the ‘path’ is used as new
training samples.
▌Very fast convergence and higher accuracy reached easily.
Sandbox case – MNIST (cont.)
0 0 0 6 6 6 6 6 6 6
distance of FC layer from labelled image
(blue) and new image (red)
I. Aganj, J. E. Iglesias, M. Reuter, M. R. Sabuncu,
and B. Fischl, "Mid-space-independent deformable
image registration”

▌Train Model
▌Find ‘glaring error’ Y according to auditor/customer
▌Find labeled nearest neighbor(s) X in “last FC” space.
▌Create a legitimate transform X->Y
▌Label images along the transform
▌Re-train, repeat
Summary of the explainability=>model improvement cycle
7 7 7 7 7 7 X X 1 1

▌We seek to segment via match to similar training patch
▌We apply a deformable DNN to ‘reach’ sample
patch from training patch
▌If we can’t find a train+transform
path, the pixel is not on a blood vessel.
28X28 patch in
new sample
28X28 patch in
training set
Deformable
DNN
Sample Medical case - blood vessel segmentation

Blood vessel segmentation

Up Close, segmentation is not so easy

The ‘Human explanation’
▌The Ophthalmologist knows blood vessels are connected and continuous.
▌Sees the whole picture (pattern of vessel) and ‘completes’ by intuition
▌DNN sees a region, has no built-in concept of connectedness or larger scale.

Good augmentation can use a small training dataset
2D UNET Data
Each layer contains two 3 × 3 convolutions each
followed by a leaky rectified linear unit
(LeakyReLU), and then a 2 × 2 max pooling with
strides of two in each dimension. In the synthesis
path, each layer consists of an upconvolution of 2
× 2 by strides of two in each dimension, followed
by two 3 × 3 convolutions each followed by a
LeakyReLU. Shortcut connections from layers of
equal resolution in the analysis path provide the
essential high-resolution features to the synthesis
path. In the last layer a 1×1 convolution reduces
the number of output channels to the number of
label.
Train data – 2 images
Val data – 57 images
Test data – 20 images
Data Augmentation:
1. Elastic Distortions
2. Perspective Transforms
3. Size Preserving Rotations
4. Size Preserving Shearing
5. Cropping
6. Rotate
7. Flip
8. Zoom
9. Color
10. Contrast
Outcome Visualization
Network Result

Details of the solution (I)
▌Due to the combinatorial explosion, we search for the minimum size window
in which it seems a DNN can still reach SOTA results.
▌For 2D BV Segmentation it seems region of 64X64->24X24 has a plateau in
performance.
▌In addition to ‘default’ augmentations (rotation, scaling etc.) we apply
bicubic spline based deformable image registrations, e.g. Eppenhof, K.
A., & Pluim, J. P. (2018). Pulmonary ct registration through supervised learning with
convolutional neural networks. IEEE transactions on medical imaging.

Details of the solution (II)
▌For the new ‘test’ image, we find for each patch the most similar ‘existing
past case’. For that we use Locality-Similar Hashing, working on the actual
patch grayscale values as the input vector
▌With a 28X28=784 element vector, a hash size of 36 and 10 hash tables, we
do an LSH search with Euclidean distance measure.
▌For the first N=5 candidates, we compute the normalized cross correlation of
the training patch and the test patch and choose highest one.
▌An NCC >0.85 is considered a match.
▌If needed, for an even stronger match (NCC => 1.0) match, one can use
DNN based unsupervised deformable image registration.
de Vos, Bob D., et al. "End-to-end unsupervised deformable image registration with a
convolutional neural network." Deep Learning in Medical Image Analysis and Multimodal
Learning for Clinical Decision Support. Springer, Cham, 2017. 204-212.

Sanity checks
Train Sample – all white (explainable) Artificially Added digits – not explainable

Xplain-able AI in Cyber
▌Purpose: Automatic Anomaly Detection in SOC logs.
▌SOC engineers need to trust AI alerts
▌Solution: retrieve relevant training samples based on AutoEncoders, LIME to mark log
entries

Xplain-able AI in Edge AI
▌Purpose: Pedestrians/electric mobility analytics
▌Failure modes must be understood and predictable PRIOR to use
▌Solution: component-based detection + Synthetic Sample Generation

Xplain-able AI in Medical Diagnostics

AI in space

tsvi.lev@necam.com

Tsvi Lev. Practical Explainability for AI - with examples

Tsvi Lev. Practical Explainability for AI - with examples

Recommended

Recommended

More Related Content

Similar to Tsvi Lev. Practical Explainability for AI - with examples

Similar to Tsvi Lev. Practical Explainability for AI - with examples (20)

More from Lviv Startup Club

More from Lviv Startup Club (20)

Recently uploaded

Recently uploaded (20)

Tsvi Lev. Practical Explainability for AI - with examples