Slides from my talk at Deep Learning World 2020. The talk covered use cases, special challenges and solutions for building Interpretable and Secure AI systems using Pytorch. - Tools for building Interpretable models - How to build secure, privacy preserving AI models with Pytorch - Use cases and insights from the field

1. BUILDING
INTERPRETABLE &
SECURE AI
SYSTEMS USING
PYTORCH
GEETA CHAUHAN
AI PARTNER ENGINEERING
FACEBOOK AI

2. AGENDA 01
INTERPRETABLE AI
02
SECURE & PRIVACY PRESERVING AI
03
REFERENCES

3. I N T E R P R E T A B I L I T Y

4. WHAT IS MODEL INTERPRETABILIT Y ?
“THE ABILIT Y TO DESCRIBE AI MODEL INTERNALS AND PREDICTIONS IN HUMAN
UNDERSTANDABLE TERMS*”
* LH Gilpin, et. al., Explaining explanations: An overview of interpretability of machine learning in IEEE 5th International Conference on data science and advanced analytics (DSAA), 2018

5. I N C R E A S E D
T R A N S P A R E N C Y
D E B U G G I N GB E T T E R
U N D E R S TA N D I N G
MODEL INTERPRETABILIT Y
Screenshot of the tool
Attributing to dog
Attribution Magnitudes

6. MODEL INTERPRETABILIT Y LIBRARY FOR PY TORCH
M U LT I M O D A L E A S Y T O U S EE X T E N S I B L E
class MyAttribution(Attribution):
def attribute(self, input, ...):
attributions = self._compute_attrs(input, ... )
# <Add any logic necessary for attribution>
return attributions
visualize_image_attr(attr_algo.attribute(input), ...)
captum.ai

7. GradientSHAP
DeepLiftSHAP
SHAP Methods Integrated Gradients
Saliency
GuidedGradCam
Attribute model output (or internal neurons) to input
features
LayerGradientSHAP
LayerDeepLiftSHAP
SHAP Methods
LayerConductance
InternalInfluence
GradCam
Attribute model output to the layers of the model
DeepLift
NoiseTunnel (Smoothgrad, Vargrad, Smoothgrad Square)
LayerActivation
LayerGradientXActivationLayerDeepLiftFeatureAblation /
FeaturePermutation
GuidedBackprop /
Deconvolution
AT TRIBUTION ALGORITHMS
Input * Gradient LayerFeatureAblation
LayerIntegratedGradients
Occlusion
Shapely Value Sampling
Gradient
Perturbation
Other

8. attributions = Attribution(forward_func, ...).attribute(inputs, ...)*
* Check out our Getting Started docs and API:
https://github.com/pytorch/captum
https://captum.ai/api/
GradientAttribution PerturbationAttribution
IntegratedGradients FeatureAblation
... ...

9. attributions = Attribution(forward_func, ...).attribute(inputs, ...)*
* Check out our Getting Started docs and API:
https://github.com/pytorch/captum
https://captum.ai/api/
the importance of
inputs to forward_func
model's forward function or
any modification of it

10. EXPL AINING WITH
INTEGRATED GRADIENTS
FEATURE 0
FEATURE 1
FEATURE 2
TARGET 0
TARGET 1
from captum.attr import IntegratedGradients
attr_algo = IntegratedGradients(model)
input = torch.rand(1, 3)
attributions = attr_algo.attribute(input, target=0)

11. EXPL AINING WITH
INTEGRATED GRADIENTS
from captum.attr import IntegratedGradients
attr_algo = IntegratedGradients(model)
input = torch.rand(1, 3)
attributions = attr_algo.attribute(input, target=0)
FEATURE 0
FEATURE 1
FEATURE 2
TARGET 0
TARGET 1

12. EXPL AINING WITH
INTEGRATED GRADIENTS
from captum.attr import IntegratedGradients
attr_algo = IntegratedGradients(model)
input = torch.rand(1, 3)
attributions = attr_algo.attribute(input, target=0)
FEATURE 0
FEATURE 1
FEATURE 2
TARGET 0
TARGET 1
OUTPUT
attributions: tensor([[-0.41, 0.54, 0.88]])

13. EXPL AINING WITH
INTEGRATED GRADIENTS
from captum.attr import IntegratedGradients
attr_algo = IntegratedGradients(model)
input = torch.rand(1, 3)
attributions, delta = attr_algo.attribute(input,
target=0,
return_convergence_delta=True)
FEATURE 0
FEATURE 1
FEATURE 2
TARGET 0
TARGET 1
OUTPUT
attributions: tensor([[-0.41, 0.54, 0.88]])
delta: 0.0190

14. EXPL AINING WITH
INTEGRATED GRADIENTS
from captum.attr import IntegratedGradients
attr_algo = IntegratedGradients(model)
input = torch.rand(1, 3)
baseline = torch.rand(1, 3)
attributions, delta = attr_algo.attribute(input,
target=0,
return_convergence_delta=True,
n_steps=5000,
baselines=baselines)
FEATURE 0
FEATURE 1
FEATURE 2
TARGET 0
TARGET 1
OUTPUT
attributions: tensor([[0.0, 0.88, -2.45]])
convergence delta: 1.5497e-06

15. ORIGINAL IMAGE ATTRIBUTING* TO DOG ATTRIBUTING* TO CAT
* MATTHEW D ZEILER, ROB FERGUS, OCCLUSION: VISUALIZING AND UNDERSTANDING CONVOLUTIONAL NETWORKS, IN SPRINGER INTERNATIONAL PUBLISHING SWITZERLAND, 2014
VISUALIZATIONS USING RESNET152 MODEL

16. VISUALIZING
EXPL ANATIONS OF
A TEXT CL ASSIFICATION
MODEL USING IMDB
DATASET
WITH
CAPTUM INSIGHTS

17. VISUALIZING
EXPL ANATIONS OF
MULTIMODAL VQA
MODELS
WITH
CAPTUM INSIGHTS

18. VISUALIZING
EXPL ANATIONS OF
A 3-L AYER MLP MODEL
USING TITANIC DATASET
WITH
CAPTUM INSIGHTS

19. CASE STUDY FOR BERT MODELS

20. EXPL AINING BERT MODELS
+ Fine-tuning BERT model for Question Answering on SQUAD dataset
+ Evaluating on Dev Set
Exact Match: 78%
F1-Score: 86%
+ Understanding the importance of different types of word tokens, layers and neurons
+ Already existing research in understanding and visualizing attention heads
+ What Does BERT Look At? An Analysis of BERT's Attention, Clark, et. al. 2019, BlackBoxNLP@ACL
+ ExBERT: A Visual Analysis Tool to Explore Learned Representations in Transformers Models, Hoover, et. al., 2019,

21. EXPL AINING BERT MODELS FOR QUESTION ANSWERING
text = 'It is important to us to include, empower and support humans of all kinds.'
question = 'What is important to us?'
[CLS]
tokens
what [SEP]to isimportant ?is us it important to us
to include em, and support humans of all kinds .##power
P(Start Position) = 0.72 P(End Position) = 0.73
[SEP]

22. # explaining layers
for i in range(model.config.num_hidden_layers):
lc = LayerConductance(squad_pos_forward_func,
model.bert.encoder.layer[i])
layer_attributions_start = lc.attribute(
input_embed, baselines=ref_emb, ..., 0))
layer_attributions_end = lc.attribute(
input_embed, baselines=ref_emb, ..., 1))
EXPL AINING BERT MODELS FOR
QUESTION ANSWERING

23. AT TRIBUTION HEAT MAP OF ALL TOKENS ACROSS ALL 12 BERT L AYERS FOR START POSITION
PREDICTION

24. AT TRIBUTION HEAT MAP OF ALL TOKENS ACROSS ALL 12 BERT L AYERS FOR END POSITION
PREDICTION

25. THE LIMITATIONS OF AT TRIBUTIONS
+ Attributions do not capture feature correlations and interactions
+ Finding good baselines is challenging
+ They are difficult to evaluate
+ Attributions do not explain the model globally

26. FUTURE DIRECTIONS
+ captum.robust
+ adversarial robustness and attacks
+ studying the connections between
model robustness and interpretability
+ captum.metrics
+ model interpretability, sensitivity, trust, infidelity
and robustness related metrics
+ captum.benchmarks
+ benchmarks for different datasets and methodologies
+ sanity checks
+ captum.optim
+ optimization-based visualizations
...

27. S E C U R E & P R I V A C Y
P R E S E R V I N G A I

28. IS IT POSSIBLE TO:
answer questions using
data we cannot see?

29. What do handwritten
digits look like?
◆ Step 1: Download data
◆ Step 2: Download SOTA training script
◆ Step 3: Run script.

30. Source: Wikipedia Commons
What do tumors
look like in humans?
◆ Step -1: Persuade a VC.
◆ Step 0: Buy a dataset from a hospital.
◆ Step 1: Download millions of tumor images.

31. Getting access to
private data is HARD!

32. We SOLVE tasks which
are accessible:
✓ ImageNet
✓ MNIST
✓ CIFAR-10
✓ Librispeech
✓ WikiText-103
✓ WMT
◆ Cancer
◆ Alzheimers
◆ Dementia
◆ Depression
◆ Anxiety
◆ … Covid-19 Cure?
… but what about?

33. TOOLS
+ Remote Execution
+ OpenMined PySyft
+ Search and Example Data
+ OpenMined PyGrid
+ Differential Privacy
+ OpenMined PyDP
+ Secure Multi-Party Communication
+ CrypTen.ai

34. INTRODUCING

35. CRYPTEN import crypten
import torch
crypten.init() # sets up communication
x = torch.tensor([1.0, 2.0, 3.0])
x_enc = crypten.cryptensor(x) # encrypts tensor
x_dec = x_enc.get_plain_text() # decrypts tensor
assert torch.all_close(x_dec, x) # this passes!
y_enc = crypten.cryptensor([2.0, 3.0, 4.0])
xy_enc = x_enc + y_enc # adds encrypted tensors
xy_dec = xy_enc.get_plain_text()
assert torch.all_close(xy_dec, x + y) # this passes!
z = torch.tensor([4.0, 5.0, 6.0])
xz_enc = x_enc + z # adds FloatTensor to CrypTensor
xz_dec = xz_enc.get_plain_text()
assert torch.all_close(xz_dec, x + z) # this passes!
K E Y F E AT U R E S :
• Tensors and CrypTensors coexist and can be mixed
and matched
• Uses standard eager execution — No compilers! Easy
debugging and learning
• Support for Secure multi-party computation (MPC)
A platform for research in machine learning using
secure-computation techniques

36. B

37. HELLO
CRYPTENSOR
1. CrypTensor wraps an implementation that does:
1. Arithmetic secret sharing.
2. XOR secret sharing.
3. Conversions between both secret sharings.
4. A large number of operations.
2. CrypTensor exposes these via a PyTorch-like API.
PyTorch LongTensor
Binary (XOR) Sharing
CrypTensor
Arithmetic Sharing
Trusted Party
Numerical Library
Secure Computation Primitives
Secure Computation Protocol
Protocol-Independent Layer
uses
abstracts
uses
AutogradCrypTensor
MPCTensor
B2A/A2B
Conversion
uses
Automatic di erentiation
User-level code Neural networks, etc.
uses
Parties

38. 1. Make a CrypTen Model.
2. Encrypt Data
3. Train!
ENCRYPTED TRAINING
import crypten
crypten.init() # sets up communication
class LogisticRegression(crypten.nn.Module):
def __init__(self):
super().__init__()
self.linear = crypten.nn.Linear(28 * 28, 10)
def forward(self, x):
return self.linear(x)
model = LogisticRegression().encrypt() # encrypts tensor

39. 1. Join Encrypted Data
2. Encrypt Model
3. Train!
Training Across Par ties
import crypten
crypten.init() # sets up communication
alice_images_enc = crypten.load("/tmp/data/alice_images.pth", src=ALICE)
bob_labels_enc = crypten.load("/tmp/data/bob_labels.pth", src=BOB)
model = LogisticRegression().encrypt()
train_model(model, alice_images_enc, bob_labels_enc)

40. 1. Create a PyTorch or ONNX model.
2. Import model into CrypTen.
3. All computations are now encrypted.
PY TORCH / ONNX
INTEGRATION
import torchvision.datasets as datasets
import torchvision.models as models
# download and set up ImageNet dataset:
transform = transforms.ToTensor()
dataset = datasets.ImageNet(
imagenet_folder,
transform=transform,
)
# download pre-trained ResNet-18 model and encrypt it:
model = models.resnet18(pretrained=True)
encrypted_model = crypten.nn.from_pytorch(
model, dataset[0],
)
# do inference on encrypted images with encrypted model:
encrypted_image = crypten.cryptensor(dataset[1])
encrypted_output = encrypted_model(encrypted_image)
output = encrypted_output.get_plain_text() # this works

41. USE CASES
+ COVID-19 Sols
+ Cancer Research
+ Integrity (eg PhotoDNA project)
+ Federated AI across Enterprise Silos
+ What problems will you solve?

42. • Captum: https://captum.ai/
• Captum Blog: https://bit.ly/2vHBxJI
• Captum Algorithms Matrix: https://captum.ai/docs/algorithms_comparison_matrix
• Interpreting MultiModal models: https://captum.ai/tutorials/Multimodal_VQA_Interpret
• Interpretable ML Book: https://christophm.github.io/interpretable-ml-book/
• Crypten: https://crypten.ai/
• CrypTen Tutorials: https://github.com/facebookresearch/CrypTen#how-crypten-works
• OpenMined: https://www.openmined.org/
• OpenMined for Covid-19 Apps: https://blog.openmined.org/providing-opensource-privacy-for-covid19/
• Udacity Course: https://www.udacity.com/course/secure-and-private-ai--ud185
• Active Federated Learning Paper: https://arxiv.org/pdf/1909.12641.pdf
• Microsoft PhotoDNA Project: https://www.microsoft.com/en-us/photodna
REFERENCES

43. QUESTIONS?
Contact:
Email: gchauhan@fb.com
Linkedin: https://www.linkedin.com/in/geetachauhan/