Youtube: https://www.youtube.com/watch?v=ttmm2yo74z8 Moustafa Alzantot is a Ph.D. Candidate in Computer Science at UCLA. His research interests include machine learning, privacy, and mobile computing. He is an inventor of two US patents and the recipient of several awards including the COMESA 2014 innovation award. He worked as an intern at Google, Facebook, and Qualcom. Yash Sharma is a visiting scientist at Cornell who recently graduated with a Bachelors and Masters in Electrical Engineering. His research has focused on adversarial examples, namely pushing the state-of-the-art in attacks in both limited access settings and challenging domains. He is interested in finding more principled solutions for resolving the robustness problem, as well as studying other practical issues which are inhibiting us from achieving AGI.

2. Practical adversarial
attacks in challenging
environments
Presenters:
Moustafa Alzantot (UCLA), Yash Sharma (Cornell)
Joint work done with:
Mani Srivastava (UCLA), Supriyo Chkraborty (IBM Research)
Ananthram Swami (ARL), Ahmed Elgohary (UMD),
Bharathan Balaji (UCLA), Bo-Jhang Ho (UCLA), Kai-Wei Chang (UCLA)

3. Artificial Intelligence

4. Machine Learning
Training data
Training Algorithm Model
prediction
new data

5. `Adversarial Examples
Panda
School bus

6. Adversarial Examples
Machines are also getting better at pattern recognition tasks
Panda
School bus

7. School bus Ostrich
Adversarial Examples

8. Adversarial Examples
2014
Szegedy et al: intriguing properties of neural networks
small changes in input can lead to significant changes in model
output
Goodfellow et al: Explaining and harnessing adversarial examples
Introduced FGSM to compute
adversarial examples

9. Adversarial Attacks
• Current (2018):
Many attacks: PGD, DeepFool, C&W/EAD, Houdini, others.
Software libraries: Cleverhans, IBM ART.
Competitions: NIPS 2017, CAAD 2018.
Adversarial Patch,
T. B. Brown, et al.

10. However, there remain many open
challenges…

11. A few open challenges
Attacking models with limited access
Attacking natural language models
Physical world attacks for speech

12. Generating Adversarial
Examples
where:
x: original image
l: target label
r: added noise
[Goodfellow et al, 2014]
introduced the Fast Gradient Sign Method.
While successful, gradient-based methods work
only under “white-box” settings

13. Black-box Attacks
[Papernot et al, 2016]
• Query victim model to train “substitute model”.
• Attack substitute model, hope to transfer to the victim model.
[Chen et al, 2017]
• Estimate gradient using finite differences (zeroth order optimization)
These methods are not efficient as they
require a HUGE number of queries!

14. GenAttack
• Black-box attack: attacker knows nothing about the model architecture
and parameters.
• Attacker can only query the model as a blackbox function.
Idea: Rely on gradient-free optimization (i.e. genetic
algorithms) to avoid having to compute the gradient.

15. GenAttack
Initialize Population
Fitness Scoring
Selection
Crossover
Mutation
done ?

16. Attacking CIFAR-10 Model
Predicted Label
OriginalLabel

17. Karatoe Galerita Trolly bus
Attacking ImageNet Models
Against Inception-v3:

18. Evaluation
(Targeted) Attack success rate:
ZOO GenAttack
MNIST 100% 100%
CIFAR-10 95% 100%
ImageNet 18% 100%

19. Evaluation
Query Efficiency
ZOO GenAttack
MNIST 2,118,222 996 (2,126X)
CIFAR-10 2,064,798 804 (2,568X)
ImageNet 2,611,456 97,493 (27X)

20. Adversarial Training
Caveats (Madry et al, 2017)
• Increase Model Capacity
• Use strong (iterative) Adversary
Methods (Tramer et al, 2017)
• Standard: Generate adversarial examples using model currently
training
• Ensemble: Generate adversarial examples using model sampled
from an ensemble already trained.

21. Attacking ImageNet Defense
Sea anemone Park meter
Against ensemble adversarially trained Inception-V3

22. Obfuscated Gradients
Found 7 ICLR 2018 defenses relied on this phenomenon
• Shattered Gradients: Defense renders gradient to be nonexistent
or incorrect
• Stochastic Gradients: Randomized defenses
• Exploding/Vanishing Gradients
Methods to Circumvent
• BPDA: Replace non-differentiable component
• EOT: Optimize through randomization
Athalye et al, 2018 (ICML 2018 Best Paper)
Methods are White-Box!

23. Attacking Gradient Obfuscation
(Targeted) ASR
ZOO GenAttack
Bit Depth 8% 100%
JPEG 0% 86%
TVM — 70%

24. A few open challenges
Attacking models with limited access
Attacking natural language models
Physical world attacks for speech

25. Natural Language Domain
• Words in text are discrete unlike image pixels
which are continuous.
• Changing a single word can drastically change
the sentence meaning.
• Have to satisfy the language’s grammar
constraints.

26. Black-box
Initialize Population
Fitness Scoring
Selection
Crossover
Mutation
done ?

27. Mutation
• Compute the N nearest neighbors of the selected word in the
(counter-fitted GloVe) embedding space
• Use the (Google 1 Billion words) language model to filter out words
that do not fit within the context; Keep the top K words
• Pick word that will maximize the target label prediction probability
• Perform replacement -> Return resulting sentence

28. Attacking Sentiment Analysis

29. Attacking Textual Entailment

30. vAttacking Speech Recognition

31. Audio Attacks
[Alzantot et al, 2017]
• Black-box Attack on Speech Command Recognition
• Method: Genetic Algorithms
[Carlini et al, 2018]
• White-box Attack on Speech-to-Text Recognition
• Method: Iterative Optimization

32. Attacking Smart Speakers
Attack success rate
9%
1%
90%
Source Target Other
Human Evaluation

33. Physical-World Attacks
• Images: [Kurakin et al, 2016; Athalye et al, 2017]
Over-the-air Adversarial Audio?

34. A few open challenges
Attacking models with limited access (!)
Attacking natural language models (!)
Physical world attacks for speech (?)