This document proposes a new black-box adversarial attack method called Prior Convictions that utilizes gradient priors and online convex optimization techniques. It formulates black-box attack as an online learning problem and estimates gradients using a sphere sampling estimator within a bandit convex optimization framework. The algorithm iteratively improves its gradient estimation by incorporating temporal and spatial gradient priors. Experiments show the method generates adversarial examples for black-box models more effectively than prior gradient-free approaches.

1. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL
ATTACKS WITH BANDITS AND PRIORS
Andrew Ilyas 1 Logan Engstrom 1 Aleksander Madry 1
1MIT
presentation by yangjq, 2019.4
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 1 / 22

2. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Adversarial attack
Figure: indistinguishable to the human, but classified with high confidence
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 2 / 22

3. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
White box attack: gradient based
All the information of the model is exposed to the attacker, e.g. the model
architecture, parameters and it’s gradients.
Simple methods such as Fast Gradient Sign Method (FGSM) worked well.
x∗
= x + ϵ · sign (∇xJ(x, y))
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 3 / 22

4. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
White box attack: gradient based
A more powerful attack: projected gradient descent (PGD)
xt+1
= Projx+S
(
xt
+ α sgn
(
∇xt L
(
θ, xt
, y
)))
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 4 / 22

5. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
White box defense
Adversarial training: The training data is augmented with adversarial
examples to make models more robust.
Manifold projections: An input sample is projected onto a learned
data manifold.
Stochasticity: Certain inputs or hidden activations are shuffled or
randomized.
Preprocessing: Inputs or hidden activations are quantized, projected
into a different representation or are otherwise preprocessed.
Historically all defense have failed on new attack methods.
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 5 / 22

6. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Without gradients
Utilize transferability
Approximate with differentiable function.
Under white box setting, correct optimization method usually worked.
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 6 / 22

7. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Black Box Attack
In most real-world situations, expecting complete access is not realistic.
Black Box setting: only output of the model is available.
Basic idea: estimate the gradient and use white box attack.
Directional derivative:
Dvf(x) = ⟨∇xf(x), v⟩ ≈ (f(x + δv) − f(x))/δ
query complexity is proportional to the dimension
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 7 / 22

8. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
imperfect estimator of gradients
Figure: The fraction of correctly estimated coordinates
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 8 / 22

9. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
The gradient estimation problem
g∗
= ∇xL(x, y)
the goal is to maximize
E
[
gT
g∗
]
This problem can be reformulate as
Ag∗
= y
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 9 / 22

10. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
The least squares method
min
g
∥g∥2 s.t. Ag = y
NES and Least Squares equivalence.
The minimum-variance unbiased estimator.
We have reached the limit of query-efficiency of black-box adversarial
attacks. (?)
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 10 / 22

11. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Gradient Prior: Time-dependent priors
⟨∇xL (xt, y) , ∇xL (xt+1, y)⟩
∥∇xL (xt, y)∥2 ∥∇xL (xt+1, y)∥2
t ∈ {1 . . . T − 1}
Figure: Cosine similarity between the gradients at the current and previous steps
along the optimization trajectory
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 11 / 22

12. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Gradient Prior: Data-dependent priors
If (i, j) and (k, l) is close then
∇xL(x, y)ij ≈ ∇xL(x, y)kl
Figure: Cosine similarity of “tiled” image gradient with original image gradient
versus the length of the square tiles
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 12 / 22

13. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Online convex optimization (OCO)
regretT (A) = sup
{f1,...,fT }⊆F
{ T∑
t=1
ft (xt) − min
x∈K
T∑
t=1
ft(x)
}
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 13 / 22

14. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Online gradient descent
Nearly optimal regret bound.
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 14 / 22

15. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Bandit Convex Optimization (BCO)
No gradient information available.
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 15 / 22

16. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
point-wise gradient estimator
The estimator should satisfy
E [gt] ≈ ∇ft
One dimensional example
g(x) =
{
f(x+δ)
δ , with probability 1
2
−f(x−δ)
δ , with probability 1
2
General case: The sphere sampling estimator
ˆfδ(x) = E
v∈B
[f(x + δv)]
E
u∈S
[f(x + δu)u] =
δ
n
∇ ˆfδ(x)
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 16 / 22

17. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
BCO with sphere sampling estimator
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 17 / 22

18. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
As a BCO problem
vt is the latent vector and gradient estimator.
gt = Proj(vt)
ℓt(g) = −
⟨
∇L(x, y),
g
∥g∥
⟩
We have a good start point vt and needs to find better estimator at time
t + 1.
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 18 / 22

19. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
The Algorithm
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 19 / 22

20. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
The Algorithm
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 20 / 22

21. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Experiments
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 21 / 22

22. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Think
1. A framework for optimization on black box model ?
PRIOR CONVICTIONS: BLACK-BOX ADVERSARIAL ATTACKS WITH BANDITS AND2019.4.8 22 / 22