Interesting ICML2018 papers

1. ICML 2018 (July 11 – July 16)
PART 2
BEST PAPER (adversarial attack)
Graph Embedding / Manifold Alignment
2016-20873 Segwang Kim 1

2. ① Best Paper
Obfuscated Gradients Give a False Sense of Security:
Circumventing Defenses to Adversarial Examples
② Graph Embedding
③ Manifolds
Overview
2

3. 3
Obfuscated Gradients Give a False Sense of Security:
Circumventing Defenses to Adversarial Examples
Ref: Synthesizing Robust Adversarial Examples

4. Faulty defenses against adv attacks
Takeaways
4
ICLR 2018 adv. defense papers
Obfuscated Gradients
Shattered gradients
Backward Pass Differentiable Approximation
Stochastic gradients
Expectation Over Transformation
Vanishing/exploding gradients
reparameterization & optimize over good space

5. Induces nonexistent or incorrect gradients
Intentionally, non-differentiable operators are used
Unintentionally, numeric instability caused
Backward Pass Differentiable Approximation
Shattered Gradients
f (NN)
g (filter)
f (NN)
f ∘g
(faulty defensed NN)
Shattered
gradients
True
gradients

6. Time-randomness
Randomized filters or classifiers
Expectation over Transformation
Stochastic Gradients
f (NN)
t (filter)
f (NN)
f ∘ t
(faulty defensed NN)
Randonmized
gradients

7. Input transformation loop prior to being fed
Loop optimization vanishes or explodes gradients
Reparameterization
Vanishing/exploding gradients
f (NN)
t (filter)
f (NN)
f ∘ t
(faulty defensed NN)
Exploding / vanishing
gradients

8. 8
Graph Embedding

9. Overview-Graph Embedding
9
• Graph Embedding in Euclidean Space
• Out-of-sample extension of graph adjacency spectral embedding
• Tree Edit Distance Learning via Adaptive Symbol Embeddings
• Spectrally approximating large graphs with smaller graphs
• Improved large-scale graph learning through ridge spectral sparsification
• Low-Rank Riemannian Optimization on Positive Semidefinite Stochastic
Matrices with Applications to Graph Clustering
• Representation Learning on Graphs with Jumping Knowledge Networks
• Graph Embedding in Hyperbolic space
• Representation Tradeoffs for Hyperbolic Embeddings
• Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
• Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic
Geometry

10. Out-of-sample extension of graph.
Computing over for the added graph is prohibited.
Allow to access to
True connectivity between given vertice and new vertex
Similarities between learned vectors and new embedding vector
Out-of-sample extension of graph adjacency spectral embedding
10
(Fixed) Embedding
?
New node is added
1
2

11. Out-of-sample extension of graph adjacency spectral embedding
11
Two basic ways for OOS extension
Linear Least Square OOS Extension
Maximum Likelihood OOS Extension
Then, how close OOS extension to insampling embedding?
For random dot product graph, they are good!
𝑋1, … , 𝑋 𝑛 ∼ 𝐹 where 𝐹 is dist. over ℝ 𝑑
1 2
adjacency spectral embedding
For adjacency matrix A, select d
eigenvectors with the largest d
eigenvalues.

12. Out-of-sample extension of graph adjacency spectral embedding
12
Up to rotation
∼ 𝐹

13. Hyperbolic Embedding
13
Recall
Hyperbolic space is GOOD for hierarchical embedding
From the geometric nature, hyperbolic model is suitable for tree-like
structure
• Parsimonious (needs relatively low dimension)
• Space becomes compressed as going far from the origin
Embedding
Tree distance=4
Embedded distance~ 4

14. Representation Tradeoffs for Hyperbolic Embeddings
14
Analysis 1
Therefore, it is powerful for wide tree
Analysis 2
But, not too deep tree.

15. Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
15
Cone can capture hierarchy
But, in n dimensions, no n+1 distinct cone regions can
simultaneously have unbounded disjoint sub-volumes
Entailment cones in Poincare ball can do
Axial symmetry
Rotation invariance
Continuity
Transitivity of nested angular cones

16. Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
16
Therefore, objective function is
Previous Now

17. Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic
Geometry
17
Lorentz Model
Mathematically same, but fewer inverse operations for optimization
Optimize on Lorentz model, visualize on Poincare ball model

18. 18
Manifold Alignment

19. Overview-Manifolds
19
• GAN
• Geometry Score: A Method For Comparing GANs
• MAGAN: Aligning Biological Manifolds
• ETC
• Stochastic Wasserstein Barycenters
• On the Generalization of Equivariance and Convolution in Neural networks
to the Action of Compact Groups

20. Geometry Score: A Method For Comparing GANs
20
Robust comparing criteron between data and fake distribution
Not necessarily images
Not for training
Comparing topology of underlying manifolds of data and fake
distribution
Number of loops and higher dimensional holes
Not geometry, topology  Easier to approximate
≅
Homeomorphic

21. Geometry Score: A Method For Comparing GANs
21
Using simplexes from randomly sampled vertices (landmarks)
with varying radius, approximate topology
0-dimensional hole
(Connected component)
1-dimensional hole
(loop)

22. Geometry Score: A Method For Comparing GANs
22
Observe birth and death of k-th dimensional holes as time
(radius) goes by
At relative time (radius) 𝛼, the k-th Betty number
Relative time being k-th Betty number persisted (=𝑖)
landmarks

23. Geometry Score: A Method For Comparing GANs
23
Since , we can consider it as distribution

24. MAGAN: Aligning Biological Manifolds
24
Same system, different measurements
Dual GAN needs supervised paired examples  impractical
Unsupervised, unpaired data ?
Aligning rather than superimposing
Tissue
Single-cell RNA
sequencing
Mass cytometry

25. MAGAN: Aligning Biological Manifolds
25
Same system, different measurements
𝐺1
𝐺2
𝐷1
𝐷2

26. MAGAN: Aligning Biological Manifolds
26
Depending on problem, properly define correspondence loss
Unsupervised correspondence
• E.g.)
Semi-supervised correspondence
Tissue
Single-cell RNA
sequencing
Mass cytometry
16 dim
20 dim
15 dim

27. MAGAN: Aligning Biological Manifolds
27
Aligning works well

28. On the Generalization of Equivariance and Convolution in Neural
Networks to the Action of Compact Groups
28
Generalize (1) rotation equivariant networks, (2) spherical
CNNs, (3) Message passing NNs
A feed forward neural network 𝒩 is equivariant to the action of a compact group G
Each layer of 𝒩 implements a generalized form of convolution derived from
where 𝑓, 𝑔 ∶ 𝐺 → ℂ
Topological group with compact topology
e.g. SO(3), SL(3) … (NOT ℝ 𝑛
)

29. On the Generalization of Equivariance and Convolution in Neural
Networks to the Action of Compact Groups
29
…
𝜒0
𝑉0
𝜒1
𝑉0
𝜒 𝐿
𝑉𝐿