CC mmds talk 2106

calculation | consulting
why deep learning works:
perspectives from theoretical chemistry
(TM)
c|c
(TM)
charles@calculationconsulting.com

calculation|consulting
MMDS 2016
why deep learning works:
perspectives from theoretical chemistry
(TM)

calculation | consulting why deep learning works
Who Are We?
c|c
(TM)
Dr. Charles H. Martin, PhD
University of Chicago, Chemical Physics
NSF Fellow in Theoretical Chemistry
Over 10 years experience in applied Machine Learning
Developed ML algos for Demand Media; the ﬁrst $1B IPO since Google
Tech: Aardvark (now Google), eHow, GoDaddy, …
Wall Street: BlackRock
Fortune 500: Big Pharma, Telecom, eBay
www.calculationconsulting.com
(TM)
3

Data Scientists are Different
c|c
(TM)
theoretical physics
machine learning specialist
(TM)
4
experimental physics
data scientist
engineer
software, browser tech, dev ops, …
not all techies are the same

c|c
(TM)
Problem: How can SGD possibly work?
Aren’t Neural Nets non-Convex ?!
(TM)
5
can Spin Glass models suggest why ?
what other models are out there ?
expected observed ?

c|c
(TM)
(TM)
6
Outline 
Random Energy Model (REM)
Temperature, regularization and the glass transition
extending REM: Spin Glass of Minimal Frustration
protein folding analogy: Funneled Energy Landscapes
example: Dark Knowledge
Recent work: Spin Glass models for Deep Nets

c|c
(TM)
(TM)
7
Warning 
condensed matter theory is about qualitative analogies
we may seek a toy model
a mean ﬁeld theory
a phenomenological description

c|c
(TM)
What problem is Deep Learning solving ?
(TM)
8
minimize cross-entropy
https://www.ics.uci.edu/~pjsados.pdf

c|c
(TM)
Problem: What is a good theoretical
model for deep networks ?
(TM)
9
p-spin spherical glass
LeCun … 2015
L Hamiltonian (Energy function)
X Gaussian random variables
w real valued (spins) , spherical constraint
H >= 3 (p)
can be solved analytically, simulated easily

c|c
(TM)
What is a spin glass ?
(TM)
10
Frustration: constraints that can not be satisﬁed
J = X = weights
S = w = spins
Energetically: all spins should be paired

c|c
(TM)
why p-spin spherical glass ?
(TM)
11
crudely: deep networks (effectively) have no local minima !
local minima
k=1 critical points
ﬂoor / ground state
k = 2 critical points
k = 3 critical points
the critical points are ordered
saddle points

c|c
(TM)
why p-spin spherical glass ?
(TM)
12
crudely: deep networks (effectively) have no local minima !
http://cims.nyu.edu/~achoroma/NonFlash/Papers/PAPER_AMMGY.pdf
ap

c|c
(TM)
(TM)
13
any local minima will do; the ground state is a state of overtraining
good generalization
overtraining
Early Stopping: to avoid the ground state ?

c|c
(TM)
(TM)
14
it’s easy to ﬁnd the ground state; it’s hard to generalize ?
Early Stopping: to avoid the ground state ?

c|c
(TM)
Current Interpretation 
(TM)
15
•ﬁnding the ground state is easy (sic); generalizing is hard
•ﬁnding the ground state is irrelevant: any local minima will do
•the ground state is a state over training

c|c
(TM)
recent p-spin spherical glass results
(TM)
16
actually: recent results (2013) on the behavior
(distribution of critical points, concentration of the means)
of an isotropic random function on a high dimensional manifold
require: the variables actually concentrate on their means
the weights are drawn from isotropic random function
related to: old results TAP solutions (1977)
# critical points ~ TAP complexity
avoid local minima? : increase Temperature
harder problem: low Temp behavior of spin glass

c|c
(TM)
(TM)
17
minimize cross-entropy of output layer
entropic effects : not just min energy
more like min free energy (divergence)
Statistical Physics and InformationTheory: Neri Merhav
i.e. variational auto encoders

c|c
(TM)
(TM)
18
Restricted Boltzmann Machine
can deﬁne free energy directly
A Practical Guide toTraining Restricted Boltzmann Machines, Hinton

c|c
(TM)
(TM)
19
Restricted Boltzmann Machine
trade off between energy and entropy
min free energy directly
A Practical Guide toTraining Restricted Boltzmann Machines, Hinton

c|c
(TM)
(TM)
20
https://web.stanford.edu/~montanar/RESEARCH/BOOK/partB.pdf
inﬁnite limit of p-spin spherical glass
A related approach: Random Energy Model (REM)

c|c
(TM)
Random Energy Model (REM)
(TM)
21
ground state is governed by ExtremeValue Statistics
http://guava.physics.uiuc.edu/~nigel/courses/563/essays2000/pogorelov.pdf
http://scitation.aip.org/content/aip/journal/jcp/111/14/10.1063/1.479951
old result from protein folding theory

c|c
(TM)
REM: What is Temperature ?
(TM)
22
We can use statistical mechanics to analyze known algorithms
I don’t mean in the traditional sense of algorithmic analysis
take Ej as the objective = loss function + regularizer
study Z: form a mean ﬁeld theory;
take limits N -> inf, T -> 0

c|c
(TM)
REM: What is Temperature ?
(TM)
23
let E(T) by the effective energy
E(T) = E/T ~ sum of weights*activations
as T -> 0, E(T) effective energies diverge; weights explode
Temperature is a proxy for weight constraints
T sets the Energy Scale

c|c
(TM)
Temperature: as Weight Constraints
(TM)
24
•traditional weight regularization
•max norm constraints (i.e. w/dropout)
•batch norm regularization (2015)
we avoid situations when the weights explode
in deep networks, we temper the weights
and the distribution of the activations (i.e local entropy)

c|c
(TM)
REM: a toy model for real Glasses 
(TM)
25
but it is believed that entropy collapse ‘drives’ the glass transition
the glass transition is not well understood

c|c
(TM)
what is a real (structural) Glass ? 
(TM)
26
Sand + Fire = Glass

c|c
(TM)
what is a real (structural) Glass ? 
(TM)
27
all liquids can be made into glasses
if we cool then fast enough
the glass transition is not a normal phase transition
not the melting point
arrangement of atoms is amorphous; not completely random
different cooling rates produce different glassy states
universal phenomena; not universal physics
molecular details affect the thermodynamics

c|c
(TM)
REM: the Glass Transition 
(TM)
28
Entropy collapses when T <~ Tc
Phase Diagram: entropy density
energy density
free energy density
https://web.stanford.edu/~montanar/RESEARCH/BOOK/partB.pdf

c|c
(TM)
REM: Dynamics on the Energy Landscape 
(TM)
29
let us assume some states trap the solver for some time;
of course, there is a great effort to design solvers that can avoid traps

c|c
(TM)
Energy Landscapes: and Protein Folding  
(TM)
30
let us assume some states trap the solver in state E(j) for a short time
and the transitions E(j) -> E(j-1) are governed by ﬁnite, reversible transitions
(i.e. SGD oscillates back and forth for a while)
classic result(s): for T near the glass Temp (Tc)
the traversal times are slower than exponential !
in a physical system, like a protein or polymer,
it would take longer than the known lifetime of the universe
to ﬁnd the ground (folded) state

c|c
(TM)
Protein Folding: the Levinthal Paradox  
(TM)
31
folding could take longer than the known lifetime of the universe ?

c|c
(TM)
(TM)
32
http://arxiv.org/pdf/cond-mat/9904060v2.pdf
Old analogy between Protein folding and Hopﬁeld Associative Memories
Natural pattern recognition could
• use a mechanism with a glass Temp (Tc) that is as low as possible
• avoid the glass transition entirely, via energetics
Nature (i.e. folding) can not operate this way !
Protein Folding: around the Levinthal Paradox

c|c
(TM)
Spin Glasses: Minimizing Frustration 
(TM)
33
http://www.nature.com/nsmb/journal/v4/n11/pdf/nsb1197-871.pdf

c|c
(TM)
Spin Glasses: Minimizing Frustration 
(TM)
34
http://www.nature.com/nsmb/journal/v4/n11/pdf/nsb1197-871.pdf

c|c
(TM)
Spin Glasses: vs Disordered FerroMagnets 
(TM)
35
http://arxiv.org/pdf/cond-mat/9904060v2.pdf

c|c
(TM)
the Spin Glass of Minimal Frustration  
(TM)
36
REM + strongly correlated ground state = no glass transition
https://arxiv.org/pdf/1312.7283.pdf

c|c
(TM)
the Spin Glass of Minimal Frustration  
(TM)
37
Training a model induces an energy gap, with few local minima
http://arxiv.org/pdf/1312.0867v1.pdf

c|c
(TM)
Energy Funnels: Entropy vs Energy  
(TM)
38
there is a tradeoff between Energy and Entropy minimization

c|c
(TM)
Energy Landscape Theory of Protein Folding
(TM)
39
there is a tradeoff between Energy and Entropy minimization

c|c
(TM)
(TM)
40
Avoids the glass transition by having more favorable energetics
Levinthal paradox
glassy surface
vanishing gradients
Energy Landscape Theory of Protein Folding
funneled landscape
rugged convexity
energy / entropy tradeoff

c|c
(TM)
RBMs: Entropy Energy Tradeoff
(TM)
41
RBM on MNIST Aligned for comparison: entropy - 200
Entropy drops off much faster than total Free Energy

c|c
(TM)
Dark Knowledge: an Energy Funnel ? 
(TM)
42
784 -> 800 -> 800 -> 10MLP on MNIST
Distilled
10,000 test cases, 10 classes
99 errors
same entropy (capacity); better loss function
ﬁt to ensemble soft-max probabilities
146 errors
784 -> 800 -> 800 -> 10

c|c
(TM)
Adversarial Deep Nets: an Energy Funnel ? 
(TM)
43
Discriminator learns a complex loss function
Generator: fake data
Discriminator: fake vs real ?
http://soumith.ch/eyescream/

c|c
(TM)
(TM)
44
Summary 
Random Energy Model (REM): simpler theoretical model
Glass Transition: temperature ~ weight constraints
extending REM: Spin Glass of Minimal Frustration
possible examples: Dark Knowledge
Funneled Energy Landscapes
Adversarial Deep Nets

(TM)
c|c
(TM)
c | c

CC mmds talk 2106

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to CC mmds talk 2106

Similar to CC mmds talk 2106 (20)

More from Charles Martin

More from Charles Martin (10)

Recently uploaded

Recently uploaded (20)

CC mmds talk 2106