WeightWatcher: Data Free Diagnostics for Deep Learning

WeightWatcher
Data Free Diagnostics for Deep Learning
TII Jan 2025
charles@calculatedcontent.com

c|c
(TM)
calculation | consulting weightwatcher.ai
Who am I?
Dr. Charles H. Martin, PhD
University of Chicago, Chemical Physics
NSF Fellow in Theoretical Chemistry
Over 20 years experience in Applied Machine Learning (ML) and AI
Developed ML algos for Demand Media; the
fi
rst $1B IPO since Google
Lean Start Ups: Aardvark (acquired by Google), …
Internet Giants: eHow, eBay, GoDaddy,WalmartLabs …
Wall Street: Barclays, BlackRock, GLG, …
Fortune 500: Big Pharma, Telecom, …
https://weightwatcher.ai
https://calculatedcontent.com

c|c
(TM)
WeightWatcher: open-source, open-science
https://weightwatcher.ai 115K+ downloads; 1000 stars

c|c
(TM)
weightwatcher (HTSR) layer quality metric:

c|c
(TM)
Research: Implicit Self-Regularization in Deep Learning
• Predicting trends in the quality of state-of-the-art neural networks without
access to training or testing data
(JMLR 2021)
(Nature Communications 2021)
• Implicit Self-Regularization in Deep Neural Networks: Evidence from
Random Matrix Theory and Implications for Learning.
• SETOL: SemiEmpirical Theory of Learning
(Invited submission, Philosophical Magazine)

c|c
(TM)
Problem: How do NN layers converge ?
Under
fi
t
Over
fi
t
Random
Ideal Fit

c|c
(TM)
6.0
1.5
8.0
2.0
Solution: weightwatcher (HTSR) layer quality metric

c|c
(TM)
HTSR layer quality metric
Llama
Falcon
LLMs: ‘under & over
fi
t’ Layers

c|c
(TM)
WeightWatcher: analyzes the (tail of the) ESD
(eigenvalues) of the layer weight matrices
2.0
4.0

c|c
(TM)
Theory-based: Layer Quality metrics
(2025)
HTSR SETOL
(2021)

c|c
(TM)
Theoretical analysis: LLMs have under
fi
t layers
> 6
Llama Falcon vs Falcon2

c|c
(TM)
Theoretical predictions:
fi
ne-tuned updates
Falcon 3 Instruct
No under
fi
t layers
> 6
Few over
fi
t layers
< 2

c|c
(TM)
Theoretical analysis:
fi
ne-tuned updates follow
predictions of the theory
Llama-3.1 Mistral Qwen2.5

c|c
(TM)
Theoretical prediction: better base model
(smaller alphas) are easier to
fi
ne-tune
Llama-3.1 70B base vs 70B Instruct

c|c
(TM)
Over
fi
tting is good : Segment Anything Models

c|c
(TM)
Over
fi
tting is bad : Llama Guard models

c|c
(TM)
TruthfulQA
metric
LLM Hallucinations: Gauge Truthfulness of Base Models

c|c
(TM)
Applications of HTSR: Student Projects
low-memory
fi
ne-tuning
optimize layer
learning rates
compres LLMs

c|c
(TM)
Random Matrix Theory: Marcenko Pastur
plus Tracy-Widom
fl
uctuations
very crisp edges
Q
RMT says if W is a simple random Gaussian matrix,
then the ESD will have a very simple , known form
Shape depends on Q=N/M
(and variance ~ 1)
Eigenvalues tightly bounded
a few spikes may appear

c|c
(TM)
Random Matrix Theory: Heavy Tailed
But if W is heavy tailed, the ESD will also have heavy tails
(i.e. its all spikes, bulk vanishes)
If W is strongly correlated , then the ESD can be modeled as if W is drawn
from a heavy tailed distribution
Nearly all pre-trained DNNs display heavy tails…as shall soon see

c|c
(TM)
AlexNet,
VGG11,VGG13, …
ResNet, …
DenseNet,
BERT, RoBERT, …
GPT, GPT2, …
Flan-T5
Bloom
Llama
Falcon
…
Heavy-Tailed: Self-Regularization
All large, well trained, modern DNNs exhibit heavy tailed self-regularization
scale free
HTSR

c|c
(TM)
Heavy-Tails: in Latent Semantic Analysis
HeavyTails do not solely come from of SGD training
LSA on 20newsgroups; great PL
fi
t; alpha = 2.2

c|c
(TM)
HTSR Theory: Heavy Tailed Self Regularization
DNN training induces breakdown of Gaussian random structure
and the onset of a new kind of heavy tailed self-regularization
Gaussian
random
matrix
Bulk+
Spikes
Heavy
Tailed
Small, older NNs
Large, modern DNNs
and/or
Small batch sizes

c|c
(TM)
HTSR Theory: 5+1 Phases of Training
Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning
Charles H. Martin, Michael W. Mahoney; JMLR 22(165):1−73, 2021.

c|c
(TM)
Heavy Tailed RMT: Universality Classes
The familiar Wigner/MP Gaussian class is not the only Universality class in RMT
Charles H. Martin, Michael W. Mahoney; JMLR 22(165):1−73, 2021.

c|c
(TM)
Heavy Tailed RMT: Universality Classes

c|c
(TM)
Statistical Mechanics
Why this works…

c|c
(TM)
calculation | consulting heavy-tails in DNNs
Classic Set Up: Student-Teacher model
Statistical Mechanics of Learning from the 1990s
MultiLayer Feed Forward Network Perceptron

c|c
(TM)
Set Up: Student-Teacher model

Quality (generalization accuracy)
Perceptron
(vector model)
NN Layer
(matrix model)

c|c
(TM)
Set Up: Matrix-generalized Student-Teacher
Heavy-Tailed correlation matrix
Student-Teacher matrix overlap
(2 forms)

c|c
(TM)
Set Up: Matrix-generalized Student-Teacher
Free Energy Quality (squared) / Generating Function
HCIZ Integral

c|c
(TM)
HCIZ Integrals : Evaluation “Tricks”
X
Need to Change Measure

c|c
(TM)
HCIZ Integrals : Evaluation “Tricks”
X
Need Determinant = 1

c|c
(TM)
HCIZ Integrals : Renormalization Group Theory
Let H be the Hamiltonian for the Quality
Volume Preserving or Scale-Invariant Transformation

c|c
(TM)
HCIZ Integrals : Effective Correlation Space

c|c
(TM)
HCIZ Integrals : Effective Space Free Energy
Volume Preserving or Scale-Invariant Transformation

c|c
(TM)
Layer Quality Metrics : SemiEmpirical Theory
“Generalized Norm”
simple, functional form
can infer from empirical
fi
t
Eigenvalues of Teacher
empirical
fi
t: R-transform:
“Asymptotics of HCZI integrals …” Tanaka (2008)
free cumulants*
WeightWatcher
Layer Quality metric ~

c|c
(TM)
R-Transform : Heavy Tails
Alpha smaller =>
Heavy tail = >
Larger higher-order cumulants =>
Better generalization…
(down to alpha=2)

c|c
(TM)
R-Transform : Branch Cut at the ECS
start of ECS
Inverse-Wishart model

c|c
(TM)
New Principle of Learning:
Volume Preserving Transformation
As the correlations concentrate into the tail
The change of measure must satisfy
a volume preserving transformation
Which sets a condition on the eigenvalues in the tail

c|c
(TM)
New Principle of Ideal Learning:
Volume Preserving Transformation
As alpha -> 2, the |det X|=1 condition holds!

c|c
(TM)
New Principle of Ideal Learning:
Wilson Exact Renormalization Group

c|c
(TM)
Evidence of Ideal Learning:
As alpha -> 2, SVD generalizing components concentrate in the tail
MLP3 on
MNIST,
varying LR
Effective Correlation Space (ECS)

c|c
(TM)
As alpha < 2, layers appear to be over
fi
t
Layer Quality Metrics : Detecting Over
fi
tting

c|c
(TM)
A result of spuriously large learning rates
Layer Quality Metrics : Correlation Traps

c|c
(TM)
As alpha < 2, layer FC1 appears to be over
fi
t
Why ?: IZ Free Energy becomes non-extensive
Layer Quality Metrics : Detecting Over
fi
tting
MLP3 on MNIST: Reduced capacity
Train FC1, freeze FC2, FC3

c|c
(TM)
Heavy Tailed Correlations: in Neuroscience
Spiking (i.e real) neurons exhibit power law behavior
weightwatcher supports several PL
fi
ts
from experimental neuroscience
plus totally new shape metrics
we have invented (and published)
Spiking (i.e real) neurons exhibit power law behavior
weightwatcher supports several PL
fi
ts
from experimental neuroscience
plus totally new shape metrics
we have invented (and published)

c|c
(TM)
WeightWatcher: why Power Law
fi
ts ?
Spiking (i.e real) neurons exhibit (truncated) power law behavior
The Critical Brain Hypothesis
Evidence of Self-Organized Criticality (SOC)
Per Bak (How Nature Works)
As neural systems become more complex
they exhibit power law behavior
and then truncated power law behavior
We see exactly this behavior in DNNs
and it is predictive of learning capacity

c|c
(TM)
WeightWatcher: open-source, open-science
We are looking for early adopters and collaborators
https://weightwatcher.ai 115K+ downloads; 1000 stars

(TM)
c|c
(TM)
c | c

WeightWatcher: Data Free Diagnostics for Deep Learning

More Related Content

Similar to WeightWatcher: Data Free Diagnostics for Deep Learning

More from Charles Martin

Recently uploaded

WeightWatcher: Data Free Diagnostics for Deep Learning