“Principal Component Analysis” or "Singular Value Decomposition"? I compare them from a mathematical and historical point of view!

1. Is “Principal Component Analysis” different from SVD?
A historical view
Hamed Zakerzadeh
Hamed Zakerzadeh PCA or SVD? 1 / 4

2. What is Principal Component Analysis (PCA)?
Karl Pearson, in 1901, introduced PCA as the
“best” plane describing n-dimensional data
points.
“. . . best-fitting plane is perpendicular
to the greatest axis of the ellipsoid
of residuals and the minimum mean
square residual varies inversely as the
length of this axis.”
Example:
X =
⎡
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎢
⎣
−1 −1
0 2
−2 −1
−3 −2
1 1
2 1
3 2
⎤
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎥
⎦
We use PCA module from
sklearn.decomposition
Hamed Zakerzadeh PCA or SVD? 2 / 4

3. What is Singular Value Decomposition (SVD)?
SVD is factorization of matrix X ∈ Rm×n
:
X = U Σ V t
U ∈ Rm×m
and V ∈ Rn×n
are unitary and
Σ is diagonal.
SVD has been derived by Beltrami
(1873) and Jordan (1874), for
bilinear forms
Columns of V are the principal
components, so SVD gives PCA!
Example: Check the previous example
with linalg.svd, and get the same
result!
Low-rank approximation [Schmidt, 1907]
SVD Ô⇒ best approximation of any rank
Xk =
k
∑
i=1
σi ui vt
i is the best approximation of
X with rank k
Hamed Zakerzadeh PCA or SVD? 3 / 4

4. Conclusion
PCA (1901) used SVD (1873), rather implicitly.
SVD came to birth before PCA; however, it was Schmidt in 1907 who related SVD
directly to PCA, using the low-rank approximation.
Karl Pearson. (1901). LIII. On lines and planes of closest fit to systems of points in
space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of
Science, 2(11), 559–572.
Camille Jordan. (1874). Mémoire sur les formes bilinéaires. Journal de
Mathématiques Pures et Appliquées, 19, 35–54.
Erhard Schmidt. (1907). Zur Theorie der linearen und nichtlinearen
Integralgleichungen. I. Teil: Entwicklung willkürlicher Funktionen nach Systemen
vorgeschriebener. Mathematische Annalen, 63 (1907), pp. 433–476.
Gilbert Stewart. (1993). On the early history of the singular value decomposition.
SIAM Review, 35(4), 551–566.
Hamed Zakerzadeh PCA or SVD? 4 / 4