Lecture 9: Dimensionality Reduction, Singular Value Decomposition (SVD), Principal Component Analysis (PCA). (ppt,pdf) Appendices A, B from the book “Introduction to Data Mining” by Tan, Steinbach, Kumar.

1. DATA MINING
LECTURE 8
Dimensionality Reduction
PCA -- SVD
Subrata Kumer Paul
Assistant Professor, Dept. of CSE, BAUET
sksubrata96@gmail.com

2. The curse of dimensionality
• Real data usually have thousands, or millions of
dimensions
• E.g., web documents, where the dimensionality is the
vocabulary of words
• Facebook graph, where the dimensionality is the
number of users
• Huge number of dimensions causes problems
• Data becomes very sparse, some algorithms become
meaningless (e.g. density based clustering)
• The complexity of several algorithms depends on the
dimensionality and they become infeasible.

3. Dimensionality Reduction
• Usually the data can be described with fewer
dimensions, without losing much of the meaning
of the data.
• The data reside in a space of lower dimensionality
• Essentially, we assume that some of the data is
noise, and we can approximate the useful part
with a lower dimensionality space.
• Dimensionality reduction does not just reduce the
amount of data, it often brings out the useful part of the
data

4. Dimensionality Reduction
• We have already seen a form of dimensionality
reduction
• LSH, and random projections reduce the
dimension while preserving the distances

5. Data in the form of a matrix
• We are given n objects and d attributes describing
the objects. Each object has d numeric values
describing it.
• We will represent the data as a nd real matrix A.
• We can now use tools from linear algebra to process the
data matrix
• Our goal is to produce a new nk matrix B such that
• It preserves as much of the information in the original matrix
A as possible
• It reveals something about the structure of the data in A

6. Example: Document matrices
n
documents
d terms
(e.g., theorem, proof, etc.)
Aij = frequency of the j-th
term in the i-th document
Find subsets of terms that bring documents
together

7. Example: Recommendation systems
n
customers
d movies
Aij = rating of j-th
product by the i-th
customer
Find subsets of movies that capture the
behavior or the customers

8. Linear algebra
• We assume that vectors are column vectors.
• We use 𝑣𝑇 for the transpose of vector 𝑣 (row vector)
• Dot product: 𝑢𝑇𝑣 (1𝑛, 𝑛1 → 11)
• The dot product is the projection of vector 𝑣 on 𝑢 (and vice versa)
• 1, 2, 3
4
1
2
= 12
• 𝑢𝑇𝑣 = 𝑣 𝑢 cos(𝑢, 𝑣)
• If ||𝑢|| = 1 (unit vector) then 𝑢𝑇𝑣 is the projection length of 𝑣 on 𝑢
• −1, 2, 3
4
−1
2
= 0 orthogonal vectors
• Orthonormal vectors: two unit vectors that are orthogonal

9. Matrices
• An nm matrix A is a collection of n row vectors and m column
vectors
𝐴 =
| | |
𝑎1 𝑎2 𝑎3
| | |
𝐴 =
− 𝛼1
𝑇
−
− 𝛼2
𝑇
−
− 𝛼3
𝑇
−
• Matrix-vector multiplication
• Right multiplication 𝐴𝑢: projection of u onto the row vectors of 𝐴, or
projection of row vectors of 𝐴 onto 𝑢.
• Left-multiplication 𝑢𝑇
𝐴: projection of 𝑢 onto the column vectors of 𝐴, or
projection of column vectors of 𝐴 onto 𝑢
• Example:
1,2,3
1 0
0 1
0 0
= [1,2]

10. Rank
• Row space of A: The set of vectors that can be
written as a linear combination of the rows of A
• All vectors of the form 𝑣 = 𝑢𝑇𝐴
• Column space of A: The set of vectors that can be
written as a linear combination of the columns of A
• All vectors of the form 𝑣 = 𝐴𝑢.
• Rank of A: the number of linearly independent row (or
column) vectors
• These vectors define a basis for the row (or column) space
of A

11. Rank-1 matrices
• In a rank-1 matrix, all columns (or rows) are
multiples of the same column (or row) vector
𝐴 =
1 2 −1
2 4 −2
3 6 −3
• All rows are multiples of 𝑟 = [1,2, −1]
• All columns are multiples of 𝑐 = 1,2,3 𝑇
• External product: 𝑢𝑣𝑇 (𝑛1 , 1𝑚 → 𝑛𝑚)
• The resulting 𝑛𝑚 has rank 1: all rows (or columns) are
linearly dependent
• 𝐴 = 𝑟𝑐𝑇

12. Eigenvectors
• (Right) Eigenvector of matrix A: a vector v such
that 𝐴𝑣 = 𝜆𝑣
• 𝜆: eigenvalue of eigenvector 𝑣
• A square matrix A of rank r, has r orthonormal
eigenvectors 𝑢1, 𝑢2, … , 𝑢𝑟 with eigenvalues
𝜆1, 𝜆2, … , 𝜆𝑟.
• Eigenvectors define an orthonormal basis for the
column space of A

13. Singular Value Decomposition
𝐴 = 𝑈 Σ 𝑉𝑇 = 𝑢1, 𝑢2, ⋯ , 𝑢𝑟
𝜎1
𝜎2
0
0
⋱
𝜎𝑟
𝑣1
𝑇
𝑣2
𝑇
⋮
𝑣𝑟
𝑇
• 𝜎1, ≥ 𝜎2 ≥ ⋯ ≥ 𝜎𝑟: singular values of matrix 𝐴 (also, the square roots
of eigenvalues of 𝐴𝐴𝑇 and 𝐴𝑇𝐴)
• 𝑢1, 𝑢2, … , 𝑢𝑟: left singular vectors of 𝐴 (also eigenvectors of 𝐴𝐴𝑇
)
• 𝑣1, 𝑣2, … , 𝑣𝑟: right singular vectors of 𝐴 (also, eigenvectors of 𝐴𝑇
𝐴)
𝐴 = 𝜎1𝑢1𝑣1
𝑇
+ 𝜎2𝑢2𝑣2
𝑇
+ ⋯ + 𝜎𝑟𝑢𝑟𝑣𝑟
𝑇
[n×r] [r×r] [r×m]
r: rank of matrix A
[n×m] =

14. Symmetric matrices
• Special case: A is symmetric positive definite
matrix
𝐴 = 𝜆1𝑢1𝑢1
𝑇
+ 𝜆2𝑢2𝑢2
𝑇
+ ⋯ + 𝜆𝑟𝑢𝑟𝑢𝑟
𝑇
• 𝜆1 ≥ 𝜆2 ≥ ⋯ ≥ 𝜆𝑟 ≥ 0: Eigenvalues of A
• 𝑢1, 𝑢2, … , 𝑢𝑟: Eigenvectors of A

15. Singular Value Decomposition
• The left singular vectors are an orthonormal basis
for the row space of A.
• The right singular vectors are an orthonormal
basis for the column space of A.
• If A has rank r, then A can be written as the sum
of r rank-1 matrices
• There are r “linear components” (trends) in A.
• Linear trend: the tendency of the row vectors of A to align
with vector v
• Strength of the i-th linear trend: ||𝐴𝒗𝒊|| = 𝝈𝒊

16. An (extreme) example
• Document-term matrix
• Blue and Red rows (colums) are linearly dependent
• There are two prototype documents (vectors of words): blue
and red
• To describe the data is enough to describe the two prototypes, and the
projection weights for each row
• A is a rank-2 matrix
𝐴 = 𝑤1, 𝑤2
𝑑1
𝑇
𝑑2
𝑇
A =

17. An (more realistic) example
• Document-term matrix
• There are two prototype documents and words but
they are noisy
• We now have more than two singular vectors, but the
strongest ones are still about the two types.
• By keeping the two strongest singular vectors we obtain most
of the information in the data.
• This is a rank-2 approximation of the matrix A
A =

18. Rank-k approximations (Ak)
Uk (Vk): orthogonal matrix containing the top k left (right)
singular vectors of A.
k: diagonal matrix containing the top k singular values of A
Ak is an approximation of A
n x d n x k k x k k x d
Ak is the best approximation of A

19. SVD as an optimization
• The rank-k approximation matrix 𝐴𝑘 produced by
the top-k singular vectors of A minimizes the
Frobenious norm of the difference with the matrix
A
𝐴𝑘 = arg max
𝐵:𝑟𝑎𝑛𝑘 𝐵 =𝑘
𝐴 − 𝐵 𝐹
2
𝐴 − 𝐵 𝐹
2
=
𝑖,𝑗
𝐴𝑖𝑗 − 𝐵𝑖𝑗
2

20. What does this mean?
• We can project the row (and column) vectors of
the matrix A into a k-dimensional space and
preserve most of the information
• (Ideally) The k dimensions reveal latent
features/aspects/topics of the term (document)
space.
• (Ideally) The 𝐴𝑘 approximation of matrix A,
contains all the useful information, and what is
discarded is noise

21. Latent factor model
• Rows (columns) are linear combinations of k
latent factors
• E.g., in our extreme document example there are two
factors
• Some noise is added to this rank-k matrix
resulting in higher rank
• SVD retrieves the latent factors (hopefully).

22. A VT

U
=
objects
features
significant
noise
noise
noise
significant sig.
=
SVD and Rank-k approximations

23. Application: Recommender systems
• Data: Users rating movies
• Sparse and often noisy
• Assumption: There are k basic user profiles, and
each user is a linear combination of these profiles
• E.g., action, comedy, drama, romance
• Each user is a weighted cobination of these profiles
• The “true” matrix has rank k
• What we observe is a noisy, and incomplete version
of this matrix 𝐴
• The rank-k approximation 𝐴𝑘 is provably close to 𝐴𝑘
• Algorithm: compute 𝐴𝑘 and predict for user 𝑢 and
movie 𝑚, the value 𝐴𝑘[𝑚, 𝑢].
• Model-based collaborative filtering

24. SVD and PCA
• PCA is a special case of SVD on the centered
covariance matrix.

25. Covariance matrix
• Goal: reduce the dimensionality while preserving the
“information in the data”
• Information in the data: variability in the data
• We measure variability using the covariance matrix.
• Sample covariance of variables X and Y
𝑖
𝑥𝑖 − 𝜇𝑋
𝑇(𝑦𝑖 − 𝜇𝑌)
• Given matrix A, remove the mean of each column
from the column vectors to get the centered matrix C
• The matrix 𝑉 = 𝐶𝑇𝐶 is the covariance matrix of the
row vectors of A.

26. PCA: Principal Component Analysis
• We will project the rows of matrix A into a new set
of attributes (dimensions) such that:
• The attributes have zero covariance to each other (they
are orthogonal)
• Each attribute captures the most remaining variance in
the data, while orthogonal to the existing attributes
• The first attribute should capture the most variance in the data
• For matrix C, the variance of the rows of C when
projected to vector x is given by 𝜎2 = 𝐶𝑥
2
• The right singular vector of C maximizes 𝜎2!

27. 4.0 4.5 5.0 5.5 6.0
2
3
4
5
PCA
Input: 2-d dimensional points
Output:
1st (right)
singular vector
1st (right) singular vector:
direction of maximal variance,
2nd (right)
singular
vector
2nd (right) singular vector:
direction of maximal variance,
after removing the projection of
the data along the first singular
vector.

28. Singular values
1: measures how much of the
data variance is explained by
the first singular vector.
2: measures how much of the
data variance is explained by
the second singular vector.
1
4.0 4.5 5.0 5.5 6.0
2
3
4
5
1st (right)
singular vector
2nd (right)
singular
vector

29. Singular values tell us something about
the variance
• The variance in the direction of the k-th principal
component is given by the corresponding singular value
σk
2
• Singular values can be used to estimate how many
components to keep
• Rule of thumb: keep enough to explain 85% of the
variation:
85
.
0
1
2
1
2





n
j
j
k
j
j



30. Example
𝐴 =
𝑎11 ⋯ 𝑎1𝑛
⋮ ⋱ ⋮
𝑎𝑛1 ⋯ 𝑎𝑛𝑛
𝐴 = 𝑈Σ𝑉𝑇
• First right singular vector 𝑣1
• More or less same weight to all drugs
• Discriminates heavy from light users
• Second right singular vector
• Positive values for legal drugs, negative for illegal
students
drugs
legal illegal
𝑎𝑖𝑗: usage of student i of drug j
Drug 2
Drug 1

31. Another property of PCA/SVD
• The chosen vectors are such that minimize the sum of square
differences between the data vectors and the low-dimensional
projections
4.0 4.5 5.0 5.5 6.0
2
3
4
5
1st (right)
singular vector

32. Application
• Latent Semantic Indexing (LSI):
• Apply PCA on the document-term matrix, and index the
k-dimensional vectors
• When a query comes, project it onto the k-dimensional
space and compute cosine similarity in this space
• Principal components capture main topics, and enrich
the document representation

33. SVD is “the Rolls-Royce and the Swiss
Army Knife of Numerical Linear
Algebra.”*
*Dianne O’Leary, MMDS ’06