Spectral clustering works by creating an affinity matrix from a similarity matrix and then applying dimensionality reduction before clustering in the reduced space. It represents the data as an undirected graph and uses the graph Laplacian matrix to perform the dimensionality reduction. The number of clusters can be determined using the eigengap heuristic or by setting k equal to the logarithm of the number of data points. The Gaussian kernel is commonly used to create the affinity matrix from the similarity matrix.

1. Presented by Danushka Bollegala

2.  Spectrum = the set of eigenvalues
 By looking at the spectrum we can know about
the graph itself!
 A way of normalizing data (canonical form)
and then perform clustering (e.g. via k-
means) on this normalized/reduced space.
 Input: A similarity matrix
 Output: A set of (non-overlapping/hard)
clusters.

3.  UndirectedGraph G(V, E)
 V: set of vertices (nodes in the network)
 E: set of edges (links in the network)
▪ Weight wij is the weight of the edge connecting vertex I
and j (represented by the affinity matrix.)
 Degree: sum of weights on outgoing edges of a
vertex.
 Measuring the size of a subset A ofV

4.  How to create the affinity matrixW from the
similarity matrix S?
 ε-neighborhood graph
▪ Connect all vertices that have similarity greater than ε
 k-nearest neighbor graph
▪ Connect the k-nearest neighbors of each vertex.
▪ Mutual k-nearest neighbor graphs for asymmetric S.
 Fully connected graph
▪ Use the Gaussian similarity function (kernel)

5.  L = D –W
 D: degree matrix. A diagonal matrix diag(d1,...,dn)
 Properties
 For every vector
 L is symmetric and positive semi-definite
 The smallest eigenvalue of L is zero and the
corresponding eigenvector is 1 = (1,...,1)T
 L has n non-negative, real-valued eigenvalues

6.  Two versions exist
 Lsym = D-1/2LD-1/2 = I - D-1/2WD-1/2
 Lrw = D-1L = I - D-1W

11.  The partition (A1,...,Ak) induces a cut on the graph
 Two types of graph cuts exist
 Spectral clustering solves a relaxed version of the
mincut problem (therefore it is an approximation)

12. By the Rayleigh-Ritz
theorem it follows that the
second eigenvalue is the
minimum.

14.  Transition probability matrix and Laplacian
are related!
 P = D-1W
 Lrw = I - P

15.  Lrw based spectral clustering (Shi &
Malik,2000) is better (especially when the
degree distribution is uneven).
 Use k-nearest neighbor graphs
 How to set the number of clusters:
 k=log(n)
 Use the eigengap heuristic
 If using Gaussian kernel how to set sigma
 Mean distance of a point to its log(n)+1 nearest
neighbors.

16.  Eckart-YoungTheorem
 The low-rank approximation B for a matrix A s.t.
rank(B) = r < rank(A) is given by,
 B = USV*, where A = UZV* and S is the same as Z
except the (r+1) and above singular values of Z are
set to zero.
 Approximation is done by minimizing the
Frobenius norm
▪ minB||A – B||F, subject to rank(B) = r