Six Myths about Ontologies: The Basics of Formal Ontology
Β
Principal component analysis and matrix factorizations for learning (part 2) ding - icml 2005 tutorial - 2005
1. Part 2. Spectral Clustering from
Matrix Perspective
A brief tutorial emphasizing recent developments
(More detailed tutorial is given in ICMLβ04 )
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 56
2. From PCA to spectral clustering
using generalized eigenvectors
Consider the kernel matrix: Wij = Ο ( xi ),Ο ( x j )
In Kernel PCA we compute eigenvector: Wv = Ξ»v
Generalized Eigenvector: Wq = Ξ»Dq
D = diag (d1,L, dn ) di = βw j ij
This leads to Spectral Clustering !
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 57
3. Indicator Matrix Quadratic Clustering
Framework
Unsigned Cluster indicator Matrix H=(h1, β¦, hK)
Kernel K-means clustering:
max Tr( H T WH ), s.t. H T H = I , H β₯ 0
H
K-means: W = XT X; Kernel K-means W = (< Ο ( xi ),Ο ( x j ) >)
Spectral clustering (normalized cut)
max Tr( H T WH ), s.t. H T DH = I , H β₯ 0
H
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 58
4. Brief Introduction to Spectral Clustering
(Laplacian matrix based clustering)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 59
5. Some historical notes
β’ Fiedler, 1973, 1975, graph Laplacian matrix
β’ Donath & Hoffman, 1973, bounds
β’ Hall, 1970, Quadratic Placement (embedding)
β’ Pothen, Simon, Liou, 1990, Spectral graph
partitioning (many related papers there after)
β’ Hagen & Kahng, 1992, Ratio-cut
β’ Chan, Schlag & Zien, multi-way Ratio-cut
β’ Chung, 1997, Spectral graph theory book
β’ Shi & Malik, 2000, Normalized Cut
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 60
6. Spectral Gold-Rush of 2001
9 papers on spectral clustering
β’ Meila & Shi, AI-Stat 2001. Random Walk interpreation of
Normalized Cut
β’ Ding, He & Zha, KDD 2001. Perturbation analysis of Laplacian
matrix on sparsely connected graphs
β’ Ng, Jordan & Weiss, NIPS 2001, K-means algorithm on the
embeded eigen-space
β’ Belkin & Niyogi, NIPS 2001. Spectral Embedding
β’ Dhillon, KDD 2001, Bipartite graph clustering
β’ Zha et al, CIKM 2001, Bipartite graph clustering
β’ Zha et al, NIPS 2001. Spectral Relaxation of K-means
β’ Ding et al, ICDM 2001. MinMaxCut, Uniqueness of relaxation.
β’ Gu et al, K-way Relaxation of NormCut and MinMaxCut
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 61
7. Spectral Clustering
min cutsize , without explicit size constraints
But where to cut ?
Need to balance sizes
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 62
8. Graph Clustering
min between-cluster similarities (weights)
sim(A,B) = ββ wij
iβ A jβB
Balance weight
Balance size
Balance volume
sim(A,A) = ββ wij
iβ A jβ A
max within-cluster similarities (weights)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 63
9. Clustering Objective Functions
s(A,B) = ββ w ij
β’ Ratio Cut iβ A jβB
s(A,B) s(A,B)
J Rcut (A,B) = +
|A| |B|
β’ Normalized Cut dA = βd i
iβA
s ( A, B) s ( A, B)
J Ncut ( A, B) = +
dA dB
s ( A, B) s ( A, B)
= +
s ( A, A) + s ( A, B) s(B, B) + s ( A, B)
β’ Min-Max-Cut
s(A,B) s(A,B)
J MMC(A,B) = +
s(A,A) s(B,B)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 64
11. A simple example
2 dense clusters, with sparse connections
between them.
Adjacency matrix Eigenvector q2
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 66
12. K-way Spectral Clustering
Kβ₯2
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 67
13. K-way Clustering Objectives
β’ Ratio Cut
β s (C k ,Cl ) s (C k ,Cl ) β s (C k ,G β C k )
J Rcut (C1 , L , C K ) = β β
β |C | + |C | β =
< k ,l > β k l
β
β
β
k
|C k|
β’ Normalized Cut
β s (C k ,Cl ) s (C k ,Cl ) β s (C k ,G β C k )
J Ncut (C1 , L , C K ) =
< k ,l >
ββ
β d
β k
+
dl
β=
β
β
β
k
dk
β’ Min-Max-Cut
β s (C k ,Cl ) s (C k ,Cl ) β s (C k ,G β C k )
J MMC (C1 , L , C K ) = β
< k ,l > β
β k k l l β
β
β s (C , C ) + s (C , C ) β = β k
s (C k , C k )
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 68
14. K-way Spectral Relaxation
h1 = (1L1,0 L 0,0 L 0)T
Unsigned cluster indicators:
h2 = (0L 0,1L1,0 L 0)T
LLL
Re-write: hk = (0 L 0,0L 0,1L1)T
h1 ( D β W )h1
T
hk ( D β W )hk
T
J Rcut (h1 , L, hk ) = T
+L+ T
h1 h1 hk hk
h1 ( D β W )h1
T
hk ( D β W )hk
T
J Ncut (h1 , L, hk ) = T
+L+ T
h1 Dh1 hk Dhk
h1 ( D β W )h1
T
hk ( D β W )hk
T
J MMC (h1 , L , hk ) = T
+L+ T
h1 Wh1 hk Whk
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 69
15. K-way Normalized Cut Spectral Relaxation
Unsigned cluster indicators: nk
}
y k = D1/ 2 (0 L 0,1L1,0L 0)T / || D1/ 2 hk ||
Re-write: ~ ~
J Ncut ( y1 , L , y k ) = T
y1 ( I β W ) y1 + L + y k ( I β W ) y k
T
~ ~
= Tr (Y T ( I β W )Y ) W = D β1/ 2WD β1/ 2
~
Optimize : min Tr (Y ( I β W )Y ), subject to Y T Y = I
T
Y
By K. Fanβs theorem, optimal solution is
~
eigenvectors: Y=(v1,v2, β¦, vk), ( I β W )vk = Ξ»k vk
( D β W )u k = Ξ»k Du k , u k = D β1/ 2 vk
Ξ»1 + L + Ξ»k β€ min J Ncut ( y1 ,L , y k ) (Gu, et al, 2001)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 70
16. K-way Spectral Clustering is difficult
β’ Spectral clustering is best applied to 2-way
clustering
β positive entries for one cluster
β negative entries for another cluster
β’ For K-way (K>2) clustering
β Positive and negative signs make cluster
assignment difficult
β Recursive 2-way clustering
β Low-dimension embedding. Project the data to
eigenvector subspace; use another clustering
method such as K-means to cluster the data (Ng
et al; Zha et al; Back & Jordan, etc)
β Linearized cluster assignment using spectral ordering and
cluster crossing
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 71
17. Scaled PCA: a Unified Framework
for clustering and ordering
β’ Scaled PCA has two optimality properties
β Distance sensitive ordering
β Min-max principle Clustering
β’ SPCA on contingency table β Correspondence Analysis
β Simultaneous ordering of rows and columns
β Simultaneous clustering of rows and columns
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 72
18. Scaled PCA
similarity matrix S=(sij) (generated from XXT)
D = diag(d1 ,L, d n ) di = si.
~ β1 β1 ~
Nonlinear re-scaling: S = D SD , sij = sij /(si.s j. )
2 2 1/ 2
~
Apply SVD on Sβ
~ 1 β‘ Tβ€
S = D S D = D β zk Ξ»k z k D = D β’β qk Ξ»k qk β₯ D
1
1 1
2 2
2 T 2
k β£k β¦
qk = D-1/2 zk is the scaled principal component
Subtract trivial component Ξ» = 1, z = d 1/ 2 /s.., q =1
0 0 0
β S β dd T /s.. = D β qk Ξ»k qT D
k
k =1 (Ding, et al, 2002)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 73
19. Scaled PCA on a Rectangle Matrix
β Correspondence Analysis
~ β1 β1 ~
Nonlinear re-scaling: P = D 2 PD 2 , p = p /( p p )1/ 2
r c ij ij i. j.
~
Apply SVD on P Subtract trivial component
P β rc / p.. = Dr β f k Ξ»k g Dc
T T r = ( p1.,L, pn. )
T
k
k =1
β1 β1
c = ( p.1,L, p.n ) T
fk = D u , gk = D v
r
2
k
2
c k
are the scaled row and column principal
component (standard coordinates in CA)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 74
21. Clustering of Bipartite Graphs (rectangle matrix)
Simultaneous clustering of rows and columns
of a contingency table (adjacency matrix B )
Examples of bipartite graphs
β’ Information Retrieval: word-by-document matrix
β’ Market basket data: transaction-by-item matrix
β’ DNA Gene expression profiles
β’ Protein vs protein-complex
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 76
23. Spectral Clustering of Bipartite Graphs
Simultaneous clustering of rows and columns
(adjacency matrix B )
s ( BR1 ,C2 ) = β βb
ri βR1c j βC 2
ij
min between-cluster sum of
xyz weights: s(R1,C2), s(R2,C1)
max within-cluster sum of xyz
cut xyz weights: s(R1,C1), s(R2,C2)
s ( BR1 ,C2 ) + s ( B R2 ,C1 ) s ( B R1 ,C2 ) + s ( B R2 ,C1 )
J MMC (C1 , C 2 ; R1 , R2 ) = +
2 s ( B R1 ,C1 ) 2 s ( B R2 ,C2 )
(Ding, AI-STAT 2003)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 78
24. Internet Newsgroups
Simultaneous clustering
of documents and words
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 79
25. Embedding in Principal Subspace
Cluster Self-Aggregation
(proved in perturbation analysis)
(Hall, 1970, βquadratic placementβ (embedding) a graph)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 80
26. Spectral Embedding: Self-aggregation
β’ Compute K eigenvectors of the Laplacian.
β’ Embed objects in the K-dim eigenspace
(Ding, 2004)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 81
27. Spectral embedding is not
topology preserving
700 3-D data points form
2 interlock rings
In eigenspace, they
shrink and separate
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 82
28. Spectral Embedding
Simplex Embedding Theorem.
Objects self-aggregate to K centroids
Centroids locate on K corners of a simplex
β’ Simplex consists K basis vectors + coordinate origin
β’ Simplex is rotated by an orthogonal transformation T
β’T are determined by perturbation analysis
(Ding, 2004)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 83
29. Perturbation Analysis
Wq = Ξ»Dq WΛ z = ( D β1 / 2WD β1 / 2 ) z = Ξ»z q = D β1 / 2 z
Assume data has 3 dense clusters sparsely connected.
C2
β‘W W W β€
11 12 13 C1
W = β’ 21 W22 W23β₯
β’W β₯
β’ 31 W32 W33β₯
β£W β¦ C3
Off-diagonal blocks are between-cluster connections,
assumed small and are treated as a perturbation
(Ding et al, KDDβ01)
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 84
32. Similarity matrix W 1st order Perturbation: Example 1
1st order
solution
Connectivity
Ξ»2 = 0.300, Ξ»2 = 0.268
matrix
Between-cluster connections suppressed
Within-cluster connections enhanced
Effects of self-aggregation
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 87
33. Optimality Properties of Scaled PCA
Scaled principal components have optimality properties:
Ordering
β Adjacent objects along the order are similar
β Far-away objects along the order are dissimilar
β Optimal solution for the permutation index are given by
scaled PCA.
Clustering
β Maximize within-cluster similarity
β Minimize between-cluster similarity
β Optimal solution for cluster membership indicators given
by scaled PCA.
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 88
34. Spectral Graph Ordering
(Barnard, Pothen, Simon, 1993), envelop reduction of sparse
matrix: find ordering such that the envelop is minimized
min β max j | i β j | wij β min β ( xi β x j ) wij 2
i ij
(Hall, 1970), βquadratic placement of a graphβ:
Find coordinate x to minimize
J= β ij
( xi β x j ) 2 wij = x T ( D β W ) x
Solution are eigenvectors of Laplacian
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 89
36. Distance Sensitive Ordering
J (Ο ) = β (i β j ) wΟ i ,Ο j = β (i β j ) wΟ i ,Ο j
2 2
ij Ο i ,Ο j
= β (Ο β Ο ) wi , j
i
β1 β1 2
j
ij
n2 Ο iβ1 β( n +1) / 2 Ο β1 β( n +1) / 2 2
=
8 ij
β( n/2 β j
n/2 ) wi , j
Define: shifted and rescaled inverse permutation indexes
Ο iβ1 β (n + 1) /2 1β n 3 β n n β1
qi = ={ , ,L, }
n /2 n n n
J (Ο ) = n2
8 β (qi β q j ) wij = q ( D β W )q
2 n2
4
T
ij
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 91
37. Distance Sensitive Ordering
Once q2 is computed, since
q2 (i ) < q2 ( j ) β Ο i
β1
<Ο β1
j
Ο i
β1
can be uniquely recovered from q2
Implementation: sort q2 induces Ο
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 92
38. Re-ordering of Genes and Tissues
J (Ο )
r=
J (random)
r = 0.18
J d =1 (Ο )
rd =1=
J d =1 ( random )
rd =1 = 3.39
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 93
39. Spectral clustering vs Spectral ordering
β’ Continuous approximation of both integer
programming problems are given by the same
eigenvector
β’ Different problems could have the same
continuous approximate solution.
β’ Quality of the approximation:
Ordering: better quality: the solution relax
from a set of evenly spaced discrete values
Clustering: less better quality: solution relax
from 2 discrete values
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 94
40. Linearized Cluster Assignment
Turn spectral clustering to 1D clustering problem
β’ Spectral ordering on connectivity network
β’ Cluster crossing
β Sum of similarities along anti-diagonal
β Gives 1-D curve with valleys and peaks
β Divide valleys and peaks into clusters
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 95
41. Cluster overlap and crossing
Given similarity W, and clusters A,B.
β’ Cluster overlap s(A,B) = ββ w
iβ A jβB
ij
β’ Cluster crossing compute a smaller fraction of cluster
overlap.
β’ Cluster crossing depends on an ordering o. It sums
weights cross the site i along the order
m
Ο (i ) = β wo (iβ j ),o (i+ j )
j =1
β’ This is a sum along anti-diagonals of W.
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 96
42. cluster crossing
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 97
43. K-way Clustering Experiments
Accuracy of clustering results:
Method Linearized Recursive 2-way Embedding
Assignment clustering + K-means
Data A 89.0% 82.8% 75.1%
Data B 75.7% 67.2% 56.4%
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 98
44. Some Additional
Advanced/related Topics
β’ Random talks and normalized cut
β’ Semi-definite programming
β’ Sub-sampling in spectral clustering
β’ Extending to semi-supervised classification
β’ Greenβs function approach
β’ Out-of-sample embeding
PCA & Matrix Factorizations for Learning, ICML 2005 Tutorial, Chris Ding 99