November, 15th, 2012 Séminaire LIPN, Université Paris 13

1. Clustered graph, visualization and hierarchical
visualization
Nathalie Villa-Vialaneix
http://www.nathalievilla.org
nathalie.villa@univ-paris1.fr
Séminaire LIPN, Université Paris 13
November, 15th, 2012
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 1 / 35

2. An overview on graph visualization and clustering
Framework
A graph (network) G = (V, E, W) with
• n vertices (nodes) V = {x1, . . . , xn};
• edges, E, weighted by Wij = Wji ≥ 0 (Wii = 0).
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 2 / 35

3. An overview on graph visualization and clustering
Network mining through visualization
A standard approach for network mining: using a force directed
placement algorithm (FDP) to display the graph; e.g.,
[Fruchterman and Reingold, 1991]
• attractive forces: along the edges, analogous to springs;
• repulsive forces : between all pairs of nodes, analogous to electric
forces.
The algorithm starts from an initial (random) position and iterates until the
layout is stabilized.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 3 / 35

4. An overview on graph visualization and clustering
Network mining through visualization
A standard approach for network mining: using a force directed
placement algorithm (FDP) to display the graph; e.g.,
[Fruchterman and Reingold, 1991]
• attractive forces: along the edges, analogous to springs;
• repulsive forces : between all pairs of nodes, analogous to electric
forces.
The algorithm starts from an initial (random) position and iterates until the
layout is stabilized.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 3 / 35

5. An overview on graph visualization and clustering
Network mining through visualization
A standard approach for network mining: using a force directed
placement algorithm (FDP) to display the graph; e.g.,
[Fruchterman and Reingold, 1991]
• attractive forces: along the edges, analogous to springs;
• repulsive forces : between all pairs of nodes, analogous to electric
forces.
The algorithm starts from an initial (random) position and iterates until the
layout is stabilized.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 3 / 35

6. An overview on graph visualization and clustering
Drawbacks of FDP algorithms
• slow (hard to use for very large graphs);
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 4 / 35

7. An overview on graph visualization and clustering
Drawbacks of FDP algorithms
• slow (hard to use for very large graphs);
• are more oriented toward aesthetic than toward an interpretable
layout:
• tendency: short edges with uniform lengths;
• negative consequence: hubs are clustered in the center of the figure.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 4 / 35

8. An overview on graph visualization and clustering
Drawbacks of FDP algorithms
• slow (hard to use for very large graphs);
• are more oriented toward aesthetic than toward an interpretable
layout:
• tendency: short edges with uniform lengths;
• negative consequence: hubs are clustered in the center of the figure.
What the user usually prefers:
1 understanding the macroscopic structure of the graph, i.e., find out
“communities” and their relations;
2 focus on details for clusters that seem to be of interest.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 4 / 35

9. An overview on graph visualization and clustering
Emphasizing “communities” in the layout
1 global approach: displaying all vertices while modifying the forces in
such a way that the dense areas are emphasized: [Noack, 2007]
(LinLog algorithm)
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 5 / 35

10. An overview on graph visualization and clustering
Emphasizing “communities” in the layout
1 global approach: displaying all vertices while modifying the forces in
such a way that the dense areas are emphasized: [Noack, 2007]
(LinLog algorithm)
2 clustering the vertices and then using a simplified representation of
the graph [Herman et al., 2000]
• partition the nodes into clusters V1, . . . , VC ;
• display the clustered graph: nodes V1, . . . , VC (surface proportional
to |Vj|) and edges width proportional to xk ∈Vi ,xk ∈Vj
Wij
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 5 / 35

11. An overview on graph visualization and clustering
Emphasizing “communities” in the layout
1 global approach: displaying all vertices while modifying the forces in
such a way that the dense areas are emphasized: [Noack, 2007]
(LinLog algorithm)
2 clustering the vertices and then using a simplified representation of
the graph [Herman et al., 2000]
• partition the nodes into clusters V1, . . . , VC ;
• display the clustered graph: nodes V1, . . . , VC (surface proportional
to |Vj|) and edges width proportional to xk ∈Vi ,xk ∈Vj
Wij
Main issue: Modify FDP to allows us to display nodes with different
sizes.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 5 / 35

12. An overview on graph visualization and clustering
Emphasizing “communities” in the layout
1 global approach: displaying all vertices while modifying the forces in
such a way that the dense areas are emphasized: [Noack, 2007]
(LinLog algorithm)
2 clustering the vertices and then using a simplified representation of
the graph
alternative approach: displaying while clustering as in
Self-Organizing Maps [Boulet et al., 2008],
[Rossi and Villa-Vialaneix, 2010] and [Olteanu et al., 2013]
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 5 / 35

13. An overview on graph visualization and clustering
Emphasizing “communities” in the layout
1 global approach: displaying all vertices while modifying the forces in
such a way that the dense areas are emphasized: [Noack, 2007]
(LinLog algorithm)
2 clustering the vertices and then using a simplified representation of
the graph
3 combined approach: hierarchical representations where finer
details are provided to the user
[Auber et al., 2003, Auber and Jourdan, 2005, Seifi et al., 2010]
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 5 / 35

14. An overview on graph visualization and clustering
Outline of this talk
• self-organizing maps based on kernels and dissimilarities;
• modularity based representations:
• combined with a map;
• used hierarchically.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 6 / 35

15. Self-organizing maps approaches
Outline
1 Self-organizing maps approaches
2 Modularity based approaches
3 Soft modularity
Hierarchical clustering and visualization
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 7 / 35

16. Self-organizing maps approaches
Basic ideas about SOM
Project the graph on a squared grid (each square of the grid is a cluster)
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 8 / 35

17. Self-organizing maps approaches
Basic ideas about SOM
Project the graph on a squared grid (each square of the grid is a cluster)
such that:
• the nodes in a same cluster are highly connected
• the nodes in two close clusters are also (less) connected
• the nodes in two distant clusters are (almost) not connected
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 8 / 35

18. Self-organizing maps approaches
Basics on Self-Organizing Maps (for multidimensional
data)
• the map is made of neurons (visually symbolized by, e.g.,
rectangles), 1...M, with which prototypes pi are associated (a
prototype is a “representer” of the neuron in the original dataset);
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 9 / 35

19. Self-organizing maps approaches
Basics on Self-Organizing Maps (for multidimensional
data)
• the map is made of neurons (visually symbolized by, e.g.,
rectangles), 1...M, with which prototypes pi are associated (a
prototype is a “representer” of the neuron in the original dataset);
• the map is equipped with a neighborhood relationship, i.e., a
“distance” (actually a dissimilarity) between neurons, D;
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 9 / 35

20. Self-organizing maps approaches
Basics on Self-Organizing Maps (for multidimensional
data)
• the map is made of neurons (visually symbolized by, e.g.,
rectangles), 1...M, with which prototypes pi are associated (a
prototype is a “representer” of the neuron in the original dataset);
• the map is equipped with a neighborhood relationship, i.e., a
“distance” (actually a dissimilarity) between neurons, D;
• goal: find the best mapping f(xi) ∈ {1, . . . , M} of the data xi in the
different neurons by minimizing the energy
E =
n
i=1
M
j=1
h(D(f(xi), j)) xi − pi
2
.
i.e., each data is assigned to a neuron so that:
• the neuron’s prototype is “close” to the data;
• the neighboring prototypes are also “close” to the data;
• distant prototypes are “distant” of the data.
(topology preservation)
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 9 / 35

21. Self-organizing maps approaches
SOM, dissimilarity SOM and kernel SOM (batch)
Original SOM algorithm (batch): x1, . . . , xn ∈ Rd
1: Initialization: randomly set p0
1
,...,p0
M
in Rd
2: for l = 1 → L do
3: for all i = 1 → n do Assignment
4: fl
(xi) ← arg minj=1,...,M xi − pl−1
j Rd
5: end for
6: for all j = 1 → M do Representation
7: pl
j
← arg minp∈Rd
n
i=1 hl
(D(fl
(xi), j)) xi − p 2
Rd
8: end for
9: end for
Problems with graphs: xi are nodes so 1/ how to define the prototypes?
and 2/ which distance to use between nodes?
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 10 / 35

22. Self-organizing maps approaches
SOM, dissimilarity SOM and kernel SOM (batch)
Dissimilarity SOM (batch): xi ∈ G defined by a dissimilarity relation:
δ(xi, xj)
1: Initialization: randomly set p0
1
,...,p0
M
in (xi)i
2: for l = 1 → L do
3: for all i = 1 → n do Assignment
4: fl
(xi) ← arg minj=1,...,M δ(xi, pl−1
j
)
5: end for
6: for all j = 1 → M do Representation
7: pl
j
← arg minp∈(xi)i
n
i=1 hl
(D(fl
(xi), j))δ(xi, p)
8: end for
9: end for
[Kohohen and Somervuo, 1998, Kohonen and Somervuo, 2002]
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 10 / 35

23. Self-organizing maps approaches
SOM, dissimilarity SOM and kernel SOM (batch)
Dissimilarity SOM (batch): xi ∈ G defined by a dissimilarity relation:
δ(xi, xj)
1: Initialization: randomly set p0
j
← γ0
ji
xi (symbolic)
2: for l = 1 → L do
3: for all i = 1 → n do Assignment
4: fl
(xi) ← arg minj=1,...,M δ2
(xi, pl−1
j
) = ∆γl−1
j i
− 1
2 (γl−1
j
)T
∆γl−1
j
where ∆ = (δ(xk , xk ))k,k
5: end for
6: for all j = 1 → M do Representation
7: γl
j
← arg minγ∈Rn
n
i=1 hl
(D(fl
(xi), j))δ2
xi, n
k=1 γk xk
8: end for
9: end for
[Rossi et al., 2007]
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 10 / 35

24. Self-organizing maps approaches
SOM, dissimilarity SOM and kernel SOM (batch)
Kernel SOM (batch): xi ∈ G defined by a kernel relation: K(xi, xj) ⇒
∃ φ : G → (H, ., . H ):K(x, x ) = φ(x), φ(x ) H
1: Initialization: randomly set p0
j
← n
i=1 γ0
ji
φ(xi)
2: for l = 1 → L do
3: for all i = 1 → n do Assignment
4: fl
(xi) ← arg minj=1,...,M φ(xi) − pl−1
j H where φ(xi) − pl−1
j H =
n
k=1 γl−1
jk
γl−1
jk
K(xk , xk ) − 2 n
k=1 γl−1
jk
K(xi, xk )
5: end for
6: for all j = 1 → M do Representation
7: γl
jk
← arg minγ∈Rn
n
i=1 hl
(D(fl
(xi), j)) φ(xi) − n
k=1 γk φ(xk ) 2
H
8: end for
9: end for
[Villa and Rossi, 2007, Boulet et al., 2008]
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 10 / 35

25. Self-organizing maps approaches
Dissimilarity SOM (stochastic)
(Online relational SOM) [Olteanu et al., 2013]
1: Initialization: randomly set γ0
ji
in R
2: for l = 1 → L do
3: Randomly chose an input xi
4: Assignment ft
(xi) ← arg minj=1,...,M γl−1
j
∆
i
− 1
2 γl−1
j
∆(γl−1
j
)T
5: for all j = 1 → M do Update of the prototypes
6: γl
j
← γl−1
j
+ αl
hl
(D(fl
(xi), j)) 1i − γl−1
j
where 1i is a vector with a
single non null coefficient at the ith position, equal to one
7: end for
8: end for
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 11 / 35

26. Self-organizing maps approaches
Which dissimilarities/kernels for graphs?
Laplacian [Kondor and Lafferty, 2002]
For a graph with vertices V = {x1, . . . , xn} and weights (wi,j)i,j=1,...,n
(positive, symmetric), the Laplacian is: L = (Li,j)i,j=1,...,n where
Li,j =
−wi,j if i j
di = j i wi,j if i = j
;
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 12 / 35

27. Self-organizing maps approaches
Which dissimilarities/kernels for graphs?
Laplacian [Kondor and Lafferty, 2002]
For a graph with vertices V = {x1, . . . , xn} and weights (wi,j)i,j=1,...,n
(positive, symmetric), the Laplacian is: L = (Li,j)i,j=1,...,n where
Li,j =
−wi,j if i j
di = j i wi,j if i = j
;
1 Diffusion matrix [Kondor and Lafferty, 2002]: for β > 0,
Kβ = e−βL
= +∞
k=1
(−βL)k
k! heat kernel (or diffusion kernel);
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 12 / 35

28. Self-organizing maps approaches
Which dissimilarities/kernels for graphs?
Laplacian [Kondor and Lafferty, 2002]
For a graph with vertices V = {x1, . . . , xn} and weights (wi,j)i,j=1,...,n
(positive, symmetric), the Laplacian is: L = (Li,j)i,j=1,...,n where
Li,j =
−wi,j if i j
di = j i wi,j if i = j
;
1 Diffusion matrix [Kondor and Lafferty, 2002]: for β > 0,
Kβ = e−βL
= +∞
k=1
(−βL)k
k! heat kernel (or diffusion kernel);
2 Generalized inverse of the Laplacian [Fouss et al., 2007] :
K = L+;
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 12 / 35

29. Self-organizing maps approaches
Which dissimilarities/kernels for graphs?
Laplacian [Kondor and Lafferty, 2002]
For a graph with vertices V = {x1, . . . , xn} and weights (wi,j)i,j=1,...,n
(positive, symmetric), the Laplacian is: L = (Li,j)i,j=1,...,n where
Li,j =
−wi,j if i j
di = j i wi,j if i = j
;
1 Diffusion matrix [Kondor and Lafferty, 2002]: for β > 0,
Kβ = e−βL
= +∞
k=1
(−βL)k
k! heat kernel (or diffusion kernel);
2 Generalized inverse of the Laplacian [Fouss et al., 2007] :
K = L+;
3 Dissimilarity: length of the shortest path between two nodes.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 12 / 35

30. Self-organizing maps approaches
A first example: a medieval social network
Example from [Boulet et al., 2008]
, http://graphcomp.univ-tlse2.fr/ In the “Archive départementales
du Lot” (Cahors, France), big corpus of 5000 transactions (mostly land
charters)
• coming from 4 “seigneuries” (about 25 little villages) in South West of
France;
• being established between 1240 and 1520 (just before and after the
hundred years’ war).
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 13 / 35

31. Self-organizing maps approaches
Simplification of this network by kernel SOM
nodes: individuals ( 600)
named in the transactions, re-
stricted to transactions estab-
lished before the HYW; edges:
the fact that two individuals are
named in a common transac-
tion or have a common lord
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 14 / 35

32. Self-organizing maps approaches
Simplification of this network by kernel SOM
Kernel SOM with heat kernel
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 14 / 35

33. Self-organizing maps approaches
A brief comparison with spectral clustering
Number of clusters: 35 50
Maximum size of the clusters: 255 268
Modularity: 0.597 0.420
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 15 / 35

34. Self-organizing maps approaches
A brief comparison with spectral clustering
Number of clusters: 35 29
Maximum size of the clusters: 255 325
Modularity: 0.597 0.433
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 15 / 35

35. Self-organizing maps approaches
Online relational SOM (faster)
Description:
• nodes: 105 American political books;
• edges weighted by the number of co-purchasing of the two books on
the internet (Amazon.com).
FDP representation
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 16 / 35

36. Self-organizing maps approaches
Online relational SOM (faster)
Description:
• nodes: 105 American political books;
• edges weighted by the number of co-purchasing of the two books on
the internet (Amazon.com).
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 16 / 35

37. Modularity based approaches
Outline
1 Self-organizing maps approaches
2 Modularity based approaches
3 Soft modularity
Hierarchical clustering and visualization
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 17 / 35

38. Modularity based approaches
Modularity [Newman and Girvan, 2004]
Popular quality measure for graph clustering: a partition of the vertices
in C clusters, (Ck )k=1,...,C has modularity:
Q(C) =
1
2m
C
k=1 i,j∈Ck
(Wij − Pij)
where Pij are weights corresponding to a “null model” where the weights
only depend on the nodes properties and not on the cluster they belong to.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 18 / 35

39. Modularity based approaches
Modularity [Newman and Girvan, 2004]
Popular quality measure for graph clustering: a partition of the vertices
in C clusters, (Ck )k=1,...,C has modularity:
Q(C) =
1
2m
C
k=1 i,j∈Ck
(Wij − Pij)
where Pij are weights corresponding to a “null model” where the weights
only depend on the nodes properties and not on the cluster they belong to.
More precisely,
Pij =
didj
2m
with di = 1
2 j i Wij is the degree of a vertex xi.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 18 / 35

40. Modularity based approaches
Modularity [Newman and Girvan, 2004]
Popular quality measure for graph clustering: a partition of the vertices
in C clusters, (Ck )k=1,...,C has modularity:
Q(C) =
1
2m
C
k=1 i,j∈Ck
(Wij − Pij)
where Pij are weights corresponding to a “null model” where the weights
only depend on the nodes properties and not on the cluster they belong to.
More precisely,
Pij =
didj
2m
with di = 1
2 j i Wij is the degree of a vertex xi.
A “good” clustering should maximize Q.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 18 / 35

41. Modularity based approaches
Interpretation
• Q increases when (xi, xj) are in a same cluster and have true
weight Wij greater than the ones expected in the null model, Pij
• Q increases when (xi, xj) are in a two different clusters and have
true weight Wij smaller than the ones expected in the null model, Pij
because
Q(C) +
1
2m
k k i∈Ck , j∈Ck
(Wij − Pij) = 0.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 19 / 35

42. Modularity based approaches
Interpretation
• Q increases when (xi, xj) are in a same cluster and have true
weight Wij greater than the ones expected in the null model, Pij
• Q increases when (xi, xj) are in a two different clusters and have
true weight Wij smaller than the ones expected in the null model, Pij
because
Q(C) +
1
2m
k k i∈Ck , j∈Ck
(Wij − Pij) = 0.
• Contrary to the minimization of the number of edges between
clusters, modularity can help to separate nodes with high degrees
into different clusters more easily
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 19 / 35

43. Modularity based approaches
Drawing optimized clustering
Combine:
• high modularity to ensure high intra clusters density and low
external connectivity
• little edge crossing
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 20 / 35

44. Modularity based approaches
Drawing optimized clustering
Combine:
• high modularity to ensure high intra clusters density and low
external connectivity
• little edge crossing by:
• Classic solution: relying on graph drawing algorithm after maximization
of the modularity
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 20 / 35

45. Modularity based approaches
Drawing optimized clustering
Combine:
• high modularity to ensure high intra clusters density and low
external connectivity
• little edge crossing by:
• Classic solution: relying on graph drawing algorithm after maximization
of the modularity
• Extend the modularity to a criterium adapted to a prior structure (like a
grid)
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 20 / 35

46. Soft modularity
Outline
1 Self-organizing maps approaches
2 Modularity based approaches
3 Soft modularity
Hierarchical clustering and visualization
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 21 / 35

47. Soft modularity
Self Organizing Map principle
For data in Rd
, SOM minimizes (over the clustering and the prototypes
(pk ))
M
j=1
n
i=1
Sf(xi),j xi − pj
2
Rd
where Skl encodes the prior structure: close to 1 for close clusters and
close to 0 for distant clusters
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 22 / 35

48. Soft modularity
Self Organizing Map principle
For data in Rd
, SOM minimizes (over the clustering and the prototypes
(pk ))
M
j=1
n
i=1
Sf(xi),j xi − pj
2
Rd
where Skl encodes the prior structure: close to 1 for close clusters and
close to 0 for distant clusters
This corresponds to a soft membership: xi belongs to Cj with
membership Sf(xi),j.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 22 / 35

49. Soft modularity
Organized modularity [Rossi and Villa-Vialaneix, 2010]
Same idea: encode a prior structure via a matrix S.
Maximize:
SQ =
1
2m
i,j
Sf(i)f(j)(Wij − Pij)
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 23 / 35

50. Soft modularity
Organized modularity [Rossi and Villa-Vialaneix, 2010]
Same idea: encode a prior structure via a matrix S.
Maximize:
SQ =
1
2m
i,j
Sf(i)f(j)(Wij − Pij)
Hence:
• if a pair of vertices (xi, xj) is such that Wij > Pij, SQ increases with the
closeness of f(xi) and f(xj) in the prior structure
• if a pair of vertices (xi, xj) is such that Wij < Pij, SQ increases if f(xi)
and f(xj) are distant in the prior structure
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 23 / 35

51. Soft modularity
Optimization
The clustering is represented by a n × C assignment matrix M with
Mik = δf(i)=k . The goal is then to maximize
SQ = F(M) =
1
2m
i,j k,l
Mik SklMlj(Wij − Pij)
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 24 / 35

52. Soft modularity
Optimization
The clustering is represented by a n × C assignment matrix M with
Mik = δf(i)=k . The goal is then to maximize
SQ = F(M) =
1
2m
i,j k,l
Mik SklMlj(Wij − Pij)
Combinatorial problem is NP-complet ⇒ use of deterministic algorithm
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 24 / 35

53. Soft modularity
Comparison on the co-appearance network from “Les
Misérables”
Co-appearance network from “Les Misérables” [Knuth, 1993]
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
Myriel
Napoleon
MlleBaptistine
MmeMagloire
CountessDeLo
Geborand
Champtercier
Cravatte
Count
OldMan
Labarre
Valjean
Marguerite
MmeDeR
Isabeau
Gervais
Tholomyes
Listolier
Fameuil
Blacheville
Favourite
Dahlia
Zephine
Fantine
MmeThenardier
Thenardier
Cosette
Javert
Fauchelevent
Bamatabois
Perpetue
Simplice
Scaufflaire
Woman1
Judge
Champmathieu
Brevet
ChenildieuCochepaille
PontmercyBoulatruelle
Eponine
Anzelma
Woman2
MotherInnocent
Gribier
Jondrette
MmeBurgon
Gavroche
Gillenormand
Magnon
MlleGillenormand
MmePontmercy
MlleVaubois
LtGillenormand
Marius
BaronessT
Mabeuf
Enjolras
Combeferre
Prouvaire
Feuilly
Courfeyrac
BahorelBossuet
Joly
Grantaire
MotherPlutarch
GueulemerBabet
Claquesous
Montparnasse
Toussaint
Child1Child2
Brujon
MmeHucheloup
77 nodes
density = 8.7%
transitivity = 49.9 %
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 25 / 35

54. Soft modularity
Methodology
Comparison of:
• Kernel SOM with various kernels: heat kernel, generalized inverse of
the Laplacian, modularity kernel (i.e., the positive part of W − P which
mimics the optimization of the modularity) and spectral SOM (based
on the first M eigenvectors of the Laplacian)
• SQ optimization
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 26 / 35

55. Soft modularity
Methodology
Comparison of:
• Kernel SOM with various kernels: heat kernel, generalized inverse of
the Laplacian, modularity kernel (i.e., the positive part of W − P which
mimics the optimization of the modularity) and spectral SOM (based
on the first M eigenvectors of the Laplacian)
• SQ optimization
Parameters varied:
• size of the prior grid or number of clusters
• for organized clusterings, type of neighborhood on the grid
• for SOM, random or PCA initialization and kernel parameter for the
heat kernel
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 26 / 35

56. Soft modularity
Methodology
Comparison of:
• Kernel SOM with various kernels: heat kernel, generalized inverse of
the Laplacian, modularity kernel (i.e., the positive part of W − P which
mimics the optimization of the modularity) and spectral SOM (based
on the first M eigenvectors of the Laplacian)
• SQ optimization
Parameters varied:
• size of the prior grid or number of clusters
• for organized clusterings, type of neighborhood on the grid
• for SOM, random or PCA initialization and kernel parameter for the
heat kernel
Selection of the solutions: Pareto points according to modularity and
number of edge crossing
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 26 / 35

57. Soft modularity
A brief comment on the kernel SOM solutions
qq qq qqq qq qq q qq qq q q
q
qq
qqqq
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0100200300400500
Generalized inverse
Modularity
Crossingedges
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
qq
q
q
qq
q
q
q
q
q
q
q
q
q
q q
q
qqq
qq
q
qq
qqq
q
q
q
q
q
Size 3
Size 4
Size 5
Pareto point
qqq qqqqq qqqqq qqqqq qqqqq q qqqq
q
qqqq
q
qqqq
q
qq
q
q
q
q
q
q
q
q
q q
qqq qq
q qq q
qqqq q q
qq
q q q
q q
qqqq qq qqq qq qq qqq qqq qq qq q qq q
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0100200300400500
Heat kernel
Modularity
Crossingedges
q
q
q
q
q
q
q
q
q
q
q
qq
q qq
q q
q q
q
qq
q
q
qqq
q q
q
q
q
q q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q q
q
qq
qqq
q
q
qq
q
q
qq
q
q
q
qq
q
q
q qqq
q
qq
q
q
qq qq qqq qq
q
qqqq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
q
q
q
q
q
q q
q
qq
q
q
q
q
qq
q
q
q
q
qq
q
q
q
qq
q
qqq
q
qq
qq
q
qq
qq
q
q
q
Size 3
Size 4
Size 5
Pareto point
qq
q
qq qq
qq
q
qq qq q qq qq q qqq
q q
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0100200300400500
Modularity kernel
Modularity
Crossingedges
qq qq
q
q
q q
q
q
q
q qq
q
q
q
q
q
q q
q
q
q
q
q
q
qq
q q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
qq
q
q
q
q
q
Size 3
Size 4
Size 5
qqqqq qq
q
qq
q
q qqq
q
qq qq
q
q qqq
0.0 0.1 0.2 0.3 0.4 0.5 0.6
0100200300400500
Spectral SOM
Modularity
Crossingedges
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q q
q
q
q
qq
q
q
q q
q
q
Size 3
Size 4
Size 5
Spectral SOM and Modularity kernel obtain poor results
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 27 / 35

58. Soft modularity
Analysis of the Pareto points for “Les Misérables”
Method Number Modularity Nb of pairs
of clusters of cut edges
Organized mod. 42
(7) 0.5638 1
Organized mod. 52
(7) 0.5652 3
32
(6) 0.5472 0
kernel SOM (HK) 52
(22) 0.5327 27
32
(8) 0.5276 2
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 28 / 35

59. Soft modularity
Analysis of the Pareto points for “Les Misérables”
Method Number Modularity Nb of pairs
of clusters of cut edges
Organized mod. 42
(7) 0.5638 1
Organized mod. 52
(7) 0.5652 3
32
(6) 0.5472 0
kernel SOM (HK) 52
(22) 0.5327 27
32
(8) 0.5276 2
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
qq
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
q
qq
q
q
q
q qq
q
q
qq
q
q
q
q q
q
q
q
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 28 / 35

60. Soft modularity Hierarchical clustering and visualization
Outline
1 Self-organizing maps approaches
2 Modularity based approaches
3 Soft modularity
Hierarchical clustering and visualization
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 29 / 35

61. Soft modularity Hierarchical clustering and visualization
Global overview
[Rossi and Villa-Vialaneix, 2011] 2 combined steps:
• Find out a clustering hierarchy (by repeating modularity optimization
in clusters) + test of the significativity of the partition at each step;
• Display the different levels of the hierarchy by a modified force
directed algorithm.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 30 / 35

62. Soft modularity Hierarchical clustering and visualization
A hierarchy of clustering
Aim: Limiting the resolution default of modularity (see [Fortunato, 2010].
How to do so? Iterate the modularity optimization in each cluster. The
modularity is optimized by a greedy algorithm with multi-levels refinement
similar to that of [Noack and Rotta, 2009].
Step 1
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 31 / 35

63. Soft modularity Hierarchical clustering and visualization
A hierarchy of clustering
Aim: Limiting the resolution default of modularity (see [Fortunato, 2010].
How to do so? Iterate the modularity optimization in each cluster. The
modularity is optimized by a greedy algorithm with multi-levels refinement
similar to that of [Noack and Rotta, 2009].
Step 2
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 31 / 35

64. Soft modularity Hierarchical clustering and visualization
A hierarchy of clustering
Aim: Limiting the resolution default of modularity (see [Fortunato, 2010].
How to do so? Iterate the modularity optimization in each cluster. The
modularity is optimized by a greedy algorithm with multi-levels refinement
similar to that of [Noack and Rotta, 2009].
Step 3
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 31 / 35

65. Soft modularity Hierarchical clustering and visualization
Stopping criterion
A clustering algorithm always provides a solution, relevant or not!
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 32 / 35

66. Soft modularity Hierarchical clustering and visualization
Stopping criterion
A clustering algorithm always provides a solution, relevant or not!
Significativity of a node clustering:
1 Generate random graphs with the same degree distribution than the
original graph;
Simulation process: MCMC algorithm of [Roberts Jr., 2000] which
permutes edges on random couples of pairs of connected nodes;
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 32 / 35

67. Soft modularity Hierarchical clustering and visualization
Stopping criterion
A clustering algorithm always provides a solution, relevant or not!
Significativity of a node clustering:
1 Generate random graphs with the same degree distribution than the
original graph;
Simulation process: MCMC algorithm of [Roberts Jr., 2000] which
permutes edges on random couples of pairs of connected nodes;
After Q|E| permutations, the obtained graph is random for the uniform
distribution on the set of graphs with the same degre distribution.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 32 / 35

68. Soft modularity Hierarchical clustering and visualization
Stopping criterion
A clustering algorithm always provides a solution, relevant or not!
Significativity of a node clustering:
1 Generate random graphs with the same degree distribution than the
original graph;
2 Optimize the modularity on these random graphs;
3 Find out the p-value of the observed modularity compared to the
empirical distribution on random graphs;
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 32 / 35

69. Soft modularity Hierarchical clustering and visualization
Stopping criterion
A clustering algorithm always provides a solution, relevant or not!
Significativity of a node clustering:
1 Generate random graphs with the same degree distribution than the
original graph;
2 Optimize the modularity on these random graphs;
3 Find out the p-value of the observed modularity compared to the
empirical distribution on random graphs;
4 If the new clustering is not found to be significant, the algorithm is
stopped.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 32 / 35

70. Soft modularity Hierarchical clustering and visualization
Display a clustering hierarchy
Basics
• start from the first step of the clustering;
• expand the clusters by order of minimal decrease in modularity.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 33 / 35

71. Soft modularity Hierarchical clustering and visualization
Display a clustering hierarchy
Basics
• start from the first step of the clustering;
• expand the clusters by order of minimal decrease in modularity.
Issues
1 taking into account the size of the clusters: [Tunkelang, 1999] for a
modification of the FDR algorithm to nodes that have different sizes;
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 33 / 35

72. Soft modularity Hierarchical clustering and visualization
Display a clustering hierarchy
Basics
• start from the first step of the clustering;
• expand the clusters by order of minimal decrease in modularity.
Issues
1 taking into account the size of the clusters: [Tunkelang, 1999] for a
modification of the FDR algorithm to nodes that have different sizes;
2 estimating in advance the place needed to represent a cluster when it
is expanded.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 33 / 35

73. Soft modularity Hierarchical clustering and visualization
Another medieval network...
From the same corpus of medieval documents:
• nodes: transactions and active individuals. 3 918 individuals and
6 455 transactions (total: 10 373 sommets);
• edges model the active involvement of an individual in a transaction.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

74. Soft modularity Hierarchical clustering and visualization
Another medieval network...
From the same corpus of medieval documents:
• nodes: transactions and active individuals. 3 918 individuals and
6 455 transactions (total: 10 373 sommets);
• edges model the active involvement of an individual in a transaction.
Modularity optimization: 48 clusters having from 10 to 740 nodes.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

75. Soft modularity Hierarchical clustering and visualization
Another medieval network...
From the same corpus of medieval documents:
• nodes: transactions and active individuals. 3 918 individuals and
6 455 transactions (total: 10 373 sommets);
• edges model the active involvement of an individual in a transaction.
Modularity optimization: 48 clusters having from 10 to 740 nodes.
Hierarchy : 4 levels, 89 classes on the latest levels.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

76. Soft modularity Hierarchical clustering and visualization
Another medieval network...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

77. Soft modularity Hierarchical clustering and visualization
Another medieval network...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

78. Soft modularity Hierarchical clustering and visualization
Another medieval network...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

79. Soft modularity Hierarchical clustering and visualization
Another medieval network...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

80. Soft modularity Hierarchical clustering and visualization
Another medieval network...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

81. Soft modularity Hierarchical clustering and visualization
Another medieval network...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

82. Soft modularity Hierarchical clustering and visualization
Another medieval network...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

83. Soft modularity Hierarchical clustering and visualization
Another medieval network...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 34 / 35

84. Soft modularity Hierarchical clustering and visualization
Conclusion
Mining a graph from a clustering
• clustering can be used to provide a simplified representation of the
network and to help the user understand its macroscopic structure;
• optimizing the modularity seems to provide better results than
approaches based on the Laplacian (it helps for separating hubs and
thus results in more balanced clusters);
• approaches presented here are almost fully automated: solutions to
tune the parameters are provided in the corresponding articles.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 35 / 35

85. Soft modularity Hierarchical clustering and visualization
Conclusion
Mining a graph from a clustering
• clustering can be used to provide a simplified representation of the
network and to help the user understand its macroscopic structure;
• optimizing the modularity seems to provide better results than
approaches based on the Laplacian (it helps for separating hubs and
thus results in more balanced clusters);
• approaches presented here are almost fully automated: solutions to
tune the parameters are provided in the corresponding articles.
Perspectives: improve the representation of the hierarchy, incorporate
additional information on nodes and edges in the clustering...
Merci pour votre attention...
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 35 / 35

86. Soft modularity Hierarchical clustering and visualization
References
Auber, D., Chiricota, Y., Jourdan, F., and Melançon, G. (2003).
Multiscale visualization of small world networks.
In INFOVIS’03.
Auber, D. and Jourdan, F. (2005).
Interactive refinement of multi-scale network clusterings.
In International Conference on Information Visualisation, pages 703–709, Los Alamitos, CA, USA. IEEE Computer Society.
Boulet, R., Jouve, B., Rossi, F., and Villa, N. (2008).
Batch kernel SOM and related laplacian methods for social network analysis.
Neurocomputing, 71(7-9):1257–1273.
Fortunato, S. (2010).
Community detection in graphs.
Physics Reports, 486:75–174.
Fouss, F., Pirotte, A., Renders, J., and Saerens, M. (2007).
Random-walk computation of similarities between nodes of a graph, with application to collaborative recommendation.
IEEE Trans Knowl Data En, 19(3):355–369.
Fruchterman, T. and Reingold, B. (1991).
Graph drawing by force-directed placement.
Software Pract Exper, 21:1129–1164.
Herman, I., Melançon, G., and Scott Marshall, M. (2000).
Graph visualization and navigation in information visualisation.
6(1):24–43.
Knuth, D. (1993).
The Stanford GraphBase: A Platform for Combinatorial Computing.
Addison-Wesley, Reading, MA.
Kohohen, T. and Somervuo, P. (1998).
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 35 / 35

87. Soft modularity Hierarchical clustering and visualization
Self-Organizing maps of symbol strings.
Neurocomputing, 21:19–30.
Kohonen, T. and Somervuo, P. (2002).
How to make large self-organizing maps for nonvectorial data.
Neural Networks, 15(8):945–952.
Kondor, R. and Lafferty, J. (2002).
Diffusion kernels on graphs and other discrete structures.
In Proceedings of the 19th International Conference on Machine Learning, pages 315–322.
Newman, M. and Girvan, M. (2004).
Finding and evaluating community structure in networks.
Phys Rev E, 69:026113.
Noack, A. (2007).
Energy models for graph clustering.
J Graph Algorithms Appl, 11(2):453–480.
Noack, A. and Rotta, R. (2009).
Multi-level algorithms for modularity clustering.
In SEA ’09: Proceedings of the 8th International Symposium on Experimental Algorithms, pages 257–268, Berlin, Heidelberg.
Springer-Verlag.
Olteanu, M., Villa-Vialaneix, N., and Cottrell, M. (2013).
On-line relational som for dissimilarity data.
In Estevez, P., Principe, J., Zegers, P., and Barreto, G., editors, Advances in Self-Organizing Maps (Proceedings of WSOM
2012), volume 198 of AISC, pages 13–22, Springer Verlag, Berlin, Heidelberg.
To appear.
Roberts Jr., J. M. (2000).
Simple methods for simulating sociomatrices with given marginal totals.
Social Networks, 22(3):273 – 283.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 35 / 35

88. Soft modularity Hierarchical clustering and visualization
Rossi, F., Hasenfuss, A., and Hammer, B. (2007).
Accelerating relational clustering algorithms with sparse prototype representation.
In 6th International Workshop on Self-Organizing Maps (WSOM), Bielefield, Germany. Neuroinformatics Group, Bielefield
University.
Rossi, F. and Villa-Vialaneix, N. (2010).
Optimizing an organized modularity measure for topographic graph clustering: a deterministic annealing approach.
Neurocomputing, 73(7-9):1142–1163.
Rossi, F. and Villa-Vialaneix, N. (2011).
Représentation d’un grand réseau à partir d’une classification hiérarchique de ses sommets.
Journal de la Société Française de Statistique, 152(3):34–65.
Seifi, M., Guillaume, J., Latapy, M., and Le Grand, B. (2010).
Visualisation interactive multi-échelle des grands graphes : application à un réseau de blogs.
In Atelier EGC 2010, Visualisation et Extraction de Connaissances, Hammamet, Tunisie.
Tunkelang, D. (1999).
A Numerical Optimization Approach to General Graph Drawing.
PhD thesis, School of Computer Science, Carnegie Mellon University.
CMU-CS-98-189.
Villa, N. and Rossi, F. (2007).
A comparison between dissimilarity SOM and kernel SOM for clustering the vertices of a graph.
In 6th International Workshop on Self-Organizing Maps (WSOM), Bielefield, Germany. Neuroinformatics Group, Bielefield
University.
Graph mining (Séminaire LIPN) Nathalie Villa-Vialaneix Paris, 11/15 2012 35 / 35