Multiple kernel Self-Organizing Maps

Multiple kernel Self-Organizing Maps
Madalina Olteanu(2), Nathalie Villa-Vialaneix(1,2), Christine
Cierco-Ayrolles(1)
http://www.nathalievilla.org
nathalie.villa@univ-paris1.fr, madalina.olteanu@univ-paris1.fr,
christine.cierco@toulouse.inra.fr
Groupe de travail SAMM-Graph (25/01/2013)
(1) (2)
MK-SOM (SAMM-Graph) Nathalie Villa-Vialaneix Toulouse, 25 Janvier 2013 1 / 25

Introduction
Outline
1 Introduction
2 MK-SOM
3 Applications

Introduction
Data, notations and objectives
Data: D datasets (xd
i
)i=1,...,n,d=1,...,D all measured on the same individuals
or on the same objects, {i, . . . , n}, each taking values in an arbitrary space
Gd.

Introduction
i
Gd.
Examples:
• (xd
i
)i is p numeric variables;
• (xd
i
)i is n nodes of a graph;
• (xd
i
)i is p factors;
• (xd
i
)i is a text...

Introduction
i
Gd.
Examples:
• (xd
i
)i is p numeric variables;
• (xd
i
)i is n nodes of a graph;
• (xd
i
)i is p factors;
• (xd
i
)i is a text...
Purpose: Combine all datasets to obtain a map of individuals/objects:
self-organizing maps for clustering {i, . . . , n} using all datasets

Introduction
Particular case [Villa-Vialaneix et al., 2013]
Data: A weighted undirected network represented by a graph G with n
nodes x1, . . . , xn with weight matrix W: Wij = Wji and Wii = 0.

Introduction
For every node, additional information is provided: either numerical
variables, factors, textual information... or a combination of all of them.

Introduction
variables, factors, textual information... or a combination of all of them.
Examples: Gender in a social network, Functional group of a gene in a
gene interaction network...

Introduction
information, factors, textual information... or a combination of all of them.
Examples: Weight of people in a social network, Number of visits of a
web page in WWW...

Introduction
Purpose
Node clustering: ﬁnd communities, i.e., groups of “close” nodes in the
graph; close meaning:
• densely connected and sharing (comparatively) a few links with the
other groups (“communities”);

Introduction
Purpose
• but also having similar labels.

Introduction
Purpose
Here: self-organizing map approach to produce a map of the nodes
(clutering+visualization).

Introduction
Purpose
Here: self-organizing map approach to produce a map of the nodes
(clutering+visualization). Multiple sources of information are handled by
combining kernels.

MK-SOM
Outline
1 Introduction
2 MK-SOM
3 Applications

MK-SOM
Basics on SOM
Project the data on a squared grid (each square of the grid is a cluster)

MK-SOM
Basics on SOM
Project the data on a squared grid (each square of the grid is a cluster)
such that:
• the nodes in a same cluster are highly connected
• the nodes in two close clusters are also (less) connected
• the nodes in two distant clusters are (almost) not connected

MK-SOM
Basics on SOM
• the map is made of neurons (visually symbolized by, e.g.,
rectangles), 1...M, each associated to a prototype pi (a prototype is a
“representer” of the neuron in the original dataset);

MK-SOM
Basics on SOM
• the map is equipped with a neighborhood relationship, i.e., a
“distance” (actually a dissimilarity) between neurons, δ;

MK-SOM
Basics on SOM
• the map is equipped with a neighborhood relationship, i.e., a
“distance” (actually a dissimilarity) between neurons, δ;
• goal: ﬁnd the best mapping f(xi) ∈ {1, . . . , M} of the observations xi
on the map minimizing the energy
E =
n
i=1
M
j=1
h(δ(f(xi), j)) xi − pi
2
.
i.e., each data is assigned to a neuron so that xi is:
• “close” to the prototype of f(xi);
• “close” (to a lesser extent) to the prototypes of the neighboring neurons
of f(xi);
• “distant” to the prototypes of the neurons that are distant of f(xi).
(topology preservation)

MK-SOM
on-line SOM
Data: x1, . . . , xn ∈ Rd
1: Initialization: randomly set p0
1
,...,p0
M
in Rd
2: for t = 1 → T do
3: Randomly choose i ∈ {1, . . . , n}
4: Assignment
ft
(xi) ← arg min
j=1,...,M
xi − pt−1
j Rd
5: for all j = 1 → M do Representation
6: pt
j
← pt−1
j
+ µt ht
(δ(ft
(xi), j)) 1i − pt−1
j
7: end for
8: end for
where ht
is a decreasing function which reduces the neighborhood when t
increases and µt is generally of order 1/t.

MK-SOM
on-line SOM
Data: x1, . . . , xn ∈ Rd
1: Initialization: randomly set p0
1
,...,p0
M
in Rd
2: for t = 1 → T do
4: Assignment
ft
(xi) ← arg min
j=1,...,M
xi − pt−1
j Rd
6: pt
j
← pt−1
j
+ µt ht
(δ(ft
(xi), j)) 1i − pt−1
j
7: end for
8: end for
where ht
is a decreasing function which reduces the neighborhood when t
increases and µt is generally of order 1/t.
Problem with non numerical data: deﬁnitions of . and pj???

MK-SOM
Kernels/Multiple kernel
What is a kernel?
(xi) ∈ G, (K(xi, xj))ij st: K(xi, xj) = K(xj, xi) and
∀ (αi)i, ij αiαjK(xi, xj) ≥ 0. In this case [Aronszajn, 1950],
∃ (H, ., . ) , Φ : G → H st : K(xi, xj) = Φ(xi), Φ(xj)

MK-SOM
Kernels/Multiple kernel
What is a kernel?
(xi) ∈ G, (K(xi, xj))ij st: K(xi, xj) = K(xj, xi) and
∀ (αi)i, ij αiαjK(xi, xj) ≥ 0. In this case [Aronszajn, 1950],
∃ (H, ., . ) , Φ : G → H st : K(xi, xj) = Φ(xi), Φ(xj)
Examples:
• nodes of a graph: Heat kernel K = e−βL
or K = L+ where L is the
Laplacian [Kondor and Lafferty, 2002, Smola and Kondor, 2003,
Fouss et al., 2007]
• numerical variables: Gaussian kernel K(xi, xj) = e−β xi−xj
2
;
• text: [Watkins, 2000]...

MK-SOM
Kernel SOM
[Mac Donald and Fyfe, 2000, Andras, 2002, Villa and Rossi, 2007]
(xi)i ⊂ G described by a kernel (K(xi, xi ))ii . Prototypes are deﬁned in the
feature space: pj = i γjiφ(xi); energy calculated calculated in H:
1: Prototypes initialization: randomly set p0
j
= n
i=1 γ0
ij
Φ(xi) st γ0
ij
≥ 0
and i γ0
ij
= 1
2: for t = 1 → T do
4: Assignment
ft
(xi) ← arg min
j=1,...,M
xi − pt−1
j H
6: γt
j
← γt−1
j
+ µt ht
(δ(ft
(xi), j)) 1i − γt−1
j
7: end for
8: end for
Usually, µt ∼ 1/t.

MK-SOM
Kernel SOM
[Mac Donald and Fyfe, 2000, Andras, 2002, Villa and Rossi, 2007]
(xi)i ⊂ G described by a kernel (K(xi, xi ))ii . Prototypes are deﬁned in the
feature space: pj = i γjiφ(xi); energy calculated calculated in H:
j
= n
i=1 γ0
ij
Φ(xi) st γ0
ij
≥ 0
and i γ0
ij
= 1
2: for t = 1 → T do
4: Assignment
ft
(xi) ← arg min
j=1,...,M
ll
γt−1
jl γt−1
jl Kt
(xl, xl ) − 2
l
γt−1
jl Kt
(xl, xi)
6: γt
j
← γt−1
j
+ µt ht
(δ(ft
(xi), j)) 1i − γt−1
j
7: end for
8: end for
Usually, µt ∼ 1/t.

MK-SOM
Combining kernels
Suppose: each (xd
i
)i is described by a kernel Kd, kernels can be
combined (e.g., [Rakotomamonjy et al., 2008] for SVM):
K =
D
d=1
αdKd,
with αd ≥ 0 and d αd = 1.

MK-SOM
Combining kernels
Suppose: each (xd
i
)i is described by a kernel Kd, kernels can be
combined (e.g., [Rakotomamonjy et al., 2008] for SVM):
K =
D
d=1
αdKd,
with αd ≥ 0 and d αd = 1.
Remark: Also useful to integrate different types of information coming
from different kernels on the same dataset.

MK-SOM
multiple kernel SOM
j
= n
i=1 γ0
ji
Φ(xi) st γ0
ji
≥ 0 and
i γ0
ji
= 1
2: Kernel initialization: set (α0
d
) st α0
d
≥ 0 and d αd = 1 (e.g., α0
d
= 1/D);
K0
← d α0
d
Kd
3: for t = 1 → T do
5: Assignment
ft
(xi) ← arg min
j=1,...,M
xi − pt−1
j
2
H(Kt )
7: γt
j
← γt−1
j
+ µt ht
(δ(ft
(xi), j)) 1i − γt−1
j
8: end for
9: empty state ;-)
10: end for

MK-SOM
multiple kernel SOM
j
= n
i=1 γ0
ji
Φ(xi) st γ0
ji
≥ 0 and
i γ0
ji
= 1
d
) st α0
d
≥ 0 and d αd = 1 (e.g., α0
d
= 1/D);
K0
← d α0
d
Kd
3: for t = 1 → T do
5: Assignment
ft
(xi) ← arg min
j=1,...,M
ll
γt−1
jl γt−1
jl Kt
(xl, xl ) − 2
l
γt−1
jl Kt
(xl, xi)
7: γt
j
← γt−1
j
+ µt ht
(δ(ft
(xi), j)) 1i − γt−1
j
8: end for
9: empty state ;-)
10: end for

MK-SOM
Tuning (αd)d on-line
Purpose: minimize over (γji)ji and (αd)d the energy
E((γji)ji, (αd)d) =
n
i=1
M
j=1
h (f(xi), j) φα
(xi) − pα
j (γj)
2
α
,

MK-SOM
n
i=1
M
j=1
h (f(xi), j) φα
(xi) − pα
j (γj)
2
α
,
KSOM picks up an observation xi whose contribution to the energy is:
E|xi
=
M
j=1
h (f(xi), j) φα
(xi) − pα
j (γj)
2
α
(1)

MK-SOM
n
i=1
M
j=1
h (f(xi), j) φα
(xi) − pα
j (γj)
2
α
,
KSOM picks up an observation xi whose contribution to the energy is:
E|xi
=
M
j=1
h (f(xi), j) φα
(xi) − pα
j (γj)
2
α
(1)
Idea: Add a gradient descent step based on the derivative of (1):
∂E|xi
∂αd
=
M
j=1
h(f(xi), j)

Kd(xd
i , xd
i ) − 2
n
l=1
γjlKd(xd
i , xd
l )+
n
l,l =1
γjlγjl Kd(xd
l , xd
l )

 =
M
j=1
h(f(xi), j) φ(xi) − pd
j (γj) 2
Kd

MK-SOM
adaptive multiple kernel SOM
j
= n
i=1 γ0
ji
Φ(xi) st γ0
ji
≥ 0 and
i γ0
ji
= 1
d
) st α0
d
≥ 0 and d αd = 1 (e.g., α0
d
= 1/D);
K0
← d α0
d
Kd
3: for t = 1 → T do
5: Assignment
ft
(xi) ← arg min
j=1,...,M
xi − pt−1
j
2
H(Kt )
7: γt
j
← γt−1
j
+ µt ht
(δ(ft
(xi), j)) 1i − γt−1
j
8: end for
9:
10: end for

MK-SOM
adaptive multiple kernel SOM
j
= n
i=1 γ0
ji
Φ(xi) st γ0
ji
≥ 0 and
i γ0
ji
= 1
d
) st α0
d
≥ 0 and d αd = 1 (e.g., α0
d
= 1/D);
K0
← d α0
d
Kd
3: for t = 1 → T do
5: Assignment
ft
(xi) ← arg min
j=1,...,M
xi − pt−1
j
2
H(Kt )
7: γt
j
← γt−1
j
+ µt ht
(δ(ft
(xi), j)) 1i − γt−1
j
8: end for
9: Kernel update αt
d
← αt−1
d
+ νt
∂E|xi
∂αd
and Kt
← d αt
d
Kd
10: end for

Applications
Outline
1 Introduction
2 MK-SOM
3 Applications

Applications
Example1: simulated data
Graph with 200 nodes classiﬁed in 8 groups:
• graph: Erdös Reyni models: groups 1 to 4 and groups 5 to 8 with
intra-group edge probability 0.3 and inter-group edge probability 0.01;
• numerical data: nodes labelled with 2-dimensional Gaussian
vectors: odd groups N
0
0
,
0.3 0
0 0.3
;
• factor with two levels: groups 1, 2, 5 and 7: ﬁrst level; other groups:
second level.
Only the knownledge
on the three datasets
can discriminate all 8
groups.

Applications
Experiment
Kernels: graph: L+, numerical data: Gaussian kernel; factor: another
Gaussian kernel on the disjunctive recoding

Applications
Experiment
Kernels: graph: L+, numerical data: Gaussian kernel; factor: another
Gaussian kernel on the disjunctive recoding
Comparison: on 100 randomly generated datasets as previously
described:
• multiple kernel SOM with all three data;
• kernel SOM with a single dataset;
• kernel SOM with two datasets or all three datasets in a single
(Gaussian) kernel.

Applications
An example
MK-SOM All in one kernel
Graph only Numerical variables only

Applications
Numerical comparison
(over 100 simulations) with mutual information
ij
|Ci ∩ ˜Cj|
200
log
|Ci ∩ ˜Cj|
|Ci| × | ˜Cj|
(adjusted version, equal to 1 if partitions are identical
[Danon et al., 2005]).

Applications
Example 2: Data coming from a medieval corpus
Example from [Boulet et al., 2008]
http://graphcomp.univ-tlse2.fr/ In the “Archive départementales
du Lot” (Cahors, France), big corpus of 5000 transactions (mostly land
charters)
• coming from 4 “seigneuries” (about 25 little villages) in South West of
France;
• being established between 1240 and 1520 (just before and after the
hundred years’ war).

Applications
Graph description
Graph:
• nodes: 1446 individuals directly involved in the transactions (1 446
individuals);
• edges: the two individuals are involved in the same transaction (3 192
edges).

Applications
Graph description
Graph:
individuals);
edges).
Labels on nodes:
• numerical: average date of activity;
• text: family name of the individual.

Applications
Graph description
Graph:
individuals);
edges).
Labels on nodes:
• numerical: average date of activity;
• text: family name of the individual.
Kernels:
• L+ for the graph;
• linear kernel for the dates;
• spectral string kernel for the family names
[Karatzoglou and Feinerer, 2010] (uses common strings having a
length larger than 4).

Applications
Date and graph maps

Applications
Name map

Applications
Conclusion
Summary
• ﬁnding communities in graph, while taking into account labels;
• uses multiple kernel and automatically tunes the combination;
• the method gives relevant communities according to all sources of
information and a well-organized map.

Applications
Conclusion
Summary
Possible developments
• currently studying a similar approach for dissimilarity/relational SOM;
• Main issue: computational time; currently studying a sparse version.

Applications
Conclusion
Summary
Possible developments
• currently studying a similar approach for dissimilarity/relational SOM;
• Main issue: computational time; currently studying a sparse version.
Thank you for your attention...

Applications
References
Andras, P. (2002).
Kernel-Kohonen networks.
International Journal of Neural Systems, 12:117–135.
Aronszajn, N. (1950).
Theory of reproducing kernels.
Transactions of the American Mathematical Society, 68(3):337–404.
Boulet, R., Jouve, B., Rossi, F., and Villa, N. (2008).
Batch kernel SOM and related laplacian methods for social network analysis.
Neurocomputing, 71(7-9):1257–1273.
Danon, L., Diaz-Guilera, A., Duch, J., and Arenas, A. (2005).
Comparing community structure identiﬁcation.
Journal of Statistical Mechanics, page P09008.
Fouss, F., Pirotte, A., Renders, J., and Saerens, M. (2007).
Random-walk computation of similarities between nodes of a graph, with application to collaborative recommendation.
IEEE Trans Knowl Data En, 19(3):355–369.
Karatzoglou, A. and Feinerer, I. (2010).
Kernel-based machine learning for fast text mining in R.
Comput Statist Data Anal, 54:290–297.
Kondor, R. and Lafferty, J. (2002).
Diffusion kernels on graphs and other discrete structures.
In Proceedings of the 19th International Conference on Machine Learning, pages 315–322.
Mac Donald, D. and Fyfe, C. (2000).
The kernel self organising map.
In Proceedings of 4th International Conference on knowledge-based intelligence engineering systems and applied technologies,
pages 317–320.

Applications
Rakotomamonjy, A., Bach, F., Canu, S., and Grandvalet, Y. (2008).
SimpleMKL.
J Mach Learn Res, 9:2491–2521.
Smola, A. and Kondor, R. (2003).
Kernels and regularization on graphs.
In Warmuth, M. and Schölkopf, B., editors, Proceedings of the Conference on Learning Theory (COLT) and Kernel Workshop,
Lecture Notes in Computer Science, pages 144–158.
Villa, N. and Rossi, F. (2007).
A comparison between dissimilarity SOM and kernel SOM for clustering the vertices of a graph.
In 6th International Workshop on Self-Organizing Maps (WSOM), Bielefield, Germany. Neuroinformatics Group, Bielefield
University.
Villa-Vialaneix, N., Olteanu, M., and Cierco-Ayrolles, C. (2013).
Carte auto-organisatrice pour graphes étiquetés.
In Actes des Ateliers FGG (Fouille de Grands Graphes), colloque EGC (Extraction et Gestion de Connaissances), Toulouse,
France.
Watkins, C. (2000).
Dynamic alignment kernels.
In Smola, A., Bartlett, P., Schölkopf, B., and Schuurmans, D., editors, Advances in Large Margin Classifiers, pages 39–50,
Cambridge, MA, USA. MIT P.

Multiple kernel Self-Organizing Maps

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Multiple kernel Self-Organizing Maps

Similar to Multiple kernel Self-Organizing Maps (17)

More from tuxette

More from tuxette (20)

Recently uploaded

Recently uploaded (20)

Multiple kernel Self-Organizing Maps