Comparing estimation algorithms for block clustering models, by Gilles Celeux, INRIA 2nd talk at BigMC seminar on 06/01/2011.

1. Comparing estimation algorithms for block
clustering models
Gilles Celeux
Projet SELECT INRIA Saclay-Île-de-France
January 6, 2011 - BIG’MC seminar

2. Block clustering setting
Block clustering of (binary) data
Let y = {(yij ); i ∈ I, j ∈ J} be a dimension n × d binary
matrix, where I is a set n objets and J a set of d variables
Permuting the lines and columns of y to discover a
clustering structure on I × J.
Getting a simple summary of the data matrix y.
Many applications : recommendation systems, genomic
data analysis, text mining, archeology, ...

3. Example
1 2 3 4 5 6 7 1 2 3 4 5 6 7 1 4 3 5 7 2 6
A A A
B C C
C H H
D B B
E F F I II III
F J J a
G D D b
H G G c
I I I
J E E
(1) (2) (3) (4)
(1) Binary data matrix
(2) A partition on I
(3) A couple of partitions on I and J
(4) Summary of the binary matrix

4. Model-based clustering framework
Assume that the data are arising from a finite mixture of
parametrised densities.
A cluster is made by observations arising from the same
density.
In a block clustering model, clusters are defined on blocks
∈ I × J.
In a block clustering model, data of a block are modelled
by the same unidimensional density.

5. Latent block mixture model
Density of the observed data is supposed to be
f (y|g, m, φ, α) = p(u|g, m, φ)f (y|g, m, u, α)
u∈U
where u is the indicator block vector.
It is assumed that uijb = zik wj , z (resp.w) being the row (resp.
column) cluster indicator vector.
Assuming that the n × d variables Yij are conditionnally
independent knowing z and w leads to the model
z wj
f (y|g, m, π, ρ, α) = πk ik ρ ϕ(yij |g, m, αk )
z,w∈Z×W i,k j, i,j,k ,

6. An exemple : Bernoulli latent block model
Mixing proportions
For fixed g, the mixing proportions for the row are π1 , . . . , πg .
For fixed m, the mixing proportions for the col. are ρ1 , . . . , ρm .
The Bernoulli density per block
ϕ(yij ; αk ) = (αk )yij (1 − αk )1−yij
where αk ∈ (0, 1).
The mixture density is
z wj
f (y|g, m, π, ρ, α) = πk ik ρ (αk )yij (1−αk )1−yij .
z,w∈Z×W i,k j, i,j,k ,
The parameters to be estimated are the πs, the ρs and the αs.

7. Maximum likelihood estimation
The loglikelihood of the model parameter is
L(θ) = f (y|g, m, π, ρ, α) (g and m fixed)
L(θ) = log p(y, w, z|g, m, θ) − log p(w, z|y; g, m, θ)
= IE[log p(y, w, z; θ)|y; π (c) , θ(c) ] − IE[log p(w, z|y; θ)|y; θ(c) ]
= Q(θ|θ(c) ) − H(θ|θ(c) )
˜
If θ ∈ arg maxθ Q(θ|θ(c) )
˜ ˜ ˜
L(θ)−L(θ(c) ) = Q(θ|θ(c) )−Q(θ|θ(c) )+H(θ(c) |θ(c) )−H(θ|θ(c) ) ≥ 0
EM algorithm
E step : computing the conditional expectation of the
complete loglikelihood Q(θ|θ(c) )
˜
M step : maximising Q(θ|θ(c) ) in θ, θ(c) → θ

8. Conditional expectation of the complete loglikelihood
For the latent block model, it is
(c) (c) (c)
Q(θ|θ(c) ) = sik log πk + tj log ρ + ei,j,k , log ϕ(xij ; αk )
i,k j, i,j,k ,
where
(c) (c)
sik = P(Zik = 1|θ(c) , y), tj = P(Wj = 1|θ(c) , y)
and
(c)
ei,j,k , = P(Zik Wj = 1|θ(c) , y).
(c)
→ Difficulty to compute ei,j,k , ... Approximations are needed.

9. Variational interpretation of EM
From the identity
L(θ) = log p(y, z, w|θ) − log p(z, w|y, θ), we get
p(y, z, w|θ)
L(θ) = IEqzw log + KL(qzw ||p(z, w|y; θ))
qzw (w, z)
= F(qzw , θ) + KL(qzw ||p(z, w|y; θ))
EM as an alterned optimisation algorithm of F(qzw , θ)
E step : Maximising F(qzw , θ(c) ) in qzw (.) with θ(c) fixed, leads to
p(z, w|y; θ(c) ) = arg min KL(qzw ||p(z, w|y; θ(c) ))
qzw
(c) (c)
M step : Maximising F(qzw , θ) in θ with qzw (.) fixed : it amounts
to find
arg max Q(θ|θ(c) ).
θ

10. Variational approximation of EM (VEM)
Restricting qwz to a function set for which the E step is easily
tractable. It is assumed that qzw (z, w, θ) = qz (z)qw (w)
(c) (c)
sik = Pqz (Zik = 1|θ(c) , x), tj = Pqw (Wj = 1|θ(c) , x),
(c) (c) (c)
ei,j,k , = sik wj
Govaert and Nadif (2008)
1. E step : Maximising the free energy F(qzw , θ(c) ) until
convergence
(c)
1.1 computing sik with fixed wjl and θ(c)
(c+1)
1.2 computing wjl with fixed sik and θ(c)
→ s(c+1) and w (c+1)
2. M step : Updating θ(c+1)

11. Some characteristics of VEM
The optimised free energy F(qzw , θ) is a lower bound of
the observed loglikelihood.
The parameter maximising the free energy could be
expected to be a good, if not consistent, approximation of
the maximum likelihood estimator.
Since VEM is minimising KL(qzw ||p(z, w|y; θ)) rather than
KL(p(z, w|y; θ)||qzw ), it is expected to be sensitive to
starting values.

12. The SEM-Gibbs algorithm
SEM
The SEM algorithm (Celeux, Diebolt 1985 ) : After the E step, a
S step is introduced to simulate the missing data according to
the distribution p(z, w|x; θ(c) ).
A difficulty for the latent block model is to simulate p(z, w|x; θ).
Gibbs sampling
The distribution p(z, w|x; θ(c) ) is simulated using a Gibbs
sampler. Repeat
Simulate z(t+1) according to p(z|x, w(t) ; θ(c) )
Simulate w(t+1) according to p(w|x, z(t+1) ; θ(c) )
→ The stationary distribution of the Markov chain is
p(z, w|x; θ(c) )

13. SEM-Gibbs for Bernoulli latent block model
1. E and S steps :
1.1 computation of p(z|y, w(t) ; θ(c) ), then simulation of z(t+1)
πk ψk (yi· , αk · )
p(zi = k |yi· , w(c) ) = , k = 1, . . . g
k πk ψk (yi· , αk · )
u −ui (c) (c)
ψk (yi· , αk · ) = αk i (1−αk )d , ui = wj yij , d = wj
j j
1.2 computation of p(w|y, z(t+1) ; θ(c) ), then simulation of w(t+1)
→ w (c+1) and z (c+1)
2. M step :
(c+1) (c+1)
(c+1) i zik (c+1) j wj
πk = ,ρ =
n d
and
(c+1) (c+1)
(c+1) ij zik wj yij
αk = (c+1) (c+1)
ij zik wj

14. SEM features
SEM is not increasing the loglikelihood at each iteration.
SEM is generating an irreductible Markov chain with a
unique stationary distribution.
The parameter estimates fluctuate around the ml estimate
→ A natural estimator of θ, z, w is the mean of
(θ(c) , z(c) , w(c) ; c = B, . . . , B + C) get after a burn-in period.
How many Gibbs iterations inside the E-S step ?
→ default version : one Gibbs sampler iteration.

15. Numerical experiments
Simulation design
n = 100 rows, d = 60 columns,
g = 3 components for I, m = 2 components for J,
equal proportions on I and J.
The parameters α have the form :
 
1− 1−
α= 1− 
1−
where is defining the overlap between the mixture
components.

16. Comparing VEM and SEM-Gibbs
Criteria of comparison
Estimate parameter values / actual parameter values for θ.
Distance between MAP partition / actual partition,
where the distance between two couples of partitions
u = (z, w) and u = (z , w ) is the relative frequency of
disagreements
1
δ(u, u ) = 1 − zik wjl zik wjl .
nd
i,j,k ,l

17. SEM Convergence
n=100, d=60 , π = (0.43, 0.36, 0.21), ρ = (0.53, 0.47),
α11 = 0.6, α21 = 0.4, α31 = 0.6, α12 = 0.6, α22 = 0.6, α32 = 0.4
0.5 0.7
rho1
0.4 rho2
0.6
0.3 pi1
pi2 0.5
0.2 pi3
0.4
0.1
0
0 500 1000 1500 2000 0 500 1000 1500 2000
0.65 0.7
0.6 0.6
0.55 a11 0.5 a12
a21 a22
0.5 a31 0.4 a32
0.45 0.3
0.4 0.2
0 500 1000 1500 2000 0 500 1000 1500 2000

18. SEM variance from a unique starting position
n=100, d=60 , π = (0.30, 0.34, 0.36), ρ = (0.53, 0.47),
δSEM = 0.18(0.01), δVEM = 0.18
0.55
0.5
0.45
0.4
0.35
0.3
0.25
1 2 3 4 5

19. Comparing VEM and SEM with starting position on θ0
The comparison is made on 100 different samples
δVEM = 0.28(0.17), δSEM = 0.34(0.17)
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
1 2 3 4 5 6 7 8 9 10 11 12
VEM
VEM
VEM
VEM
VEM
VEM
SEM
SEM
SEM
SEM
SEM
SEM

20. VEM and SEM with random starting positions
Comparisons made on a sample from 100 different positions
δVEM = 0.49(0.16), δSEM = 0.17(0.02)
!
kl
0.65
0.6
0.55
0.5
0.45
0.4
1 2 3 4 5 6 7 8 9 10 11 12
BEM
BEM
BEM
BEM
BEM
BEM
SEM
SEM
SEM
SEM
SEM
SEM

21. Same comparison : less noisy case
Comparisons made on a sample from 100 different positions
δVEM = 0.20(0.23), δSEM = 0.045(0.004)
!
kl
0.65
0.6
0.55
0.5
0.45
0.4
0.35
1 2 3 4 5 6 7 8 9 10 11 12
BEM
BEM
BEM
BEM
BEM
BEM
SEM
SEM
SEM
SEM
SEM
SEM

22. Discussion : VEM vs. SEM
Numerical comparisons lead to the conclusions
VEM leads rapidly to reasonable parameter estimates
when its initial position is near enough the ml estimation.
VEM is quite sensitive to starting values.
SEM-Gibbs is (essentially) unsensitive to starting values.
→ Coupling SEM and VEM should be beneficial to derive
sensible ml estimates for the latent block model.

23. Difficulties with Maximum likelihood
Those difficulties concern the computation of information
criteria for model selection.
The likelihood remains difficult to be computed.
What is the sample size in a latent block model ?
There are many combinations (g, m) to be considered to
choose a relevant number of blocks.
→ Bayesian inference could be thought of as attractive for the
latent block model.

24. Bayesian inference : choosing the priors
Choosing conjugate priors is essential for the latent block
model.
The choice is easy in the binary case : the priors for π, ρ
and α are D(1, . . . , 1) or D(1/2, . . . , 1/2). They are non
informative priors.
In the continuous case, the conjugate priors for α = (µ, σ 2 )
are weakly informative.
Priors for the number of clusters
This sensitive choice jeopardizes Bayesian inference for
mixtures (Aitken 2000).
It seems that choosing truncated Poisson P(1) priors over the
range 1, . . . , gmax and 1, . . . , mmax is often a reasonable
choice (Nobile 2005).

25. Bayesian inference : Reversible Jump sampler
A possible advantage of Bayesian inference could be to make
use of a RJMCMC sampler to choose relevant values for g and
m since the likelihood is unavailable.
But, in the latent block context, the standard RJMCMC is
(remains ?...) unattractive since there is a couple of
clusters to deal with.
Fortunately, the allocation sampler of Nobile and Fearnside
(2007) could be used instead.

26. The allocation sampler : collapsing
The point of allocation sampler is to use a (RJ)MCMC algorithm
on a collapsed model.
Collapsed joint posterior
Using conjugacy properties, we get by integrating the full
posterior with respect to π, ρ and α
g m
P(g, m, z, w|y) = P(g)P(m)CF (.) Mk
k =1 =1
where CF (.) is a closed form function made of Gamma
functions and
Mk = P(αk ) p(yij |αk )dαk .
i/zi =k j/wj =

27. The allocation sampler : MCMC moves
Moves with fixed numbers of clusters
Updating the label of row i in cluster k :
m +i −i
nk + 1 Mk Mk
˜
P(zi = k ) ∝ , k = k.
nk Mk Mk
=1
Other moves are possible (Nobile and Fearnside 2007).
Moves to split or combine clusters
Two reversible moves to split a cluster or combine two clusters
analogous to the RJMCMC moves of R & G’97 are defined.
But, thanks to collapsing, those moves are of fixed dimension.
Integrating out the parameters leads to reduce the sampling
variability.

28. The allocation sampler : label switching
Following Nobile, Fearnside (2007), Friel and Wyse (2010)
used a post-processing procedure with the cost function
T −1 n
(t) (T )
C(k1 , k2 ) = I zi = k1 , zi = k2 .
t=1 i=1
1 The z(t) MCMC sequence has been rearranged such that
for s < t, z(s) uses less or the same number of
components than z(t) .
2 An algorithm returns the permutation σ(.) of the labels in
g
z(T ) which minimises the total cost k T −1 C(k , σ(k )).
=1
3 z(T ) is relabelled using the permutation σ(.).

29. Remarks on the procedure to deal with label switching
Due to collapsing, the cost function does not involve
sampled model parameters.
The row and columns allocations are post-processed
separately.
Simple algebra lead to an efficient on-line post-processing
procedure.
When g and m are large, g! and m! are tremendous.

30. Summarizing MCMC output
Most visited model : for each (k , ), its posterior probability
is estimated by the relative frequency of visits after post
processing to undo label switching.
MAP cluster model : it is the visited (g, m, z, w) having
highest probability a posteriori from the MCMC samples.

31. Simulated data
A 200 × 200 binary table. The posterior model probability of the
generating model was respectively (from left to right and from
top to bottom) : .96, .95, .90 ; .93, .89, .84 ; .80, .30, .15.

32. Congressional voting data
The data set records the votes of 435 members (267
democrats, 168 republicans) of the 98th on 16 different key
issues.
Voting data collapsed LBM BEM2

33. An example on microarray experiments
The data consist of the expression level of 419 genes under 70
conditions.
Weakly informative hyperprior parameters have been chosen.
The sampler has been run 220,000 iterations with 20,000 for
burn-in.
Hereunder is a detail of the posterior distribution of block
clusters models.
columns
rows 3 4 5
24 .064 .071 .042
25 .102 .120 .070
26 .037 .046 .023
Most visited model : (25, 4)
MAP cluster model : (26, 4).

34. References
Govaert, G. and Nadif, M. (2007) Block clustering with
Bernoulli mixture models : Comparison of differents
approaches. Computationanl Statistics and Data Analysis,
52, 3233-3245.
Nobile, A. and Fearnside, A. T. (2007) Bayesian finite
mixtures with an unknown number of components : The
allocation sampler. Statistics and Computing, 17, 147-162.
Wyse, J. and Friel, N. (2010) Block clustering with
collapsed latent block models. In revision at Statistics and
Computing (http ://arxiv.org/abs/1011.2948).