This presentation introduces a new method for sequentially extracting local independent component analysis (ICA) structures from mixed signal data containing multiple ICA clusters. The method uses beta divergence to estimate the mean and covariance matrices and assigns beta weights to separate clusters sequentially. Simulation results on both artificial and natural image datasets demonstrate the method can separate multiple sub-Gaussian, super-Gaussian, and sub-super Gaussian signal mixtures and outperforms existing local ICA methods in terms of not requiring prior knowledge of cluster numbers or source types and having faster execution time. The method may help analyze gene expression microarray data in the future.

1. WELCOME TO THE PRESENTATION
1
on
By
Md. Shahjaman1 and Md.Nurul Haque Mollah2
1Dept. of Statistics, Begum Rokeya University, Rangpur, Bangladesh.
email: shahjaman_brur@yahoo.com
2 Dept. of Statistics, University of Rajshahi,
Rajshahi, Bangladesh.
Organized by
Institute of Statistical Research and Training (ISRT)
December 27-29, 2014, University of Dhaka, Dhaka, Bangladesh

2. OUTLINE OF THE PRESENTATION
2
1. Introduction to Independent Component Analysis (ICA)
2. Principles of ICA Algorithms
3. Local ICA by Minimum Beta Divergence method
4. Sequential Extraction of Local ICA (New Proposal)
5. Simulation Results
6. Conclusion

3. 1. INTRODUCTION TO ICA
Independent Component Analysis (ICA):
1 11 12 1 1
2 21 22 2 2
1
( ) ( )
n
n
m m mn n
x a a a s
x a a a s
x t As t
x a a s
     
     
        
     
     
          
b
ICA is a statistical method for recovering original sources from the mixture
data. The general statistical model for ICA is defined by
3
S (t) n× 1 original signals whose components are assumed to be mutually
independent and non-Gaussian with zero mean vector (Unknown).
X (t) m × 1 mixed signals (known / observed)
A n × n mixing matrix (unknown)
b Bias vector (unknown)
Problem: Estimate A and b to recover S=A-1(X(t)-b)= W(X(t)-b) from X, where, W=A-1

4. 2. PRINCIPLES OF ICA ALGORITHMS
(1) ICA by Maximizing Non-Gaussianity
Kurtosis:
Entropy:
Neg-entropy:
4 2 2 0 Gaussianity
abs[kurt( )] abs[ { } 3( { }) ]
ve Non Gaussianity
y E y E y

   
  





J(y)
J(y)
yHyHyJ gauss
MaximizingbyincreasesyGaussianitNon
MinimizingbyincreasesyGaussianit
)()()(
Gaussianity increases by Maximizing
( ) ( )log ( )
Non Gaussianity increases by Minimizing
H(y)
H y f y f y dy
H(y)

   


(2) ICA by Minimizing Mutual Information
Mutual Information is defined by
It measures the dependency among the variables.
0)()(),...,,(
1
321  
yHyHyyyI i
n
i
4

5. Classical ICA model,
Local ICA Model
.
.
.
.
.
.
xt = Ast + bxt Є D
( t = 1, 2,.., n )
S
S1
Sj
Sc
xi1=(A1si1+b1) Є D 1,
xij=(Ajsij+bj) Є Dj,
( i = 1, 2, …, nj )
xic=(Acsic+bc) Є Dc,
Xt Є D,
t = 1, 2,…, n;
n=(n1+…+nc)
3 LOCAL ICA BY THE MINIMUM BETA-DIVERGENCE METHOD

6. There are many algorithms to separate ICA structures when there is only one cluster in the
data set and they works well. But when more than one ICA clusters (Local ICA) presents in
the entire data space then those methods gives misleading results. However in this case a
few Method attempts to Separate ICA clusters such as (a) ICA mixture model by Lee et al.
,2002 and (b) Mixture ICA model by Mollah et al. ,2007.
Problems in (a):
1) The no. of cluster should be known in advance
2) Non-Robust
3) Long execution time for large data set
Problems in (b):
1) Types of sources should be known in advance as sub-Gaussian or supper
Gaussian
2) If some data points of a ICA cluster falls in the neighborhood of other ICA cluster
then for recovering vector, some times it creates problem and cannot separate
the sources properly.
Thus an attempt has been made to overcome these problems in our proposed
method by sequentially extraction of Local ICA using beta-weight function.
6
3.1 LOCAL ICA BY THE MINIMUM BETA-DIVERGENCE METHOD

7. The minimum beta-divergence estimators for the mean vector μ and covariance
matrix V obtained (Mollah et.al ,2007) iteratively as follows :
is known as beta-weight function.
It produces larger weights corresponding to recovered ICA cluster and smaller
weights for the unrecovered ICA clusters
1
1
1
1
1
1
1
1
( | , )
and (1)
( | , )
( | , )( )( )
(2)
(1 ) ( | , )
where, ( | , ) exp ( ) ( )
2
n
i t t ii
t n
i t ti
n T
i t t i t i ti
t n
i t ti
T
V
V
V
V
V
v V








 

 









 


 
    
 







  

 
x x
x
x x x
x
x x x (3)
3.2 LOCAL ICA BY THE MINIMUM BETA-DIVERGENCE METHOD
7

8. 4 SEQUENTIAL EXTRACTION OF LOCAL ICA
(NEW PROPOSAL)
We separate the c cluster sequentially by rule Then
Then we apply D(k) within FastICA algorithm to recover LocalICA structures
Algorithm
I. select an appropriate β by cross validation.
II. compute μ(k) and V(k) for μ and V using (1) and (2)
III. calculate weights for each of the data points using β-weight
function (3)
IV. separate the first ICA cluster from the dataset using D(k) in (4) and
remove it from the entire data set
V. separate the second ICA cluster from the dataset using D(k) in (4)
and remove it from the entire data set
VI. repeat (i) to (v) until all ICA cluster are recovered.
8
 ( ) , ( ) ( )
ˆ| ( ; , ) 0.2 ; 1, 2,.. , 1, 2,.. (4)k t t k kD x D x x t n k c     

9. 5 SIMULATION RESULTS
2 dimensional 3 sub-Gaussian mixtures
Recovered Components by Proposed Method
(a)
(b)
(c)
Original Components
Observed Components
(d)
(e)
(f)
(g)
(h)
(i)
Step 1 Step 2 Step 3

10. 5.1 SIMULATION RESULTS
2 dimensional 3 supper-Gaussian mixtures
Recovered Components by Proposed Method
(a)
(b)
(c)
Original Components
Observed Components
(d)
(e)
(f)
(g)
(h)
(i)
Step 1 Step 2 Step 3

11. 5.2 SIMULATION RESULTS
2 dimensional 1 sub,1 supper and 1 sub-supper Gaussian mixtures
Recovered Components by Proposed Method
(a)
(b)
(c)
Original Components
Observed Components
(d)
(e)
(f)
(g)
(h)
(i)
Step 1 Step 2 Step 3

12. 5.3 SIMULATION RESULTS
2 dimensional 1 sub,1 supper and 1 sub-supper Gaussian mixtures
With Outliers
Original Components
Observed Components
12

13. 5.3 SIMULATION RESULTS
2 dimensional 1 sub,1 supper and 1 sub-supper Gaussian mixtures
With Outliers
Recovered Components by Lee Recovered Components by Proposed Method
Step 1 Step 2 Step 3 Step 1 Step 2 Step 3 Step 4

14. 5.4 SIMULATION RESULTS
2 dimensional 1 sub-sub (a-b), 1 supper-supper (d-e), 1sub-supper
(g-h) Gaussian artificial signals
Original Signals Mixed Signals

15. 5.4 SIMULATION RESULTS
Recovered 2 dimensional 1 supper-supper (a-b), 1 sub-supper (d,f),
1sub-sub (g-h)-Gaussian artificial signals
Weights for each signals by Lee Recovered Signals by Lee
Step1Step2Step3

16. 5.4 SIMULATION RESULTS
Recovered 2 dimensional 1 supper-supper (a-b), 1 sub-supper (d,f),
1sub-sub (g-h)-Gaussian artificial signals
Weights for each signals by Our Method Recovered Signals by Proposed Method
Step1Step2Step3

17. 5.3 SIMULATION RESULTS
Simulation With Natural Images
Tree Image Mixed Image1
Moon Image
Gaussian Noise
Mixed Image2
Recovered by Our Method
Clustered Images
17
Step 1 Step 2

18. 5 CONCLUSION
In this presentation we proposed a method using beta-divergence, which is able to
extract Local ICA structures sequentially. A ICA cluster is separated from the
dataset using a weight function known as beta-weight function by considering the
other part of the dataset as outliers and do the same task again to the remaining
data points in the dataset for sequential extraction of Local ICA structures.
The main advantages of this method over the existing methods (Mollah et.
al,2006 and Lee et. Al. 2002) is that the no. of cluster and the type of sources
need not be known in advance and it also reduces the execution time.
Our simulation and natural image results shows that our method performs
significantly over the other existing methods.
Our next target is to apply our method in Micro-array gene-expression data set.
18

19. ?????