Upcoming SlideShare
×

2012　mdsp pr10 ica

217
-1

Published on

Published in: Technology, Education
0 Likes
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here
• Be the first to comment

• Be the first to like this

Views
Total Views
217
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
4
0
Likes
0
Embeds 0
No embeds

No notes for slide

2012　mdsp pr10 ica

1. 1. Course Calendar Class DATE Contents 1 Sep. 26 Course information & Course overview 2 Oct. 4 Bayes Estimation 3 〃 11 Classical Bayes Estimation - Kalman Filter - 4 〃 18 Simulation-based Bayesian Methods 5 〃 25 Modern Bayesian Estimation ：Particle Filter 6 Nov. 1 HMM(Hidden Markov Model) Nov. 8 No Class 7 〃 15 Bayesian Decision 8 〃 29 Non parametric Approaches 9 Dec. 6 PCA(Principal Component Analysis) 10 〃 13 ICA(Independent Component Analysis) 11 〃 20 Applications of PCA and ICA 12 〃 27 Clustering, k-means et al. 13 Jan. 17 Other Topics 1 Kernel machine. 14 〃 22(Tue) Other Topics 2
2. 2. Lecture Plan Independent Component Analysis -1．Whitening by PCA 1. Introduction Blind Sourｃe Separation（BSS） 2. Problem Formulation and independence 3. Whitening + ICA Approach 4. Non-Gaussianity Measure References: [1] A. Hyvarinen et al. “Independent Component Analysis”Wiley-InterScience, 2001
3. 3. 3 - １. Whitening by PCA (Preparation for ICA approach) Whitened := Uncorrelatedness + Unity variance* PCA is a very useful tool to transform a random vector x to an uncorrelated or whitened z. Matrix V is not uniquely defined, so we have free parameter. (In 2-d case: rotation parameter) * Here we assume all random variables are zero mean PCA gives one solution of whitening issue. z Vx n-vector matrix (unity matrix)z z Vx n n   C = I n-vector covariance matrix x n n Fig. 1
4. 4. 4 PCA whitening method: - Define the covariance matrix - {Eigenvalue, Eigenvector} pairs of Cx - Representation of Cx (*) - Whitening matrix transformation * The matrix E is orthogonal matrix that satisfies EET=ETE=I  T x E xxC 1 2 1 2 T T T x n T n                              e e C e e e EΛE e   { , 1 }i i i n e   1 1 2 2 1 2 1 1 : ( , , )T T z x diag E         z Vx , V = EΛ E C zz I  (1) (2) (3)
5. 5. 5 １. Introduction Blind Source Separation (BSS) Ex. Source signals Mixed signals BSS Problem: Recover or separate source signals with no prior information on mixing matrix [ aij ]. Typical BSS problem in real world is known as “Cocktail party problem” - Independent Component Analysis (ICA) utilizes the independence of source signals to solve BSS issue.    1 3s t s t     3 1 ( 1,2,3)i ij j j x t a s t i    Fig.2
6. 6. mic1 mic2 Source 1 s1(t) Source 2 s2(t) y2(t) y1(t) ICA Solution of BSS ? Mixing Process Separation Process Independency degree Fig.3
7. 7. 7 2. Problem Formulation and Independence - Source signals (zero mean): sj (t), j=1~n - Recorded signals : xi (t), i=1~n - Linear Mixing Process (No delay model *) (* In real environment, the arrival time differences between microphones should be involved in the mixing model) Recover the sources sj (t), j=1~n from mixed signals xi (t) i=1~n - aij are unknown - We want to get both aij and si (t) (-aij , -si (t)) Under the following assumptions         1 1 1 ( 1 ) Vector-Marix form: = where ( constant matrix) , , , , , n i ij j j ij T T n n x t a s t i n a n n x x s s           x As A x s (4) (5)
8. 8. 8 Assumption 1: Source waveforms are statistically independent Assumption 2: The sources have non-Gaussian distribution Assumption 3: Matrix A is square and invertible (for simplicity) Estimated A is used to recover the original signals by inverse (de- mixing) operation. s=Bx where B=A-1 Possible Solutions by ICA -To ambiguities- - The Variance(amplitude) of recovered signals Because, if a pair of is the solution of the underlying BSS problem, then is also the other solution. Variance of source signals are assumed to be unity: - The order of recovered signals cannot be determined (Permutation)   ,ij ja s t 2 1jE s      1 ,ij jKa s t K       (6)
9. 9. Basics Independence and Uncorrelatedness Statistical independence of two random variables x and y Knowing the value of x does not provide information about the distribution of y . Example: Uncorrelatedness of two random variables x and y their covariance is zero, i.e. 9          , ,y y x y x yp y x p y p x y p x p y     1 0 - variables , , are uncorrelated then we have is diagonal matrix n T x E xy x x E     C xx Fig.4 (7)
10. 10. Statistical Independence     0x yE x m y m   Uncorrelatedness      x,y x yp x,y p x p y - Independence means uncorrelatedness - Gaussian density case Independence = Uncorrelatedness
11. 11. 11 Examples of Sources and Mixed Signals [Uniform densities (Sub-Gaussian)] - Two independent components s1, s2 that have the same uniform density - Mixing matrix        1 2 1 2 1 3 ,2 3 0 Variance of 1 , i i i s p s otherwise s p s s p s p s         1 2 1 1 2 2 5 10 where 10 2 5 10 10 2 x As A x s s s s                        a a 2a 1a Fig.5 Fig.6 (8)
12. 12. 12 [Super-Gaussian densities] - Two independents s1, s2 have super-Gaussian like Fig. ?? - - Mixing signals 2a 1a Super Gaussian Source signals joint distribution Mixed signals distribution Fig.8 Fig.7
13. 13. 13 3. Whitening(PCA) + ICA approach Observed signals This means y is also whitened signal. Conclusion: Whitening gives the independent components only up to an orthogonal transformation. whitening ICA x z s 1 2 T  z ED E x =Vx New Mixing matrix is Orthogonal matrix **( )   z Vx =VAs = As A VA Question*: Is this unique solution?     Ans* ( orthogonal matrix) T T T T y U E E   y = Uz C yy Uzz U UIU = I    ** T T T T E zz AE ss A AA I  
14. 14. 14 Why Gaussian variables are forbidden (Assumption 2) Suppose two independent signals s1 and s2 are Gaussian and we obtain Joint Gaussian distribution is invariant with respect to orthogonal transformation. This means that we cannot find (identify) orthogonal matrix A from the mixed data.   2 2 1 2 1 2 2 2 1 2 1 1 , exp exp 2 22 2 1 1 1 exp exp 2 2 2 2 T s s p s s s s                                 s s   1 1 1 2 ( =orthogonal matrix, ) 1 1 1 1 , exp exp 2 2 2 2 T T T T p z z                   z = As A A = A , s = A z z AA z z z The same density is observed = no information arises by the orthonormal transformation (9)
15. 15. 15 We need to answer the following two questions. 1) How can the non-Gaussianity of y is measured? 2) How can we compute the values of B that maximize the measure? Maximization of Non-Gaussianity For a given density p(x), we define a measure of non-Gaussianity NG(p(x)) (non-negative, =0 if p(x) is Gaussian ) NG [Intuitive interpretation of ICA as non-Gaussianity minimization] 0 Gaussian p(x) Non-Gaussian 1 1 is non-Gaussian : ( ) is unknown ( ( )) as a function of ( ) is maximuzed when s i T T n T T T i i x As s b s A x Bx x b y b x b As q s b q Ab p y b q y q                    NG (10) (11)
16. 16. 16 NG(py)0 y=q1s1 +q2s2 y=q1s1 y=q2s2 Reduce NG by mixing Maximization by alternating b: As y →qisi NG tends to be maximized 4. Measure of non-Gaussianity Kurtosis is a classical measure of non-Gaussianity The absolute value of the kurtosis can be used as s measure of non-Gaussian. Optimization problem is solved as an ICA solution.      2 4 2 ( ) : 3Kurt p y E y E y    0 Gaussian super-Gaussian  ( )Kurt p y sub-Gaussian      , : ( ) b Max J b J b Kurt p y (12) (13)
1. A particular slide catching your eye?

Clipping is a handy way to collect important slides you want to go back to later.