Explicit Signal to Noise Ratio in Reproducing Kernel Hilbert Spaces.pdf

857 views
790 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
857
On SlideShare
0
From Embeds
0
Number of Embeds
112
Actions
Shares
0
Downloads
21
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Explicit Signal to Noise Ratio in Reproducing Kernel Hilbert Spaces.pdf

  1. 1. Explicit Signal to Noise Ratio in Reproducing Kernel Hilbert Spaces Luis Gómez-Chova1 Allan A. Nielsen2 Gustavo Camps-Valls1 1 Image Processing Laboratory (IPL), Universitat de València, Spain. luis.gomez-chova@uv.es , http://www.valencia.edu/chovago2 DTU Space - National Space Institute. Technical University of Denmark. IGARSS 2011 – Vancouver, Canada * IPL Image Processing Laboratory
  2. 2. Intro SNR KMNF Results ConclusionsOutline 1 Introduction 2 Signal-to-noise ratio transformation 3 Kernel Minimum Noise Fraction 4 Experimental Results 5 Conclusions and Open questionsL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 1/23
  3. 3. Intro SNR KMNF Results ConclusionsMotivation Feature Extraction Feature selection/extraction is essential before classification or regression to discard redundant or noisy components to reduce the dimensionality of the data Create a subset of new features by combinations of the existing ones Linear Feature Extraction Linear methods offer Interpretability ∼ knowledge discovery PCA: projections maximizing the data set variance PLS: projections maximally aligned with the labels ICA: non-orthogonal projections with maximal independent axesL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 2/23
  4. 4. Intro SNR KMNF Results ConclusionsMotivation Feature Extraction Feature selection/extraction is essential before classification or regression to discard redundant or noisy components to reduce the dimensionality of the data Create a subset of new features by combinations of the existing ones Linear Feature Extraction Linear methods offer Interpretability ∼ knowledge discovery PCA: projections maximizing the data set variance PLS: projections maximally aligned with the labels ICA: non-orthogonal projections with maximal independent axes Drawbacks 1 Most feature extractors disregard the noise characteristics! 2 Linear methods fail when data distributions are curved (nonlinear relations)L. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 2/23
  5. 5. Intro SNR KMNF Results ConclusionsObjectives Objectives New nonlinear kernel feature extraction method for remote sensing data Extract features robust to data noise Method Based on the Minimum Noise Fraction (MNF) transformation Explicit Kernel MNF (KMNF) Noise is explicitly estimated in the reproducing kernel Hilbert space Deals with non-linear relations between the noise and signal features jointly Reduces the number of free parameters in the formulation to one Experiments PCA, MNF, KPCA, and two versions of KMNF (implicit and explicit) Test feature extractors for real hyperspectral image classificationL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 3/23
  6. 6. Intro SNR KMNF Results Conclusions 1 Introduction 2 Signal-to-noise ratio transformation 3 Kernel Minimum Noise Fraction 4 Experimental Results 5 Conclusions and Open questionsL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 4/23
  7. 7. Intro SNR KMNF Results ConclusionsSignal and noise Signal vs noise Signal: magnitude generated by an inaccesible system, si Noise: magnitude generated by the medium corrupting the signal, ni Observation: signal corrupted by noise, xi Notation Observations: xi ∈ RN , i = 1, . . . , n Matrix notation: X = [x1 , . . . , xn ] ∈ Rn×N Centered data sets: assume X has zero mean 1 Empirical covariance matrix: Cxx = n X X Projection matrix: U (size N × np ) → X = XU (np extracted features)L. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 5/23
  8. 8. Intro SNR KMNF Results ConclusionsPrincipal Component Analysis Transformation Principal Component Analysis (PCA) Find projections of X = [x1 , . . . , xN ] maximizing the variance of data XU PCA: maximize: Trace{(XU) (XU)} = Trace{U Cxx U} subject to: U U=I Including Lagrange multipliers λ, this is equivalent to the eigenproblem Cxx ui = λi ui → Cxx U = UD ui are the eigenvectors of Cxx and they are orthonormal, ui uj = 0L. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 6/23
  9. 9. Intro SNR KMNF Results ConclusionsPrincipal Component Analysis Transformation Principal Component Analysis (PCA) Find projections of X = [x1 , . . . , xN ] maximizing the variance of data XU PCA: maximize: Trace{(XU) (XU)} = Trace{U Cxx U} subject to: U U=I Including Lagrange multipliers λ, this is equivalent to the eigenproblem Cxx ui = λi ui → Cxx U = UD ui are the eigenvectors of Cxx and they are orthonormal, ui uj = 0 PCA limitations 1 Axes rotation to the directions of maximum variance of data 2 It does not consider noise characteristics: Assumes noise variance is low → last eigenvectors with low eigenvalues Maximum variance directions may be affected by noiseL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 6/23
  10. 10. Intro SNR KMNF Results ConclusionsMinimum Noise Fraction Transformation The SNR transformation Find projections maximizing the ratio between signal and noise variances:  ff U Css U SNR: maximize: Tr U Cnn U subject to: U Cnn U = I Unknown signal and noise covariance matrices Css and CnnL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 7/23
  11. 11. Intro SNR KMNF Results ConclusionsMinimum Noise Fraction Transformation The SNR transformation Find projections maximizing the ratio between signal and noise variances:  ff U Css U SNR: maximize: Tr U Cnn U subject to: U Cnn U = I Unknown signal and noise covariance matrices Css and Cnn The MNF transformation Assuming additive X = S + N and orthogonal S N = N S = 0 noise Maximizing SNR is equivalent to Minimizing NF = 1/(SNR+1):  ff U Cxx U MNF: maximize: Tr U Cnn U subject to: U Cnn U = I This is equivalent to solving the generalized eigenproblem: Cxx ui = λi Cnn ui → Cxx U = Cnn UDL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 7/23
  12. 12. Intro SNR KMNF Results ConclusionsMinimum Noise Fraction Transformation The MNF transformation Minimum Noise Fraction equivalent to solve the generalized eigenproblem: Cxx ui = λi Cnn ui → Cxx U = Cnn UD Since U Cnn U = I, eigenvalues λi are the SNR+1 in the projected space Need estimates of signal Cxx = X X and noise Cnn ≈ N N covariancesL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 8/23
  13. 13. Intro SNR KMNF Results ConclusionsMinimum Noise Fraction Transformation The MNF transformation Minimum Noise Fraction equivalent to solve the generalized eigenproblem: Cxx ui = λi Cnn ui → Cxx U = Cnn UD Since U Cnn U = I, eigenvalues λi are the SNR+1 in the projected space Need estimates of signal Cxx = X X and noise Cnn ≈ N N covariances The noise covariance estimation Noise estimate: diff. between actual value and a reference ‘clean’ value N = X − Xr Xr from neighborhood assuming a spatially smoother signal than the noise Assume stationary processes in wide sense: Differentiation: ni ≈ xi − xi −1 1 PM Smoothing filtering: ni ≈ xi − M k=1 wk xi −k Wiener estimates Wavelet domain estimatesL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 8/23
  14. 14. Intro SNR KMNF Results Conclusions 1 Introduction 2 Signal-to-noise ratio transformation 3 Kernel Minimum Noise Fraction 4 Experimental Results 5 Conclusions and Open questionsL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 9/23
  15. 15. Intro SNR KMNF Results ConclusionsKernel methods for non-linear feature extraction Kernel methods Input features space Kernel feature space Φ 1 Map the data to a high-dimensional feature space, H (dH → ∞) 2 Solve a linear problem there Kernel trick No need to know dH → ∞ coordinates for each mapped sample φ(xi ) Kernel trick: “if an algorithm can be expressed in the form of dot products, its non-linear (kernel) version only needs the dot products among mapped samples, the so-called kernel function:” K (xi , xj ) = φ(xi ), φ(xj ) Using this trick, we can implement K-PCA, K-PLS, K-ICA, etcL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 10/23
  16. 16. Intro SNR KMNF Results ConclusionsKernel Principal Component Analysis (KPCA) Kernel Principal Component Analysis (KPCA) Find projections maximizing variance of mapped data [φ(x1 ), . . . , φ(xN )] KPCA: maximize: Tr{(ΦU) (ΦU)} = Tr{U Φ ΦU} subject to: U U=I The covariance matrix Φ Φ and projection matrix U are dH × dH !!! KPCA through kernel trick Apply the representer’s theorem: U = Φ A where A = [α1 , . . . , αN ] KPCA: maximize: Tr{A ΦΦ ΦΦ A} = Tr{A KKA} subject to: U U = A ΦΦ A = A KA = I Including Lagrange multipliers λ, this is equivalent to the eigenproblem KKαi = λi Kαi → Kαi = λi αi Now matrix A is N × N !!! (eigendecomposition of K) Projections are obtained as ΦU = ΦΦ A = KAL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 11/23
  17. 17. Intro SNR KMNF Results ConclusionsKernel MNF Transformation KMNF through kernel trick Find projections maximizing SNR of mapped data [φ(x1 ), . . . , φ(xN )] Replace X ∈ Rn×N with Φ ∈ Rn×NH Replace N ∈ Rn×N with Φn ∈ Rn×NG Cxx U = Cnn UD ⇒ Φ ΦU = Φn Φn UD Not solvable: matrices Φ Φ and Φn Φn are NH × NH and NG × NG Left multiply both sides by Φ, and use representer’s theorem, U = Φ A: ΦΦ ΦΦ A = ΦΦn Φn Φ AD → Kxx Kxx A = Kxn Kxn AD Now matrix A is N × N !!! (eigendecomposition of Kxx wrt Kxn ) Kxx = ΦΦ is symmetric with elements K (xi , xj ) Kxn = ΦΦn = Knx is non-symmetric with elements K (xi , nj ) Easy and simple to program! Potentially useful when signal and noise are nonlinearly relatedL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 12/23
  18. 18. Intro SNR KMNF Results ConclusionsSNR in Hilbert spaces Implicit KMNF: noise estimate in the input space Estimate the noise directly in the input space: N = X − Xr Signal-to-noise kernel: Kxn = ΦΦn → K (xi , nj ) with Φn = [φ(n1 ), . . . , φ(nn )] Kernels Kxx and Kxn dealing with objects of different nature → 2 params Two different kernel spaces → eigenvalues have no longer meaning of SNR Explicit KMNF: noise estimate in the feature space Estimate the noise explicitly in the Hilbert space: Φn = Φ − Φr Signal-to-noise kernel: Kxn = ΦΦn = Φ(Φ − Φr ) = ΦΦ − ΦΦr = Kxx − Kxr Again it is not symmetric K (xi , rj ) = K (ri , xj ) Advantage: same kernel parameter for Kxx and KxnL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 13/23
  19. 19. Intro SNR KMNF Results ConclusionsSNR in Hilbert spaces Explicit KMNF: nearest reference Differentiation in feature space: φni ≈ φ(xi ) − φ(xi ,d ) x xr (Kxn )ij ≈ φ(xi ), φ(xj ) − φ(xj,d ) = K (xi , xj ) − K (xi , xj,d ) Explicit KMNF: averaged reference Difference to a local average in feature space (e.g. 4-connected xr 1 PD xr x xr neighboring pixels): φni ≈ φ(xi ) − φ(xi ,d ) D d =1 xr D D 1 X 1 X (Kxn )ij ≈ φ(xi ), φ(xj ) − φ(xj,d ) = K (xi , xj ) − K (xi , xj,d ) D D d =1 d =1 Explicit KMNF: autoregression reference Weight the relevance of each kernel in the summation: D X D X (Kxn )ij ≈ φ(xi ), φ(xj ) − wd φ(xj,d ) = K (xi , xj ) − wd K (xi , xj,d ) d =1 d =1L. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 14/23
  20. 20. Intro SNR KMNF Results Conclusions 1 Introduction 2 Signal-to-noise ratio transformation 3 Kernel Minimum Noise Fraction 4 Experimental Results 5 Conclusions and Open questionsL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 15/23
  21. 21. Intro SNR KMNF Results ConclusionsExperimental results Data material AVIRIS hyperspectral image (220-bands): Indian Pine test site 145 × 145 pixels, 16 crop types classes, 10366 labeled pixels The 20 noisy bands in the water absorption region are intentionally kept Experimental setup PCA, MNF, KPCA, and two versions of KMNF (implicit and explicit) The 220 bands transformed into a lower dimensional space of 18 featuresL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 16/23
  22. 22. Intro SNR KMNF Results ConclusionsVisual inspection: extracted features in descending order of relevance Features 1–3 4–6 7–9 10–12 13–15 16–18 PCA MNF KPCA implicit KMNF explicit KMNFL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 17/23
  23. 23. Intro SNR KMNF Results ConclusionsAnalysis of the eigenvalues: signal variance and SNR of transformed data Analysis of the eigenvalues: Signal variance of the transformed data for PCA SNR of the transformed data for MNF and KMNF 150 PCA MNF KMNF Eigenvalue / SNR 100 50 0 0 5 10 15 20 # feature The proposed approach provides the highest SNR!L. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 18/23
  24. 24. Intro SNR KMNF Results ConclusionsLDA classifier: land-cover classification accuracy 0.6 0.6 Kappa statistic (κ) Kappa statistic (κ) 0.5 0.5 0.4 0.4 0.3 0.3 PCA PCA MNF MNF 0.2 KPCA 0.2 KPCA KMNFi KMNFi KMNF KMNF 0.1 0.1 2 4 6 8 10 12 14 16 18 2 4 6 8 10 12 14 16 18 # features # features Original hyperspectral image Multiplicative random noise (10%) Best results: linear MNF and the proposed KMNF The proposed KMNF method outperforms MNF when the image is corrupted with non additive noiseL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 19/23
  25. 25. Intro SNR KMNF Results ConclusionsLDA classifier: land-cover classification maps LDA-MNF LDA-KMNFL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 20/23
  26. 26. Intro SNR KMNF Results Conclusions 1 Introduction 2 Signal-to-noise ratio transformation 3 Kernel Minimum Noise Fraction 4 Experimental Results 5 Conclusions and Open questionsL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 21/23
  27. 27. Intro SNR KMNF Results ConclusionsConclusions and open questions Conclusions Kernel method for nonlinear feature extraction maximizing the SNR Good theoretical and practical properties for extracting noise-free features Deals with non-linear relations between the noise and signal The only parameter is the width of the kernel Knowledge about noise can be encoded in the method Simple optimization problem → eigendecomposition of the kernel matrix Noise estimation in the kernel space with different levels of sophistication Simple feature extraction toolbox (SIMFEAT) soon at http://isp.uv.es Open questions and Future Work Pre-images of transformed data in the input space Learn kernel parameters in an automatic way Test KMNF in more remote sensing applications: denoising, unmixing, ...L. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 22/23
  28. 28. Intro SNR KMNF Results Conclusions Explicit Signal to Noise Ratio in Reproducing Kernel Hilbert Spaces Luis Gómez-Chova1 Allan A. Nielsen2 Gustavo Camps-Valls1 1 Image Processing Laboratory (IPL), Universitat de València, Spain. luis.gomez-chova@uv.es , http://www.valencia.edu/chovago 2 DTU Space - National Space Institute. Technical University of Denmark. IGARSS 2011 – Vancouver, Canada * IPL Image Processing LaboratoryL. Gómez-Chova et al. Explicit Kernel Signal to Noise Ratio IGARSS 2011 – Vancouver 23/23

×