Sparse representation and compressive sensing

11,461 views

Published on

Published in: Education, Technology

Sparse representation and compressive sensing

  1. 1. Advanced Signal ProcessingSparse Representation and Compressive Sensing Dr. M. Sabarimalai Manikandan Assistant Professor Center for Excellence in Computational Engineering and Networking Amrita University, Coimbatore Campus E-mail: msm.sabari@gmail.com September 16, 2011 Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  2. 2. Advanced Signal ProcessingWhy Signal Processing? ◮ Most natural signals are non-stationary and have highly complex time-varying spectro-temporal characteristics. ◮ Mixture of many sources ◮ Composition of mixed events ◮ Various kinds of noise and artifacts ◮ The SP is challenging task because the natural signals are typically having different shapes, amplitudes, durations and frequency content, which are not known in many different applications and systems Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  3. 3. Advanced Signal ProcessingSignal Representation Using Basis Functions ◮ A set {ψn }N is called a orthonormal basis for RN if the n=1 vectors in the set span RN and are linearly independent ◮ Let x ∈ RN×1 be the input signal that is spanned by N basis functions {ψn }N . Then, a discrete-time signal x can be n=1 represented as N x= αn ψ n = Ψα (1) n=1 where α = [α1 , α2 , α3 , ......αN ] is the transform coefficients vector that is computed as αn = x, ψ n . ◮ For some transform matrix, the transform coefficients vector α has a small number of large amplitude coefficients and a large number of small amplitude coefficients Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  4. 4. Advanced Signal ProcessingSome of Representation or Transform Matrices ◮ Fourier transform matrix ◮ discrete cosines (DCT matrix) and discrete sines (DST matrix) ◮ Haar transform matrix ◮ wavelet and wavelet packets matrices ◮ Gabor filters ◮ curvelets, ridgelets, contourlets, bandelets, shearlets ◮ directionlets, grouplets, chirplets ◮ Hermite polynomials, and so on Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  5. 5. Advanced Signal ProcessingLimitations of Fixed Representation Matrix ◮ The Fourier transform is suitable for analysis of the steady-state sinusoidal signals but it fails to capture the sharp changes and discontinuities in the signals. ◮ In the STFT-based methods, the choices for widths of the time-window affect the frequency and time resolution. ◮ The common problem in well-known wavelet transform-based methods is which mother wavelet function and characteristic scale provides the best time-frequency resolution for detection of transients and non-transients. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  6. 6. Advanced Signal ProcessingSparse Representation/Recovery ◮ Definition: The sparse representation theory has shown that sparse signals can be exactly reconstructed from a small number of elementary signals (or atoms). ◮ The sparse representation of natural signals can be achieved by exploiting its sparsity or compressibility. ◮ A natural signal is said to be sparse signal if that can be compactly expressed as a linear combination of a few small number of basis vectors. ◮ Sparse representation has become an invaluable tool as compared to direct time-domain and transform-domain signal processing methods. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  7. 7. Advanced Signal ProcessingSparse Representation: Applications ◮ audio/image/video processing tasks (compression, denoising, deblurring, inpainting, and superresolution) ◮ speech enhancement and recognition ◮ signal detection and classification ◮ face recognition, array processing, blind source separation ◮ sensor networks and cognitive radios ◮ power quality disturbances ◮ underwater acoustic communications ◮ data acquisition and imaging technologies Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  8. 8. Advanced Signal ProcessingSparse Signal: Sparsity ◮ Definition: A signal can be sparse or compressible in some transform matrix Ψ when the transform coefficients vector α has a small number of large amplitude coefficients and a large number of small amplitude coefficients. ◮ Observations: Most of the energy is concentrated in a few transform coefficients in a vector α ◮ The other N − K coefficients have less contribution in representing a signal vector x ∈ RN×1 . ◮ The insignificant coefficients are set to zero in coding scheme. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  9. 9. Advanced Signal ProcessingSparse Signal: Sparsity ◮ The value of K is computed as K = α 0 , where . 0 denotes the ℓ0 -norm which counts the number of non-zero entries in α. ◮ Concluding Remarks: A sparse signal x can be exactly represented or approximated by the linear combination of K basis functions with shorter transform coefficients vector. ◮ In such a reconstruction process, the reconstruction error by a K -term representation decays exponentially as K increases. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  10. 10. Advanced Signal ProcessingSparse Representation: Dictionary Learning ◮ Need for Dictionary Learning: In practice, a signal is composed of impulsive and oscillatory transients, spikes and low-frequency components. ◮ Nature: The composite signal may not exhibit sparsity in one transform basis matrix because some of its components are sparse in one domain while other components are sparse in another domain. ◮ The signals may exhibit sparsity in either time-domain or frequency-domain. ◮ For example, the 50 Hz powerline signal is sparse in the frequency-domain and the impulse or spikes component is sparse in the time-domain. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  11. 11. Advanced Signal ProcessingSparse Representation: Dictionary Learning ◮ Problem with Fixed Basis: In practice, the composite signal (spike is superimposed on powerline signal) exhibits sparsity in neither time-domain nor frequency-domain. ◮ In such cases, a fixed orthogonal basis functions are not flexible enough to capture the complex local waves of a signal. ◮ For example, a fixed elementary cosine waveforms of discrete cosine transform (DCT) matrix fails to capture transient parts of biosignals. ◮ Detection and suppression of impulsive noise in speech waveform. ◮ Compression of slow varying signals with spikes. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  12. 12. Advanced Signal ProcessingSparse Representation: Need for Best Basis Functions ◮ Remedies: To improve the sparsity of composite signals, one has to construct a transform matrix with the best basis functions. ◮ One way to process such signal is to work with an large dictionary matrix. ◮ A best basis set from a dictionary matrix used to sparsify the data may yield highly compact representations of many natural signals. ◮ Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  13. 13. Advanced Signal ProcessingSparse Representation: What is Dictionary? ◮ Dictionary: A dictionary is a collection of elementary waveforms or prototype atoms or basis functions. ◮ A dictionary matrix D of dimension N × L can be represented as D = {ψ 1 |ψ 2 |ψ 3 |..........|ψ L }. ◮ The column vectors {ψ l }L of an dictionary D are l=1 discrete-time elementary signals of length N × 1, called dictionary atoms or basis functions. ◮ The atoms in the pre-defined dictionary may be pairwise orthogonal, linear independent, linear dependent, or not orthogonal. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  14. 14. Advanced Signal ProcessingSparse Representation: Classification of Dictionaries ◮ Based on the number of atoms L and the signal length N, the pre-defined dictionary, D ∈ RN×M , could be classified into three categories: ◮ (i ) when L > N, D is called overcomplete, or redundant dictionary. ◮ (ii ) when L < N, D is called undercomplete dictionary. ◮ (iii ) D is said to be complete dictionary if L = N. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  15. 15. Advanced Signal ProcessingSome of Sparse Transform Matrices ◮ dirac and heaviside functions ◮ Fourier transform matrix and Fourier, short-time Fourier transform (STFT) ◮ discrete cosines (DCT matrix) and discrete sines (DST matrix) ◮ Haar transform matrix ◮ wavelet and wavelet packets matrices ◮ Gabor filters ◮ curvelets, ridgelets, contourlets, bandelets, shearlets ◮ directionlets, grouplets, chirplets ◮ Hermite polynomials, and so on Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  16. 16. Advanced Signal ProcessingSparse Representation: Research Problems ◮ The dirac dictionary can be used to detect the spikes in a signal and the discrete cosines dictionary can provide sinusoidal waveforms ◮ The SR from redundant dictionaries may provide better ways to reveal/capture the structures in nonstationary environments ◮ The SR may offer better performance in signal modeling and classification problems ◮ An efficient and flexible dictionary matrix has to be built for separation of mixtures of events ◮ Many researchers have attempted to build dictionary for specific signal processing tasks ◮ How to learn the dictionary from the training datasets Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  17. 17. Advanced Signal ProcessingThe CS Measurement System ◮ Performing reduction/compression when sensing analog signals ◮ The CS is a new data acquisition theory ◮ The number of measurements is typically below the number of samples obtained from the Nyquist sampling theorem ◮ The nonadaptive linear measurements of the input signal vector are computed as y = Φx (2) where y is an M × 1 measurement vector, M ≪ N and Φ is an M × N measurement/sensing matrix. ◮ Measurements using a second basis matrix Φ ∈ RM×N that is incoherent with the sparsity basis matrix Ψ ∈ RN×N Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  18. 18. Advanced Signal ProcessingThe CS System: Reduction and Information ◮ The measurement system actually performs dimensionality reduction ◮ Measurements are able to completely capture the useful information content embedded in a sparse signal ◮ Measurements are information of the signals and thus can be used as features for signal modeling ◮ If the Φ consists of elementary sinusoid waveforms, then α is a vector of Fourier coefficients. ◮ If the Φ consists of Dirac delta functions, then α is a vector of sampled values of continuous time signal x(t). Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  19. 19. Advanced Signal ProcessingThe CS Recovery: Issues ◮ The y can be written as y = ΦΨα (3) ◮ We define the matrix D = ΦΨ with a size of M × N. ◮ The major problem associated with CS concept is that we have to solve an underdetermined system of equations to recover the original signal x from the measurement vector y . ◮ This system has infinitely many solutions since the number of equations is less than the number of unknowns ◮ It is necessary to impose constraints such as “sparsity” and “incoherence” that are introduced for for this signal recovery to be efficient Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  20. 20. Advanced Signal ProcessingCS Reconstruction: Incoherence ◮ the CS recovery relies on two basic principles [?]: ◮ (i ) the row vectors of the measurement matrix Φ cannot sparsely represent the column vectors of the sparsity matrix Ψ, and vice versa ◮ (ii ) the number of measurements M is greater than N O(cKlog ( K )) ◮ these conditions can ensure that it is possible to recover the set of nonzero elements of sparse vector α from measurements y. ◮ the input signal x can be reconstructed by the linear transformation of α: . Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  21. 21. Advanced Signal ProcessingCS Reconstruction: Incoherence ◮ Sparsity basis matrix Ψ is orthonormal and the sensing matrix Φ consists of M row vectors drawn randomly from some basis ˜ matrix Φ ∈ RN×N ◮ The mutual coherence is computed as: √ µ(Φ, Ψ) = N max | Φk , Ψj | (4) 1≤k,j≥N ◮ It measures the largest correlation between any two elements √ of Φ and Ψ and will take a value between 1 and N. ◮ The value of coherence is large when the elements of Φ and Ψ are highly correlated and thus CS system requires more measurements. ◮ The smaller value µ(Φ, Ψ) indicates maximally incoherent bases and hence, the number of measurements will be less Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  22. 22. Advanced Signal ProcessingCS Recovery: How many measurements? ◮ The recovery performance is perfect and optimal when the bases are perfectly incoherent, and unavoidably decreases when the mutual coherence µ increases. ◮ The number of measurements M required for perfect signal reconstruction can be computed as: M ≥ c · K · µ2 (Φ, Ψ) · log(N) (5) where c is positive constant, µ is the mutual coherence, K is the sparsity factor, and N is the length of the input vector. ◮ The value of coherence is large when the elements of Φ and Ψ are highly correlated and thus CS system requires more measurements. ◮ The smaller value µ(Φ, Ψ) indicates maximally incoherent bases and hence, the number of measurements in (4) can be the smallest Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  23. 23. Advanced Signal ProcessingCS Recovery: How many measurements? ◮ Under very low mutual coherence value, k-sparse signal can be reconstructed from k.log (N) measurements using basis pursuit ◮ Examples of such pairs (maximal mutual incoherence) are: Φ is the spike basis and Ψ is the Fourier basis ◮ Φ is the noiselet basis and Ψ is the wavelet basis. Noiselets are also maximally incoherent with spikes and incoherent with the Fourier basis. ◮ Φ is a random matrix and Ψ is any fixed basis Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  24. 24. Advanced Signal ProcessingCompressive Sensing/Measurement Matrices ◮ The the entries of Φ are: (i ) samples of independent and identically distributed (iid) Gaussian or Bernoulli entries ◮ (ii ) randomly selected rows of an orthogonal N × N matrix ◮ The RIP says that D acts as an approximate isometry on the set of vectors that are K -sparse, and a matrix D satisfies the K −restricted isometry property if there exists the smallest number, δs ∈ [0 1], such that (1 − δs ) α 2 ≤ Dα 2 ≤ (1 + δs ) α 2 . 2 2 2 The constant δs depends on K , Φ, and α. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  25. 25. Advanced Signal ProcessingCS Recovery by ℓ1 -norm Optimization ◮ The goal of a sparse recovery algorithm is to obtain an estimate of α given only y and D = ΦΨ ◮ The recovery of the K -sparse signal x from the measurements y is ill-posed since M < N ◮ The CS system of equations is underdetermined ◮ the sparest vector is computed by solving the well-known underdetermind problem with sparsity constraint, ˆ α = arg min α 0 subject to y = ΦΨα = Dα (6) α where • denotes ℓ0 -norm that counts the number of nonzero entries in a vector. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  26. 26. Advanced Signal ProcessingCS Recovery by ℓ1 -norm Optimization ◮ By allowing a certain degree of reconstruction error given by the magnitude of the noise ◮ the optimization constraint is now relaxed: λ α = arg min{ α ˆ 0 + y − ΦΨα 2 } 2 (7) α 2 where λ ∈ R+ , which controls the relative importance applied to the reconstruction error term and the sparseness term. ◮ the solution needs a combinatorial search among all possible sparse α, which is infeasible for most problems of interest Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  27. 27. Advanced Signal ProcessingCS Recovery by ℓ1 -norm Optimization ◮ To overcome this problem, many nonlinear optimization-based methods have been proposed to obtain sparest vector α by converting (6) into a convex problem which relaxes the ℓ0 -norm to an ℓ1 -norm problem α = arg min α ˆ 1 subject to y = ΦΨα = Dα (8) α λ α = arg min{ α ˆ 1 + y − ΦΨα 2 } 2 (9) α 2 ◮ which can be solved by linear programming such as BP, MP and OMP ◮ The solution to equation (8) is exact or optimal if the number of measurements K is large enough compared to the sparsity factor K , K < M < N and the measurements are chosen uniformly at random Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  28. 28. Advanced Signal ProcessingSR Applications: Transients Detection 2 1 amplitude 0 −1 −2 0 20 40 60 80 100 (a) 1 0 amplitude −1 1 0 −1 0 20 40 60 80 (b) 1 amplitude 0 −1 −2 0 20 40 60 80 100 120 140 (c) sample number Figure: Examples of measured transients with 50 Hz power supply waveforms: (a) spike, (b) microinterruption, and (c) oscillatory transient. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  29. 29. Advanced Signal ProcessingSR Applications: Transients Detection ◮ The over-complete dictionary Ψ with size of N × 2N is constructed as Ψ = [ I C] (10) where I is the N × N identity (or spike-like) matrix, and C is the N × N DCT matrix. ◮ 1 √ , i = 0, 0 ≤ j ≤ N −1 M Cij = 2 π(2j+1)i (11) M cos( 2N ), 1 ≤ i ≤ N − 1, 0 ≤ j ≤ N −1 and the spike like matrix is constructed as   1 0 ··· 0 0 0 1 ··· 0 0   Iij = 0 0 1 0 0 (12)  . . .  . . . ..  . . . . 0 0 0 0 ··· 1 N×N Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  30. 30. Advanced Signal ProcessingSR Applications: Transients Detection ◮ the signal x can be written as x ≈ Ψα = [ I ˜ C]˜ = Id + Ca. α (13) and can be rewritten as N N y= dn I n + an Cn . (14) n=1 n=1 ◮ The common problem in well-known wavelet transform-based methods is which mother wavelet function and characteristic scale provides the best time-frequency resolution for detection of transients and non-transients. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  31. 31. Advanced Signal ProcessingSR Applications: Transients Detection 1. Input: N × 1 input signal vector y . 2. Specify the value of regularization parameter λ. 3. Read the N × 2N over-complete dictionary matrix Ψ. 4. Solve the ℓ1 -norm minimization problem: α = arg minα { Ψα − y 2 + λ α 1 } ˜ 2 5. Obtain the detail and approximation coefficient vectors. 6. Process detail vector for detecting boundaries of transient event. 7. Output: time-instants and transient portions Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  32. 32. Advanced Signal ProcessingSR Applications: Impulsive Transients Powerline with impulsive noise 1 original 0 −1 0.5 component detail 0 −0.5 approximation 1 component 0.5 0 −0.5 0 0.02 0.04 0.06 0.08 0.1 0.12 Time (sec) Figure: Illustrates the detail and approximation components extracted by using the proposed method. The power supply waveform is corrupted by spikes. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  33. 33. Advanced Signal ProcessingSR Applications: Transients Detection 50 Hz powerline with microinterruption 1 original 0 −1 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 0.8 component 0.6 detail 0.4 0.2 0 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 1 approximation component 0 −1 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1 Time (sec) Figure: Illustrates the detail and approximation components extracted by using the proposed method. The The power supply waveform with microinterruption. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  34. 34. Advanced Signal ProcessingSR Applications: Transients Detection low−amplitude transients high−amplitude transients original signal 1 orignal signal 1 0 0 −1 −1 0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.01 0.02 0.03 0.04 0.05 0.06 (a) (b) (detail signal extracted) (detail signal extracted) 1 1 detected transient detected transient 0.5 0.5 0 0 −0.5 −0.5 −1 −1 −1.5 0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.01 0.02 0.03 0.04 0.05 0.06 (c) (d) signal extracted signal extracted approximation approximation 1 1 0 0 −1 −1 0 0.01 0.02 0.03 0.04 0.05 0.06 0 0.01 0.02 0.03 0.04 0.05 0.06 Time (sec) Time (sec) (e) (f) Figure: Example of waveforms S1 and S2 illustrates signals corrupted by low-amplitude transient S1 and high-amplitude transient S2 due to capacitor switching, respectively. The detected transient events by using our method. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  35. 35. Advanced Signal ProcessingSR Applications: Transients Detection 0.4 with noise signal with 0.2 1 spike spike 0 0 −0.2 −1 0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05 (a1) (a2) wavelet method wavelet method 0.4 (First Detail) (First Detail) 0.2 1 0 0 −0.2 −1 0 0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05 (b1) (b2) wavelet method wavelet method (Second Detail) (Second Detail) 0.4 1 0.2 0 0 −0.2 0 0.01 0.02 0.03 0.04 −1 (c1) 0 0.01 0.02 0.03 0.04 0.05 (c2) 0.2 by our method detected spike by our method detected spike 1 0.1 0 0 −1 0.01 0.02 0.03 0.04 0.05 0 0.01 0.02 0.03 0.04 0.05 Time (sec) Time (sec) (d1) (d2) Figure: Example of transient signals S3 and S4 : (a1) the spike buried in strong noise with SNR value of -10 dB; (a2) the 50 Hz sinusoidal signal affected by a superimposed spike; Plots are the outputs from the wavelet-based methods and the proposed method. Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  36. 36. Advanced Signal ProcessingSR Applications: Removal of Powerline original ECG signal original ECG signal 0.5 0.5 a m p litu d e am plitude 0 0 -0.5 -0.5 200 400 600 800 1000 1200 1400 1600 1800 2000 200 400 600 800 1000 1200 1400 1600 1800 2000 Time (sec) Time (sec) original ECG signal plus powerline (10 degree) original ECG signal plus powerline (86 degree) 0.5 0.5 am plitude a m p litu d e 0 0 -0.5 -0.5 200 400 600 800 1000 1200 1400 1600 1800 2000 200 400 600 800 1000 1200 1400 1600 1800 2000 Time (sec) Time (sec) Output of CS-based approach Output of CS-based approach 0.5 0.5 am plitude a m p litu d e 0 0 -0.5 -0.5 200 400 600 800 1000 1200 1400 1600 1800 2000 200 400 600 800 1000 1200 1400 1600 1800 2000 Time (sec) Time (sec) Figure: Removal of Powerline from ECG Signal Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  37. 37. Advanced Signal ProcessingSR Applications: Removal of Artifacts original ECG signal 0.5 amplitude 0 -0.5 -1 500 1000 1500 2000 2500 3000 T ime (sec) original ECG signal plus powerline 0.5 amplitude 0 -0.5 -1 500 1000 1500 2000 2500 3000 T ime (sec) Output of CS-based approach 0.5 amplitude 0 -0.5 500 1000 1500 2000 2500 3000 T ime (sec) Output of CS-based baseline wander removal 0.4 amplitude 0.2 0 -0.2 -0.4 500 1000 1500 2000 2500 3000 T ime (sec) Figure: Simultaneous removal of Powerline and LF artifact from ECG Signal Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing
  38. 38. Advanced Signal ProcessingSparse Representation and Compressive Sensing Thanks for your Attention! Dr. M. Sabarimalai Manikandan Sparse Representation and Compressive Sensing

×