Successfully reported this slideshow.
Upcoming SlideShare
×

# Robust Kernel-Based Regression Using Orthogonal Matching Pursuit

739 views

Published on

Presentetion for the IEEE International Workshop on MACHINE LEARNING FOR SIGNAL PROCESSING MLSP 2013 conference.

• Full Name
Comment goes here.

Are you sure you want to Yes No

Are you sure you want to  Yes  No
• Be the first to like this

### Robust Kernel-Based Regression Using Orthogonal Matching Pursuit

1. 1. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP) G. Papageorgiou, P. Bouboulis, S. Theodoridis Department of Informatics and Telecommunications, National and Kapodistrian University of Athens, Greece MLSP SEP. 22-25, 2013, Southampton, UK G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
2. 2. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Outline 1 Introduction 2 Problem formulation 3 Unravelling KROMP 4 Experimental Results 5 Conclusions and future work G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
3. 3. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Introduction Let (yk, xk), k = 1, 2, ..., n, with yk ∈ R, xk ∈ Rm. A typical regression task is to estimate the input-output relation satisfying: yk = f (xk) + ηk, k = 1, 2, ..., n. The problem is solved: f linear: Least Squares in the Euclidean space. f nonlinear: Assume that f ∈ H, where H will be assumed to be a RKHS and the minimizer of min f n k=1 yk − f (xk) 2 + µ||f ||2 H, µ > 0 admits a representation ˆf (x) = n i=1 αi κ(x, xi ) (LS in H). G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
4. 4. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Problem formulation What if the noise samples contain outliers? G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
5. 5. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Problem formulation What if the noise samples contain outliers? Is it possible to remove other type of noise too? G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
6. 6. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Problem formulation What if the noise samples contain outliers? Is it possible to remove other type of noise too? Let uk be impulse noise samples of unknown “energy”, assumed to be outliers, hence sparse. Since LS is not robust against outliers we reformulate: Problem model yk = f (xk) + uk + ηk, k = 1, 2, ..., n, G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
7. 7. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Problem formulation Assuming that the unknown function is expressed as a linear combination of kernel functions of the form f = n k=1 αkκ(·, xk) + c, we intend to solve: Minimization Task min α,c,u u 0 s.t. Kα + c1 + u − y 2 2 + λ α 2 2 + λc2 ≤ ε G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
8. 8. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Problem formulation Minimization Task min α,c,u ||u||0 s.t. ||Kα + c1 + u − y||2 2 + λ||α||2 2 + λc2 ≤ ε Remarks: Keeps the square error low. Regularization guards against overﬁtting. Use of nonconvex function. Exploits greedy sparse approximation algorithms from the family of (GA). G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
9. 9. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Sparse approximation (denoising) algorithms l0 (pseudo)-norm minimization: min x ||x||0 s.t. ||Ax − b||2 ≤ Greedy Algorithms (GA) (MP) (OMP) (OLS) (LARS) G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
10. 10. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Sparse approximation (denoising) algorithms l1 norm minimization: min x ||x||1 s.t. ||Ax − b||2 ≤ or minx λ||x||1 + 1 2||Ax − b||2 2 for appropriate Lagrange multiplier λ = λ(A, b, ) Homotopy (TNIPM) Gradient Projection (GP) (IRLS) Iterative Shrinkage- Thresholding Proximal Gradient (PG) (CoSaMP) Augmented Lagrangian Methods G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
11. 11. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Column selection from matrix A We recast the optimization task in matrix notation: min z ||u||0 s.t. ||Az − y||2 2 + λ||Bz||2 2 ≤ , where A = K 1 In , z =   α c u   , B =   In 0 On 0T 1 0T On 0 On   , meaning that only the last part of z is sparse. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
12. 12. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Column selection from matrix A Initially (k := 0) we set A(0) = [K 1] and B(0) = In+1. At each step: We ﬁnd one position (index) for an outlier (say 1 ≤ jk ≤ n). Then, A is augmented by the column jk of In and B is augmented by zeros in order to match the appropriate dimensions (actually this is a projection matrix B2 = B). Finally, we solve minz A(k) z − y 2 2 + λ B(k) z 2 2. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
13. 13. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Column selection from matrix A Green columns: columns from matrix A, which have not been selected yet. Red columns: columns that have already been selected at previous steps. These participate in the regularized minimization problem. Step k := 0 : A(0) = [K 1] Step k := 1 : A(1) = [K 1 e2] Step k := 2 : A(2) = [K 1 e2 e3] G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
14. 14. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Column selection from matrix A Green columns: columns from matrix A, which have not been selected yet. Red columns: columns that have already been selected at previous steps. These participate in the regularized minimization problem. Step k := 0 : A(0) = [K 1] Step k := 1 : A(1) = [K 1 e2] Step k := 2 : A(2) = [K 1 e2 e3] G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
15. 15. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Column selection from matrix A Green columns: columns from matrix A, which have not been selected yet. Red columns: columns that have already been selected at previous steps. These participate in the regularized minimization problem. Step k := 0 : A(0) = [K 1] Step k := 1 : A(1) = [K 1 e2] Step k := 2 : A(2) = [K 1 e2 e3] G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
16. 16. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Column selection from matrix A Green columns: columns from matrix A, which have not been selected yet. Red columns: columns that have already been selected at previous steps. These participate in the regularized minimization problem. Step k := 0 : A(0) = [K 1] Step k := 1 : A(1) = [K 1 e2] Step k := 2 : A(2) = [K 1 e2 e3] G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
17. 17. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Kernel Regularized OMP (KROMP): Initialization Select the kernel for the kernel matrix K, the regularization parameter λ and the error threshold . Step k := 0 : Solve the regularized LS problem and compute the residual z(0) := arg minz ||A(0)z − y||2 2 + λ||B(0)z||2 2, r(0) = y − A(0)z(0), where S (0) act = {1, ..., n + 1}, A(0) = [K 1], B(0) = In+1. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
18. 18. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Kernel Regularized OMP (KROMP): Iteration cycle Iteration: While ||r(k−1)||2 > 1 k := k + 1. 2 Find the most correlated direction jk := arg maxj /∈Sact |r (k−1) j |. 3 Update the set: S (k) act := S (k−1) act ∪ {jk } A(k) := [A(k−1) ejk ], B(k) := B(k−1) 0 0T 0 . 4 Compute current solution and residual: z(k) := arg minz ||A(k) z − y||2 2 + λ||B(k) z||2 2, r(k) = y − A(k) z(k) . Remark: The residual obtained at each cycle is strictly decreasing. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
19. 19. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Reducing Complexity As complexity at each step is O(n + k)3, k << n, there is a need for cost reduction. Initially, the matrix C(0) = [A(0)T A(0) + λB(0)] is inverted, in order to solve the minimization task. Similarly, at step k the matrix C(k) = [A(k)T A(k) + λB(k)] is inverted. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
20. 20. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Reducing Complexity However, as large part of the inverted matrix at each step remains unchanged, we could avoid the inversion by applying the matrix inversion lemma (MIL): C(k)−1 = C(k−1)−1 + 1 t C(k−1)−1 αT jk αjk C(k−1)−1 −1 t C(k−1)−1 αT jk −1 t αjk C(k−1)−1 1 t , where αjk = eT jk A(k−1), t = 1 − αjk C(k−1)−1 αT jk . Achieved computational cost: O(n + k)2 per step. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
21. 21. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work sinc: 1-D case −1 −0.5 0 0.5 1 −40 −20 0 20 40 Noisy data versus estimated data with KROMP −1 −0.5 0 0.5 1 −10 0 10 20 Original data versus estimated data with MSE=0.05671 (a) KROMP with λ = 1. −1 −0.5 0 0.5 1 −40 −20 0 20 40 Noisy data versus estimated data with ADM −1 −0.5 0 0.5 1 −10 0 10 20 Original data versus ADM estimated data with MSE=0.09278 (b) ADM with λ = 1 and µ = 0.1. Figure: Reconstruction of y = 20sinc(2πx) over the presence of 20dB Gaussian noise and 10% outliers with values taken randomly in [−20, 20]. (a) using KROMP and (b) using ADM (Giannakis-Mateos 2012). The MSE is computed over 10000 data sets of 201 points. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
22. 22. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work sinc: 2-D case −1 0 1 −1−0.8−0.6−0.4−0.200.20.40.60.81 −40 −20 0 20 40 60 Noisy data versus estimated data −1 0 1 −1−0.8−0.6−0.4−0.200.20.40.60.81 −20 0 20 40 60 Original data versus estimated data Figure: Reconstruction of f (x, y) = 50sinc(π x2 + y2) over the presence of 15dB Gaussian noise and 10% outliers with values taken randomly in [−20, 20]. KROMP attains a MSE = 1.022, while ADM attains a MSE = 2.205 (computed over 10000 data sets of 21 × 21 = 421 points). G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
23. 23. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Application in image denoising: Lena Original Image Noisy Image, PSNR=20.39dB Figure: The original image of Lena (512 × 512 pixels) and its noisy counterpart. Image corrupted with 20dB Gaussian noise and impulse noise uniformly distributed with values ±100. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
24. 24. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Application in image denoising: Lena Restored Image with KROMP, PSNR=31.2dB Restored Image with ADM, PSNR=27.07dB Figure: Left: reconstructed image using KROMP attains a PSNR = 31.2dB. Right: reconstructed image using ADM attains a PSNR = 27.07dB. In both algorithms parameters are set using cross-correlation so that the MSE is minimized. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
25. 25. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Application in image denoising: Boat Original Image Noisy Image, PSNR=20.2dB Figure: The original image of Boat (512 × 512 pixels) and the noisy image. Image corrupted with 20dB Gaussian noise and impulse noise uniformly distributed with values ±100. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
26. 26. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Application in image denoising: Boat Restored Image with KROMP, PSNR=29.32dB Restored Image with ADM, PSNR=26.23dB Figure: Left:reconstructed image using KROMP attains a PSNR = 29.32dB. Right:reconstructed image using ADM attains a PSNR = 26.23dB. In both algorithms parameters are set using cross-correlation so that the MSE is minimized. G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
27. 27. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Conclusions An (OMP)-based algorithm is proposed for use in nonlinear estimation The performance of the algorithm exceeds its predecessor (ADM) in numerous examples The residual obtained at each cycle is proved to be strictly decreasing Recent applications in image-denoising seem to verify our claim towards better performance The heavy computational cost is reduced using MIL; other popular techniques (e.g., Cholesky and QR factorization) seem to work also (OMP) is a well studied algorithm, hence rich properties are there to reinforce our work G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
28. 28. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Future work 1 Construction of a more suitable stopping criterion 2 Complexity reduction KROMP using QR decomposition KROMP using Cholesky factorization 3 Theoretical results for stability of the method Conditions on the recovery of the exact support of the outlier vector Bounds on the approximation of the solution 4 Applications of KROMP as an image or audio denoising algorithm Codes and more at: http://bouboulis.mysch.gr/kernels.html G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)
29. 29. Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work Thank you Thank you! Questions please G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)