1.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
ROBUST KERNEL-BASED REGRESSION
USING ORTHOGONAL MATCHING PURSUIT
(OMP)
G. Papageorgiou, P. Bouboulis, S. Theodoridis
Department of Informatics and Telecommunications, National and Kapodistrian
University of Athens, Greece
MLSP SEP. 22-25, 2013,
Southampton, UK
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

2.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Outline
1 Introduction
2 Problem formulation
3 Unravelling KROMP
4 Experimental Results
5 Conclusions and future work
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

3.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Introduction
Let (yk, xk), k = 1, 2, ..., n, with yk ∈ R, xk ∈ Rm. A typical
regression task is to estimate the input-output relation satisfying:
yk = f (xk) + ηk, k = 1, 2, ..., n.
The problem is solved:
f linear: Least Squares in the Euclidean space.
f nonlinear: Assume that f ∈ H, where H will be assumed to
be a RKHS and the minimizer of
min
f
n
k=1
yk − f (xk)
2
+ µ||f ||2
H, µ > 0
admits a representation ˆf (x) = n
i=1 αi κ(x, xi ) (LS in H).
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

4.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Problem formulation
What if the noise samples contain outliers?
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

5.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Problem formulation
What if the noise samples contain outliers?
Is it possible to remove other type of noise too?
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

6.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Problem formulation
What if the noise samples contain outliers?
Is it possible to remove other type of noise too?
Let uk be impulse noise samples of unknown “energy”, assumed to
be outliers, hence sparse. Since LS is not robust against outliers
we reformulate:
Problem model
yk = f (xk) + uk + ηk, k = 1, 2, ..., n,
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

7.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Problem formulation
Assuming that the unknown function is expressed as a linear
combination of kernel functions of the form
f = n
k=1 αkκ(·, xk) + c, we intend to solve:
Minimization Task
min
α,c,u
u 0
s.t. Kα + c1 + u − y 2
2 + λ α 2
2 + λc2 ≤ ε
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

8.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Problem formulation
Minimization Task
min
α,c,u
||u||0 s.t. ||Kα + c1 + u − y||2
2 + λ||α||2
2 + λc2
≤ ε
Remarks:
Keeps the square error low.
Regularization guards against overfitting.
Use of nonconvex function.
Exploits greedy sparse approximation algorithms from the
family of (GA).
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

9.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Sparse approximation (denoising) algorithms
l0 (pseudo)-norm minimization:
min
x
||x||0 s.t. ||Ax − b||2 ≤
Greedy Algorithms (GA)
(MP) (OMP) (OLS) (LARS)
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

10.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Sparse approximation (denoising) algorithms
l1 norm minimization:
min
x
||x||1 s.t. ||Ax − b||2 ≤
or minx λ||x||1 + 1
2||Ax − b||2
2 for appropriate Lagrange
multiplier λ = λ(A, b, )
Homotopy
(TNIPM)
Gradient
Projection (GP)
(IRLS)
Iterative
Shrinkage-
Thresholding
Proximal
Gradient (PG)
(CoSaMP)
Augmented
Lagrangian
Methods
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

11.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Column selection from matrix A
We recast the optimization task in matrix notation:
min
z
||u||0 s.t. ||Az − y||2
2 + λ||Bz||2
2 ≤ ,
where A = K 1 In , z =


α
c
u

 , B =


In 0 On
0T
1 0T
On 0 On

 , meaning
that only the last part of z is sparse.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

12.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Column selection from matrix A
Initially (k := 0) we set A(0) = [K 1] and B(0) = In+1.
At each step:
We find one position (index) for an outlier (say 1 ≤ jk ≤ n).
Then, A is augmented by the column jk of In and B is
augmented by zeros in order to match the appropriate
dimensions (actually this is a projection matrix B2
= B).
Finally, we solve minz A(k)
z − y 2
2 + λ B(k)
z 2
2.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

13.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Column selection from matrix A
Green columns: columns from matrix A, which have not been
selected yet.
Red columns: columns that have already been selected at previous
steps. These participate in the regularized minimization problem.
Step k := 0 : A(0) = [K 1]
Step k := 1 : A(1) = [K 1 e2]
Step k := 2 : A(2) = [K 1 e2 e3]
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

14.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Column selection from matrix A
Green columns: columns from matrix A, which have not been
selected yet.
Red columns: columns that have already been selected at previous
steps. These participate in the regularized minimization problem.
Step k := 0 : A(0) = [K 1]
Step k := 1 : A(1) = [K 1 e2]
Step k := 2 : A(2) = [K 1 e2 e3]
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

15.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Column selection from matrix A
Green columns: columns from matrix A, which have not been
selected yet.
Red columns: columns that have already been selected at previous
steps. These participate in the regularized minimization problem.
Step k := 0 : A(0) = [K 1]
Step k := 1 : A(1) = [K 1 e2]
Step k := 2 : A(2) = [K 1 e2 e3]
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

16.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Column selection from matrix A
Green columns: columns from matrix A, which have not been
selected yet.
Red columns: columns that have already been selected at previous
steps. These participate in the regularized minimization problem.
Step k := 0 : A(0) = [K 1]
Step k := 1 : A(1) = [K 1 e2]
Step k := 2 : A(2) = [K 1 e2 e3]
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

17.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Kernel Regularized OMP (KROMP): Initialization
Select the kernel for the kernel matrix K, the regularization
parameter λ and the error threshold .
Step k := 0 : Solve the regularized LS problem and compute
the residual z(0) := arg minz ||A(0)z − y||2
2 + λ||B(0)z||2
2,
r(0) = y − A(0)z(0), where S
(0)
act = {1, ..., n + 1},
A(0) = [K 1], B(0) = In+1.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

18.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Kernel Regularized OMP (KROMP): Iteration cycle
Iteration: While ||r(k−1)||2 >
1 k := k + 1.
2 Find the most correlated direction jk := arg maxj /∈Sact
|r
(k−1)
j |.
3 Update the set: S
(k)
act := S
(k−1)
act ∪ {jk }
A(k)
:= [A(k−1)
ejk
], B(k)
:=
B(k−1)
0
0T
0
.
4 Compute current solution and residual:
z(k)
:= arg minz ||A(k)
z − y||2
2 + λ||B(k)
z||2
2, r(k)
= y − A(k)
z(k)
.
Remark: The residual obtained at each cycle is strictly decreasing.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

19.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Reducing Complexity
As complexity at each step is O(n + k)3, k << n, there is a
need for cost reduction.
Initially, the matrix C(0) = [A(0)T
A(0) + λB(0)] is inverted, in
order to solve the minimization task.
Similarly, at step k the matrix C(k) = [A(k)T
A(k) + λB(k)] is
inverted.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

20.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Reducing Complexity
However, as large part of the inverted matrix at each step
remains unchanged, we could avoid the inversion by applying
the matrix inversion lemma (MIL):
C(k)−1
=
C(k−1)−1
+ 1
t C(k−1)−1
αT
jk
αjk
C(k−1)−1
−1
t C(k−1)−1
αT
jk
−1
t αjk
C(k−1)−1 1
t
,
where αjk
= eT
jk
A(k−1), t = 1 − αjk
C(k−1)−1
αT
jk
.
Achieved computational cost: O(n + k)2 per step.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

21.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
sinc: 1-D case
−1 −0.5 0 0.5 1
−40
−20
0
20
40
Noisy data versus estimated data with KROMP
−1 −0.5 0 0.5 1
−10
0
10
20
Original data versus estimated data with MSE=0.05671
(a) KROMP with λ = 1.
−1 −0.5 0 0.5 1
−40
−20
0
20
40
Noisy data versus estimated data with ADM
−1 −0.5 0 0.5 1
−10
0
10
20
Original data versus ADM estimated data with MSE=0.09278
(b) ADM with λ = 1 and µ = 0.1.
Figure: Reconstruction of y = 20sinc(2πx) over the presence of 20dB
Gaussian noise and 10% outliers with values taken randomly in [−20, 20].
(a) using KROMP and (b) using ADM (Giannakis-Mateos 2012). The
MSE is computed over 10000 data sets of 201 points.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

22.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
sinc: 2-D case
−1
0
1
−1−0.8−0.6−0.4−0.200.20.40.60.81
−40
−20
0
20
40
60
Noisy data versus estimated data
−1
0
1
−1−0.8−0.6−0.4−0.200.20.40.60.81
−20
0
20
40
60
Original data versus estimated data
Figure: Reconstruction of f (x, y) = 50sinc(π x2 + y2) over the
presence of 15dB Gaussian noise and 10% outliers with values taken
randomly in [−20, 20]. KROMP attains a MSE = 1.022, while ADM
attains a MSE = 2.205 (computed over 10000 data sets of
21 × 21 = 421 points).
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

23.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Application in image denoising: Lena
Original Image Noisy Image, PSNR=20.39dB
Figure: The original image of Lena (512 × 512 pixels) and its noisy
counterpart. Image corrupted with 20dB Gaussian noise and impulse
noise uniformly distributed with values ±100.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

24.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Application in image denoising: Lena
Restored Image with KROMP, PSNR=31.2dB Restored Image with ADM, PSNR=27.07dB
Figure: Left: reconstructed image using KROMP attains a
PSNR = 31.2dB. Right: reconstructed image using ADM attains a
PSNR = 27.07dB. In both algorithms parameters are set using
cross-correlation so that the MSE is minimized.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

25.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Application in image denoising: Boat
Original Image Noisy Image, PSNR=20.2dB
Figure: The original image of Boat (512 × 512 pixels) and the noisy
image. Image corrupted with 20dB Gaussian noise and impulse noise
uniformly distributed with values ±100.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

26.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Application in image denoising: Boat
Restored Image with KROMP, PSNR=29.32dB Restored Image with ADM, PSNR=26.23dB
Figure: Left:reconstructed image using KROMP attains a
PSNR = 29.32dB. Right:reconstructed image using ADM attains a
PSNR = 26.23dB. In both algorithms parameters are set using
cross-correlation so that the MSE is minimized.
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

27.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Conclusions
An (OMP)-based algorithm is proposed for use in nonlinear
estimation
The performance of the algorithm exceeds its predecessor
(ADM) in numerous examples
The residual obtained at each cycle is proved to be strictly
decreasing
Recent applications in image-denoising seem to verify our
claim towards better performance
The heavy computational cost is reduced using MIL; other
popular techniques (e.g., Cholesky and QR factorization)
seem to work also
(OMP) is a well studied algorithm, hence rich properties are
there to reinforce our work
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

28.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Future work
1 Construction of a more suitable stopping criterion
2 Complexity reduction
KROMP using QR decomposition
KROMP using Cholesky factorization
3 Theoretical results for stability of the method
Conditions on the recovery of the exact support of the outlier
vector
Bounds on the approximation of the solution
4 Applications of KROMP as an image or audio denoising
algorithm
Codes and more at: http://bouboulis.mysch.gr/kernels.html
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)

29.
Introduction Problem formulation Unravelling KROMP Experimental Results Conclusions and future work
Thank you
Thank you!
Questions please
G. Papageorgiou, P. Bouboulis, S. Theodoridis DI, University of Athens, Greece
ROBUST KERNEL-BASED REGRESSION USING ORTHOGONAL MATCHING PURSUIT (OMP)