Upcoming SlideShare
×

# Adaptive Signal and Image Processing

912 views

Published on

Slide of a course for the summer school "Ecole Analyse multirésolution pour l'image", June 20-22, 2012, Auxerre, France.

Published in: Education
1 Comment
1 Like
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Thankyou Gabriel

Are you sure you want to  Yes  No
Views
Total views
912
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
139
1
Likes
1
Embeds 0
No embeds

No notes for slide

### Adaptive Signal and Image Processing

1. 1. Adaptive Signaland Image Processing Gabriel Peyré www.numerical-tours.com
2. 2. Natural Image PriorsFourier decomposition Wavelet decomposition
3. 3. Natural Image PriorsFourier decomposition Wavelet decomposition
4. 4. Overview•Sparsity for Compression and Denoising•Geometric Representations•L0 vs. L1 Sparsity•Dictionary Learning
5. 5. Sparse Approximation in a Basis
6. 6. Sparse Approximation in a Basis
7. 7. Sparse Approximation in a Basis
8. 8. Image and Texture ModelsUniformly smooth C image. Fourier, Wavelets: ||f fM ||2 = O(M ). | f |2
9. 9. Image and Texture ModelsUniformly smooth C image. Fourier, Wavelets: ||f fM ||2 = O(M ). | f |2Discontinuous image with bounded variation. Wavelets: ||f fM ||2 = O(M 1 ). | f|
10. 10. Image and Texture ModelsUniformly smooth C image. Fourier, Wavelets: ||f fM ||2 = O(M ). | f |2Discontinuous image with bounded variation. Wavelets: ||f fM ||2 = O(M 1 ). | f|C -geometrically regular image. 2 Curvelets: ||f fM ||2 = O(log3 (M )M 2 ). | f|
11. 11. Image and Texture ModelsUniformly smooth C image. Fourier, Wavelets: ||f fM ||2 = O(M ). | f |2Discontinuous image with bounded variation. Wavelets: ||f fM ||2 = O(M 1 ). | f|C -geometrically regular image. 2 Curvelets: ||f fM ||2 = O(log3 (M )M 2 ). | f|
12. 12. Image and Texture ModelsUniformly smooth C image. Fourier, Wavelets: ||f fM ||2 = O(M ). | f |2Discontinuous image with bounded variation. Wavelets: ||f fM ||2 = O(M 1 ). | f|C -geometrically regular image. 2 Curvelets: ||f fM ||2 = O(log3 (M )M 2 ). | f|More complex images: needs adaptivity.
13. 13. Compression by Transform-coding forwardf transform a[m] = ⇥f, m⇤ R Image f Zoom on f
14. 14. Compression by Transform-coding forward f transform a[m] = ⇥f, m⇤ R q[m] Z bin T ⇥ 2T |a[m]| 2T T T a[m]Quantization: q[m] = sign(a[m]) Z T a[m] ˜ Image f Zoom on f Quantized q[m]
15. 15. Compression by Transform-coding forward codi f transform a[m] = ⇥f, m⇤ R q[m] Z ng bin T ⇥ 2T |a[m]| 2T T T a[m]Quantization: q[m] = sign(a[m]) Z T a[m] ˜Entropic coding: use statistical redundancy (many 0’s). Image f Zoom on f Quantized q[m]
16. 16. Compression by Transform-coding forward codi f transform a[m] = ⇥f, m⇤ R q[m] Z n g bin T ng c odi de q[m] Z ⇥ 2T |a[m]| 2T T T a[m]Quantization: q[m] = sign(a[m]) Z T a[m] ˜Entropic coding: use statistical redundancy (many 0’s). Image f Zoom on f Quantized q[m]
17. 17. Compression by Transform-coding forward codi f transform a[m] = ⇥f, m⇤ R q[m] Z n g bin T ng c odi dequantization de a[m] ˜ q[m] Z ⇥ 2T |a[m]| 2T T T a[m]Quantization: q[m] = sign(a[m]) Z T a[m] ˜Entropic coding: use statistical redundancy (many 0’s). ⇥ 1Dequantization: a[m] = sign(q[m]) |q[m] + ˜ T 2 Image f Zoom on f Quantized q[m]
18. 18. Compression by Transform-coding forward codi f transform a[m] = ⇥f, m⇤ R q[m] Z n g bin T ng c odi backward dequantization defR = a[m] ˜ m a[m] ˜ q[m] Z transform ⇥ 2T m IT |a[m]| 2T T T a[m]Quantization: q[m] = sign(a[m]) Z T a[m] ˜Entropic coding: use statistical redundancy (many 0’s). ⇥ 1Dequantization: a[m] = sign(q[m]) |q[m] + ˜ T 2 Image f Zoom on f Quantized q[m] f , R =0.2 bit/pixel
19. 19. Compression by Transform-coding forward codi f transform a[m] = ⇥f, m⇤ R q[m] Z n g bin T ng c odi backward dequantization defR = a[m] ˜ m a[m] ˜ q[m] Z transform ⇥ 2T m IT |a[m]| 2T T T a[m]Quantization: q[m] = sign(a[m]) Z T a[m] ˜Entropic coding: use statistical redundancy (many 0’s). ⇥ 1Dequantization: a[m] = sign(q[m]) |q[m] + ˜ T 2 “Theorem:” ||f fM ||2 = O(M ) =⇥ ||f fR ||2 = O(log (R)R ) Image f Zoom on f Quantized q[m] f , R =0.2 bit/pixel
20. 20. Wavelets…JPEG-2000 vs. JPEG JPEG2k: exploit the statistical redundancy of coe cients. + embedded coder. chunks of large coe cients. neighboring coe cients are not independent. Compression JPEG Compression EZW Compressionorm Image f JPEG, R = .19bit/pxl JPEG2k, R = .15bit/pxl
21. 21. Denoising (Donoho/Johnstone)
22. 22. Denoising (Donoho/Johnstone) N 1 thresh. ˜ f= f, m⇥ m f= f, m⇥ m m=0 | f, m ⇥|>T
23. 23. Denoising (Donoho/Johnstone) N 1 thresh. ˜ f= f, m⇥ m f= f, m⇥ m m=0 | f, m ⇥|>TTheorem: if ||f0 f0,M ||2 = O(M ), In practice: ˜E(||f f0 || ) = O( 2 2 +1 ) for T = 2 log(N ) T 3
24. 24. Overview•Sparsity for Compression and Denoising•Geometric Representations•L0 vs. L1 Sparsity•Dictionary Learning
25. 25. Piecewise Regular Functions in 1D Theorem: If f is C outside a ﬁnite set of discontinuities: O(M 1 ) (Fourier), ||f fM || = 2 O(M 2 ) (wavelets).For Fourier, linear non-linear, sub-optimal.For wavelets, linear non-linear, optimal.
26. 26. Piecewise Regular Functions in 2D Theorem: If f is C outside a set of ﬁnite length edge curves, O(M 1/2 ) (Fourier), ||f fM || = 2 O(M 1 ) (wavelets).Fourier Wavelets.Wavelets: same result for BV functions (optimal).
27. 27. Piecewise Regular Functions in 2D Theorem: If f is C outside a set of ﬁnite length edge curves, O(M 1/2 ) (Fourier), ||f fM || = 2 O(M 1 ) (wavelets).Fourier Wavelets.Wavelets: same result for BV functions (optimal).Regular C edges: sub-optimal (requires anisotropy).
28. 28. Geometrically Regular ImagesGeometric image model: f is C outside a set of C edge curves. BV image: level sets have ﬁnite lengths. Geometric image: level sets are regular. | f| Geometry = cartoon image Sharp edges Smoothed edges
29. 29. Curvelets for Cartoon ImagesCurvelets: [Candes, Donoho] [Candes, Demanet, Ying, Donoho]
30. 30. Curvelets for Cartoon ImagesCurvelets: [Candes, Donoho] [Candes, Demanet, Ying, Donoho]Redundant tight frame (redundancy 5): not e cient for compression.Denoising by curvelet thresholding: recovers edges and geometric textures.
31. 31. Approximation with Triangulation Vertices V = {vi }M .Triangulation (V, F): i=1 Faces F {1, . . . , M }3 .
32. 32. Approximation with Triangulation Vertices V = {vi }M .Triangulation (V, F): i=1 Faces F {1, . . . , M }3 . MPiecewise linear approximation: fM = m ⇥m m=1 = argmin ||f µm ⇥m || µ m ⇥m (vi ) = m i is a ne on each face of F.
33. 33. Approximation with Triangulation Vertices V = {vi }M .Triangulation (V, F): i=1 Faces F {1, . . . , M }3 . MPiecewise linear approximation: fM = m ⇥m m=1 = argmin ||f µm ⇥m || µ m ⇥m (vi ) = m i is a ne on each face of F.Theorem: There exists (V, F) such that Regular areas: M/2 equilateral triangles. 2 ||f fM || Cf M 1/2 M Singular areas: M 1/2 M/2 anisotropic triangles.
34. 34. Approximation with Triangulation Vertices V = {vi }M .Triangulation (V, F): i=1 Faces F {1, . . . , M }3 . MPiecewise linear approximation: fM = m ⇥m m=1 = argmin ||f µm ⇥m || µ m ⇥m (vi ) = m i is a ne on each face of F.Theorem: There exists (V, F) such that Regular areas: M/2 equilateral triangles. 2 ||f fM || Cf M 1/2 MOptimal (V, F): NP-hard. Singular areas:Provably good greedy schemes: M 1/2 M/2 anisotropic triangles. [Mirebeau, Cohen, 2009]
35. 35. Greedy Triangulation Optimization Bougleux, Peyr´, Cohen, ICCV’09 eM = 200M = 600 Anisotropic triangulation JPEG2000
36. 36. Geometric Multiscale Processing f Image f R N Wavelet transform jWavelet coe cients fj [n] = f, j,n fj
37. 37. Geometric Multiscale Processing f Image f R N Wavelet transform j Wavelet coe cients fj [n] = f, j,n fjGeometry Geometric transform Geometric coe cients
38. 38. Geometric Multiscale Processing f Image f R N Wavelet transform j Wavelet coe cients fj [n] = f, j,n fjGeometry Geometric transform Geometric coe cients 2 j b[j, , n] = fj , b⇤,n ⇥ ⇥ 2
39. 39. Geometric Multiscale Processing f Image f R N Wavelet transform j Wavelet coe cients fj [n] = f, j,n fjGeometry Geometric transform C cartoon image: Geometric coe cients 2 j ||f fM ||2 = O(M ) b[j, , n] = fj , b⇤,n ⇥ ⇥ M = Mband + M 2
40. 40. Overview•Sparsity for Compression and Denoising•Geometric Representations•L0 vs. L1 Sparsity•Dictionary Learning
41. 41. Image Representation Q 1Dictionary D = {dm }m=0 of atoms dm RN . Q 1Image decomposition: f = xm dm = Dx m=0 dm xm dm = f D x
42. 42. Image Representation Q 1Dictionary D = {dm }m=0 of atoms dm RN . Q 1Image decomposition: f = xm dm = Dx m=0 dmImage approximation: f Dx xm dm = f D x
43. 43. Image Representation Q 1Dictionary D = {dm }m=0 of atoms dm RN . Q 1Image decomposition: f = xm dm = Dx m=0 dmImage approximation: f Dx xm dmOrthogonal dictionary: N = Q = xm = f, dm f D x
44. 44. Image Representation Q 1Dictionary D = {dm }m=0 of atoms dm RN . Q 1Image decomposition: f = xm dm = Dx m=0 dmImage approximation: f Dx xm dmOrthogonal dictionary: N = Q = xm = f, dm f D xRedundant dictionary: N Q Examples: TI wavelets, curvelets, . . . x is not unique.
45. 45. Sparsity Q 1Decomposition: f= xm dm = Dx m=0Sparsity: most xm are small. Example: wavelet transform. Image f Coe cients x
46. 46. Sparsity Q 1Decomposition: f= xm dm = Dx m=0Sparsity: most xm are small. Example: wavelet transform. Image fIdeal sparsity: most xm are zero. J0 (x) = | {m xm = 0} | Coe cients x
47. 47. Sparsity Q 1Decomposition: f= xm dm = Dx m=0Sparsity: most xm are small. Example: wavelet transform. Image fIdeal sparsity: most xm are zero. J0 (x) = | {m xm = 0} |Approximate sparsity: compressibility ||f Dx|| is small with J0 (x) M. Coe cients x
48. 48. Sparse Coding Q 1Redundant dictionary D = {dm }m=0 , Q N. non-unique representation f = Dx.Sparsest decomposition: min J0 (x) f =Dx
49. 49. Sparse Coding Q 1Redundant dictionary D = {dm }m=0 , Q N. non-unique representation f = Dx.Sparsest decomposition: min J0 (x) f =Dx 1Sparsest approximation: min ||f Dx|| + J0 (x) 2 x 2 Equivalence min ||f Dx|| M ⇥ J0 (x) M min J0 (x) ||f Dx||
50. 50. Sparse Coding Q 1Redundant dictionary D = {dm }m=0 , Q N. non-unique representation f = Dx.Sparsest decomposition: min J0 (x) f =Dx 1Sparsest approximation: min ||f Dx|| + J0 (x) 2 x 2 Equivalence min ||f Dx|| M ⇥ J0 (x) M min J0 (x) ||f Dx||Ortho-basis D: ⇤ Pick the M largest f, dm ⇥ if |xm | 2 coe cients xm = 0 otherwise. in { f, dm ⇥}m
51. 51. Sparse Coding Q 1Redundant dictionary D = {dm }m=0 , Q N. non-unique representation f = Dx.Sparsest decomposition: min J0 (x) f =Dx 1Sparsest approximation: min ||f Dx|| + J0 (x) 2 x 2 Equivalence min ||f Dx|| M ⇥ J0 (x) M min J0 (x) ||f Dx||Ortho-basis D: ⇤ Pick the M largest f, dm ⇥ if |xm | 2 coe cients xm = 0 otherwise. in { f, dm ⇥}mGeneral redundant dictionary: NP-hard.
52. 52. Convex Relaxation: L1 Prior J0 (x) = | {m xm = 0} | J0 (x) = 0 null image.Image with 2 pixels: J0 (x) = 1 sparse image. J0 (x) = 2 non-sparse image. d1 d0 q=0
53. 53. Convex Relaxation: L1 Prior J0 (x) = | {m xm = 0} | J0 (x) = 0 null image.Image with 2 pixels: J0 (x) = 1 sparse image. J0 (x) = 2 non-sparse image. d1 d0 q=0 q = 1/2 q=1 q = 3/2 q=2 q priors: Jq (x) = |xm |q (convex for q 1) m
54. 54. Convex Relaxation: L1 Prior J0 (x) = | {m xm = 0} | J0 (x) = 0 null image.Image with 2 pixels: J0 (x) = 1 sparse image. J0 (x) = 2 non-sparse image. d1 d0 q=0 q = 1/2 q=1 q = 3/2 q=2 q priors: Jq (x) = |xm |q (convex for q 1) mSparse 1 prior: J1 (x) = ||x||1 = |xm | m
55. 55. Inverse ProblemsDenoising/approximation: = Id.
56. 56. Inverse ProblemsDenoising/approximation: = Id.Examples: Inpainting, super-resolution, compressed-sensing
57. 57. Regularized InversionDenoising/compression: y = f0 + w RN . Sparse approximation: f = Dx where 1 x ⇥ argmin ||y Dx||2 + ||x||1 x 2 Fidelity
58. 58. Regularized InversionDenoising/compression: y = f0 + w RN . Sparse approximation: f = Dx where 1 x ⇥ argmin ||y Dx||2 + ||x||1 x 2 Fidelity Replace D by DInverse problems y = f0 + w RP . 1 x ⇥ argmin ||y Dx|| + ||x||1 2 x 2
59. 59. Regularized InversionDenoising/compression: y = f0 + w RN . Sparse approximation: f = Dx where 1 x ⇥ argmin ||y Dx||2 + ||x||1 x 2 Fidelity Replace D by DInverse problems y = f0 + w RP . 1 x ⇥ argmin ||y Dx|| + ||x||1 2 x 2Numerical solvers: proximal splitting schemes. www.numerical-tours.com
60. 60. Inpainting Results
61. 61. Overview•Sparsity for Compression and Denoising•Geometric Representations•L0 vs. L1 Sparsity•Dictionary Learning
62. 62. Dictionary Learning: MAP EnergySet of (noisy) exemplars {yk }k . 1Sparse approximation: min ||yk Dxk || + ||xk ||1 2 xk 2
63. 63. Dictionary Learning: MAP EnergySet of (noisy) exemplars {yk }k . 1Sparse approximation: min min ||yk Dxk || + ||xk ||1 2 D C xk k 2 Dictionary learning
64. 64. Dictionary Learning: MAP EnergySet of (noisy) exemplars {yk }k . 1Sparse approximation: min min ||yk Dxk || + ||xk ||1 2 D C xk k 2 Dictionary learning Constraint: C = {D = (dm )m m, ||dm || 1} Otherwise: D + , X 0
65. 65. Dictionary Learning: MAP EnergySet of (noisy) exemplars {yk }k . 1Sparse approximation: min min ||yk Dxk || + ||xk ||1 2 D C xk k 2 Dictionary learning Constraint: C = {D = (dm )m m, ||dm || 1} Otherwise: D + , X 0Matrix formulation: 1 min f (X, D) = ||Y DX|| + ||X||1 2 X⇥R Q K 2 D⇥C RN Q
66. 66. Dictionary Learning: MAP EnergySet of (noisy) exemplars {yk }k . 1Sparse approximation: min min ||yk Dxk || + ||xk ||1 2 D C xk k 2 Dictionary learning Constraint: C = {D = (dm )m m, ||dm || 1} Otherwise: D + , X 0Matrix formulation: 1 min f (X, D) = ||Y DX|| + ||X||1 2 X⇥R Q K 2 min f (X, D) D⇥C R N Q X Convex with respect to X. Convex with respect to D. D Non-onvex with respect to (X, D). Local minima
67. 67. Dictionary Learning: AlgorithmStep 1: k, minimization on xk 1 min ||yk Dxk || + ||xk ||1 2 xk 2 Convex sparse coding. D, initialization
68. 68. Dictionary Learning: AlgorithmStep 1: k, minimization on xk 1 min ||yk Dxk || + ||xk ||1 2 xk 2 Convex sparse coding.Step 2: Minimization on D D, initialization min ||Y DX|| 2 D C Convex constraint minimization.
69. 69. Dictionary Learning: Algorithm Step 1: k, minimization on xk 1 min ||yk Dxk || + ||xk ||1 2 xk 2 Convex sparse coding. Step 2: Minimization on D D, initialization min ||Y DX|| 2 D C Convex constraint minimization. Projected gradient descent:D ( +1) = ProjC D ( ) (D ( ) X Y )X
70. 70. Dictionary Learning: Algorithm Step 1: k, minimization on xk 1 min ||yk Dxk || + ||xk ||1 2 xk 2 Convex sparse coding. Step 2: Minimization on D D, initialization min ||Y DX|| 2 D C Convex constraint minimization. Projected gradient descent:D ( +1) = ProjC D ( ) (D ( ) X Y )XConvergence: toward a stationary point of f (X, D). D, convergence
71. 71. Patch-based Learning Learning DExemplar patches yk Dictionary D [Olshausen, Fields 1997] State of the art denoising [Elad et al. 2006]
72. 72. Patch-based Learning Learning DExemplar patches yk Dictionary D [Olshausen, Fields 1997] State of the art denoising [Elad et al. 2006] Learning D Sparse texture synthesis, inpainting [Peyr´ 2008] e
73. 73. Comparison with PCAPCA dimensionality reduction: ⇥ k, min ||Y D(k) X|| D (k) = (dm )m=0 k 1 DLinear (PCA): Fourier-like atoms. RUBINSTEIN et al.: al.: DICTIONARIES FOR SPARSE REPRESENTATION RUBINSTEIN et DICTIONARIES FOR SPARSE REPRESENTATION 1980 by by Bast 1980 Bastiaa fundamental prop fundamental p A basic 1-D G A basic 1-D forms forms © © G = = n, G ⇤ ⇤ DCT PCA where w(·) is is where w(·) a Fig. 1. 1.Left: A fewfew £ 12 12 DCT atoms. Right: The ﬁrst 40 KLT atoms, (typically a Gau Fig. Left: A 12 12 £ DCT atoms. Right: The ﬁrst 40 KLT atoms, (typically a G trained using 12 £ 12 12 image patches from Lena. trained using 12 £ image patches from Lena. frequency resolu frequency reso matical foundatio matical founda late 1980’s by by late 1980’s D B. B. Non-Linear Revolution and Elements Modern Dictionary Non-Linear Revolution and Elements of of Modern Dictionary who studied thet who studied Design Design and by by Feichti and Feichting In In statistics research, the 1980’s saw the rise of new generalized group statistics research, the 1980’s saw the rise of a a new generalized gro powerful approach known as as robust statistics. Robust statistics powerful approach known robust statistics. Robust statistics
74. 74. Comparison with PCAPCA dimensionality reduction: ⇥ k, min ||Y D(k) X|| D (k) = (dm )m=0 k 1 DLinear (PCA): Fourier-like atoms. RUBINSTEIN et al.: al.: DICTIONARIES FOR SPARSE REPRESENTATION RUBINSTEIN et DICTIONARIES FOR SPARSE REPRESENTATIONSparse (learning): Gabor-like atoms. 1980 by by Bast 1980 Bastiaa fundamental prop fundamental p A basic 1-D G A basic 1-D forms forms © © 4 G = = n, G ⇤ ⇤ 4 DCT PCA where w(·) is is where w(·) a Fig. 1. 1.Left: A fewfew £ 12 12 DCT atoms. Right: The ﬁrst 40 KLT atoms, (typically a Gau Fig. Left: A 12 12 £ DCT atoms. Right: The ﬁrst 40 KLT atoms, 0.15 (typically a G 0.15 trained using 12 £ 12 12 image patches from Lena. trained using 12 £ image patches from Lena. frequency resolu 0.1 frequency reso 0.1 matical foundatio matical founda 0.05 0.05 0 0 late 1980’s by by late 1980’s D B. B. Non-Linear Revolution and Elements Modern Dictionary Non-Linear Revolution and Elements of of Modern Dictionary -0.05 -0.05 who studied thet who studied Design Design -0.1 -0.1 and by by Feichti -0.15 and Feichting -0.15 In In statistics research, the 1980’s saw the rise of new generalized group statistics research, the 1980’s saw the rise of a a new generalized gro -0.2 -0.2 Gabor Learned powerful approach known as as robust statistics. Robust statistics powerful approach known robust statistics. Robust statistics
75. 75. Patch-based DenoisingNoisy image: f = f0 + w.Step 1: Extract patches. yk (·) = f (zk + ·) yk[Aharon & Elad 2006]
76. 76. Patch-based DenoisingNoisy image: f = f0 + w.Step 1: Extract patches. yk (·) = f (zk + ·)Step 2: Dictionary learning. 1 min ||yk Dxk || + ||xk ||1 2 D,(xk )k 2 k yk[Aharon & Elad 2006]
77. 77. Patch-based DenoisingNoisy image: f = f0 + w.Step 1: Extract patches. yk (·) = f (zk + ·)Step 2: Dictionary learning. 1 min ||yk Dxk || + ||xk ||1 2 D,(xk )k 2 kStep 3: Patch averaging. yk = Dxk ˜ ˜ f (·) ⇥ yk (· zk ) ˜ k yk ˜ yk[Aharon & Elad 2006]
78. 78. Learning with Missing DataInverse problem: y = f0 + w LEARNING MULTISCALE AND S 1 1 min ||y f || + 2 ||pk (f ) Dxk || + ⇥||xk ||1 2f,(xk )k 2 2 kD C f0Patch extractor: pk (f ) = f (zk + ·) pk LEARNING MULTISCALE AND SPARSE REPRESENTATIONS 237 (a) Original y (a) Original (b) Damaged
79. 79. Learning with Missing DataInverse problem: y = f0 + w LEARNING MULTISCALE AND S 1 1 min ||y f || + 2 ||pk (f ) Dxk || + ⇥||xk ||1 2f,(xk )k 2 2 kD C f0Patch extractor: pk (f ) = f (zk + ·) pk Step 1: k, minimization on xk LEARNING MULTISCALE AND SPARSE REPRESENTATIONS 237 Convex sparse coding. (a) Original y (a) Original (b) Damaged
80. 80. Learning with Missing DataInverse problem: y = f0 + w LEARNING MULTISCALE AND S 1 1 min ||y f || + 2 ||pk (f ) Dxk || + ⇥||xk ||1 2f,(xk )k 2 2 kD C f0Patch extractor: pk (f ) = f (zk + ·) pk Step 1: k, minimization on xk LEARNING MULTISCALE AND SPARSE REPRESENTATIONS 237 Convex sparse coding. Step 2: Minimization on D (a) Original y Quadratic constrained. (a) Original (b) Damaged
81. 81. Learning with Missing DataInverse problem: y = f0 + w LEARNING MULTISCALE AND S 1 1 min ||y f || + 2 ||pk (f ) Dxk || + ⇥||xk ||1 2f,(xk )k 2 2 kD C f0Patch extractor: pk (f ) = f (zk + ·) pk Step 1: k, minimization on xk LEARNING MULTISCALE AND SPARSE REPRESENTATIONS 237 Convex sparse coding. Step 2: Minimization on D (a) Original y Quadratic constrained. Step 3: Minimization on f Quadratic. (a) Original (b) Damaged
82. 82. Inpainting Example LEARNING MULTISCALE AND SPARSE REPRESENTATIONS LEARNING MULTISCALE AND SPARSE REPRESENTATIONS 237 237 (a) Original (b) Damaged Image f0 (a) Original (a) Original Observations (c) Restored, N = 1 (b) Damaged (b) Damaged Regularized f (d) Restored, N = 2 y = using + w Fig. 14. Inpainting f0 N = 2 and n = 16 × 16 (bottom-right image), or N = 1 and n = 8 × 8 (bottom-left). J = 100 iterations were performed, producing an adaptive dictionary. During the learning, 50% of the patches were used. A sparsity factor L = 10 has been used during the learning process and L = 25 for the ﬁnal reconstruction. The damaged image was created by removing 75% of the data from the original image. The initial PSNR is 6.13dB. The resulting PSNR for N = 2 is[Mairal et al. 2008] 33.97dB and 31.75dB for N = 1.
83. 83. Adaptive Inpainting and Separation Wavelets Local DCT Wavelets Local DCT Learned [Peyr´, Fadili, Starck 2010] e
84. 84. OISING ALGORITHM WITH 256 ATOMS OF SIZE 7 7 3 FOR TS ARE THOSE GIVEN BY MCAULEY AND AL [28] WITH THEIR “3 D DICTIONARY. THE BOTTOM-RIGHT ARE THE IMPROVEMENTS OBTAINED WITH THE ADAPTIVE APPRO SENTATION FOR COLOR IMAGE RESTORATION EST RESULTS FOR EACH GROUP. AS CAN BE SEEN, OUR PROPOSED TECHNIQUE CONSISTENTLY PRODUC K-SVD ALGORITHM [2] ON EACH CHANNEL SEPARATELY WITH 8 8 ATOMS. THE BOTTOM-LEFT AR s are reduced with our proposed technique ( Higher Dimensional Learningmples of color artifacts while reconstructing a damaged version of the image (a) without the improvement here proposed ( in the new metric). in our proposed new metric). Both images have been denoised with the same global dictionary.bserves a bias effect in the color from the castle and in some part of the water. What is more, the color of the sky is piecewise constant when MAIRAL rs), which is another artifact our approach corrected. (a) Original. (b) Original algorithm, dB. dB. (c) Proposed algorithm, et al.: SPARSE REPRESENTATION FOR COLOR IMAGE RESTORATIONnaries with 256 atoms learned on a generic database of natural images, with two different sizes of patches. Note the large number of color-less atoms. can have negative values, the vectors are presented scaled and shifted to the [0,255] range per channel: (a) 5 5 3 patches; (b) 8 8 3 patches.OLOR IMAGE RESTORATION 61 Fig. 7. Data set used for evaluating denoising experiments. Learning D (a) Training Image; (b) resulting dictionary; (b) is the dictionary learned in the image in (a). The dictionary is more colored than the global one. TABLE I Fig. 7. Data set used for evaluating denoising experiments. les of color artifacts while reconstructing a damaged version of the image (a) without the improvement here proposedatoms learned on new metric). Fig. 2. Dictionaries with 256 ( in the a generic database of natural images, with two dare reduced with our proposed technique ( TABLE I our proposed new metric). Both images have been denoised negative values, the vectors are presented scaled and shifted to the [0,2 in Since the atoms can have with the same global dictionary.ervesITH 256 ATOMS OF SIZE castle and in3 FOR of the water. What is more, the color of the sky is . EACH constant IS DIVIDED IN FOUR W a bias effect in the color from the 7 7 some part AND 6 6 3 FOR piecewise CASE whenEN BY is another artifact our approach corrected. (a)HEIR “3(b) Original algorithm, HE TOP-RIGHT RESULTS ARE THOSE OBTAINED BY), which MCAULEY AND AL [28] WITH T Original. 3 MODEL.” T dB. (c) Proposed algorithm, 3 MODEL.” THE TOP-RIGHT RESUL dB. AND 6M [2] ON EACH CHANNEL SEPARATELY WITH 8 8 ATOMS. THE BOTTOM-LEFT ARE OUR RESULTS OBTAINEDE BOTTOM-RIGHT ARE THE IMPROVEMENTS OBTAINED WITH THE ADAPTIVE APPROACH WITH 20 ITERATIONS.EACH GROUP. AS CAN BE SEEN, OUR PROPOSED TECHNIQUE CONSISTENTLY PRODUCES THE BEST RESULTS 6 3 FOR . EAC
85. 85. OISING ALGORITHM WITH 256 ATOMS OF SIZE 7 7 3 FOR TS ARE THOSE GIVEN BY MCAULEY AND AL [28] WITH THEIR “3 D DICTIONARY. THE BOTTOM-RIGHT ARE THE IMPROVEMENTS OBTAINED WITH THE ADAPTIVE APPRO SENTATION FOR COLOR IMAGE RESTORATION EST RESULTS FOR EACH GROUP. AS CAN BE SEEN, OUR PROPOSED TECHNIQUE CONSISTENTLY PRODUC K-SVD ALGORITHM [2] ON EACH CHANNEL SEPARATELY WITH 8 8 ATOMS. THE BOTTOM-LEFT AR Higher Dimensional Learning O NLINE L EARNING FOR M ATRIX FACTORIZATION AND FACTORIZATION AND S PARSE C ODING O NLINE L EARNING FOR M ATRIX S PARSE C ODINGmples of color artifacts while reconstructing a damaged version of the image (a) without the improvement here proposed ( s are reduced with our proposed technique ( in the new metric). in our proposed new metric). Both images have been denoised with the same global dictionary.bserves a bias effect in the color from the castle and in some part of the water. What is more, the color of the sky is piecewise constant when MAIRAL rs), which is another artifact our approach corrected. (a) Original. (b) Original algorithm, dB. dB. (c) Proposed algorithm, et al.: SPARSE REPRESENTATION FOR COLOR IMAGE RESTORATIONnaries with 256 atoms learned on a generic database of natural images, with two different sizes of patches. Note the large number of color-less atoms. can have negative values, the vectors are presented scaled and shifted to the [0,255] range per channel: (a) 5 5 3 patches; (b) 8 8 3 patches.OLOR IMAGE RESTORATION 61 Fig. 7. Data set used for evaluating denoising experiments. Learning D (a) Training Image; (b) resulting dictionary; (b) is the dictionary learned in the image in (a). The dictionary is more colored than the global one. TABLE I Fig. 7. Data set used for evaluating denoising experiments. les of color artifacts while reconstructing a damaged version of the image (a) without the improvement here proposedatoms learned on new metric). Fig. 2. Dictionaries with 256 ( in the a generic database of natural images, with two dare reduced with our proposed technique ( TABLE I our proposed new metric). Both images have been denoised negative values, the vectors are presented scaled and shifted to the [0,2 in Since the atoms can have with the same global dictionary.ervesITH 256 ATOMS OF SIZE castle and in3 FOR of the water. What is more, the color of the sky is . EACH constant IS DIVIDED IN FOUR W a bias effect in the color from the 7 7 some part AND 6 6 3 FOR piecewise CASE whenEN BY is another artifact our approach corrected. (a)HEIR “3(b) Original algorithm, HE TOP-RIGHT RESULTS ARE THOSE OBTAINED BY), which MCAULEY AND AL [28] WITH T Original. 3 MODEL.” T dB. (c) Proposed algorithm, 3 MODEL.” THE TOP-RIGHT RESUL dB. AND 6M [2] ON EACH CHANNEL SEPARATELY WITH 8 8 ATOMS. THE BOTTOM-LEFT ARE OUR RESULTS OBTAINEDE BOTTOM-RIGHT ARE THE IMPROVEMENTS OBTAINED WITH THE ADAPTIVE APPROACH WITH 20 ITERATIONS. InpaintingEACH GROUP. AS CAN BE SEEN, OUR PROPOSED TECHNIQUE CONSISTENTLY PRODUCES THE BEST RESULTS 6 3 FOR . EAC
86. 86. Movie Inpainting