Upcoming SlideShare
×

# Talk icml

433 views

Published on

1 Like
Statistics
Notes
• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

Views
Total views
433
On SlideShare
0
From Embeds
0
Number of Embeds
43
Actions
Shares
0
12
0
Likes
1
Embeds 0
No embeds

No notes for slide

### Talk icml

1. 1. Online Dictionary Learning for Sparse Coding Julien Mairal1 Francis Bach1 Jean Ponce2 Guillermo Sapiro3 1 INRIA–Willow project 2 Ecole Normale Supérieure 3 University of Minnesota ICML, Montréal, June 2008
2. 2. What this talk is about Learning eﬃciently dictionaries (basis set) for sparse coding. Solving a large-scale matrix factorization problem. Making some large-scale image processing problems tractable. Proposing an algorithm which extends to NMF, sparse PCA,. . .
3. 3. 1 The Dictionary Learning Problem2 Online Dictionary Learning3 Extensions
4. 4. 1 The Dictionary Learning Problem2 Online Dictionary Learning3 Extensions
5. 5. The Dictionary Learning Problem y = xorig + w measurements original image noise
6. 6. The Dictionary Learning Problem[Elad & Aharon (’06)] Solving the denoising problem Extract all overlapping 8 × 8 patches xi . Solve a matrix factorization problem: n 1 min ||x − Dαi ||2 + λ||αi ||1 , αi ,D∈C i=1 2 i 2 sparsity reconstruction with n > 100, 000 Average the reconstruction of each patch.
7. 7. The Dictionary Learning Problem[Mairal, Bach, Ponce, Sapiro & Zisserman (’09)] Denoising result
8. 8. The Dictionary Learning Problem[Mairal, Sapiro & Elad (’08)] Image completion example
9. 9. The Dictionary Learning ProblemWhat does D look like?
10. 10. The Dictionary Learning Problem n 1 min ||xi − Dαi ||2 + λ||αi ||1 2 α∈R i=1 2 k×n D∈C C = {D ∈ Rm×k s.t. ∀j = 1, . . . , k, ||dj ||2 ≤ 1}. Classical optimization alternates between D and α. Good results, but very slow!
11. 11. 1 The Dictionary Learning Problem2 Online Dictionary Learning3 Extensions
12. 12. Online Dictionary Learning Classical formulation of dictionary learning 1 n min fn (D) = min l(xi , D), D∈C D∈C n i=1 where 1 l(x, D) = mink ||x − Dα||2 + λ||α||1 . 2 α∈R 2
13. 13. Online Dictionary Learning Which formulation are we interested in? 1 n min f (D) = Ex [l(x, D)] ≈ lim l(xi , D) D∈C n→+∞ n i=1
14. 14. Online Dictionary Learning Online learning can handle potentially inﬁnite datasets, adapt to dynamic training sets, be dramatically faster than batch algorithms [Bottou & Bousquet (’08)].
15. 15. Online Dictionary LearningProposed approach 1: for t=1,. . . ,T do 2: Draw xt 3: Sparse Coding 1 αt ← arg min ||xt − Dt−1 α||2 + λ||α||1 , 2 α∈Rk 2 4: Dictionary Learning 1 t 1 Dt ← arg min ||xi −Dαi ||2 +λ||αi ||1 , 2 D∈C t i=1 2 5: end for
16. 16. Online Dictionary LearningProposed approach Implementation details Use LARS for the sparse coding step, Use a block-coordinate approach for the dictionary update, with warm restart, Use a mini-batch.
17. 17. Online Dictionary LearningProposed approach Which guarantees do we have? Under a few reasonable assumptions, ˆ we build a surrogate function ft of the expected cost f verifying ˆ lim ft (Dt ) − f (Dt ) = 0, t→+∞ Dt is asymptotically close to a stationary point.
18. 18. Online Dictionary LearningExperimental results, batch vs online m = 8 × 8, k = 256
19. 19. Online Dictionary LearningExperimental results, batch vs online m = 12 × 12 × 3, k = 512
20. 20. Online Dictionary LearningExperimental results, batch vs online m = 16 × 16, k = 1024
21. 21. Online Dictionary LearningExperimental results, ODL vs SGD m = 8 × 8, k = 256
22. 22. Online Dictionary LearningExperimental results, ODL vs SGD m = 12 × 12 × 3, k = 512
23. 23. Online Dictionary LearningExperimental results, ODL vs SGD m = 16 × 16, k = 1024
24. 24. Online Dictionary LearningInpainting a 12-Mpixel photograph
25. 25. Online Dictionary LearningInpainting a 12-Mpixel photograph
26. 26. Online Dictionary LearningInpainting a 12-Mpixel photograph
27. 27. Online Dictionary LearningInpainting a 12-Mpixel photograph
28. 28. 1 The Dictionary Learning Problem2 Online Dictionary Learning3 Extensions
29. 29. Extension to NMF and sparse PCA NMF extension 1n min ||xi − Dαi ||2 s.t. αi ≥ 0, 2 D ≥ 0. α∈R i=1 2 k×n D∈C SPCA extension n 1 min ||xi − Dαi ||2 + λ||α1 ||1 2 α∈Rk×n i=1 2 D∈C C = {D ∈ Rm×k s.t. ∀j ||dj ||2 + γ||dj ||1 ≤ 1}. 2
30. 30. Extension to NMF and sparse PCAFaces: Extended Yale Database B (a) PCA (b) NNMF (c) DL
31. 31. Extension to NMF and sparse PCAFaces: Extended Yale Database B (d) SPCA, τ = 70% (e) SPCA, τ = 30% (f) SPCA, τ = 10%
32. 32. Extension to NMF and sparse PCANatural Patches (a) PCA (b) NNMF (c) DL
33. 33. Extension to NMF and sparse PCANatural Patches (d) SPCA, τ = 70% (e) SPCA, τ = 30% (f) SPCA, τ = 10%
34. 34. Conclusion Take-home message Online techniques are adapted to the dictionary learning problem. Our method makes some large-scale image processing tasks tractable—. . . . . . — and extends to various matrix factorization problems.