【NeurIPS 2018 読み会 in 京都】 Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds https://papers.nips.cc/paper/8120-theoretical-linear-convergence-of-unfolded-ista-and-its-practical-weights-and-thresholds

1. 1

2. n
• ~2013.3, PhD@ ,
• 2013.4~2016.3, @IBM
• 2016.4~2017.8, @ ERATO, NII
• 2017.9~, @ ,
n
•
ECML’11
AISTATS’15,17
•
AISTATS’18
AAAI’17,18
2

3. Cont.
n
• ℓ"
n
• Lasso
min
&
1
2
) − +, -
+ / , "
3

4. Cont.
n
• ℓ"
n
• Lasso
min
&
1
2
) − +, -
+ / , "
n
• Lasso
4

5. n
• Sparse Coding
Learned ISTA (LISTA)
n Sparse Coding
• Lasso
n LISTA ISTA
• ISTA: Iterative Soft-Threshold Algorithm
proximal gradient descent
5
[Gregor and LeCun, ICML’10]

6. Sparse Coding ISTA
n /
n ISTA Lasso
6
ISTA

7. Sparse Coding ISTA
n /
n ISTA Lasso
7
ISTA

8. Sparse Coding Learned ISTA
n Learned ISTA (LISTA) [Gregor and LeCun, ICML’10]
ISTA
n LISTA
•
•
•
8
MNIST LISTA

9. n
n
• ISTA
• Learned ISTA (LISTA)
n
n LISTA
n
n
9

10. Iterative Soft-Thresholding Algorithm (ISTA)
n
• !∗ ∈ ℝ% ←
n
• & ∈ ℝ' ←
• / ( ∈ ℝ'×% ←
!∗ & = (!∗ + ,
• !∗
n Lasso
-! = argmin
4
1
2
& − (! 8 + 9 ! :
10

11. Iterative Soft-Thresholding Algorithm (ISTA)
n
!" = argmin
*
1
2
- − /" 0 + 2 " 3
n ISTA Lasso
• " ← 06
• "
3
0
- − /" 0 "
• " ← " − 7 /8/" − /8- = 96 − 7/8/ " + 7/8-
"
• " ← ℎ;< " = sign " ⊙ max {0, abs " − 72}
11→

12. Learned ISTA (LISTA) [Gregor and LeCun, ICML’10]
n ISTA
• !" = ℎ%&
'()
→ !+ = ℎ%,
-"!" + '")
→ !/ = ℎ%0
-+!+ + '+)
→ …
n ISTA
12
-2 = 34 − 6787
'2 = 678
92 = :;
!"
)
!+ !/-" ℎ%, -+ ℎ%0
'(
'" '+
←
↑
ℎ%&
ISTA 3
4 DNN

13. Learned ISTA (LISTA) [Gregor and LeCun, ICML’10]
n LISTA
• !" = $"(&; (, *, +) K+1 DNN
• ISTA -! DNN !"
(, *, + DNN
13
-!
(, *, +
DNN
!.
/
!0 !1(. ℎ34 (0 ℎ35
*6
*. *0
←
↑
ℎ37

14. Learned ISTA (LISTA) [Gregor and LeCun, ICML’10]
n LISTA -
• ! "#
{ !%, "#% }%()
*
• K+1 DNN #+ = -+(!; 0, 1, 2)
•
min
7,8,9
1
2
<=, "> "# − -+(!; 0, 1, 2) @
n LISTA -
• #+ = -+(!; 0, 1, 2)
DNN
14
SGD
MNIST LISTA

15. n
n
• ISTA
• Learned ISTA (LISTA)
n
n LISTA
n
n
15

16. Research Question
LISTA
n 10~20
n
• ! = #$∗ + ' (($∗, ')
• (($∗
, ') !′ ISTA
-$′
16
ISTA 100

17. n
• !∗ #
!∗
$ ≤ &, !∗
( ≤ ), # * ≤ +
n
• ,- = / − 1-2 i.e. !-3* = ℎ56
!- + 1- 8 − 2!-
• s 9 !-
:, ; > 0 1-, >-
!- − !∗
? ≤ )&exp −:9 + ;+
17
s
8 = 2!∗
+ #
!-3* = ℎ56
,!- + 1-8

18. n
• !∗ #
!∗
$ ≤ &, !∗
( ≤ ), # * ≤ +
n
• ,- = / − 12- i.e. !34* = ℎ67
!3 + 2- 9 − 1!3
• : !3 ;, < > 0
2-, ?-
!3 − !∗
@ ≤ )&exp −;: + <+
s
9 = 1!∗
+ #
18
LISTA DNN

19. n
• !" = $ − &"' i.e. (")* = ℎ,-
(" + &" / − '("
• s 0 ("
1, 3 > 0 &", 6"
(" − (∗
8 ≤ :;exp −10 + 3?
n @ = supp (∗
• @ = supp (∗ : = D: (E
∗
≠ 0
n 2
• &E, 6E
1. (" E = 0, ∀D ∉ @, ∀0
2. (" − (∗
* 0
19

20. n Notation
• !
! " #
! $ %
• &
&"' (#, *)
&",: #
&:,' *
&:,$ %
• -. = 01 sup
5∗,7
8. − 8∗
: + <=>
01: = max
"B'
C. ",:D:,' <=: = max
",'
C. "'
20

21. 1.
n DNN
• !"#$ = ℎ'(
!" + *" + − -!"
n !" . = 0, ∀2 ∉ 4
• + = -!∗ + 6 ∀2 ∉ 4
!"#$ . = ℎ'(
!" . + *" .,:- !∗ − !" + *" .,:6
= ℎ'(
*" .,:-:,8 !∗ − !" 8 + *" .,:6
n 9" = :; sup
?∗,@
!" − !∗
$ + ABC
• :;: = max
.GH
*" .,:-:,H AB: = max
.,H
*" .H
*" .,:-:,8 !∗ − !" 8 + *" .,:6 ≤ 9" ⇒ !"#$ . = 0
21≤ :; sup
?∗,@
!" − !∗
$ ≤ ABC

22. 2. !" − !∗
% &
n ' ∈ )
!"*% + = ℎ./
!" + + 1" +,:4:,5 !∗ − !" 5 + 1" +,:6
n 1" +,:4:,+ = 1 1"
!" + + 1" +,:4:,5 !∗ − !" 5
= !" + + 1" +,:4:,5∖+ !∗ − !" 5∖+ + !∗ − !" +
= !∗
+ + 1" +,:4:,5∖+ !∗ − !" 5∖+
!"*% + − !∗
+ − 1" +,:4:,5∖+ !∗ − !" 5∖+ − 1" +,:6 ≤ :"
!"*% + − !∗
+
≤ 1" +,:4:,5∖+ !∗ − !" 5∖+ + 1" +,:6 + :"
≤ ;< !∗ − !" 5∖+ %
+ =>? + :"
22
ℎ./
:"
shrink

23. 2. !" − !∗
% &
n !"'% − !∗
%
!"'% − !∗
% ≤ ∑*∈, !"'% * − !∗
*
≤ ∑*∈, -. !∗ − !" ,∖* %
+ 123 + 4"
≤ -. 5 − 1 !∗ − !" % + 5 4" + 5 123
n sup
sup
:∗,<
!"'% − !∗
%
≤ 2-.> − -. sup
:∗,<
!∗ − !" % + 2>123
≤ 2-.> − -. "'%sup
:∗,<
!∗ − !? % + 2>123 ∑@A?
"'%
2-.> − -. @
≤ 2-.> − -. ">B + 13
≤ >Bexp −E& + 13
23
E = −log 2-.> − -.
1 =
JKLM
%'NOPJNOK2 -.> − -. < 1
=
!* R = 0, ∀U ∉ 5
4" = -. sup
:∗,<
!" − !∗
% + 123

24. n LISTA-CP (LISTA with weights coupling)
• LISTA !"#$ = ℎ'(
)"!" + +",
• LISTA-CP !"#$ = ℎ'(
!" + +" , − .!"
n
• LISTA-CP +", 0"
!" − !∗
2 ≤ 45exp −9: + ;<
n LISTA-CP LISTA LISTA
)", +", 0"
24

25. n !∗ !# − !∗
%
n LISTA ISTA &!
!∗ !# − !∗
%
n !# − &! %
• !# − &! %
%
≤ !∗ − &! %
%
+ !# − !∗
%
%
!# − &! % tight
25

26. n
n
• ISTA
• Learned ISTA (LISTA)
n
n LISTA
n
n
26

27. LISTA
n ℎ"
• ℎ" # $
= sign #$ ⊙ max #$ − /
n Support Selection ℎ"
0
• #1 #1
→ shrink
• ℎ"
0
#
$
= 2
#$
ℎ" # $
27
#
shrink
#1 3% shrink
shrink

28. LISTA
n LISTA-CPSS
• s ! "#
$%% ≥ $, (%% ≤ ( *#, +#
"# − "∗
. ≤ /0exp −$%%! + (%%5
28
Weights Coupling Weights Coupling
Support Selection LISTA
"#67 = ℎ:;
<#"# + *#=
LISTA-CP
"#67 = ℎ:;
"# + *#(= − ?"#)
Support Selection LISTA-SS
"#67 = ℎ:;
A
<#"# + *#=
LISTA-CPSS
"#67 = ℎ:;
A
"# + *#(= − ?"#)
LIST-CP

29. n
n
• ISTA
• Learned ISTA (LISTA)
n
n LISTA
n
n
29

30. n
• !∗ 500
• # 250
• $
n LISTA-CP
30
NMSE
/
ISTA, FISTA
AMP
LISTA
LISTA-CP
NMSE
/
ISTA, FISTA
AMP
LISTA
LISTA-CP
LIST-CP
https://github.com/xchen-tamu/linear-lista-cpss

31. n
• !∗ 500
• # 250
• $
n LISTA-CPSS
31
LIST-CPSS
NMSE
/
LISTA-CPSS
NMSE
LISTA-CPSS
/
https://github.com/xchen-tamu/linear-lista-cpss

32. n Research Question
• Learned ISTA (LISTA)
n
• LISTA
• LISTA weights coupling (CP), support
selection (SS)
n LISTA-CPSS
32