This document presents a method for nonconvex compressed sensing using the sum-of-squares (SoS) method. It formulates q-minimization, which requires fewer samples than l1-minimization but is nonconvex, as a polynomial optimization problem. The SoS method is then applied to obtain a pseudoexpectation operator satisfying a pseudo robust null space property, guaranteeing stable signal recovery. Specifically, it shows that for a Rademacher measurement matrix, with the number of measurements scaling quadratically in the sparsity s, the SoS method finds a solution x^ satisfying ||x^-x||_q ≤ O(σs(x)q) + ε, providing nearly q-stable recovery.

1. Nonconvex Compressed Sensing
with the Sum-of-Squares Method
Tasuku Soma (Univ. Tokyo)
Joint work with:
Yuichi Yoshida (NII&PFI)
1 / 21

2. 1 Introduction
2 Sum of Squares Method
3 Nonconvex Compressed Sensing
2 / 21

3. 1 Introduction
2 Sum of Squares Method
3 Nonconvex Compressed Sensing
3 / 21

4. Compressed Sensing
Given: A ∈ Rm×n
(m n) and y = Ax,
Task: Estimate the original sparse signal x ∈ Rn
.
y = A x
4 / 21

5. Compressed Sensing
Given: A ∈ Rm×n
(m n) and y = Ax,
Task: Estimate the original sparse signal x ∈ Rn
.
y = A x
Applications:
Image Processing, Statistics, Machine Learning...
4 / 21

6. 1 Minimization (Basis Pursuit)
0 1
min z 1 sub. to Az = y
5 / 21

7. 1 Minimization (Basis Pursuit)
0 1
min z 1 sub. to Az = y
• Convex relaxation for 0 minimization
• For a subgaussian A with m = Ω(s log n
s
),
1 minimization reconstructs x.
[Cand`es-Romberg-Tao ’06, Donoho ’06]
s: sparsity of x (maybe unknown)
5 / 21

8. Nonconvex Compressed Sensing
0 1/2 1
q minimization (0 < q ≤ 1):
[Laska-Davenport-Baraniuk ’09,Cherian-Sra-Papanikolopoulos ’11]
min z
q
q sub. to Az = y
6 / 21

9. Nonconvex Compressed Sensing
0 1/2 1
q minimization (0 < q ≤ 1):
[Laska-Davenport-Baraniuk ’09,Cherian-Sra-Papanikolopoulos ’11]
min z
q
q sub. to Az = y
• Requires fewer samples than 1 minimization
• Recovers arbitrary sparse signals as q → 0
• Nonconvex Optimization!
6 / 21

10. Stable Signal Recovery
x needs not to be sparse but close to a sparse signal.
7 / 21

11. Stable Signal Recovery
x needs not to be sparse but close to a sparse signal.
p-stable recovery (0 < p ≤ ∞): Algorithms that output
ˆx satisfying
ˆx − x p ≤ O(σs(x)p)
for any x ∈ Rn
.
7 / 21

12. Stable Signal Recovery
x needs not to be sparse but close to a sparse signal.
p-stable recovery (0 < p ≤ ∞): Algorithms that output
ˆx satisfying
ˆx − x p ≤ O(σs(x)p)
for any x ∈ Rn
. p distance to s-sparse vector
7 / 21

13. Stable Signal Recovery
x needs not to be sparse but close to a sparse signal.
p-stable recovery (0 < p ≤ ∞): Algorithms that output
ˆx satisfying
ˆx − x p ≤ O(σs(x)p)
for any x ∈ Rn
. p distance to s-sparse vector
• If A is a subgaussian matrix with m = Ω(s log n
s
),
1 minimization is 1-stable
[Cand`es-Romberg-Tao ’06,Cand`es ’08]
• For same A, q minimization is q-stable
(0 < q ≤ 1) [Cohen-Dahmen-DeVore ’09]
7 / 21

14. Our Result
Theorem
For x ∞ ≤ 1, A: Rademacher matrix, and fixed
q = 2−k
, there is a polytime algorithm for finding ˆx s.t.
ˆx − x q ≤ O(σs(x)q) + ε,
provided that m = Ω(s2/q
log n).
Aij ∼ {±1/
√
m}
• (Nearly) q-stable recovery
• #samples >> O(s log(n/s)) (Sample Complexity
Trade Off)
• Use of SoS Method and Ellipsoid Method
8 / 21

15. 1 Introduction
2 Sum of Squares Method
3 Nonconvex Compressed Sensing
9 / 21

16. SoS Method [Lasserre ’06, Parrilo ’00, Nesterov ’00, Shor ’87]
Polynomial Optimization: f, g1, . . . , gi ∈ R[z]: polynomials
min
z
f(z)
sub. to gi(z) = 0 (i = 1, . . . , m)
10 / 21

17. SoS Method [Lasserre ’06, Parrilo ’00, Nesterov ’00, Shor ’87]
Polynomial Optimization: f, g1, . . . , gi ∈ R[z]: polynomials
min
z
f(z)
sub. to gi(z) = 0 (i = 1, . . . , m)
SoS Relaxation (of degree d):
min
E
E[f(z)]
sub. to E : R[z]d → R, linear operator “pseudoexpectation”
E[1] = 1
E[p(z)2
] ≥ 0 (p ∈ R[z] : deg(p) ≤ d/2)
E[gi(z)p(z)] = 0 (p ∈ R[z] : deg(gip) ≤ d, i = 1, . . . , m)
10 / 21

18. Facts on SoS Method
• The SoS Relaxation (of degree d) reduces to Semidefinite
Programming (SDP) with nO(d)
-size matrix.
11 / 21

19. Facts on SoS Method
• The SoS Relaxation (of degree d) reduces to Semidefinite
Programming (SDP) with nO(d)
-size matrix.
• Dual View: SoS Proof System
Any (low-degree) “proof” in SoS proof system yields an
algorithm via the SoS method.
11 / 21

20. Facts on SoS Method
• The SoS Relaxation (of degree d) reduces to Semidefinite
Programming (SDP) with nO(d)
-size matrix.
• Dual View: SoS Proof System
Any (low-degree) “proof” in SoS proof system yields an
algorithm via the SoS method.
• Very Powerful Tool in Computer Science:
• Subexponential Alg. for UG [Arora-Barak-Steurer’10]
• Planted Sparse Vector [Barak-Kelner-Steurer’14]
• Sparse PCA [Ma-Wigderson’14]
11 / 21

21. 1 Introduction
2 Sum of Squares Method
3 Nonconvex Compressed Sensing
12 / 21

22. Basic Idea
Formulate q minimization as polynomial optimization:
min z
q
q sub. to Az = y
Note: |z(i)|q
is not a polynomial, but representable by lifting;
|z(i)|
4
= z(i)2
, |z(i)| ≥ 0
13 / 21

23. Basic Idea
Formulate q minimization as polynomial optimization:
min z
q
q sub. to Az = y
Note: |z(i)|q
is not a polynomial, but representable by lifting;
|z(i)|
4
= z(i)2
, |z(i)| ≥ 0
×Solutions of SoS method do not satisfy triangle inequalities:
E z + x
q
q E z
q
q + x
q
q
13 / 21

24. Basic Idea
Formulate q minimization as polynomial optimization:
min z
q
q sub. to Az = y
Note: |z(i)|q
is not a polynomial, but representable by lifting;
|z(i)|
4
= z(i)2
, |z(i)| ≥ 0
×Solutions of SoS method do not satisfy triangle inequalities:
E z + x
q
q E z
q
q + x
q
q
Add Valid Constraints!
min z
q
q s.t. Az = y, Valid Constraints
13 / 21

25. Triangle Inequalities
z + x
q
q ≤ z
q
q + x
q
q
We have to add |z(i) + x(i)|q
, but do not know x(i).
14 / 21

26. Triangle Inequalities
z + x
q
q ≤ z
q
q + x
q
q
We have to add |z(i) + x(i)|q
, but do not know x(i).
Idea: Using Grid
L: set of multiples of δ in [−1, 1]. -1 0 1δ
• new variable for |z(i) − b|q
(b ∈ L)
• triangle inequalities for
|z(i) − b|q
, |z(i) − b |q
, and |b − b |q
(b, b ∈ L)
14 / 21

27. Robust q Minimization
Instead of x, we will find xL
∈ Ln
closest to x.
Robust q Minimization
min z
q
q s.t. y − Az 2
2 ≤ η2
η = σmax(A)
√
sδ
15 / 21

28. Robust q Minimization
Instead of x, we will find xL
∈ Ln
closest to x.
Robust q Minimization
min z
q
q s.t. y − Az 2
2 ≤ η2
η = σmax(A)
√
sδ
q Robust Null Space Property
vS
q
q ≤ ρ vS
q
q + τ Av
q
2
for any v and S ⊆ [n] with |S| ≤ s.
15 / 21

29. Robust q Minimization
Instead of x, we will find xL
∈ Ln
closest to x.
Robust q Minimization
min z
q
q s.t. y − Az 2
2 ≤ η2
η = σmax(A)
√
sδ
q Pseudo Robust Null Space Property ( q-PRNSP)
E vS
q
q ≤ ρ E vS
q
q + τ E Av 2
2
q/2
for any v = z − b (b ∈ Ln
) and S ⊆ [n] with |S| ≤ s.
15 / 21

30. PRNSP =⇒ Stable Recovery
Theorem
If E satisfies q-PRNSP, then
E z − xL q
q ≤
2(1 + ρ)
1 − ρ
σs(xL
)q
q +
21+q
τ
1 − ρ
ηq
,
where xL
is the closest vector in Ln
to x.
Proof Idea:
A proof of stability only needs:
•
q
q triangle inequalities for z − xL
, x and z + xL
• 2 triangle inequality
16 / 21

31. Rounding
Extract an actual vector ˆx from a pseudoexpectation E.
ˆx(i) := argmin
b∈L
E|z(i) − b|q
(i = 1, . . . , n)
Theorem
If E satisfies PRNSP,
ˆx − xL q
q ≤ 2
2(1 + ρ)
1 − ρ
σs(xL
)q
q +
21+q
τ
1 − ρ
ηq
17 / 21

32. Imposing PRNSP
How can we obtain E satisfying PRNSP?
Idea: Follow known proofs for robust NSP!
• From Restricted Isometry Property (RIP)
[Cand`es ’08]
• From Coherence
[Gribonval-Nielsen ’03, Donoho-Elad ’03]
• From Lossless Expander [Berinde et al. ’08]
18 / 21

33. Coherence
The coherence of a matrix A = [a1 . . . an] is
µ = max
i j
| ai, aj |
ai 2 aj 2
Facts:
• If µq
< 1
2s
, q Robust NSP holds.
• If A is a Rademacher matrix with
m = O(s2/q
log n), then µq
< 1
2s
w.h.p.
19 / 21

34. Small Coherence =⇒ PRNSP
Issue: Naive import needs exponentially many
variables and constraints!
Lemma
If A is a Rademacher matrix,
• additinal variables are polynomially many
• additional constraints have a separation oracle
Thus ellipsoid methods find E with PRNSP.
20 / 21

35. Our Result
Theorem
For x ∞ ≤ 1, A: Rademacher matrix, and fixed
q = 2−k
, there is a polytime algorithm for finding ˆx s.t.
ˆx − x q ≤ O(σs(x)q) + ε,
provided that m = Ω(s2/q
log n).
Aij ∼ {±1/
√
m}
• (Nearly) q-stable recovery
• #samples >> O(s log(n/s)) (Sample Complexity
Trade Off)
• Use of SoS Method and Ellipsoid Method
21 / 21

36. Putting Things Together
Using a Rademacher matrix yields PRNSP:
E vS
q
q ≤ O(1) · E vS
q
q + O(s) · E Av 2
2
q/2
22 / 21

37. Putting Things Together
Using a Rademacher matrix yields PRNSP:
E vS
q
q ≤ O(1) · E vS
q
q + O(s) · E Av 2
2
q/2
This guarantees:
ˆx − x
q
q ≤ O(σs(xL
)q
q) + O(s) · ηq
22 / 21

38. Putting Things Together
Using a Rademacher matrix yields PRNSP:
E vS
q
q ≤ O(1) · E vS
q
q + O(s) · E Av 2
2
q/2
This guarantees:
ˆx − x
q
q ≤ O(σs(xL
)q
q) + O(s) · ηq
Theorem
If we take δ small, then the rounded vector ˆx satisfies
ˆx − x
q
q ≤ O(σs(x)q
q) + ε.
(pf) η = σmax(A)
√
sδ and σmax(A) = O(n/m)
22 / 21