The document proposes an algorithm called Robust Tensor Decomposition (RTD) to recover a low-rank tensor from observations that are corrupted by block sparse errors. RTD iteratively estimates the low-rank and sparse components using hard thresholding and tensor projections. It compares RTD to existing approaches on video sequences, finding RTD yields more accurate recovery with fewer assumptions.

3. Rn
Rn1⇥n2
Rn1⇥n2⇥n3
Rn⇥n⇥n

4. 1 2 r+ + +· · ·=
=
rX
i=1
iui ⌦ ui ⌦ ui
=
rX
i=1
iu⌦3
i

5. = +
L S
(= U⌃V T
)

6. (a) Original frames (b) Low-rank ˆL (c) Sparse ˆS (d) Low-rank ˆL (e) Sparse ˆS
Convex optimization (this work) Alternating minimization [47]
Figure 2: Background modeling from video. Three frames from a 200 frame video sequence
taken in an airport [31]. (a) Frames of original video M. (b)-(c) Low-rank ˆL and sparse
components ˆS obtained by PCP, (d)-(e) competing approach based on alternating minimization
of an m-estimator [47]. PCP yields a much more appealing result despite using less prior
knowledge.
Figure 2 (d) and (e) compares the result obtained by Principal Component Pursuit to a state-of-
the-art technique from the computer vision literature, [47].12 That approach also aims at robustly
recovering a good low-rank approximation, but uses a more complicated, nonconvex m-estimator,
which incorporates a local scale estimate that implicitly exploits the spatial characteristics of natural
images. This leads to a highly nonconvex optimization, which is solved locally via alternating
minimization. Interestingly, despite using more prior information about the signal to be recovered,
this approach does not perform as well as the convex programming heuristic: notice the large
artifacts in the top and bottom rows of Figure 2 (d).
In Figure 3, we consider 250 frames of a sequence with several drastic illumination changes.
Here, the resolution is 168 ⇥ 120, and so M is a 20, 160 ⇥ 250 matrix. For simplicity, and to
illustrate the theoretical results obtained above, we again choose = 1/
p
n1.13 For this example,
on the same 2.66 GHz Core 2 Duo machine, the algorithm requires a total of 561 iterations and 36
(a) Original frames (b) Low-rank ˆL (c) Sparse ˆS (d) Low-rank ˆL (e) Sparse ˆS
Convex optimization (this work) Alternating minimization [47]
Figure 2: Background modeling from video. Three frames from a 200 frame video sequence
taken in an airport [31]. (a) Frames of original video M. (b)-(c) Low-rank ˆL and sparse
components ˆS obtained by PCP, (d)-(e) competing approach based on alternating minimization
of an m-estimator [47]. PCP yields a much more appealing result despite using less prior
knowledge.
(a) Original frames (b) Low-rank ˆL (c) Sparse ˆS (d) Low-rank ˆL (e) Sparse ˆS
Convex optimization (this work) Alternating minimization [47]
Figure 2: Background modeling from video. Three frames from a 200 frame video sequence
taken in an airport [31]. (a) Frames of original video M. (b)-(c) Low-rank ˆL and sparse
components ˆS obtained by PCP, (d)-(e) competing approach based on alternating minimization
of an m-estimator [47]. PCP yields a much more appealing result despite using less prior
knowledge.
(a) Original frames (b) Low-rank ˆL (c) Sparse ˆS (d) Low-rank ˆL (e) Sparse ˆS
Convex optimization (this work) Alternating minimization [47]
Figure 2: Background modeling from video. Three frames from a 200 frame video sequence
taken in an airport [31]. (a) Frames of original video M. (b)-(c) Low-rank ˆL and sparse
components ˆS obtained by PCP, (d)-(e) competing approach based on alternating minimization
of an m-estimator [47]. PCP yields a much more appealing result despite using less prior
knowledge.
Figure 2 (d) and (e) compares the result obtained by Principal Component Pursuit to a state-of-
the-art technique from the computer vision literature, [47].12 That approach also aims at robustly
recovering a good low-rank approximation, but uses a more complicated, nonconvex m-estimator,
which incorporates a local scale estimate that implicitly exploits the spatial characteristics of natural
images. This leads to a highly nonconvex optimization, which is solved locally via alternating
minimization. Interestingly, despite using more prior information about the signal to be recovered,
this approach does not perform as well as the convex programming heuristic: notice the large
artifacts in the top and bottom rows of Figure 2 (d).
In Figure 3, we consider 250 frames of a sequence with several drastic illumination changes.
Here, the resolution is 168 ⇥ 120, and so M is a 20, 160 ⇥ 250 matrix. For simplicity, and to
illustrate the theoretical results obtained above, we again choose = 1/
p
n1.13 For this example,
on the same 2.66 GHz Core 2 Duo machine, the algorithm requires a total of 561 iterations and 36
(a) Original frames (b) Low-rank ˆL (c) Sparse ˆS (d) Low-rank ˆL (e) Sparse
Convex optimization (this work) Alternating minimization [4
Figure 2: Background modeling from video. Three frames from a 200 frame video seque
taken in an airport [31]. (a) Frames of original video M. (b)-(c) Low-rank ˆL and spa
components ˆS obtained by PCP, (d)-(e) competing approach based on alternating minimizat
of an m-estimator [47]. PCP yields a much more appealing result despite using less pr
knowledge.
Figure 2 (d) and (e) compares the result obtained by Principal Component Pursuit to a
the-art technique from the computer vision literature, [47].12 That approach also aims at
recovering a good low-rank approximation, but uses a more complicated, nonconvex m-e
which incorporates a local scale estimate that implicitly exploits the spatial characteristics o
images. This leads to a highly nonconvex optimization, which is solved locally via al
minimization. Interestingly, despite using more prior information about the signal to be r
this approach does not perform as well as the convex programming heuristic: notice
artifacts in the top and bottom rows of Figure 2 (d).
In Figure 3, we consider 250 frames of a sequence with several drastic illumination
Here, the resolution is 168 ⇥ 120, and so M is a 20, 160 ⇥ 250 matrix. For simplicity
illustrate the theoretical results obtained above, we again choose = 1/
p
n1.13 For this
on the same 2.66 GHz Core 2 Duo machine, the algorithm requires a total of 561 iteration
(a) Original frames (b) Low-rank ˆL (c) Sparse ˆS (d) Low-rank ˆL (e) Sparse
Convex optimization (this work) Alternating minimization [4
Figure 2: Background modeling from video. Three frames from a 200 frame video seque
taken in an airport [31]. (a) Frames of original video M. (b)-(c) Low-rank ˆL and spa
components ˆS obtained by PCP, (d)-(e) competing approach based on alternating minimizat
of an m-estimator [47]. PCP yields a much more appealing result despite using less pr
knowledge.
Figure 2 (d) and (e) compares the result obtained by Principal Component Pursuit to a
= +

7. 1 2 r+ + +· · ·
= +
T = L + S

8. 1 2 r+ + +· · ·
= +
T = L + S

9. 1 2 r+ + +· · ·
= +
T = L + S

10. T = L + S
L, S ˆL, ˆS
ˆS
ˆL Pl(T ˆS)
ˆS H⇣(T ˆL)
ˆL
ˆS
Pl
H⇣ ⇣

11. Robust Tensor Decomposition under Block Sparse Perturbation
Algorithm 1 (bL, bS) = RTD (T, , r, ): Tensor Ro-
bust PCA
1: Input: Tensor T 2 Rn⇥n⇥n
, convergence crite-
rion , target rank r, thresholding scale parameter
. Pl(A) denote estimated rank-l approximation
of tensor A, and let l(A) denote the estimated
lth
largest eigenvalue using Procedure 1. HT⇣(A)
denotes hard-thresholding, i.e. H⇣(A))ijk = Aijk
if |Aijk| ⇣ and 0 otherwise.
2: Set initial threshold ⇣0 1(T) and estimates
S(0)
= H⇣0 (T L(0)
).
3: for Stage l = 1 to r do
4: for t = 0 to ⌧ = 10 log n T S(0)
2
/ do
5: L(t+1)
= Pl(T S(t)
).
6: S(t+1)
= H⇣(T L(t+1)
).
7: ⇣t+1= ( l+1(T S(t+1)
)+ 1
2
t
l(T S(t+1)
)).
8: If l+1(L(t+1)
) < 2n , then return L(⌧)
, S(⌧)
,
else reset S(0)
= S(⌧)
9: Return: bL = L(⌧)
, bS = S(⌧)
Procedure 1 {ˆLl, ( ˆuj, j)j2[l]} =
(Gradient Ascent method)
1: Input: Symmetric tensor T
rank l, exact rank r, N1 numb
or restarts, N2 number of powe
initialization. Let T1 T.
2: for j = 1, . . . , r do
3: for i = 1, . . . , N1 do
4: ✓ ⇠ N(0, In). Compute
u of Tj(I, I, ✓). Initialize
Tj(u, u, u).
5: repeat
6: v
(t+1)
i Tj(I, v
(t)
i , v
(t)
i )
{Run power method t
ball}
7:
(t+1)
i Tj(v
(t+1)
i , v
(t+1
i
8: until t = N2
9: Pick the best:
arg maxi2[N1] Tj(v
(t+1)
i , v
(t
i
i =
(t+1)
i and vi = v
(t+1
i
10: Deflate: Tj Tj ivi ⌦

12. Robust Tensor Decomposition under Block Sparse Perturbation
Algorithm 1 (bL, bS) = RTD (T, , r, ): Tensor Ro-
bust PCA
1: Input: Tensor T 2 Rn⇥n⇥n
, convergence crite-
rion , target rank r, thresholding scale parameter
. Pl(A) denote estimated rank-l approximation
of tensor A, and let l(A) denote the estimated
lth
largest eigenvalue using Procedure 1. HT⇣(A)
denotes hard-thresholding, i.e. H⇣(A))ijk = Aijk
if |Aijk| ⇣ and 0 otherwise.
2: Set initial threshold ⇣0 1(T) and estimates
S(0)
= H⇣0 (T L(0)
).
3: for Stage l = 1 to r do
4: for t = 0 to ⌧ = 10 log n T S(0)
2
/ do
5: L(t+1)
= Pl(T S(t)
).
6: S(t+1)
= H⇣(T L(t+1)
).
7: ⇣t+1= ( l+1(T S(t+1)
)+ 1
2
t
l(T S(t+1)
)).
8: If l+1(L(t+1)
) < 2n , then return L(⌧)
, S(⌧)
,
else reset S(0)
= S(⌧)
9: Return: bL = L(⌧)
, bS = S(⌧)
Procedure 1 {ˆLl, ( ˆuj, j)j2[l]} =
(Gradient Ascent method)
1: Input: Symmetric tensor T
rank l, exact rank r, N1 numb
or restarts, N2 number of powe
initialization. Let T1 T.
2: for j = 1, . . . , r do
3: for i = 1, . . . , N1 do
4: ✓ ⇠ N(0, In). Compute
u of Tj(I, I, ✓). Initialize
Tj(u, u, u).
5: repeat
6: v
(t+1)
i Tj(I, v
(t)
i , v
(t)
i )
{Run power method t
ball}
7:
(t+1)
i Tj(v
(t+1)
i , v
(t+1
i
8: until t = N2
9: Pick the best:
arg maxi2[N1] Tj(v
(t+1)
i , v
(t
i
i =
(t+1)
i and vi = v
(t+1
i
10: Deflate: Tj Tj ivi ⌦

13. Robust Tensor Decomposition under Block Sparse Perturbation
Algorithm 1 (bL, bS) = RTD (T, , r, ): Tensor Ro-
bust PCA
1: Input: Tensor T 2 Rn⇥n⇥n
, convergence crite-
rion , target rank r, thresholding scale parameter
. Pl(A) denote estimated rank-l approximation
of tensor A, and let l(A) denote the estimated
lth
largest eigenvalue using Procedure 1. HT⇣(A)
denotes hard-thresholding, i.e. H⇣(A))ijk = Aijk
if |Aijk| ⇣ and 0 otherwise.
2: Set initial threshold ⇣0 1(T) and estimates
S(0)
= H⇣0 (T L(0)
).
3: for Stage l = 1 to r do
4: for t = 0 to ⌧ = 10 log n T S(0)
2
/ do
5: L(t+1)
= Pl(T S(t)
).
6: S(t+1)
= H⇣(T L(t+1)
).
7: ⇣t+1= ( l+1(T S(t+1)
)+ 1
2
t
l(T S(t+1)
)).
8: If l+1(L(t+1)
) < 2n , then return L(⌧)
, S(⌧)
,
else reset S(0)
= S(⌧)
9: Return: bL = L(⌧)
, bS = S(⌧)
Procedure 1 {ˆLl, ( ˆuj, j)j2[l]} =
(Gradient Ascent method)
1: Input: Symmetric tensor T
rank l, exact rank r, N1 numb
or restarts, N2 number of powe
initialization. Let T1 T.
2: for j = 1, . . . , r do
3: for i = 1, . . . , N1 do
4: ✓ ⇠ N(0, In). Compute
u of Tj(I, I, ✓). Initialize
Tj(u, u, u).
5: repeat
6: v
(t+1)
i Tj(I, v
(t)
i , v
(t)
i )
{Run power method t
ball}
7:
(t+1)
i Tj(v
(t+1)
i , v
(t+1
i
8: until t = N2
9: Pick the best:
arg maxi2[N1] Tj(v
(t+1)
i , v
(t
i
i =
(t+1)
i and vi = v
(t+1
i
10: Deflate: Tj Tj ivi ⌦

14. Robust Tensor Decomposition under Block Sparse Perturbation
Algorithm 1 (bL, bS) = RTD (T, , r, ): Tensor Ro-
bust PCA
1: Input: Tensor T 2 Rn⇥n⇥n
, convergence crite-
rion , target rank r, thresholding scale parameter
. Pl(A) denote estimated rank-l approximation
of tensor A, and let l(A) denote the estimated
lth
largest eigenvalue using Procedure 1. HT⇣(A)
denotes hard-thresholding, i.e. H⇣(A))ijk = Aijk
if |Aijk| ⇣ and 0 otherwise.
2: Set initial threshold ⇣0 1(T) and estimates
S(0)
= H⇣0 (T L(0)
).
3: for Stage l = 1 to r do
4: for t = 0 to ⌧ = 10 log n T S(0)
2
/ do
5: L(t+1)
= Pl(T S(t)
).
6: S(t+1)
= H⇣(T L(t+1)
).
7: ⇣t+1= ( l+1(T S(t+1)
)+ 1
2
t
l(T S(t+1)
)).
8: If l+1(L(t+1)
) < 2n , then return L(⌧)
, S(⌧)
,
else reset S(0)
= S(⌧)
9: Return: bL = L(⌧)
, bS = S(⌧)
which is a multilinear combination of the tensor mode-
1 fibers. Similarly T(u, v, w) 2 R is a multilinear com-
bination of the tensor entries.
A tensor T 2 Rn⇥n⇥n
has a CP rank at most r if it
can be written as the sum of r rank-1 tensors as
T =
X
i2[r]
⇤
i ui ⌦ ui ⌦ ui, ui 2 Rn
, kuik = 1, (3)
where notation ⌦ represents the outer product. We
sometimes abbreviate a ⌦ a ⌦ a as a⌦3
. Without loss
of generality, ⇤
i > 0, since ⇤
i u⌦3
i = ⇤
i ( ui)⌦3
.
RTD method: We propose non-convex algorithm
RTD for robust tensor decomposition, described in Al-
Procedure 1 {ˆLl, ( ˆuj, j)j2[l]} = Pl(T): GradAscent
(Gradient Ascent method)
1: Input: Symmetric tensor T 2 Rn⇥n⇥n
, target
rank l, exact rank r, N1 number of initializations
or restarts, N2 number of power iterations for each
initialization. Let T1 T.
2: for j = 1, . . . , r do
3: for i = 1, . . . , N1 do
4: ✓ ⇠ N(0, In). Compute top singular vector
u of Tj(I, I, ✓). Initialize v
(1)
i u. Let =
Tj(u, u, u).
5: repeat
6: v
(t+1)
i Tj(I, v
(t)
i , v
(t)
i )/kTj(I, v
(t)
i , v
(t)
i )k2
{Run power method to land in spectral
ball}
7:
(t+1)
i Tj(v
(t+1)
i , v
(t+1)
i , v
(t+1)
i )
8: until t = N2
9: Pick the best: reset i
arg maxi2[N1] Tj(v
(t+1)
i , v
(t+1)
i , v
(t+1)
i ) and
i =
(t+1)
i and vi = v
(t+1)
i .
10: Deflate: Tj Tj ivi ⌦ vi ⌦ vi.
11: for j = 1, . . . , r do
12: repeat
13: Gradient Ascent iteration: v
(t+1)
j v
(t)
j +
1
4 (1+ /
p
n)
·
⇣
T(I, v
(t)
j , v
(t)
j ) kv
(t)
j k2
v
(t)
j
⌘
.
14: until convergence (linear rate, refer Lemma 3).
15: Set buj = v
(t+1)
j , j = T(v
(t+1)
j , v
(t+1)
j , v
(t+1)
j )
16: return Estimated top l out of all the top r eigen-
pairs (buj, j)j2[l], and low rank estimate ˆLl =P
i2[l] ibuj ⌦ buj ⌦ buj.
is ˜O n4+c
r2
.
T u2
= u uT
u = 1
T u
u
s.t. kuk = 1f(u) = T u3
max
u2Rn
f(u)

16. + =
L⇤
1 2 r+ + +··· +
L
kL L⇤
kF /kL⇤
kF

17. Animashree Anandkumar, Prateek Jain, Yang Shi, U. N. Niranjan
d
10 20 30 40
Error
0.4
0.5
0.6
Figure
d
10 20 30 40
Error 0.3
0.4
0.5
0.6
0.7
0.8
Figure
d
10 20 30 40
Error
100
Figure
Nonwhiten
Whiten(random)
Whiten(true)
Matrix(slice)
Matrix(flat)
d
10 20 30 40
Error
100
Figure
(a) (b) (c) (d)
Figure 1: (a) Error comparison of di↵erent methods with deterministic sparsity, rank 5, varying d. (b) Error compari
of di↵erent methods with deterministic sparsity, rank 25, varying d. (c) Error comparison of di↵erent methods w
block sparsity, rank 5, varying d. (d) Error comparison of di↵erent methods with block sparsity, rank 25, varying
Error = kL⇤
LkF /kL⇤
kF . The curve labeled ‘T-RPCA-W(slice)’ refers to considering recovered low rank part fr
a random slice of the tensor T by using matrix non-convex RPCA method as the whiten matrix, ‘T-RPCA-W(tru
is using true second order moment in whitening, ‘M-RPCA(slice)’ treats each slice of the input tensor as a non-con
matrix-RPCA(M-RPCA) problem, ‘M-RPCA(flat)’ reshapes the tensor along one mode and treat the resultant a
matrix RPCA problem. All four sub-figures share same curve descriptions.
d
10 20 30 40
Time(s)
50
100
150
200
Figure
d
10 20 30 40
Time(s)
102
103
Figure
d
10 20 30 40
Time(s)
102
Figure
Nonwhiten
Whiten(random)
Whiten(true)
Matrix(slice)
Matrix(flat)
d
10 20 30 40
Time(s)
102
103
Figure
d
10 20 30 40
Error
0.4
0.5
0.6
d
10 20 30 40
Error
0.3
0.4
0.5
0.6
0.7
0.8
d
10 20 30 40
Error
100
Nonwhiten
Whiten(random)
Whiten(true)
Matrix(slice)
Matrix(flat)
d
10 20 30 40
Error
100
(a) (b) (c) (d)
Figure 1: (a) Error comparison of di↵erent methods with deterministic sparsity, rank 5, varying d. (b) Error compa
of di↵erent methods with deterministic sparsity, rank 25, varying d. (c) Error comparison of di↵erent methods
block sparsity, rank 5, varying d. (d) Error comparison of di↵erent methods with block sparsity, rank 25, varyi
Error = kL⇤
LkF /kL⇤
kF . The curve labeled ‘T-RPCA-W(slice)’ refers to considering recovered low rank part
a random slice of the tensor T by using matrix non-convex RPCA method as the whiten matrix, ‘T-RPCA-W(t
is using true second order moment in whitening, ‘M-RPCA(slice)’ treats each slice of the input tensor as a non-co
matrix-RPCA(M-RPCA) problem, ‘M-RPCA(flat)’ reshapes the tensor along one mode and treat the resultant
matrix RPCA problem. All four sub-figures share same curve descriptions.
d
10 20 30 40
Time(s)
50
100
150
200
Figure
d
10 20 30 40
Time(s)
102
103
Figure
d
10 20 30 40
Time(s)
102
Figure
Nonwhiten
Whiten(random)
Whiten(true)
Matrix(slice)
Matrix(flat)
d
10 20 30 40
Time(s)
102
103
Figure
(a) (b) (c) (d)
Figure 2: (a) Running time comparison of di↵erent methods with deterministic sparsity, rank 5, varying d. (b) Run
time comparison of di↵erent methods with deterministic sparsity, rank 25, varying d. (c) Running time comparis
di↵erent methods with block sparsity, rank 5, varying d. (d) Running time comparison of di↵erent methods with
sparsity, rank 25, varying d. Curve descriptions are same as in Figure 1.
is orthogonal. We can also extend to non-orthogonal
tensors L⇤
, whose components u are linearly indepen-
Synthetic datasets: The low-rank part LP
u⌦3
is generated from a factor matrix

18. Robust Tensor Decomposition under Block Sparse Perturbation
(a) (b) (c)
3: Foreground filtering or activity detection in the Curtain video dataset. (a): Original image fra
und filtered (sparse part estimated) using tensor method; time taken is 5.1s. (c): Foreground
part estimated) using matrix method; time taken is 5.7s.