The document discusses foreground detection through robust low-rank matrix decomposition combined with spatio-temporal constraints, focusing on its application in video surveillance and motion capture. It elaborates on the techniques used, including iterative reweighted least squares (IRLS) and different formulations for robust PCA, supplemented with experimental results validating the method's effectiveness. Key challenges and future works, such as the need for real-time processing improvements, are also addressed.

1. Foreground Detection via Robust Low Rank Matrix
Decomposition including spatio-temporal Constraint
C. Guyon, T. Bouwmans and E. Zahzah
MIA Laboratory (Mathematics Images & Applications), University of La Rochelle, France
—
Workshop BMC, Daejong Korea
November 5, 2012
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 1 Roche

2. Summary
1 Introduction and motivations on IRLS
2 Temporal constraint with an adapted Norm
3 Diagram flow and Spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 2 Roche

3. Introduction and motivations
Purpose
Foreground detection : Segmentation of moving objects in video
sequence acquired by a fixed camera.
Background modeling : Modelization of all that is not moving
object.
Involved applications
Surveillance camera
Motion capture
On the importance
Crucial Task : Often the first step of a full video surveillance system.
Strategy used
Eigenbackground decomposition.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 3 Roche

4. Eigenbackgrounds
Find an « ideal » subspace of the video sequence, which describes the best
as possible the (dynamic) background.
Fig.1 The common process of background subtraction via PCA (Principal Component
Analysis). At final step, an adaptative threshold is used to get a binary image.
Without a robust framework, the moving object may be absorbed in the model !
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 4 Roche

5. Data Structure Transformation
First, we consider a video sequence as a matrix A ∈ Rn×m
n is the amount of pixels in a frame (∼ 106 )
m is the number of frames considered (∼ 200)
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 5 Roche

6. IRLS : Vector version (1)
The usual IRLS (Iteratively reweighted least squares) scheme for solve
argmin ||Ax − b||α is given by
x
D (i) = diag((ε + |b − Ax (i) |)α−2 )
(1)
x (i+1) = (At D (i) A)−1 At D (i) b
that a suitable IRLS method is convergent for 1 ≤ α < 3
Other formulation,
r (i) = b − Ax (i)
D = diag((ε + |r (i) |)α−2 )
(2)
y (i) = (A DA)−1 A Dr (i)
x (i+1) = x (i) + (1 + λopt )y (i)
λopt computed in a second inner loop and convergent for 1 ≤ α < +∞
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 6 Roche

7. IRLS : Vector version (2)
For Spatio/temporal RPCA, needs to solve the following general problem :
argmin ||Ax − b||α + λ||Cx − d ||β (3)
x
By derivation, the associated IRLS scheme is,
r1 = b − Ax (i ) , r2 = d − Cx (i ) , e1 = ε + |r1 |, e2 = ε + |r2 |
α 1 β 1 −1 β−2
D1 = ( e1 ) α −1 diag(e1 ), D2 = λ( e2 ) β diag(e2 )
α−2
(i ) −1
(4)
y = (A D1 A + C D2 C ) (A D1 r1 + C D2 r2 )
x (i +1) = x (i ) + (1 + λopt )y (i )
Good news : Just few line of matlab code !
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 7 Roche

8. Matrix Version
More generally, we consider the following matrix regression problem with
two parameters norm (α, β) and a weighted matrix (W ),
n m
α 1
min ||AX − B||α,β with ||Mij ||α,β = ( ( Wij |Mij |β ) β ) α (5)
X W W
i=1 j=1
The problem is solved in the same manner on matrices with a reweighted
regression strategy,
Until X is stable, repeat on each k-columns
R ← B − AX
S ← ε + |R| (6)
α −1
β−2 β
Dk ← diag(Sik ◦ ( j (Sij ◦ Wij )) β ◦ Wik )k
Xik ← Xik +(1+Λ(max(α, β)))(At Dk A)−1 At Dk Rik
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 8 Roche

9. Exemple : Geometric median (convex problem)
Fig.1 Find the x-axis points which minimize the ||.||α distance between the 7 others
points (black stars). Minimum of Curves is corresponding to this optimal value for α =
2, 1.66, 1.33 and 1 (blue, green, red, black).
Fig.1 Same in Two Dimension. By continuously changing the minization problem (i.e.
varying α) during each iterations, this trick accelerates the convergence of IRLS.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), University of La / 27
November 5, 2012 9 Roche

10. Various RPCA formulation (only for α = 1)
PCA with fixed rank is : min ||S||F
L,S
s.t. Rank(L) = k (7)
A=L+S
R(obust)PCA is (Non convex and NP-hard ) :
min ||σ(L)||0 + λ||S||0
L,S (8)
s.t. A=L+S
Convex relaxed problem of (2) is RPCA-PCP proposed by Candès et al. [1] :
min ||σ(L)||1 + λ||S||1
L,S (9)
s.t. A=L+S
Where σ(L) means singular values of L.
A mix is Stable PCP of Zhou et al. [2] (both entry-wise and sparse noise) :
min ||σ(L)||1 + λ||S||1
L,S (10)
s.t. ||A − L − S||F < δ
All of them could be solved by Augmented Lagrangian Multipliers (ALM).
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 10 / 27
La Roche

11. Video examples
Some examples, temporal RPCA and ideal RPCA with groundtruth fitting
and optimal parameters fitting.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 11 / 27
La Roche

12. Summary
1 Introduction and motivations on IRLS
2 Temporal constraint with an adapted Norm
3 Diagram flow and Spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 12 / 27
La Roche

13. Sparse solution
In RPCA, residual error is sparse.
Using the RPCA decomposition on a synthetic low-rank random matrix
plus noise, the error looks like :
Same principle with video. Sparse noise (or outliers) are the moving objects.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 13 / 27
La Roche

14. Let’s play with norms
Varying the α, β norm → Different kind of recovering pattern error.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 14 / 27
La Roche

15. Let’s play with norms...(2)
Some issues
What is the best specific norm for temporal constrain ?
Initial assumption is ||.||2,1 . Confirmed experimentally ?
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 15 / 27
La Roche

16. Validation
If ideal eigenbakgrounds are that, best norm must be ...
Let’s denote Lopt , the ideal low-rank subspace which outliers do not contribute to PCA
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 16 / 27
La Roche

17. Experimental results
Let’s denote Lα,β , the low-rank
recovered matrix with a ||.||α,β -PCA.
The plot show the error between
||Lopt − Lα,β ||F for parameters α
and β chosen freely. The darkest
value means that the error is the
smallest here.
||S||2,1 is not optimal, but for convenience we use it.
The benefit of the ad hoc block-sparse hypothesis is confirmed by
testing its efficiency directly on video dataset.
Experimentation done on dynamic category of dataset change detection
workshop 2012 : http://www.changedetection.net/
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 17 / 27
La Roche

18. Summary
1 Introduction and motivations on IRLS
2 Temporal constraint with an adapted Norm
3 Diagram flow and Spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 18 / 27
La Roche

19. Overview and addition of the spatial constraint
Figure: Overview of the learning and evaluation process. Learning process needs
GT (Groundtruth) for better fits the eigenbackground components.
Spatial Constrain
Suppose A = L + S and L and S are computed via some kind of
RPCA technique with the addtion of Total Variation penalty on S.
This increase connected (or connexe) shapes.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 19 / 27
La Roche

20. Exemple with a synthetic 1-D signal
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 20 / 27
La Roche

21. Summary
1 Introduction and motivations on IRLS
2 Temporal constraint with an adapted Norm
3 Diagram flow and Spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 21 / 27
La Roche

22. Experimental Protocol
Optimal threshold is chosen for maximizing F-measure criterion
which is based 2 × 2 histogram of True/false/positive/negative :
TP TP 2 DR Prec
DR = , Prec = , F =
TP + FN TP + F P DR + Prec
Good performance is then obtained when the F-measure is closed to 1
Time consumption is not take into account in the evaluation process.
RPCA-LBD is compared with the following two Robust methods :
Robust Subspace Learning (RSL) De La Torre et al. [4]
Principal Component Pursuit (RPCA-PCP) [1]
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 22 / 27
La Roche

23. Quantitative Results
Here, we show experimental results on the real dataset of BMC,
Video Recall Precision F-measure PSNR Visual Results
1 0.9139 0.7170 0.8036 38.2425
2 0.8785 0.8656 0.8720 26.7721
3 0.9658 0.8120 0.8822 37.7053
4 0.9550 0.7187 0.8202 39.3699
5 0.9102 0.5589 0.6925 30.5876
6 0.9002 0.7727 0.8316 29.9994
7 0.9116 0.8401 0.8744 26.8350
8 0.8651 0.6710 0.7558 30.5040
9 0.9309 0.8239 0.8741 55.1163
Table: Quantitative results with common criterions. Last column show the
original, GT and result of the first four real video sequences.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 23 / 27
La Roche

24. Other results
Figure: First eigenBackground of the fifths sequence of Rotary (BMC) with the
norms ||.||opt , ||.||1,1 and ||.||2,1 . Last row shows the first five eigenBackground on
real dataset with ||.||2,1
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 24 / 27
La Roche

25. Summary
1 Introduction and motivations on IRLS
2 Temporal constraint with an adapted Norm
3 Diagram flow and Spatial constraint
4 Experimental Results
5 Conclusion
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 25 / 27
La Roche

26. Qualitative analysis
Advantages
Experiments on video surveillance datasets show that this approach is
more robust than RSL and RPCA-PCP in presence of dynamic
backgrounds and illumination changes.
Well suited for video with spatially spread and temporarily sparse
outliers.
Disadvantages
Not efficient on sequences with outliers always presents.
For small local variations, like « wind tree » are not (yet) well
modelized by this kind global PCA. Probably needs more eigen
components !
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 26 / 27
La Roche

27. Future Works & References
Future Works
Lack in computation time : Further research consists in developping an
incremental version to update the model at every frame and to achieve
the real-time requirements.
References
[1] E. Candes, X. Li, Y. Ma, and J. Wright, Robust principal component
analysis, International Journal of ACM, vol. 58, no. 3, May 2011.
[2] Z. Zhou, X. Li, J. Wright, E. Candes, and Y. Ma, Stable principal
component pursuit,IEEE ISIT Proceedings, pp. 1518-1522, Jun. 2010.
[3] G. Tang and A. Nehorai, Robust principal component analysis based
on low-rank and block-sparse matrix decomposition, CISS 2011, 2011.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 27 / 27
La Roche