Background subtraction is a technique used to separate foreground objects from backgrounds in video frames. It works by comparing each frame to a background model and detecting differences which indicate moving foreground objects. Recursive techniques like mixtures of Gaussians model the background pixel values over time using multiple Gaussian distributions, allowing the background model to adapt to changing lighting conditions. Adaptive background/foreground detection uses a background model that evolves over time to distinguish foreground objects from the background in a robust way.

1. Background Subtraction
12/8/2011 1

2. Background Subtraction
• As the name suggests, background subtraction
is the process of separating out foreground
objects from the background in a sequence of
video frames.
12/8/2011 2

3. Background Subtraction
• Background subtraction is a widely used
approach for detecting moving objects from
static cameras.
12/8/2011 3

4. Fundamental Logic
• Fundamental logic for detecting moving
objects from the difference between the
current frame and a reference frame, called
“background image” and this method is
known as FRAME DIFFERENCE METHOD
12/8/2011 4

5. Background Modelling
12/8/2011 5

6. Background Model
12/8/2011 6

7. After Background Filtering…
12/8/2011 7

8. Background Subtraction Principles
At ICCV 1999, MS Research presented a study, Wallflower: Principles and Practice
of Background Maintenance, by Kentaro Toyama, John Krumm, Barry
Brumitt, Brian Meyers. This paper compared many different background
subtraction techniques and came up with some principles:
P1:
P2:
P3:
P4:
12/8/2011 8

9. The Problem - Requirements
The background image is not fixed but must adapt to:
• Illumination changes
•Gradual •Sudden (such as clouds)
12/8/2011 9

10. • Motion changes
•Camera oscillations • High-frequencies background
objects (such as tree
branches, sea waves, and
similar)
12/8/2011 10

11. • Changes in the background geometry
•Parked cars
12/8/2011 11

12. Widely Used!
• Traffic monitoring (counting
vehicles, detecting & tracking
vehicles),
• Human action recognition
(run, walk, jump, squat),
• Human-computer interaction
(“human interface")
• Object tracking (watched tennis
lately),
• And in many other cool
applications of computer vision
such as digital forensics.
12/8/2011 12

13. Simple Approach
Image at time t: Background at time t:
I (x, y, t) B(x, y, t)
> Th
1. Estimate the background for time t.
2. Subtract the estimated background from the input frame.
3. Apply a threshold, Th, to the absolute difference to get the
foreground mask.
12/8/2011 13

14. The basic methods
• Frame difference:
| framei–framesi-1| > Th
• The estimated background is just the previous
frame
• It evidently works only in particular conditions
of objects’ speed and frame rate
• Very sensitive to the threshold Th
12/8/2011 14

15. Frame Differencing
• Background is estimated to be the previous
frame. Background subtraction equation then
becomes:
B(x ,y ,t) = I (x ,y ,t-1)
I (x, y, t) - I (x, y, t – 1) > Th
12/8/2011 15

16. Frame Differencing
• Depending on the object
structure, speed, frame rate and global
threshold, this approach may or may not be
useful (usually not).
> Th
12/8/2011 16

17. Frame Differencing
Th = 25 Th = 50
Th = 100 Th = 200
12/8/2011 17

18. Thresholding
• The key parameter in the thresholding process is the
choice of the threshold value.
• Several different methods for choosing a threshold
exist,
– Users can manually choose a threshold value, or
– A thresholding algorithm can compute a value
automatically, which is known as automatic thresholding
12/8/2011 18

19. Motivation
• A robust background subtraction algorithm should handle:
lighting changes, repetitive motions from clutter and long-
term scene changes.
12/8/2011 19

20. Algorithm Overview
• The values of a particular pixel is modeled as a mixture of adaptive
Gaussians.
– Why mixture? Multiple surfaces appear in a pixel.
– Why adaptive? Lighting conditions change.
• At each iteration Gaussians are evaluated using a simple heuristic to
determine which ones are mostly likely to correspond to the
background.
• Pixels that do not match with the “background Gaussians” are
classified as foreground.
• Foreground pixels are grouped using 2D connected component
analysis.
12/8/2011 20

21. Advantages vs. Shortcomings
• Advantages:
– A different “threshold” is selected for each pixel.
– These pixel-wise “thresholds" are adapting by time.
– Objects are allowed to become part of the background
without destroying the existing background model.
– Provides fast recovery.
• Disadvantages:
– Cannot deal with sudden, drastic lighting changes.
– Initializing the Gaussians is important (median filtering).
– There are relatively many parameters, and they should be
selected intelligently.
12/8/2011 21

22. Challenges background subtraction
Algorithm
• It must be robust against changes in illumination.
• It should avoid detecting non-stationary background
objects such as swinging leaves, rain, snow ,and
shadow cast by moving objects.
• Finally, its internal background model should react
quickly to changes in background such as starting
and stopping of vehicles.
12/8/2011 22

23. BACKGROUND SUBTRACTION
ALGORITHMS
Flow diagram of a generic background subtraction algorithm.
12/8/2011 23

24. Background subtraction Algorithms
• The four major steps in a background
subtraction algorithm are:
1. Preprocessing,
2. Background modeling,
3. Foreground detection, and
4. Data validation.
12/8/2011 24

25. Background Modeling
• Background modeling is at the heart of any
background subtraction algorithm.
• Background model is that which robust
against environmental changes in the
background, but sensitive enough to identify
all moving objects of interest.
12/8/2011 25

26. Background Modeling
• We classify background modeling techniques
into two broad categories
– Non-recursive and
– Recursive
12/8/2011 26

27. Non-recursive Techniques
• A non-recursive technique uses a sliding
window approach for background estimation.
• Non-recursive techniques are highly adaptive
as they do not depend on the history beyond
those frames stored in the buffer.
12/8/2011 27

28. Non-recursive Techniques
• Some of the commonly-used non-recursive
techniques are:-
– Frame differencing
– Median filter
– Mean filter
12/8/2011 28

29. Frame differencing
This we have studied in earlier slide
12/8/2011 29

30. Median filter
• The background estimate is defined to be the
median at each pixel location of all the frames
in the buffer.
• The assumption is that the pixel stays in the
background for more than half of the frames
in the buffer.
12/8/2011 30

31. Median Filter
• Assuming that the background is more likely to
appear in a scene, we can use the median of the
previous n frames as the background model:
12/8/2011 31

32. 12/8/2011 32

33. 12/8/2011 33

34. Mean Filter
• In this case the background is the mean of the
previous n frames:
12/8/2011 34

35. For n = 10:
Estimated Background Foreground Mask
For n = 20:
Estimated Background Foreground Mask
12/8/2011 35

36. For n = 50:
Estimated Background Foreground Mask
12/8/2011 36

37. Kalman filter
• Kalman filter is a widely-used recursive
technique for tracking linear dynamical
systems under Gaussian noise. Many different
versions have been proposed for background
modeling, differing mainly in the state spaces
used for tracking.
12/8/2011 37

38. Mixture of Gaussians (MoG)
• Unlike Kalman filter which tracks the evolution of a single
Gaussian, the MoG method tracks multiple Gaussian
distributions simultaneously.
• MoG maintains a density function for each pixel.
• It is capable of handling multi-modal background
distributions.
• Since MoG is parametric, the model parameters can be
adaptively updated without keeping a large buffer of video
frames.
12/8/2011 38

39. Mixture of Gaussians
• the pixel process of
a pixel Z is the set of
the last t values at
the pixel
• {Z0, Z1, ..., Zt−1}
12/8/2011 39

40. Gaussian Mixture Model (GMM)
• A GMM is a mixture
pdf which is a linear
combination of K
Gaussian pdfs.
• wi = 1
• each pixel is given
one GMM
12/8/2011 40

41. Background/Foreground Detection
• There are two types of B/F detection methods:
– Non-adaptive: It depend on certain numbers of video
frames and do not maintain a background model in the
algorithm.
• B/F detection based on two frames
• B/F detection based on three frames
– Adaptive: It maintain a background model and the
parameters of the background model evolve over time.
• Adaptive B/F detection Statistical
• B/F detection based on Gaussian model
12/8/2011 41

42. Drawback of non-adaptive methods
• The non-adaptive methods are useful only in
– high-supervised,
– short-term tracking applications without
significant changes in the video scene.
– When errors happen, it requires manual re-
initialization.
– Without re-initialization, errors in the background
accumulate over time.
12/8/2011 42

43. Adaptive Background/Foreground
Detection
• Adaptive B/F detection is chosen for more and
more VCA applications because of the limitation
of non-adaptive methods. A standard adaptive
B/F detection method maintains a background
model within the system.
12/8/2011 43

44. Video Content Analysis (VCA) is the capability of
automatically analyzing video to detect and determine
temporal events not based on a single image
12/8/2011 44

45. The algorithm of adaptive B/F
detection
• The algorithm of adaptive B/F detection is
described below.
– fi : A pixel in a current frame, where i is the frame
index.
– µ : A pixel of the background model (fi and m are
located at the same location).
– di: Absolute difference between fi and m.
– bi: B/F mask - 0: background. 0 x ff: foreground.
– T: Threshold
– α: Learning rate of the background.
12/8/2011 45

46. Adaptive B/F Detection
1. di = |fi - µ|
2. If di > T , fi belongs to the foreground; otherwise, it
belongs to the background.
Adaptive B/F Detection
12/8/2011 46

47. Examples of Adaptive B/F Detection
Examples of Adaptive B/F Detection
12/8/2011 47

48. Demo
12/8/2011 48

49. THANK YOU
12/8/2011 49