2. OUTLINE
• What is video stabilization?
• Where is it useful?
• How does it work?
• What are the problems
3. DEFINITION
• Most amateur videos are captured using hand-held cameras. They are often very shaky
and difficult to watch.
• Video stabilization techniques have been developed to smooth shaky camera motion in
videos before viewing.
• Stabilization is the process of estimating and compensating for the background image
motion occurring due to the ego-motion of the camera.
4. ELIMINATING JITTER
• Although this jitter can be eliminated by anchoring the binoculars on a tripod, this is not
always feasible.
5. EXAMPLE
Wang, Y.S., et al., Spatially and Temporally Optimized Video Stabilization. IEEE transactions on visualization and computer graphics, 2013.
http://people.cs.nctu.edu.tw/~yushuen/VideoStabilization/
6. PARALLAX
• Parallax is a displacement or difference in the apparent position of an object viewed along
two different lines of sight
7. BIOLOGICAL MOTIVATION: INSECT NAVIGATION
• Insects have relatively small nervous system with very few neurons when compared to the
human brain, they are still capable of complex tasks, such as safe landing, obstacle
avoidance.
• Behavioral research with insects suggest that insects primarily use visual information.
• Insects have immobile eyes with fixed focal length. Moreover, they do not possess
stereoscopic vision. Insect eyes possess inferior spatial acuity but their eyes sample the
world at a significantly higher rate than human eyes do.
• The study can serve as a pure motivational tool indicating that such complex tasks, such
as stabilization, can be performed real-time, with the accuracy desired. Second, this study
can lead us into the paradigm of “active vision” or “purposive vision”.
• In fact, several researchers have used such biologically inspired mechanisms for flight
control and obstacle avoidance.
Al Bovik. The Essential Guide to Video Processing
8. BIOLOGICAL MOTIVATION: INSECT NAVIGATION
• Bees that fly through holes tend to fly through the center of these holes. Bees, like most
other insects, cannot measure distances from surfaces by using stereoscopic vision.
• Recent experiments have indicated that bees balance the image motion on the lateral
portion of their two eyes as they fly through openings.
• Bees were trained to fly in narrow tunnels with certain patterns on the side walls of the
tunnels. It was shown in that bees tended to fly at the center of this tunnel when the
patterns on the side walls were stationary.
• If one of these patterned side walls was moved in the direction of the bee’s flight, thereby
reducing the image motion experienced by the bee on that side, then the bees moved
closer to that side wall. Similarly, when one of the patterned side walls was moved in the
direction opposite to the direction of the bee’s flight, the bee moved away from the moving
wall.
9. BIOLOGICAL MOTIVATION: INSECT NAVIGATION
• Collision avoidance is another task that is visually driven in most insects. When an insect
approaches an obstacle, its image expands on it’s eyes. Insects are sensitive to this
image expansion and turn away from the direction in which the image expansion occurs,
thereby avoiding collision with obstacles.
10. DIFFERENT ALGORITHMS
• Depending on the type of scenario and the type of motion involved, we have different
algorithms to achieve stabilization.
• Presence of a dominant plane in the scene
• Derotation of the image sequence
• Mosaic construction
• Presence of moving objects
11. CLASSIFICATION OF TECHNIQUES
• Techniques are classified as two categories:
• Feature-based methods: extract and match discrete features between frames and
trajectories of these features are fit to a global motion model.
• Flow-based methods: optical flow of the image sequence is an intermediate quantity
that is used in determining the global motion.
12. PHASES OF STABILIZATION
• In video stabilization, we need to analyze the image motion and obtain models for the
global motion in image sequences.
• Generally the process of stabilization have to go through two phases:
• motion estimation
• motion smoothing
14. EFFECT OF CAMERA MOTION
• The effect of camera motion can be computed using projective geometry:
15. EFFECT OF CAMERA MOTION
• Other popular global deformations mapping the projection of a point between two frames
are the similarity:
and affine transformations:
16. IMAGE FEATURES
• The basic goal in feature-based motion estimation is to use features to find maps that
relate the images taken from different view-points.
• These maps are then used to estimate the image motion by computing the parameters of
a motion model.
• Consider the case of pure rotation:
• Though various lengths, ratios, and angles formed on the images are all different, the
cross ratio remains the same. Given four collinear points A, B, C, and D on an image,
R. Hartley and A. Zisserman. Multiple View Geometry in computer vision. Cambridge University Press, Cambridge, UK, 2000.
17. IMAGE FEATURES
• this intuition leads to a map relating the two images.
• Given four corresponding points in general position in the two images, we can map any
point from one image to the other.
• Now, any point F on ABE will map to point F´ such that the cross ratio is preserved.
• This way one can map each point on one image to the other image. Such a map is called
homography.
18. IMAGE FEATURES
• In the case of planar scene:
• x1, a point on first image plane, xp, the corresponding point on the real plane, x2, the
corresponding point on the second image plane.
• Thus, homography H =H1H2 maps points from one image plane to the other.
19. IMAGE FEATURES
• On the other hand, when there are depth variations in the scene, such a homography
doesn’t exist between images formed by camera translation.
• In the case of depth variations, we can use structure from motion (SFM) approaches to
estimate the motion of the camera.
20. FEATURE BASED ALGORITHMS
• A number of features are extracted in each image and feature matching algorithms are
used to establish correspondence between the images.
• The motion parameters are found by first identifying the set of feature matches.
21. FEATURE TRAJECTORY SMOOTHING
• Let the ith trajectory be where pi and m and n are the start and the end frames of Pi,
respectively.
• Our goal is to solve an optimization problem that can minimize the acceleration of Pi in
each frame while constraining the offsets of neighboring trajectories to be consistent
within the input video.
Wang, Y.S., et al., Spatially and Temporally Optimized Video Stabilization. IEEE transactions on visualization and computer graphics, 2013.
22. BEZIER CURVES
• Bezier curves are used in computer graphics to produce curves which appear reasonably
smooth at all scales (as opposed to polygonal lines, which will not scale nicely) in which
the interpolating polynomials depend on certain control points.
23. FEATURE TRAJECTORY SMOOTHING
• each smoothed trajectory is represented using a Be’zier curve and reduce the unknown
variables from all feature positions to curve control points. This reduced model also
achieves strong stabilization because the smoothed feature positions are interpolated
from the control points. We show the details of our technique in the following subsections.
24. DELAUNAY TRIANGULATION
• In mathematics and computational geometry, a Delaunay triangulation for a set P of
points in a plane is a triangulation DT(P) such that no point in P is inside
the circumcircle of any triangle in DT(P).
• In geometry, the circumscribed circle or circumcircle of a polygon is a circle which
passes through all the vertices of the polygon..
25. SPATIAL RIGIDITY PRESERVATION
• spatial rigidity is retained when stabilizing a video in order to preserve neighboring feature
trajectories to have similar treatments.
• Specifically, we compute the neighbor relations between features in each frame using the
Delaunay triangulation and enforce each triangle to undergo a rigid transformation. That
is, triangles are allowed to move and rotate but their sizes and shapes should be retained.
• This constraint works well in most videos.
26. OBJECTIVE FUNCTION
• we search for the control points of Bezier curves that can minimize the objective function.
28. LIMITATIONS
• Although the algorithm is robust to all challenging examples, the stabilization is not
effective if there are no background features in some frames.
29. MOSAICING
• Mosaicing is the process of compositing or piecing together successive frames of the
stabilized image sequence so as to virtually increase the field of view of the camera.
• Mosaics are commonly defined only for scenes viewed by a pan/tilt camera, for which the
images can be related by a projective transformation.
30. REFERENCES
• Al Bovik. The Essential Guide to Video Processing
• Wang, Y.S., et al., Spatially and Temporally Optimized Video Stabilization. IEEE
transactions on visualization and computer graphics, 2013.
• http://www.ics.uci.edu/~eppstein/gina/delaunay.html
• http://en.wikipedia.org/wiki/Delaunay_triangulation
• http://www.math.ubc.ca/~cass/gfx/bezier.html
• http://en.wikipedia.org/wiki/B%C3%A9zier_curve
• http://people.cs.nctu.edu.tw/~yushuen/VideoStabilization/
