This document discusses mosaicing and stabilization techniques. It describes how to warp images into a common coordinate system to stabilize video or create mosaics. Chaining pairwise homographies allows stabilizing sequences where reference frames do not overlap. Mosaics from rotating cameras avoid ghosting since images are related by homographies. Blending is used to combine overlapping image regions. Spherical and cylindrical mosaics can be used to create panoramas by mapping images to non-planar surfaces.

1. Robert Collins
CSE486, Penn State
Lecture 17:
Mosaicing and Stabilization

2. Robert Collins
CSE486, Penn State
Recall: Planar Projection
Pixel coords
Internal u
params
Perspective v
projection
y
Homography
x
Point on plane
Rotation + Translation

3. Robert Collins
CSE486, Penn State
Recall: Projective (un)Warping
H
from Hartley & Zisserman
Source Image Destination image

4. Robert Collins
Recall : Planar Projection
CSE486, Penn State
H1 H2
H

5. Robert Collins
CSE486, Penn State
Applications: Stabilization
Given a sequence of video frames, warp them
into a common image coordinate system.
k-2 k-1 frame k k+1 k+2
destination
frame
This “stabilizes” the video to appear as if the
camera is not moving.

6. Robert Collins
CSE486, Penn State
Stabilization Example
VIVID project

7. Robert Collins
CSE486, Penn State
Stabilization by Chaining
What if the reference image does not overlap with all the
source images? As long as there are pairwise overlaps, we
can chain (compose) pairwise homographies.
H40 = H10 * H21 * H32 * H43
H10 H21 H32 H43
frame k k+1 k+2 k+3 k+4
destination
Not recommended for long sequences, as
alignment errors accumulate over time.

8. Robert Collins
CSE486, Penn State
Applications: Mosaicing
Source 1 Destination image Source 2
H1 H2
from Hartley & Zisserman
Mosaic

9. Robert Collins
CSE486, Penn State
Note on Planar Mosaicing
Assumes scene is roughly planar.
What if scene isn’t planar? Alignment
will not be good if significant 3D relief
“Ghosting”

10. Robert Collins
CSE486, Penn State
Ghosting Example
Source image Reference image
H

11. Robert Collins
CSE486, Penn State
Ghosting Example (cont)
Mosaic

12. Robert Collins
Mosaics from Rotating Cameras
CSE486, Penn State
However, there is a mitigating factor
in regards to ghosting…
Images taken from a rotating camera
are related by a 2D homography…
regardless of scene structure!

13. Robert Collins
Rotating Camera (top-down view)
CSE486, Penn State

14. Robert Collins
Rotating Camera (top-down view)
CSE486, Penn State

15. Robert Collins
Rotating Camera (top-down view)
CSE486, Penn State

16. Robert Collins
Rotating Camera (top-down view)
CSE486, Penn State
Rays in camera coord system are invariant!

17. Robert Collins
Rotating Camera (top-down view)
CSE486, Penn State
Rays in camera coord system are invariant!

18. Robert Collins
Rotating Camera (top-down view)
CSE486, Penn State
Rays in camera coord system are invariant!

19. Robert Collins
CSE486, Penn State
Special Case : Rotating Camera
Internal
params Projection Relative R,T
Relative Rotation
of camera Translation is 0
This is important!

20. Robert Collins
CSE486, Penn State
Special Case : Rotating Camera
Internal
params Projection Relative R,T

21. Robert Collins
CSE486, Penn State Relations among Images
Taken by Rotating Camera
Image 1
Same ray!
Image 2
-1

22. Robert Collins
CSE486, Penn State
Mosaicing Example
Original Images (from a pan/tilt camera)
Panoramic (Mosaic) View

23. Robert Collins
CSE486, Penn State
One more detail: Blending!

24. Robert Collins
CSE486, Penn State
Approaches to Blending
How to combine colors in area of overlap?
1) Straight averaging P = (P1 + P2) / 2
2) Feathering P = (w1*P1 + w2*P2 ) / (w1+w2)
With wi being distance from image border
3) Equalize intensity statistics (gain, offset)

25. Robert Collins
CSE486, Penn State
Before and After Blending

26. Robert Collins
CSE486, Penn State
360 Degree Panoramas?
Problem: Can’t just choose a reference image
to map all other images to.
Solution: Use cylindrical or spherical mosaic
surface rather than a plane.

27. Robert Collins
CSE486, Penn State
Panorama Input images
Sarnoff

28. Robert Collins
CSE486, Penn State
Spherical Panorama Result
Sarnoff

29. Robert Collins
CSE486, Penn State
Panorama Unwarped
Sarnoff

30. Robert Collins
CSE486, Penn State
Quicktime VR
This example from www.panoguide.com/gallery/
Also big list at http://www.multimedialibrary.com/diana/qtvr_sites.asp

31. Robert Collins
CSE486, Penn State
Quicktime VR

32. Robert Collins
CSE486, Penn State
Quicktime VR
This example from www.ems.psu.edu/~fraser/qtvr/