This document presents a method for multi-sensor image fusion using temporal object detection from visible and infrared video frames. The method uses a Gaussian mixture model for background subtraction to detect foreground objects in each frame. An edge detection algorithm is then applied and the resulting edge maps are fused based on local differences to generate a fused output frame that emphasizes detected objects and preserves details from the visual frame. Experimental results demonstrate the fusion of daytime and low light visible and infrared frames. Future work will add object tracking capabilities to the system.

1. Multi-Sensor Image Fusion Using
Temporal Object Detection
Mehmet Celenk and Melih Altun
School of EECS, Ohio University
Stocker Center, Athens, OH 45701 USA
ICGST Conference on Computer
Science and Engineering, CSE-11
19-21 December, 2011
Istanbul, Turkey

2. Image Processing and Pattern
Recognition Research
 Security and Surveillance Video Processing

3. Research (Cont.)
 Non Invasive Detection of Melanoma and
Other Biomedical Applications

4. Research (Cont.)
 Road Pattern and Scenery Learning for Safe
Transportation

5. More Information
http://www.ohio.edu/engineering/index.cfm
http://twitter.com/russcollege
http://www.facebook.com/pages/Russ-College-of-Engineering-and-Technology/297028071591

6.  Introduction
 Background and Foreground Selection
 Image Fusion
 Experimental Results
 Summary and Future Work
Outline

7. Introduction
Video Content Enhancement and Image Fusion
• Surveillance applications support tracking and monitoring operations
• Content enhancement includes detection of hidden and occluded
objects in a scene
• Thermal imaging provides additional information about hidden objects
• Visible imaging holds prominent visual information
• Image fusion combines multi-sensor inputs for a more informative
output
• Thermal features such as the body heat of a person, are likely to
occur at regions where changes are detected. Hence, foreground
detection becomes a major part in our fusion method.

8. Introduction
Video Content Enhancement and Image Fusion
Visual Video Frame Infrared (IR) Video Frame

9. Introduction
Video Content Enhancement and Image Fusion

10. Introduction
Video Content Enhancement and Image Fusion

11. Introduction
Video Content Enhancement and Image Fusion

12. Introduction
Video Content Enhancement and Image Fusion

13. Introduction
Video Content Enhancement and Image Fusion

14. Background and Foreground Selection
Background modeling
Gaussian Mixture Model
( ) [ ] [ ]



−Σ−
Σ
= ∑=
ii
T
i
K
i i
Di txtxwtxP µµ
π
)()(
2
1
exp
2
1
)(
1
2
wi: Associated weight for the ith Gaussian
Σi = σi2
·I: Covariance matrix,
D: number of dimensions
Sorting wi / σi2
and selecting the ones over a certain threshold gives
the most possible background pixel values

15. Background and Foreground Selection (Cont.)
Fixed learning rate (i.e., η(t)=α) leads to slow adaptation
Variable learning rate is applied to prevent low convergence
α
α
η +
−
=
i
i
c
1
(t)
ci is the number of matching Gaussians for the given pixel
wi(t) = (1-α) · wi(t-1) + α · qi
μi(t) = (1-ηi(t)) · μi(t-1) + ηi(t) · x(t)
σi
2
(t) = (1-ηi(t)) · σi
2
(t-1) + ηi(t) · (x(t)- μi(t-1))2
where qi equals 1 for matching Gaussians and 0 otherwise.
Weights and Gaussian parameters are updated as

16. Background and Foreground Selection (Cont.)

17. Background and Foreground Selection (Cont.)
Current Frame Background Frame
Difference Selected Foreground

18. Image Fusion
Edge Detection
22






∂
∂
+





∂
∂
=∇
y
I
x
I
I VV
V
Background Frame Edge Map
Selected Foreground Masked Edge Map
( ) ( ) ( )jiIjiIjiI MVVM ,,, ⋅∇=∇
Masked Edge Map

19. Image Fusion (Cont.)
Local Edge Difference
∑ ∑
+
−=
+
−=
∇−∇=∆
Mi
Mik
Mj
Mjl
BMVML lkIlkIjiE ),(),(),(
∇IBM ∇IVM ΔEL < 0

20. Image Fusion (Cont.)
Visible Frame
IR Frame
Fused Frame

21. Overall System Diagram

22. Experimental Results
Frames from daytime videos
Visible Frame
IR Frame
Fused Frame

23. Experimental Results (Cont.)
Visible frame with added occlusions Fused frame

24. Frames from low light videos
Experimental Results (Cont.)
Visible Frames IR Frames Fused Frames

25. Summary and Future Work
 A new approach to multispectral image fusion that
uses temporal changes in IR and visual video
sequences is described
 IR information is employed only when necessary such
as the cases as highly pertinent low visibility and
occlusions. Details in the visual image are preserved
in all other regions.
 Detected objects are emphasized with bounding
boxes to highlight regions of interest.

26. Summary and Future Work (Cont.)
 Foreground objects inside the bounding boxes can
be identified in each frame by means of their spectral
and spatial information.
 Such an implementation will further enhance video
contents and it will add object tracking capabilities

27. References
1. D.S. Lee, Effective Gaussian Mixture Learning for Video Background
Substraction, IEEE Transactions on Pattern Analysis and Machine Intelligence vol.
27(5), pp. 827–832, 2005.
2. C. Stauffer and W.E.L. Grimson, Adaptive Background Mixture Models for Real-
Time Tracking, Proc. Conf. Computer Vision and Pattern Recognition, vol. 2, pp.
246-252, 1999.
3. S. Theodoridis and K. Koutroumbas, Pattern Recognition, 4th ed. Elsevier, 2009.
4. J. S. Lim, Two-Dimensional Signal and Image Processing, Prentice Hall, 1990.
5. D. Dwyer, Octec Limited, http://www.octec.co.uk/
6. C. Ó Conaire, N.E. O'Connor, E. Cooke, A.F. Smeaton, Comparison of fusion
methods for thermo-visual surveillance tracking. In: International Conference on
Information Fusion , 2006.
7. J. Davis and V. Sharma, Background-Subtraction using Contour-based Fusion of
Thermal and Visible Imagery, Computer Vision and Image Understanding, Vol
106(2-3), pp. 162-182, 2007.
8. A. Toet, J.K. IJspeert, A.M. Waxman, and M. Aguilar, Fusion of Visible and
Thermal Imagery Improves Situational Awareness, Displays, vol. 18, pp. 85-95,
1997.