The document presents a study on color-guided thermal image super-resolution, utilizing a visible camera to enhance infrared images through a novel algorithm. The work involves creating a multi-sensor imaging system and dataset, applying guided filters, cost volume construction, and outlier detection methods to improve image quality. Experimental results validate the effectiveness of the proposed approach compared to existing methods, supported by grants from the National Science Foundation of China.

1. WELCOME

2. Md. Safayet Hossain
ID:1907551
Student of M.Sc
Computer Science and Engineering (KUET)
Presented By:

3. COLOR GUIDED THERMAL IMAGE SUPER
RESOLUTION
Xiaohui Chen(1) , Guangtao Zhai1 , Jia Wang (2) ,Chunjia Hu (1) and
Yuanchun Chen(2)
Institute of Image Communication and Information Processing, Shanghai Jiao Tong
University, Shanghai, China
• 1 {(fxhuichen.cn,zhaiguangtao,huchunjia.sjtug)@gmail.com},
2{(fjiawang,chenyuanchung)@sjtu.edu.cn}

4. ABSTRACT OF THIS PAPER
Use the visible camera as a guidance to supper resolution the IR
images.
Setup a prototype of the IR-color multi-sensor imaging system
Construct a dataset including videos collected from different scenes
for further research.
Present their color guided algorithm which is suitable for this kind of
super resolution problem.

5. INTRODUCTION
A registered high-quality texture image can provide significant information to
enhance the infrared image due to their strong correlation.
Use a guided filter[1] in the correlated region between IR image and color image.
Construct a cost volume of IR image values probability based on the input image.
A best cost selecting and sub-pixel refinement are taken to produce a refined IR
image.
The output image is got after a outlier detection.
1. Z. Zhang, “A flexible new technique for camera calibration,” Pattern

6. SYSTEM SETUP AND PREPROCESSING
System Setting
Camera Calibration and Color- IR Registration
Proposed Approach

7. SYSTEM SETTING
 Infrared camera(
Magnity MAG62) –
resolution 640*480)
 Visible camera(
Kedacom IPC120 )-
resolution 1280*960)
 Beam Splitter

8. CAMERA CALIBRATION AND COLOR-IR
REGISTRATION
First they tried to use mature algorithm proposed by Zhang [1].
Then finally they used the method proposed by Han [2].
1. Z. Zhang, “A flexible new technique for camera calibration,” Pattern
2. J. Han, E. Pauwels, and P. de Zeeuw, “Visible and infrared image registration employing line-based geometric analysis,” in Computational Intelligence for Multimedia
Understanding. Springer, 2011, pp. 114–125.

9. DATASETS
Fig show several images and their corresponding IR images in their datasets.

10. THE PROPOSED APPROACH

11. COST VOLUME
A coarse cost volume is first built to preserve the sub-pixel accuracy of the input
IR image based on current IR image estimation Ii
As the differences get large, the cost function should become constant in order to
make candidate IR values vary a lot because the current values are not
necessarily correct. The truncated quadric model is one of available function.
Thus the cost function can be defined as following:
 Where L is the search range,ƞ is a constant, d means a candidate IR image.

12. GUIDANCE SELECTION
The infrared image discontinuities often co-occur with color or
brightness changes within the associated camera image of the
same scene.
A color camera is a combination of three sensors: red, green, and
blue. Different channels have different correlation with IR image. To
verify this, we calculate the PSNR and SSIM between different
channels and the corresponding IR images in our database.

13. GUIDANCE SELECTION

14. THE CORRELATED REGION
To avoid wrong texture transfer, they only applied guided
filter in their correlated region. The correlated region could
be calculated by their cross-correlation in a small patch. If
the value is bigger than the threshold T, the point is regard
as one point of correlated region. The others is in
uncorrelated region.

15. GUIDED FILTER IN CORRELATED REGION
The guided image filter is a novel explicit image filter. It
not only has good edge-preserving smoothing properties
like the bilateral filter[1], but also makes the filtering output
more structured and keeps more details than other edge-
preserving filters.
1. C. Tomasi and R. Manduchi, “Bilateral filtering for gray and color

16. SUB-PIXEL ESTIMATION
As we mentioned in Cost volume, the discontinuity of cost
function and limited search range result in the
discontinuity of IR images. In order to eliminate this effect,
quadratic polynomial interpolation is adopted here to
approximate the cost function.

17. OUTLIER DETECTION
Outlier detection is used here to solve the problem that sometimes
black points may appear where tiny edges only exist in IR image.
we use outlier detection method to find the incorrect points and then
we adopt a median filter around them to eliminate outlier points.

18. THE PROPOSED APPROACH

19. EXPERIMENTAL RESULTS
Experiment results. From left to right are the suitable channel images, the ground truth IR images, the bicubic
interpolation results, the result using JBU and the results using our proposed algorithm. The result images has
been upsampled by a factor of 1:4 (in each axis).

20. EXPERIMENTAL RESULTS
PSNR(DB) AND SSIM OF IR IMAGE SUPER RESOLUTION

21. ACKNOWLEDGEMENT
• This work was supported in part by the National Science Foundation of China
under Grants 61422112, 61371146, 61521062, 61527804, and National High-tech
R&D Program of China under Grant 2015AA01590.

22. Any Question?
Thank you