COMPUTER 10: Lesson 7 - File Storage and Online Collaboration
Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering
1. Comparison between Blur Transfer and
Blur Re-Generation in Depth Image
Based Rendering
Norishige Fukushima†, Naoki Kodera†, Yutaka Ishibashi†,
Masayuki Tanimoto‡
July 2-4, 2014 Budapest, Hungary 3DTV-CON 2014
†Graduate School of Engineering, Nagoya Institute of Technology,
Japan
‡Nagoya Industrial Science Research Institute,
Japan
2. Outline
Background
Related Works
View Synthesis Methods
Blur erasing type
Blur re-generation type
Blur transfer type
Experimental Results
Conclusion and Future Works
10. View synthesis method
3 types of blur treatment
Blur erasing type
Blur re-generation type
Blur transfer type
Improved blur transfer method
Blur erasing type
Blur re-generation type
Blur transfer type
Improved blur transfer method (proposed method)
11. View synthesis method
3 types of blur treatment
Blur erasing type
Blur re-generation type
Blur transfer type
Improved blur transfer method
Blur erasing type
Blur re-generation type
Blur transfer type
Improved blur transfer method (proposed method)
14. View synthesis method
3 types of blur treatment
Blur erasing type
Blur re-generation type
Blur transfer type
Improved blur transfer method
Blur erasing type
Blur re-generation type
Blur transfer type
Improved blur transfer method (proposed method)
15. Blur re-generation type
Generating blurred region using alpha matting
Splitting input image into three images
→ foreground, background and alpha mask
Alpha blending three images after DIBR
Input image
Split
Foreground
Alpha mask
Background
17. View synthesis method
3 types of blur treatment
Blur erasing type
Blur re-generation type
Blur transfer type
Improved blur transfer method (proposed method)
23. Improved blur transfer method
(proposed method)
Generating mask by canny filter
Smoothing masked region by Gaussian filter
Generate mask
Smooth
Masked region
24. Comparison
Blur erasing type
× Blurs are broken. ◎ Fastest
Blur re-generation type
○ Blurs are reconstructed. ×Slowest
Blur transfer type
○ Blurs are kept. ○ Faster
For better boundary treatment, we compare
blur re-generation type with blur transfer type.
25. Experimental Results (1/6)
Evaluating PSNR of these methods
Basic type [1]
Blur re-generation type [2]
Blur transfer type [3]
Improved blur transfer method (proposed method)
: TeddyH
(950×750)
[1]: Y. Mori et al., Signal Process. Image Commu., vol. 24, no. 1, pp. 65-72, Jan. 2009.
[2]: N. Kodera et al., IEEE VCIP 2013.
[3]: X. Xu et al., IEEE ICASSP 2012.
: Bowling1
(671×555)
: Reindeer
(671×555)
Input depth map is ground truth
28. Experimental Results (4/6)
Reindeer: 671 x 555
Re-generationProposed
In proposed method, blurs around object
boundary blend background color.
Proposed method dilates depth value to the
unnecessary region.
29. Experimental Results (5/6)
Bowling1: 671 x 555
Re-generationProposed
Proposed method interpolates using
foreground or mixed color.
Depth maps are not dilated enough.
30. Experimental Results (6/6)
Computational cost
Proposed method ≒ Basic method (15ms) + 4ms
2 dilations for input depth map < 1ms
Canny filter for depth map on the synthesized image < 3ms
Gaussian filter with a small kernel < 1ms
Blur re-generation type > 100ms
Matting ≒ 55ms
3 times DIBR ≒ 45ms
Blending < 2ms
31. Conclusions
Comparing 3 types of DIBR method in blur treatment.
Proposed method improves subjective quality at object boundaries,
and objectively has 0.61dB improvement.
This is the second best (0.31dB lower than the blur re-generation).
Proposed method reaches the state of the arts of blur regeneration
method, and computational cost is about x5 effective.
Proposed (19 ms) vs Blur re-generation (>100 ms)
Future Work
Investigating effect of proposed method using estimated
depth maps.
Considering effect of coding distortions.
Editor's Notes
Today I introduce Comparison between Blur Transfer and Blur Re-Generation in Depth Image Based Rendering.
This is a rendering method for view synthesis.
The outline of our presentation is as follows;
At first , we introduce background and Related Works
Then, we review 3 view synthesis method.
Blur erasing type
Blur re-generation type
Blur transfer type
Next, Experimental Results are shown,
And finally, we conclude our presentation.
I bileifly introduce a ftee viewpoint rendering method of depth image based rendering DIBR.
The DIBR use RGB images and its associated depth maps.
For the free viewpoint image rendering, the pair of the input view and the depth map are warped to the desired viewpoint.
Next, oposit side of view is also warped, and then, still remaining holes are filled.
In this presentation, we improve the rendering result of this part. This is the zoomed image.
In this region, fore ground image and background image are mixed.
warping in this condition, the mixed color and the foreground object are split.
and after hole filling, the remained mixed color looks artifact.
Thus the boundary regions are degraded.
To solve the boundary region problem, we introduce 3 type of method.
1st one is Blur erasing type
2nd one is Blur re-generation type
3rd one is Blur transfer type.
In this presentation, we compare the three type of the methods.
And we make a little improvement for the last type.
The fast type is blur erasing type.
In this type, mixed color is erased by some method, for example erosion or thresholding…
The method can remove mixed color in the background side. But the shape of blur is broken. The radius of blur become small.
The 2nd type is blur regeneration type.
The type use alpha matting processing.
With alpha mating, an input natural image is split into 3 part of images, foreground, background and alpha mask.
The foreground and background image has mixed color in there images and the mixed information is represented by the alpha mask.
For DIBR processing with alpha matting, we warp these images individually. And then, we alpha-blend the warped fore and background images with warped alpha map.
The method can suppress background mixed color, and also keep the shape of blur.
Thus the method has the highest image quality. The drawback of this type is computational cost. The method requires 3 times DIBR and a additional processing of matting.
The last type is blur transfer type.
With this type, the depth map is dilated to cover the mixed region before warping.
Alter the warping blur is kept and mixed color is stiched to the foreground object.
Alter the warping, we perform holefilling and then we can obtain free viewpoint images.
The flow of the type is dilation for the depth maps and then process DIBR.
We add simple process to improve the quality of this type.
The additional processes are fundamental processing of Canny and Gaussian filtering/
In the additional process, we first perform Canny filtering to detect discontinuity regions, and then filter the region by Gaussing an filtering.
The assumption of the blur transfer type is background color is simple or flat, and shape of blur is not changed after the warping.
To keep the assumption, the smoothing of the object boundary is effective.
To review there types the characteristics of the types are as shown.
Then we compare them.
We use Middlebury stereo dataset for our evaluation. The data set has ground truth depth map and multi view images.
We render the center viewpoint between the left and right images by using the 3 types of DIBR, and then compare with the captured RGB image by using PSNR.
We use 30 pairs in the dataset.
This is a table of PSNR.
Left side of basic is blur erasing type
And next side of re-generation is blur regeneration with matting method
And transfer means with depth map dilation and proposed has additional processing of canny and Gaussian filtering for the blur transfer type.
The results show that blur regeneration type has the highest PSNR and the proposed method is the 2nd best.
But the difference between two method is about 0.3 dB
This is a rendering result. The blur regeneration and proposed method looked same and the contor artifacts are soften.
This is a bad case of proposed and blur transferred type.
The computational cost of the type are shown.
The basic method and blur transfer, proposed method have realtime performance.
But blur regeneration method does not has realtime performance, even if the method uses the fastest matting method.