Pseudo Four-Channel Image Denoising for Noisy CFA Raw Data (ICIP2015)
•
3 likes•1,353 views
Pseudo Four-Channel Image Denoising for Noisy CFA Raw Data
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (8)
Similar to Pseudo Four-Channel Image Denoising for Noisy CFA Raw Data (ICIP2015)
Similar to Pseudo Four-Channel Image Denoising for Noisy CFA Raw Data (ICIP2015) (20)
More from Masayuki Tanaka
More from Masayuki Tanaka (20)
Recently uploaded
Recently uploaded (20)
Pseudo Four-Channel Image Denoising for Noisy CFA Raw Data (ICIP2015)
- 1. Hiroki Akiyama, Masayuki Tanaka, and Masatoshi Okutomi Pseudo Four-Channel Image Denoising for Noisy CFA Raw Data
- 2. Color imaging with a single image sensor Single color image sensor Image processing pipeline Noisy CFA Raw data Noise-free color image Color imaging with a single image sensor requires color demosaicking and denoising. 1
- 3. Background: Color demosaicking Color demosaicking focuses on the color interpolation with noise-free CFA raw data. Color demosaicking Noise-free CFA raw data Full-color image Color demosaicking is still important and challenging. 2 Yesterday’s poster ELI-P3.5
- 4. Background: Denoising Denoising algorithms focus on reduction of Gaussian noise (Poisson noise). Denoising Gaussian noise is still important and challenging. Denoising Noisy image Noise-free image Neglecting effect of demosaicking Demosaicking error, Noise amplification, Correlated noise, and Non-Gaussian noise Correlated non-Gaussian noise 3
- 5. Goal: Denoising and Demosaiking Single color image sensor Denoising and Demosaicking Noisy CFA Raw data Noise-free color image Goal is to generate noise-free color image with noisy CFA raw data. In practice, we need to consider denoising and demosaicking simultaneously. 4
- 6. Three approaches 2. Denoising-after-demosaicking 3. Demosaicking-after-denoising 1. Joint denoising-and-demosaicking Demosaicking Denoising DemosaickingDenoising Denoising and demosaicking 5
- 7. Joint denoising-and-demosaicking Denoising and demosaicking Noisy CFA raw data Noise-free color image [11] K. Hirakawa, Joint demosaicing and denoising Image Processing, TIP 2006. [12] D. Paliy, Spatially Adaptive Color Filter Array Interpolation for Noiseless and Noisy Data, 2007. [13] L. Condat, Joint demosaicking and denoising by total variation minimization, ICIP 2012. [14] E. Dubois, Demosaicking of Noisy Bayer-Sampled Color Images With Least-Squares Luma-Chroma Demultiplexing and Noise Level Estimation, TIP 2013. Positives: Negatives: Performance is relatively good! It usually requires huge computational cost. It is not suitable to on- camera processing. 6
- 8. Denoising-after-demosaicking Demosaicking Denoising Noisy CFA raw data Noise-free color image Demosaicking: [2] L. Zhang, Color demosaicking via directional linear minimum mean square-error estimation, TIP 2005. [3] L. Zhang, Color demosaicking by local directional interpolation and nonlocal adaptive thresholding, JEI 2011. [4] D. Kiku, Residual Interpolation for Color Image Demosaicking, ICIP 2013. Denoising: [15] J. Mairal, Non-local sparse models for image restoration, ICCV 2009. [16] K. Dabov, Image denoising with block-matching and 3D filtering, EI 2006. Positives: Negatives: Simple and easy to understand. There are high-performance of demosaicking and denoising algorithms. The noise is amplified. Denoising of non-Gaussian noise is very challenging!
- 9. Demosaicking-after-denoising DemosaickingDenoising Noisy CFA raw data Noise-free color image CFA Raw data denoising: [8] A. Danielyan, Cross-color BM3D filtering of noisy raw data, 2009. [9] L. Zhang, PCA-based spatially adaptive denoising of CFA images for singlesensor digital cameras, TIP 2009. [10] S. H. Park, A case for denoising before demosaicking color filter array data, 2009. Positives: Negatives: Simple and easy to understand. There are high-performance of demosaicking. The CFA raw data denoising is challenging! If we can reduce the noise in the noisy CFA raw data, we can simply apply the high-performance demosaicking algorithms. We propose the CFA raw data denoising.
- 10. Outline of the proposed CFA raw data denoising Noisy CFA raw data Denoised CFA raw data PCA-based color transform Pixel intensities of pseudo four-channel image Pseudo four-channel image (Merged sub-images) Channel-by-channel denoising Inverse color transform Color-transformed domain 1. The CFAraw data is converted to pseudo four-channel image. 2. Denoising in the PCA-based color-transformed domain. 3. High-performance denoising algorithm can be applied. 9
- 11. Pseudo Four-Channel Image Noisy CFA raw data Four sub-images Pseudo four-channel image Four-channel data come from different sampling location. Four-channel data does not represent color. But, we consider four-channel data as color. We introduce the pseudo four-channel image. 10
- 12. Pseudo Four-Channel Image Noisy CFA raw data Four sub-images Pseudo four-channel image Four-channel data come from different sampling location. Four-channel data does not represent color. 11 Positives: Negatives: We can apply existing high- performance image processing algorithm to the pseudo four- channel image. It yields block artifacts due to four pixel blocks.
- 13. PCA-Based Color Transform Domain 12 [9] L. Zhang, PCA-based spatially adaptive denoising of CFA images for singlesensor digital cameras, TIP 2009. (vi is eigen vector of Σ) Color transform matrix xi : noise-free four-channel data ni: noise of four-channel data Noise variances
- 14. Noise variance in Color-Transformed Domain 13 Even if RGB noise variances are independent to channel, the noise variance in color-transformed domain depends on channel. Noise variance of each channel in color-transformed domain We apply the donising algorithm for channel depending noise variance model.
- 15. Block Artifacts 14 Pseudo four-channel image denoising Noisy CFA raw data Pseudo four-channel image Ground-truth W/O Block Artifact Reduction Block Artifact Reduction
- 16. What causes the block artifacts? 15 Demosaicking Full-color image with pattern (1,1) Noisy CFA raw data Denoised CFA raw data Denoising Noisy pseudo four-channel image with pattern (1,1) Denoised pseudo four-channel image with pattern (1,1) There are four patterns: (0,0), (1,0), (0,1) and (1,1) We can reduce the block artifacts by averaging four patterns of denoised CFA raw data.
- 17. Block Artifact Reduction 16 We can reduce the block artifacts by averaging four patterns of denoised CFA raw data. CFA raw data denoising with pattern (0,0) CFA raw data denoising with pattern (1,0) CFA raw data denoising with pattern (0,1) CFA raw data denoising with pattern (1,1) Demosaicking
- 18. Experimental Comparisons 17 Dataset: Kodak image dataset CFA denosing comparisons Full-color image comparisons BM3D[16] to CFA Zhang[9] Park[10] proposed Demosaicking only RI[5] Denoising-after-Demosaicking RI[5]->BM3D[16] Joint Denoising-and-demosaicking Paliy[12], Condat[13], Dubois[14] Demosaicking-after-Denoising (BM3D[13]toCFA)->RI[5], Zhang[9]->RI[5] Park[10]->RI[5], Proposed->RI[5]
- 19. CFA raw data denoising 18 Denoising Noisy CFA raw data Denoised CFA raw data Noise level PSNRofCFArawdata BM3D Zhang Park
- 20. Full-color image comparison 19 Noisy CFA raw data Full-color image
- 21. Visual comparisons (σ=20) 20 Proposed Ground truth RI [4] only RI [4]->BM3D[13] Paliy[8] Condat[9] BM3D[13]->RI[4] Zhang[6]->RI[4] Park[11]->RI[4]Dubois [10]
- 22. Visual comparisons (σ=40) 21 Proposed Ground truth RI [4] only RI [4]->BM3D[13] Paliy[8] Condat[9] BM3D[13]->RI[4] Zhang[6]->RI[4] Park[11]->RI[4]
- 23. Conclusion 22 DemosaickingDenoising Noisy CFA raw data Noise-free color image We have proposed the CFA raw data denoising to reconstruct full-color image from noisy CFA raw data. 1. Pseudo four-channel image 2. PCA-based color transformation 3. Block artifact reduction The matlab code is available online http://www.ok.ctrl.titech.ac.jp/res/CFADN/CFADN.html
Editor's Notes
- Next speaker is me, the title is pseudo four-channel image denoising for noisy CFA raw data. In this presentation, I will be talking about the denoising and the demosaicking.
- Color imaging with a single image sensor is widely used in digital camera. Actual output of the single color image sensor is called the CFA raw data which includes only one color component at each pixel. To obtain the color image, we need to apply image processing pipeline which includes demosaicking and denosing. The final color image quality strongly depends on these two processing.
- Color demosaicking is a technique to recover or to interpolate full color image from the CFA raw data. The Bayer CFA is de-fact of the color filter array. Many demosaicking algorithms focus for the Bayer CFA. Researches on the color demosaicking has long history and the performance of the color demosaicking has been improved year-by-year. I’d like to advertise our algorithm which we have presented yesterday, ARI. It has pretty good performance. If you have interests, please check the paper. The matlab code for it is also available online. However, these color demosaicking algorithms assume noise-free case. In practice, we need to take account noise effect.
- So, denoising. Denoising is essential and important image processing. BM3D is very famous and high-performance. However, almost all denoising algorithm is only for Gaussian noise or Poisson noise, neglecting the color demosaicking effect. Color demosaiking produce the demosaicking error, amplify the noise, makes correlation noise to signal, and changes the statistics of the noise. Reduction of this kind of correlated non-Gaussian noise is very challenging. There is a few algorithm for that purpose.
- The goal of this paper is to propose the image processing pipeline which generates the color image from noisy CFA raw data.
- There are three types of approaches to generate color image from noisy CFA raw data. First, joint denoising-and-demosaicking. It simultaneously performs denoising and demosaicking. Second, Denoising-after-demosaicking. The demosaicking is performed for the noisy CFA raw data. Then, denoising follows. Third, Demosaiking-after-denoising. The proposed algorithm adopts this. The noisy CFA raw data is denoised. Then, the demosaikcing is performed. I will briefly review these three approach.
- Joint denoising-and-demosaicking. Several algorithms has been proposed. EM algorithm or reconstruction approach is used. The performance of this approach is relatively good. However, it usually requires huge computational cost. So, this approach is not suitable to on-camera processing.
- Denoising-after-demosaicking. It is simple combination of existing algorithms. So, we can apply the existing high-performance of demosaicking and denoising algorithms. However, again, existing demosaicking algorithms neglect the noise and existing denoising is only for Gaussian noise.
- Demosaicking-after-denoising. The proposed algorithm uses this approach. If we can denoise the CFA raw data, we can simply apply the high-performance of demosaicking algorithm. But the CFA raw data denoising is challenging. So, we propose the CFA raw data denoising here.
- Here is outline of the proposed CFA raw data denoising. In the proposed algorithm, first, the CFA raw data is converted to the pseudo four-channel image. I will explain about the pseudo four-channel image later. Then, the PCA-based color transform is applied. The denoising is peformed in this color-transformed domain. The denoised data is transformed back to RGB color space. Then, we can get the denoised CFA raw data.
- Here, I introduce the pseudo four-channel image. We can get four sub-images by rearranging the pixels of the CFA raw data, like this. Then, we consider these four sub-images as a single four-channel image. Of course, four-channel data at the pixel come from different sampling location. And four-channel data does not represent color.
- It leads pros and cons. Positive side is that we can apply existing image processing algorithm to pseudo four-channel image, because we can consider it as four-color image. Negative is that it yields block artifacts due to this four pixel blocks. I will show later about the block artifacts.
- Now, we consider four-channel data as an image. So, we can apply normal image processing for it. PCA-based color transform is one of them. In RGB color case, the PCA-based color transform is related to Yuv color transformation. We follow the same approach as Zhang, considering the noise variances.
- We need to take account of the noise variances. Even if RGB noise variances are independent to channel, the noise variance in color-transform domain depends on channel. However, the noise variance of each channel in the color-transformed domain is easily estimated with the color-transformation matrix. After estimating the noise variances in the color-transformed domain, we apply the existing high-performance denoising algorithm.
- If we apply simply the pseudo four-channel image denoising, we will have the block artifact like this. This block artifacts come from these four-pixel block structure. So, we propose the block artifact reduction algorithm.
- It illustrate the block artifact in full-color image. It comes from four-pixel block when converting the four-channel image. This four-pixel block is not only one pattern. We can change the phase or start point of pixel.
- Actually, there are four patterns. Simply averaging these four pattern of denoised CFA raw data, we can reduce the block artifacts.
- Let’s move on experimental comparisons. We use Kodak image dataset. I will show two different types of evaluations. First CFA denoising comparisons. It evaluate the PSNR of the CFA raw data, not full color image. Second full-color image comparisons.
- Here, you can see that the comparisons of the CFA raw data denosing. We evaluate the PSNR of the denoised CFA raw data. The PSNR represents higher is better. Horizontal axis is noise level, and vertical axis is PSNR. As you can see, the proposed algorithm outperforms existing algorithms.
- Next, the comparisons of the restored full-color image. Dm represents demosaicking only, This is denoising-after-demosaciking approach. These three algorithms are joint denoising-and-demosaicking approach. These four are demosaicking-after-denoising. This comparison also show the proposed algorithm is the best performance among them.
- Here is the visual comparisons. But, I am afraid that it might be difficult to see the differences.
- This is another examples.
- Let me conclude as follows. We have proposed the CFA raw data denoising to restore the full-color image from the noisy CFA raw data. The keys of the proposed algorithm are, Introducing the pseudo four-channel image, Applying the PCA-based color transformation, And Block artifact reduction. The matlab code is available online. Please evaluate by yourself. Thank you. Any question?