Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[Woong Bae, Jaejun Yoo] Beyond deep residual learning for image restoration: Persistent Homology-Guided Manifold Simplification

495 views

Published on

CVPR NTIRE Workshop Challenge(SISR) 3rd ranked
Paper: https://arxiv.org/abs/1611.06345
Abstract:
The latest deep learning approaches perform better than the state-of-the-art signal processing approaches in various
image restoration tasks. However, if an image contains many patterns and structures, the performance of these CNNs is still inferior. To address this issue, here we propose a novel feature space deep residual learning algorithm that outperforms the existing residual learning. The main idea is originated from the observation that the performance of a learning algorithm can be improved if the input and/or label manifolds can be made topologically simpler by an analytic mapping to a feature space. Our extensive numerical studies using denoising experiments and NTIRE single-image super-resolution (SISR) competition demonstrate that the proposed feature space residual learning
outperforms the existing state-of-the-art approaches. Moreover, our algorithm was ranked third in NTIRE competition with 5-10 times faster computational time compared to the top ranked teams. The source code is available at https://github.com/iorism/CNN.git

Published in: Technology
  • Be the first to comment

  • Be the first to like this

[Woong Bae, Jaejun Yoo] Beyond deep residual learning for image restoration: Persistent Homology-Guided Manifold Simplification

  1. 1. Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification Paper Day for CVPR 2017 Woong Bae*, Jaejun Yoo* joint work with Yo Seob Han and Jong Chul Ye * denotes the co-first author CVPR Workshop NTIRE Competition 3rd Ranked
  2. 2. Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification Paper Day for CVPR 2017 Woong Bae*, Jaejun Yoo* joint work with Yo Seob Han and Jong Chul Ye * denotes the co-first author CVPR Workshop NTIRE Competition 3rd Ranked
  3. 3. Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification Woong Bae*, Jaejun Yoo* joint work with Yo Seob Han and Jong Chul Ye * denotes the co-first author CVPR Workshop NTIRE Competition 3rd Ranked Paper Day for CVPR 2017
  4. 4. Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification CVPR Workshop NTIRE Competition 3rd Ranked Paper Day for CVPR 2017 Woong Bae*, Jaejun Yoo* joint work with Yo Seob Han and Jong Chul Ye * denotes the co-first author
  5. 5. Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification CVPR Workshop NTIRE Competition 3rd Ranked Paper Day for CVPR 2017 Woong Bae*, Jaejun Yoo* joint work with Yo Seob Han and Jong Chul Ye * denotes the co-first author
  6. 6. Beyond Deep Residual Learning for Image Restoration: Persistent Homology-Guided Manifold Simplification CVPR Workshop NTIRE Competition 3rd Ranked Paper Day for CVPR 2017 Woong Bae*, Jaejun Yoo* joint work with Yo Seob Han and Jong Chul Ye * denotes the co-first author
  7. 7. CVPR Workshop NTIRE Competition 3rd Ranked
  8. 8. Theoretical Approach Paper Day for CVPR 2017
  9. 9. Paper Day for CVPR 2017 Image Restoration • Gaussian Denoising • Single Image Super Resolution (SISR)
  10. 10. Paper Day for CVPR 2017 Image Restoration • Gaussian Denoising • Single Image Super Resolution (SISR)
  11. 11. Paper Day for CVPR 2017 Image Restoration • Gaussian Denoising • Single Image Super Resolution (SISR)
  12. 12. Paper Day for CVPR 2017 * J Kim et al. CVPR 2016 Deep Residual Learning
  13. 13. Why it works? Intuition: Paper Day for CVPR 2017 simpler data manifold has simpler topology. learning simpler data manifold is easier.
  14. 14. Generalization Bound Paper Day for CVPR 2017  What we truly want to do: 𝐿𝐿 𝑓𝑓 = 𝐸𝐸𝐷𝐷 𝑌𝑌 − 𝑓𝑓 𝑋𝑋 2 Risk minimization where D is true distribution.  What we actually do: �𝐿𝐿𝑛𝑛 𝑓𝑓 = 1 𝑛𝑛 � 𝑖𝑖=1 𝑛𝑛 𝑌𝑌𝑖𝑖 − 𝑓𝑓 𝑋𝑋𝑖𝑖 2 Empirical Risk minimization since D is unknown.
  15. 15. Generalization Bound Paper Day for CVPR 2017  What we truly want to do: 𝐿𝐿 𝑓𝑓 = 𝐸𝐸𝐷𝐷 𝑌𝑌 − 𝑓𝑓 𝑋𝑋 2 Risk minimization where D is true distribution.  What we actually do: �𝐿𝐿𝑛𝑛 𝑓𝑓 = 1 𝑛𝑛 � 𝑖𝑖=1 𝑛𝑛 𝑌𝑌𝑖𝑖 − 𝑓𝑓 𝑋𝑋𝑖𝑖 2 Empirical Risk minimization since D is unknown.
  16. 16. Generalization Bound Paper Day for CVPR 2017 e.g. Underfitting vs Overfitting Model complexity Loss(error) Empirical Risk True Risk Complexity Penalty
  17. 17. Generalization Bound Paper Day for CVPR 2017 e.g. Underfitting vs Overfitting Model complexity Loss(error) Empirical Risk True Risk Complexity Penalty Fixed
  18. 18. Key Idea Paper Day for CVPR 2017 e.g. Underfitting vs Overfitting Model complexity Loss(error) Empirical Risk True Risk Complexity Penalty Fixed Is that all?
  19. 19. Key Idea Paper Day for CVPR 2017 Fixed “MAKE THE PROBLEM EASIER TO SOLVE!” Reduce the complexity of the data manifold such that simple neural network can be used.
  20. 20. Key Idea Paper Day for CVPR 2017 Fixed “MAKE THE PROBLEM EASIER TO SOLVE!” Reduce the complexity of the data manifold such that simple neural network can be used. 1. How can we “measure the complexity” of the data manifold?  Topological Data Analysis: Persistent homology 2. What is the proper “Transform” to reduce the complexity?  Residual (artifact) learning, Wavelet transform (Haar)
  21. 21. Topological Data Analysis (TDA) • You may have seen this famous example: Coffee cup vs. Doughnut vs. Circle Paper Day for CVPR 2017 “A topologist is a person who cannot tell the difference between a coffee mug and a doughnut”
  22. 22. Data Analysis Paper Day for CVPR 2017 Determining the global structure of a noisy point cloud is not difficult when the points are in 𝐸𝐸2 , but for clouds in higher dimensions, a planar projection is not always easy to decipher. Figure adopted from R. Ghrist “BARCODES: THE PERSISTENT TOPOLOGY OF DATA”
  23. 23. Benefits of TDA • Topology can summarize the general or global feature of the manifold (even under the high dimensional space). explain how a material's shape can be completely deformed into new one without losing its core properties called... “Topological invariants” (e.g. homology of the space) Paper Day for CVPR 2017 Hole dim. 0 : cluster Hole dim. 1 : cycle Hole dim. 2 : void * Homology = Hole
  24. 24. Persistent Homology • Examines the persistence of the homology group Watch the evolution of simplicial complex K increasing the radius, or threshold(𝝐𝝐). Paper Day for CVPR 2017
  25. 25. Paper Day for CVPR 2017 Persistent Homology
  26. 26. Paper Day for CVPR 2017 Persistent Homology
  27. 27. Paper Day for CVPR 2017 Persistent Homology simpler manifold merges faster
  28. 28. Paper Day for CVPR 2017 Results
  29. 29. Paper Day for CVPR 2017 Network Design Principle 𝑿𝑿𝑿 𝒇𝒇𝜽𝜽 𝝋𝝋(𝒚𝒚) − 𝒙𝒙𝒙 = 𝒓𝒓 𝒓𝒓 𝜑𝜑 𝑿𝑿 𝒀𝒀 = 𝝋𝝋−𝟏𝟏 (𝑿𝑿𝑿 + 𝒓𝒓) Wavelet transform Residual mapping
  30. 30. Summary Paper Day for CVPR 2017 • Simpler data manifold has simpler topology. • Learning simpler data manifold is easier. • Persistent Homology to measure the complexity of the data manifold. • Explain the reason of good performance in “Deep Residual Learning”. • Manifold simplification by using wavelet transform.
  31. 31. Practical Approach Paper Day for CVPR 2017
  32. 32. content • Contribution • Gaussian Denoising problem • Bicubic Single Image Super Resolution problem • Unknown Single Image Super Resolution problem • Behind Story Paper Day for CVPR 2017
  33. 33. What is our contribution? • We have proposed an efficient network which is possible to improve performance by changing only the input and label, leaving the network alone. • We tried to analyze the meaning of residual and wavelet decomposition on CNN using Persistent homology. Paper Day for CVPR 2017
  34. 34. What is Gaussian Denoising Denoising method [ Denoised image ][ Gaussian Noisy image ] [Gaussian Noise][Clean image] Paper Day for CVPR 2017
  35. 35. What is residual learning Label : clean image Input : noisy image output : denoised image Original learning Label : noise Denoising Input : noisy im output : estimated noise Residual learning noisy im clean im Denoising Paper Day for CVPR 2017
  36. 36. Proposed network for Gaussian Denoising 4 4320 Module unit 320 Moduleunit2 Moduleunit5 320 320320320320 X : InputNoisy Image X – Y Denoised Image L L H L L H HH Y : Label L L H L L H HH L L H L L H HH DWT IDWT 3x3 Conv, ReLU 3x3 Conv, bnorm ReLU 3x3 ConvBypass 3x3 Conv, bnorm, ReLU 320 [ Proposed Network for denoisng ] 1. Deep residual learning based on wavelet transform 2. DWT : Discrete Wavelet Transform 3. Residual learning in wavelet domain Paper Day for CVPR 2017
  37. 37. Advantages of Wavelet decomposition 1. Make more simple data 2. Reduction of patch size efficiently 1) Reduction of network run time and learning time 2) Reduction of minimal receptive (if the training data is simple ...) 3. Emphasize various frequencies due to wavelet transform (Be careful not to be biased on one frequency) [ Down-sampled image ] [ Original image ] 512 pixel 256 pixel 40 pixel 80 pixel 512 pixel [ Original image ] [ Patch size between decomposed and original] Paper Day for CVPR 2017
  38. 38. Comparison by depth of filter 1. The deeper the filter depth, the more expressive and powerful. 2. The deeper the layer, the more expressive and powerful. Paper Day for CVPR 2017
  39. 39. Comparison by method : PQF 1. Polyphase Quadrature Filter (even / odd decomposition) 2. Because the networks are exactly the same, accurate comparisons are possible. 3. Network power and learning level are important for accurate comparison. [ PQF decomposition ] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 3 9 11 2 4 10 12 5 7 13 15 6 8 14 16 PQF [ Effect of wavelet transform for denoising ] Paper Day for CVPR 2017
  40. 40. Learning for Gaussian denoising 1. L2 loss + L2 regularization 2. Initialization using improved Xavier method 3. Add Gaussian noisy every epoch  Data augmentation 4. Data augmentation : flip, rotation, random cropping 5. Pick only features that are not good at learning, rotate by 3 to 7degrees 6. Using SGD 7. 0.1 gradient clipping  The top 3% values at first epoch 8. 40x40 patch 9. Batch norm : 32 (best) 10. Training data : BSD400 + urban100 11. 128 random-sampled Sub-epoch (total 53 epoch) 12. Haar wavelet is the best. 13. Noisy mixed(𝛿𝛿=5~55) learning is also good Paper Day for CVPR 2017 [ barbara image ]
  41. 41. Result for Gaussian denoising • Firstly Barbara score of our network higher than BM3D • Noisy mixed(𝛿𝛿=5~55) learning is also good Paper Day for CVPR 2017
  42. 42. Result image for Gaussian denoising Paper Day for CVPR 2017
  43. 43. Bicubic SISR SISR Bicubic x4 down-sampling Paper Day for CVPR 2017
  44. 44. 3rd Network for NTIRE SISR for bicubic CVPR workshop competition (1) • Applying wavelet transform to each RGB channel • Long bypass 12 [ 4 x 3(RGB) ] 256 Moduleunit1 Moduleunit12 256 256 X : Input 4ch x 3ch(RGB) Bicubic upsampling Image – RGB(3ch) X – Y SISR Image L L H L L H H H Y : Residual 4ch x 3ch(RGB) L L H L L H H H L L H L L H H H DWT IDWT 3x3 Conv, ReLU 3x3 bnorm, ReLU ReLU 3x3 ConvLong Bypass 3x3 Conv, bnorm, ReLU 256 12 [ 4 x 3(RGB) ] R-4ch G-4ch B-4ch R-4ch G-4ch B-4ch Moduleunit6 Moduleunit7 256 256 256 256 [ Proposed SISR Network for test phase ] Paper Day for CVPR 2017
  45. 45. Long bypass 1. Reduction of gradient vanishing problem 2. Suitable for images due to faster than Dense net - Although it is possible to make a fast reconstruction by partition image reconstruction, It increases the redundancy at the boundary. 3. Put Long-bypass between 20 of conv  After merging, do BN + ReLU (stable gradients) Paper Day for CVPR 2017
  46. 46. Unknown SISR SISR Unknown x4 down-sampling ? Paper Day for CVPR 2017
  47. 47. Residual shuffling • Shuffling is simply sampling with even / odd. • We tried the residual of shuffling by matching the input and output domains. • Non-use wavelet  It is hard to match the input and label when the number of channels was too high Paper Day for CVPR 2017 1 2 3 4
  48. 48. 3rd Network for NTIRE SISR for unknown CVPR workshop competition (2) • Shuffled residual learning • Non wavelet decomposition 12 or 27 or 48 [ scale x scale 3(RGB) ] 320 Module unit 320 Moduleunit2 Moduleunit12 320 320320320320 X : Input scale x scale 3ch(RGB) Down Sampled Image – RGB(3ch) X – Y SISR Image L L H L L H H H Y : shuffled Residual Label scale x scale 3ch(RGB) L L H L L H H H L L H L L H H H shuffling Inverseshuffling 3x3 Conv, ReLU 3x3 Conv, bnorm ReLU 3x3 ConvBypass 3x3 Conv, bnorm, ReLU 320 12 or 27 or 48 [ scale x scale 3(RGB) ] R-ch G-ch B-ch R-ch G-ch B-ch [ Proposed SISR Network for test phase ] Paper Day for CVPR 2017
  49. 49. Learning for SISR NTIRE & 291dataset 1. If the pattern of data is insufficient (291 sets), overfitting well.  control dropout rate, regularization factor  Data augmentation : RGB augmentation, add noisy 2. Sometimes except for a patch with a low sum of 2D gradient 3. Patch : 20x20 4. L2 loss + L2 regularization 5. Data augmentation : flip, rotation, random crop 6. Using SGD 7. 0.05 gradient clipping ( The top 3% ) 8. Batch norm : 64 9. NTIRE competition data : DIV2K (2K resolution 1000 images) 10. 512 random Sub-epoch (150 epoch) 11. For Bicubic x3, x4 problem, Bicubic x2, x3, x4 dataset was learned simultaneously. 12. For Unknown SISR, 20x20 patch is not optimal.  Because we don’t know what downsampling method us. Paper Day for CVPR 2017 [ DIV2K image ]
  50. 50. Result image for SISR Paper Day for CVPR 2017
  51. 51. Result image for SISR (NTIRE-DIV2K) Paper Day for CVPR 2017
  52. 52. Behind Story… – Why use the Batch norm? • 학습이 빠르고 쉬우니깐…  잘되니깐?? regularization 효과  문제의 단순화 • 하지만 learning 결과에 제약을 거는 느낌이 있음  exponential moving avg 때문?  scale과 shift 파라미터 학습의 정확도 때문? • Batch norm를 어느정도 학습한 후에 제거해보면?  Batch norm를 convolution layer의 bias로 보낸 후 다시 러닝.  네트워크 특성에 따라 Batch norm을 쓰고 말지를 결정하는 것이 가 장 현명할듯 하다. Paper Day for CVPR 2017
  53. 53. Behind Story… – Why use the Batch norm? • Reduce dead ReLU pixel? Paper Day for CVPR 2017 bias of 1st conv for wavelet decomposition bias of 1st conv for shuffling Batch norm of 2st for wavelet decomposition Batch norm of 2st for shuffling mean variation variation mean
  54. 54. Behind Story… – Why use ReLU? • ReLU is thresholding(sparse)  dicision tree  so fast • Threashold값은 batch norm 또는 bias가 조절해 줄 것으로 믿음  따라서, relu이전의 convolution 의 bias나 batch norm의 효율성이 중요하다고 생각함 • network power가 충분하다면 유의미함  마지막 convolution이 부호를 정해줄 수 있음  많은 linear + nonlinear 조합으로 표현 가능 • pReLU 도 확인해볼 필요는 있어보임 Paper Day for CVPR 2017
  55. 55. Denoising *Bae et al. 2017 https://arxiv.org/abs/1611.06345
  56. 56. Result image for SISR (291dataset) High Res DnCNN-3 DRCN Proposed (a) Ground Truth (b) VDSR (c) DnCNN-3 (d) Proposed Paper Day for CVPR 2017
  57. 57. Single Image Super Resolution CVPR Workshop 2017 NTIRE Challenge ranked 3rd *Bae et al. 2017 https://arxiv.org/abs/1611.06345
  58. 58. Single Image Super Resolution CVPR Workshop 2017 NTIRE Challenge ranked 3rd *Bae et al. 2017 https://arxiv.org/abs/1611.06345

×