Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 1: prior embedding deep rain removal

23 views

Published on

ICME2019 Tutorial: Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 1: prior embedding deep rain removal

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 1: prior embedding deep rain removal

  1. 1. IEEE ICME-2019 8:30 – 12:00, July 08 Shanghai, China Intelligent Image/Video Editing Tutorial
  2. 2. STRUCT Group Jiaying Liu Wenhan Yang Prior Embedding Deep Rain Removal Part 1 IEEE ICME-2019
  3. 3. Outline Background / 012 Single Image Rain Streak Removal / 024 Multi-Frame Rain Streak Removal / 061 Single Image Rain Drop Removal / 111 Prior Embedding Deep Rain Removal
  4. 4. Outline Background / 012 Single Image Rain Streak Removal / 024 Multi-Frame Rain Streak Removal / 061 Single Image Rain Drop Removal / 111 Prior Embedding Deep Rain Removal
  5. 5. STRUCT Group5 Background Visual Degradation • Heavy Rain/Snow • Underwater • Low Light • Haze/Sandstorm • Downsample • Motion Blur • System Noise • Optical Distortion Degradation in Data Acquisition Degradation before Data Acquisition Degradation after Data Acquisition Data Acquisition • Scratches • Watermark • Mildew • Compression Loss
  6. 6. STRUCT Group6 Background Visual Intelligent Computing Underwater Enhancement Dehazing Text Removal Super Resolution Rain Streak Removal Denoising Low Light EnhancementRain Drop Removal
  7. 7. STRUCT Group Background Prior Embedding Deep Rain Removal12  Data-Driven Solution Low Quality Image/Video High Quality Image/Video Video Surveillance Big Data Data-Driven Solution Feature Representation Feature Mapping + • Learning to be intelligent
  8. 8. STRUCT Group Background Prior Embedding Deep Rain Removal8  Image Degradation Model k k y DHF x n  Model Formulation Noise nDownSampleDBlur HMotion Fk x ky Scene Observations Atmosphere n
  9. 9. STRUCT Group Background Prior Embedding Deep Rain Removal9  Maximum-a-posteriori (MAP) estimation • Based on Lk, estimate x • Bayesian rule • Prior (regularization term) • Preference to the solutions, statistic information         1 1 ˆ max | max log | log p p k kk k P P P     x x x x y y x x Likelihood (Fidelity) Prior(Regularization)     2 2 ˆ argmin k k k k p x x D H x y x
  10. 10. STRUCT Group Intelligent Visual Enhancement Prior Embedding Deep Rain Removal10  Without Priors  Side Priors / Joint Task  Learned Priors ModelInput Output Model Input Output Side Input Regularization ModelInput Output Regularization
  11. 11. STRUCT Group Intelligent Visual Enhancement Prior Embedding Deep Rain Removal11 Image Rain Streak Removal Raindrop Removal Side Prior Side Prior Learned Prior Joint Rain Detection and Removal GAN-Based Attention-Guided Generator Discriminator Yes/No? Video Rain Removal Recurrent Rain Removal and Reconstruction RNN RNN RNN
  12. 12. STRUCT Group Research Lists Prior Embedding Deep Rain Removal12  Image Super-Resolution  Video Super-Resolution  Image Denoising  Image and Video Deraining  Image and Video Compression  Image Processing Datasets • https://github.com/flyywh/Rain-Removal • https://github.com/flyywh/Image-Denoising-State-of-the-art • https://github.com/flyywh/Image-compression-and-video-coding • https://github.com/flyywh/Video-Super-Resolution • https://github.com/flyywh/Image-Processing-Datasets • https://github.com/flyywh/Super-Resolution.Benckmark
  13. 13. STRUCT Group Workshop & Challenges Prior Embedding Deep Rain Removal13  NTIRE-2017/2018/2019 (Joint with CVPR) • Super-Resolution • Dehazing • Spectral Image Super-Resolution • Image Denoising in Real Scenario • Raw Image Denoising • Super-Resolution in Real Scenario • Image Colorization • Image Deblurring • Video Deblurring • Video Super-Resolution
  14. 14. STRUCT Group Workshop & Challenges Prior Embedding Deep Rain Removal14  PIRM-2018 (Joint with ECCV) • Perceptual Image Restoration and Manipulation  CLIC-2018/2019 (CVPR) • Novel encoder/decoder architectures, flow control between the encoder and the decoder, and learn how to quantize (or learn to quantize) better.
  15. 15. STRUCT Group Workshop & Challenges Prior Embedding Deep Rain Removal15  UG2-2018 (Joint with CVPR)  Objective  Track • Can the application of enhancement algorithms as a pre- processing step improve image interpretability for manual analysis or automatic visual recognition to classify scene content? • Image enhancement to facilitate manual inspection • Image enhancement to improve automatic object recognition
  16. 16. STRUCT Group Workshop & Challenges Prior Embedding Deep Rain Removal16  UG2-2018 (Joint with CVPR)
  17. 17. STRUCT Group Workshop & Challenges 17  UG2-2018 (Joint with CVPR) • Low Resolution • Low Light • Noise • Blocking • Blurring • Reflection • Lens Flare • Over Exposure • Under Exposure • Turbulence • Annotation • Rain Drop Blurring Low Light Low Resolution Over-Exposure Blocking Noise & Annotation Prior Embedding Deep Rain Removal
  18. 18. STRUCT Group Workshop & Challenges 18  UG2-2018 (Joint with CVPR) • Runner-up Award on the Track “Automatic Object Recognition" • Our Solution: Sequential Restoration for Visual Recognition Code: https://github.com/yyvettey/TAMU-PKU-UG2 Report Slide: https://github.com/flyywh/flyywh.github.io/blob/master/att/UG2-Slides-v1.3.pdf Prior Embedding Deep Rain Removal
  19. 19. STRUCT Group Workshop & Challenges Prior Embedding Deep Rain Removal19  UG2+2019 (Joint with CVPR)  Detections in Degraded Conditions • (Semi-)Supervised Object Detection in Haze Conditions • (Semi-)Supervised Face Detection in Low Light Conditions • Zero-Shot Object Detection with Raindrop Occlusions
  20. 20. Outline Background / 012 Single Image Rain Streak Removal / 024 Multi-Frame Rain Streak Removal / 061 Single Image Rain Drop Removal / 111 Prior Embedding Deep Rain Removal
  21. 21. STRUCT Group21 Single Image Rain Streak Removal Visual Loss in Bad Weather  Bad weather conditions  Visibility degradations
  22. 22. STRUCT Group22 Computer Vision in Bad Weather Applying the Faster RCNN detector released from [Ren15]  Bad weather makes CV methods invalid  Lost details and irregular signal distribution Single Image Rain Streak Removal
  23. 23. STRUCT Group23 Several Concepts Rain streak Mist Rain drop Rain drop Single Image Rain Streak Removal
  24. 24. STRUCT Group24 Representative Work Single Image Rain Streak Removal Yu Luo, Yong Xu, Hui Ji, Removing rain from a single image via discriminative sparse coding, ICCV, 2015.  Image modeling  Linear additive composite model  Screen Blend Model  Discriminative sparse coding  Fidelity  Rain Model  Mutual exclusivity 2015 ICCV Discriminative Sparse Coding
  25. 25. STRUCT Group25 Representative Work Single Image Rain Streak Removal Yu Li, Robby T. Tan, Xiaojie Guo, Jiangbo Lu, and Michael S. Brown, Single Image Rain Streak Decomposition Using Layer Priors, CVPR, 2016. Input Rain Patch Background Rain Streaks Maximum a posteriori Background Prior Rain Prior 2015 ICCV Discriminative Sparse Coding 2016 CVPR Layer Prior
  26. 26. STRUCT Group26 Representative Work Single Image Rain Streak Removal 2015 ICCV Discriminative Sparse Coding 2016 CVPR Layer Prior 2017 CVPR DetailNet  Negative residual mapping  Reduce the mapping range  Learning process  Easier Xueyang Fu, Jiabin Huang, et al., Removing rain from single images via a deep detail network, CVPR, 2017.  Image detail layer  Remove background interference  Improve de-raining quality
  27. 27. STRUCT Group27 Representative Work Single Image Rain Streak Removal 2015 ICCV Discriminative Sparse Coding 2016 CVPR Layer Prior 2017 CVPR DetailNet He Zhang, Vishal M. Patel, Density-aware single image de-raining using a multi-stream dense network, CVPR, 2018. 2018 CVPR DID-MDN  Rain Prior  Rain density detection  Remove accordingly  Feature Learning  Multi-stream DenseNet  Characterize rain-streaks with different scales and shapes
  28. 28. STRUCT Group28 Representative Work Single Image Rain Streak Removal 2015 ICCV Discriminative Sparse Coding 2016 CVPR Layer Prior 2017 CVPR DetailNet Xia Li, Jianlong Wu, Zhouchen Lin, Hong Liu, Hongbin Zha, RESCAN: Recurrent Squeeze-and-Excitation Context Aggregation Net, ECCV, 2018. 2018 CVPR DID-MDN 2018 ECCV RESCAN  Stage-Wise Removal  RNNs model different stages  Feature Learning  Contextual dilated network  SE: rain layer attention
  29. 29. STRUCT Group29 Single Image Rain Streak Removal  Single Image Rain Streak Removal Deep Joint Rain Detection and Removal From a Single Image Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan, CVPR 2017
  30. 30. STRUCT Group30 Previous Works  Classification in texture feature space  Morphological Component Analysis [Kang12]  Discriminative Sparse coding [Luo15]  Rain Streak Removal Using Layer Priors (LP) [Li16]  Fail to handle heavy rain cases Light Rain Cases Heavy Rain Cases Rain image LP Rain image LP Single Image Rain Streak Removal
  31. 31. STRUCT Group31 Our Aim: Heavy Rain Cases  Heavy Rain Problems  Heavy rain  Different types of rain streaks in the same image  Mist  Distant rain accumulation, like haze Single Image Rain Streak Removal
  32. 32. STRUCT Group32 Rain Image Generation (1/3)  Traditional Rain Synthesis Model  Additive Model [Lu15, Li16] ෨𝐒 is not consistently distributed  designing prior is hard Signal separation  loss texture detail in non-rain regions , O B S = + Single Image Rain Streak Removal
  33. 33. STRUCT Group33 Rain Image Generation (2/3)  Heavy rains  Mist 1 , s t t    O B S R   1 + 1- , s t t          O B S R Α Single Image Rain Streak Removal
  34. 34. STRUCT Group34 Rain Image Generation (3/3)  Region Dependent Rain Removal  Separating streak location + rain level  Region detection enable implicit separation processing for rain / non-rain regions = +  = + [ICCV15, CVPR16] Single Image Rain Streak Removal
  35. 35. STRUCT Group35 Our Aim:  Syntheses and model rain images better from Data Generation to Model Design Degradation model Data Generation A Modeling Training Testing and refine ?  Region dependent rain synthesis  Heavy rain generation  Joint rain detection and removal  Recurrent derain and dehaze Single Image Rain Streak Removal
  36. 36. STRUCT Group36 Joint Rain Detection and Removal  Multi-Task Learning Rain Features (Ft) LR LS LB Input (O0) [Ft , Rt] [Ft, Rt, St] Background (Bt)Rain Streak (St)Rain Mask (Rt) Rain Images (Ot) Joint Rain Detection and Removal T(•) Convs Convs Convs CNNs Single Image Rain Streak Removal
  37. 37. STRUCT Group37 Rain Features (Ft) LR LS LB Bt+1 = Ot - T(Ot) Input (O0) Ot = Bt [Ft , Rt] [Ft, Rt, St] Background (Bt)Rain Streak (St)Rain Mask (Rt) Rain Images (Ot) Demist Joint Rain Detection and Removal T(•) Convs Convs Convs CNNs Recurrent Joint Derain and Demist  Recurrent Rain Removal  One type for each  Demist  Derain  Demist  Derain Single Image Rain Streak Removal
  38. 38. STRUCT Group38 Objective Evaluation Compared Methods  Proposed-, Proposed, Proposed-R  LP[Li16], DSC[Luo15], SRCNN[Dong14], CNN Rain Drop[Eigen13] Datasets  Rain12 [Li16]  Rain100L, Rain100H Normal case Heavy case Baseline Rain12 Rain100L Metric PSNR SSIM PSNR SSIM ID 27.21 0.75 23.13 0.70 DSC 30.02 0.87 24.16 0.87 LP 32.02 0.91 29.11 0.88 CNN 26.65 0.78 23.70 0.81 SRCNN 34.41 0.94 34.41 0.94 Proposed 35.86 0.96 36.11 0.97 Baseline Rain100H Metric PSNR SSIM ID 14.02 0.5239 DSC 14.26 0.4225 LP 15.66 0.5444 Proposed- 20.79 0.5978 Proposed 22.15 0.6736 Proposed-R 23.45 0.7490 Single Image Rain Streak Removal
  39. 39. STRUCT Group39 Subjective Evaluation Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Single Image Rain Streak Removal
  40. 40. STRUCT Group40 Subjective Evaluation Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Single Image Rain Streak Removal
  41. 41. STRUCT Group41 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  42. 42. STRUCT Group42 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  43. 43. STRUCT Group43 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  44. 44. STRUCT Group44 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  45. 45. STRUCT Group45 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  46. 46. STRUCT Group46 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  47. 47. STRUCT Group47 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  48. 48. STRUCT Group48 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  49. 49. STRUCT Group49 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  50. 50. STRUCT Group50 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  51. 51. STRUCT Group51 DSC[ICCV15] Ours Rain Image ID[TIP12] LP[CVPR16] Subjective Evaluation Single Image Rain Streak Removal
  52. 52. STRUCT Group52 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  53. 53. STRUCT Group53 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  54. 54. STRUCT Group54 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  55. 55. STRUCT Group55 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  56. 56. STRUCT Group56 Rain Image ID[TIP12] LP[CVPR16] DSC[ICCV15] Ours Subjective Evaluation Single Image Rain Streak Removal
  57. 57. STRUCT Group57 Single Image Rain Streak Removal  Single Image Rain Streak Removal Joint Rain Detection and Removal from a Single Image with Contextualized Deep Networks Wenhan Yang, Robby T. Tan, Jiashi Feng, Zongming Guo, Shuicheng Yan, and Jiaying Liu, TPAMI 2019
  58. 58. STRUCT Group58 Joint Rain Detection and Removal  Multi-Task Learning Single Image Rain Streak Removal
  59. 59. STRUCT Group59 Joint Rain Detection and Removal  Recurrent Learning Single Image Rain Streak Removal 1. Input and output connection 1. Input and output connection 2. Recurrent guidance 3. Intermediate variable connection 4. Feature connection
  60. 60. STRUCT Group60 Joint Rain Detection and Removal  Detail Preserving Rain Accumulation Removal Single Image Rain Streak Removal Dark scenes with accumulation Accumulation Removal • Dark scenes Detail Preserving • Decreasing lightness in synthesis • White balance: additional pure black and white patches in training
  61. 61. STRUCT Group61 Joint Rain Detection and Removal Single Image Rain Streak Removal  Detail Preserving Rain Accumulation Removal  Separate training  Joint testing (a) Training (b) Testing
  62. 62. STRUCT Group62 Objective Evaluation Objective Results Single Image Rain Streak Removal Methods Rain12 Rain100L Rain100H Rain800 ID 27.21 23.13 13.78 20.54 DSC 30.02 24.16 15.66 22.46 LP 32.02 29.11 14.26 23.68 CNN 26.65 23.7 13.21 23.95 SRCNN 34.41 32.63 18.29 25.10 DetailNet 35.31 33.50 20.12 25.22 UGSM 33.3 28.83 13.40 23.12 JCAS 33.09 29.91 14.26 22.25 DID-MDN 30.14 28.27 13.85 22.55 ID-CGAN 20.78 23.39 16.86 23.81 JORDER- 35.86 35.41 20.79 25.61 JORDER 36.02 36.11 22.15 26.03 JORDER-R 36.21 36.62 23.45 26.73 JORDER-E 36.14 37.10 24.54 27.08 Methods Rain12 Rain100L Rain100H Rain800 ID 0.7534 0.6991 0.3968 0.6739 DSC 0.8679 0.8663 0.5444 0.7060 LP 0.9082 0.8812 0.4225 0.7954 CNN 0.7829 0.8142 0.3712 0.6589 SRCNN 0.9421 0.9392 0.6124 0.8232 DetailNet 0.9485 0.9444 0.6351 0.8228 UGSM 0.9323 0.8823 0.5089 0.7675 JCAS 0.9276 0.9041 0.4837 0.7682 DID-MDN 0.8762 0.8569 0.3748 0.7639 ID-CGAN 0.8519 0.8186 0.4921 0.8072 JORDER- 0.9534 0.9632 0.5978 0.8378 JORDER 0.9612 0.9741 0.6736 0.8501 JORDER-R 0.9644 0.9820 0.7490 0.8683 JORDER-E 0.9593 0.9795 0.8024 0.8716
  63. 63. STRUCT Group63 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  64. 64. STRUCT Group64 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  65. 65. STRUCT Group65 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  66. 66. STRUCT Group66 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  67. 67. STRUCT Group67 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  68. 68. STRUCT Group68 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  69. 69. STRUCT Group69 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  70. 70. STRUCT Group70 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  71. 71. STRUCT Group71 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  72. 72. STRUCT Group72 Subjective Evaluation Subjective Results Single Image Rain Streak Removal Input JORDER-R JORDER-E
  73. 73. Outline Background / 012 Single Image Rain Streak Removal / 024 Multi-Frame Rain Streak Removal / 061 Single Image Rain Drop Removal / 111 Prior Embedding Deep Rain Removal
  74. 74. STRUCT Group074  Rain Removal in Video Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos Jiaying Liu, Wenhan Yang, Shuai Yang, Zongming Guo. "Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. Video Rain Removal
  75. 75. STRUCT Group075 Representative Work K. Garg and S.K. Nayar, Detection and Removal of Rain from Videos, CVPR, 2004.  Visual appearance  Dynamics of rain  Photometry of environment  Correlation model  Dynamics of rain  Motion blur model  Photometry of rain RainDynamicsPhotometry + = 2004 CVPR Detection and Removal Video Rain Removal
  76. 76. STRUCT Group076 Representative Work Video Rain Removal Yi-Lei Chen and Chiou-Ting Hsu, A Generalized Low-Rank Appearance Model for Spatio-Temporally Correlated Rain Streaks, ICCV, 2013. 2004 CVPR Detection and Removal 2013 ICCV Spatio-Temporally Correlated
  77. 77. STRUCT Group  Initial rain detection  Rain map refinement 077 Representative Work Jin-Hwan Kim, Jae-Young Sim, and Chang-Su Kim, Video Deraining and Desnowing Using Temporal Correlation and Low-Rank Matrix Completion, TIP, 2015.  Rain streak removal 2004 CVPR Detection and Removal 2013 ICCV Spatio-Temporally Correlated 2015 TIP TCLRM Video Rain Removal
  78. 78. STRUCT Group078 Representative Work 2004 CVPR Detection and Removal  P-MoG model  Rain streak layer  Background layer  Moving object layer 2017 ICCV Stochastic Encoding Wei Wei, Lixuan Yi, et al., Should We Encode Rain Streaks in Video as Deterministic or Stochastic?, ICCV, 2017. 2013 ICCV Spatio-Temporally Correlated 2015 TIP TCLRM Video Rain Removal
  79. 79. STRUCT Group079 Representative Work 2004 CVPR Detection and Removal 2017 ICCV Stochastic Encoding 2018 CVPR Multi-Scale Sparse Coding Minghan Li, et al., Video Rain Streak Removal By Multiscale Convolutional Sparse Coding, CVPR, 2018. 2013 ICCV Spatio-Temporally Correlated 2015 TIP TCLRM Video Rain Removal
  80. 80. STRUCT Group080 Representative Work  Priors and Regularizers  Rain Sparsity  Horizontal Direction  Vertical Direction  Temporal Direction 2018 CVPR Multi-Scale Sparse Coding 2018 TIP FastDeRain Wei Wei, Lixuan Yi, et al., Should We Encode Rain Streaks in Video as Deterministic or Stochastic?, ICCV, 2017. Video Rain Removal
  81. 81. STRUCT Group081 Representative Work 2004 CVPR Detection and Removal 2017 ICCV Stochastic Encoding 2018 CVPR Multi-Scale Sparse Coding 2018 TIP FastDeRain 2018 CVPR SpacCNN Jie Chen et al., Robust Video Content Alignment and Compensation for Rain Removal in a CNN Framework, CVPR, 2018. 2013 ICCV Spatio-Temporally Correlated 2015 TIP TCLRM Video Rain Removal
  82. 82. STRUCT Group082  Rain Removal in Video Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos Jiaying Liu, Wenhan Yang, Shuai Yang, Zongming Guo. "Erase or Fill? Deep Joint Recurrent Rain Removal and Reconstruction in Videos", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. Video Rain Removal
  83. 83. STRUCT Group083 Our Aim: Deep Learning Video Rain Removal  Motivations and contributions  Deep networks + video rain removal  Rain models  Cover visual effects of degradations in practice  Occlusions  Detect degradation type  Jointly consider rain removal and background detail reconstruction Video Rain Removal
  84. 84. STRUCT Group084 Rain Image Generation (1/5)  Traditional Rain Synthesis Model  Additive Model [Lu15, Li16]  Rain removal  signal separation problem = + Video Rain Removal
  85. 85. STRUCT Group085 Rain Image Generation (2/5)  Traditional Rain Synthesis Model  Additive Model [Lu15, Li16]  The presented streaks have similar shapes and directions  Their distributions in spatial locations are uncorrelated Video Rain Removal
  86. 86. STRUCT Group086 Rain Image Generation (3/5)  Rain Occlusions  The light transmittance of rain drop becomes low  presents identical intensities Video Rain Removal
  87. 87. STRUCT Group087 Rain Image Generation (4/5)  Occlusion Aware Hybrid Rain Model   O: observed frame with rain streaks B: background frame without rain streaks S: rain streak A: rain reliance map Video Rain Removal
  88. 88. STRUCT Group088 Rain Image Generation (5/5)  Solution  Degradation Factor Video Rain Removal
  89. 89. STRUCT Group089 Joint Recurrent Rain Removal and Reconstruction  Multi-Task Learning  Degradation Classification  Rain Removal  Background Frame Reconstruction  Joint Learning Video Rain Removal
  90. 90. STRUCT Group090 Joint Recurrent Rain Removal and Reconstruction  Rain Removal  Single frame CNN extractor  Rain removal network  Separate rain streaks based on spatial features  Make Ft good at distinguishing rain streaks and normal textures Video Rain Removal
  91. 91. STRUCT Group091 Joint Recurrent Rain Removal and Reconstruction  Recurrent Architecture  Fusion Network  CNN+GRU architecture tO tF F-NetCNN 1tH tH Conv -1tO D-Net F-Net Conv CNN D-Net Video Rain Removal
  92. 92. STRUCT Group092 Joint Recurrent Rain Removal and Reconstruction  Degradation Classification Network  Detect the degradation type of rain frames explicitly  Rain occlusion  video reconstruction ˆt D-Net Conv Conv Softmax Conv Conv Ldetect tF 1tH Video Rain Removal
  93. 93. STRUCT Group093 Joint Recurrent Rain Removal and Reconstruction  Fusion Network  This architecture updates and aggregates internal memory progressively  long-term temporal dynamics of sequential data Conv ReLU TanH Conv  F-Net Conv tz 1tH tH tH tr Sum tF Video Rain Removal
  94. 94. STRUCT Group094 Joint Recurrent Rain Removal and Reconstruction  Reconstruction Network  Fill in missing rain occlusion regions based on temporal redundancy  Joint Rain Removal and Reconstruction Network  Estimate the background frame with both kinds of information  ˆ tE B JRC-Net C-Net ConvConv ˆ tB Linpaint Lrect 1tH tH Video Rain Removal
  95. 95. STRUCT Group095 Network Training  Loss Function  Multiple Losses Video Rain Removal
  96. 96. STRUCT Group096 Objective Evaluation Compared Methods  Discriminative Sparse Coding (DSC)[Luo15]  Layer Prior (LP) [Li16]  Joint Rain Detection and Removal (JORDER) [Yang17]  Deep Detail Network (DetailNet)[Fu17]  Stochastic Encoding (SE)[Wei17]  Temporal Correlation and Low-Rank Matrix (TCLRM) [Kim15] Datasets  RainSynLight25  RainSynComplex25  RainPractical10 Video Rain Removal
  97. 97. STRUCT Group097 Objective Evaluation PSNR and SSIM Results Baseline Light Heavy Metric PSNR SSIM PSNR SSIM Rain Image 23.69 0.8058 14.67 0.4563 DetailNet 25.72 0.8572 16.50 0.5441 TCLRM 28.77 0.8693 17.31 0.4956 JORDER 30.37 0.9235 20.20 0.6335 LP 27.09 0.8566 17.65 0.5364 DSC 25.63 0.8328 17.33 0.5036 SE 26,56 0.8006 16.76 0.5293 J4RNet 32.96 0.9434 24.13 0.7163 Video Rain Removal
  98. 98. STRUCT Group098 Subjective Evaluation Input SE JORDER DetailNet TCLRM J4R Video Rain Removal
  99. 99. STRUCT Group099 J4R Input Subjective Evaluation Video Rain Removal
  100. 100. STRUCT Group0100 SEJ4R Subjective Evaluation Video Rain Removal
  101. 101. STRUCT Group101 JORDERJ4R Subjective Evaluation Video Rain Removal
  102. 102. STRUCT Group102 DetailNetJ4R Subjective Evaluation Video Rain Removal
  103. 103. STRUCT Group103 TCLRMJ4R Subjective Evaluation Video Rain Removal
  104. 104. STRUCT Group104 Input SE JORDER DetailNet TCLRM J4R Subjective Evaluation Video Rain Removal
  105. 105. STRUCT Group105 InputJ4R Subjective Evaluation Video Rain Removal
  106. 106. STRUCT Group106 SEJ4R Subjective Evaluation Video Rain Removal
  107. 107. STRUCT Group107 JORDERJ4R Subjective Evaluation Video Rain Removal
  108. 108. STRUCT Group108 DetailNetJ4R Subjective Evaluation Video Rain Removal
  109. 109. STRUCT Group109 TCLRMJ4R Subjective Evaluation Video Rain Removal
  110. 110. STRUCT Group110 Input SE JORDER DetailNet TCLRM J4R Subjective Evaluation Video Rain Removal
  111. 111. STRUCT Group111 J4R Input Subjective Evaluation Video Rain Removal
  112. 112. STRUCT Group112 J4R SE Subjective Evaluation Video Rain Removal
  113. 113. STRUCT Group113 JORDERJ4R Subjective Evaluation Video Rain Removal
  114. 114. STRUCT Group114 DetailNetJ4R Subjective Evaluation Video Rain Removal
  115. 115. STRUCT Group115 TCLRMJ4R Subjective Evaluation Video Rain Removal
  116. 116. STRUCT Group116 Input SE JORDER DetailNet TCLRM J4R Subjective Evaluation Video Rain Removal
  117. 117. STRUCT Group117 J4R Input Subjective Evaluation Video Rain Removal
  118. 118. STRUCT Group118 SEJ4R Subjective Evaluation Video Rain Removal
  119. 119. STRUCT Group119 JORDERJ4R Subjective Evaluation Video Rain Removal
  120. 120. STRUCT Group120 DetailNetJ4R Subjective Evaluation Video Rain Removal
  121. 121. STRUCT Group121 TCLRMJ4R Subjective Evaluation Video Rain Removal
  122. 122. STRUCT Group0122  Rain Removal in Video D3R-Net: Dynamic Routing Residue Recurrent Network for Video Rain Removal Jiaying Liu, Wenhan Yang, Shuai Yang, and Zongming Guo. "D3R-Net: Dynamic Routing Residue Recurrent Network for Video Rain Removal", IEEE Trans. on Image Processing (TIP), Vol.28, No.2, pp.699-712, Feb. 2019. Video Rain Removal
  123. 123. STRUCT Group0123 Dynamic Routing Residue RNN for Deraining  Rain Removal  Region Dependent  Smooth regions  strong smoothing filter  Texture/edge regions  careful operation  Side context information  Moving objects Vs. static scenes  Texture regions Vs. smooth regions  Rain Type  …… Video Rain Removal
  124. 124. STRUCT Group0124 Dynamic Routing + RNN  Recurrent Neural Network Video Rain Removal CNN CNN with input bypass connections Spatial network connected by convolutional recurrent units Vanilla CNN ResNet (CNN with feature and input bypass connections)
  125. 125. STRUCT Group0125 Dynamic Routing + RNN  Recurrent Neural Network Video Rain Removal CNN Spatial network connected by Gated Recurrent Unit Spatial network connected by convolutional recurrent units Spatial network connected by Gated Recurrent Units
  126. 126. STRUCT Group0126 Dynamic Routing + RNN Video Rain Removal CNN Dynamic CNN Dynamic RNN  Dynamic Routing Network
  127. 127. STRUCT Group0127 Dynamic Routing + RNN Video Rain Removal  Dynamic Routing Recurrent Redidue Network
  128. 128. STRUCT Group0128 Network Training  Loss Function  Multiple Losses Video Rain Removal
  129. 129. STRUCT Group0129 Objective Evaluation Compared Methods  Discriminative Sparse Coding (DSC)[Luo15]  Layer Prior (LP) [Li16]  Joint Rain Detection and Removal (JORDER) [Yang17]  Deep Detail Network (DetailNet)[Fu17]  Stochastic Encoding (SE)[Wei17]  Temporal Correlation and Low-Rank Matrix (TCLRM) [Kim15] Datasets  RainSynLight25  RainSynComplex25  RainPractical10 Video Rain Removal
  130. 130. STRUCT Group0130 Objective Evaluation PSNR and SSIM Results Baseline Light Heavy Metric PSNR SSIM PSNR SSIM Rain Image 23.69 0.8058 14.67 0.4563 DetailNet 25.72 0.8572 16.50 0.5441 TCLRM 28.77 0.8693 17.31 0.4956 JORDER 30.37 0.9235 20.20 0.6335 LP 27.09 0.8566 17.65 0.5364 DSC 25.63 0.8328 17.33 0.5036 SE 26,56 0.8006 16.76 0.5293 D3R-Net 32.96 0.9434 27.03 0.8303 Video Rain Removal
  131. 131. STRUCT Group0131 Subjective Evaluation Subjective Results Video Rain Removal Rain image TCLRM DetailNet JORDER FastDeRain DSC LP D3R-Net
  132. 132. STRUCT Group0132 Subjective Evaluation Subjective Results Video Rain Removal Rain image TCLRM DetailNet JORDER FastDeRain DSC LP D3R-Net
  133. 133. STRUCT Group0133 Subjective Evaluation Subjective Results Video Rain Removal Rain image TCLRM DetailNet JORDER FastDeRain DSC LP D3R-Net
  134. 134. STRUCT Group0134 Subjective Evaluation Subjective Results Video Rain Removal Rain image TCLRM DetailNet JORDER FastDeRain DSC LP D3R-Net
  135. 135. STRUCT Group135 Deeper Thought  Hard Detection  Soft Attention  Multi-Task Learning  Adversarial Learning • Side Prior  Learned Prior • Only focus on very hard parts • Attentive GAN Video Rain Removal
  136. 136. Outline Background / 012 Single Image Rain Streak Removal / 024 Multi-Frame Rain Streak Removal / 061 Single Image Rain Drop Removal / 111 Prior Embedding Deep Rain Removal
  137. 137. STRUCT Group137 Deep Raindrop Removal  Single Image Rain Drop Removal Attentive Generative Adversarial Network for Raindrop Removal from A Single Image Rui Qian, Robby T. Tan, Wenhan Yang, Jiajun Su, Jiaying Liu, CVPR 2018 Spotlight
  138. 138. STRUCT Group138 Our Aim: Removing Diverse Raindrops  Diverse Raindrop Problems  Physical: size, shape, …  Optical: color, transparency, … Deep Raindrop Removal
  139. 139. STRUCT Group139 Raindrop Dataset  Over 1,000 Image Pairs  Various Outdoor Conditions  Various Background Deep Raindrop Removal
  140. 140. STRUCT Group140 Raindrop Image Formation  Raindrop Image Modeling  B is the background  R is the raindrop layer  M is the mask Deep Raindrop Removal I B Rain Region
  141. 141. STRUCT Group141 Attentive GAN for Raindrop Removal (1/4)  Overall Network Architecture  Attentive-recurrent network  Context autoencoder  Discriminative network Deep Raindrop Removal
  142. 142. STRUCT Group142 Attentive GAN for Raindrop Removal (2/4)  Attentive-Recurrent Network  Generate attention map Mask(M) Loss Loss Loss Deep Raindrop Removal
  143. 143. STRUCT Group143 Attentive GAN for Raindrop Removal (3/4)  Contextual Autoencoder  Multi-scale loss LM + perceptual loss LP Deep Raindrop Removal
  144. 144. STRUCT Group144 Attentive GAN for Raindrop Removal (4/4)  Discriminative Network  Input: Attention map + generated image / ground truth image  Output: Real / Fake O Deep Raindrop Removal
  145. 145. STRUCT Group145 Network Training  Loss Function  Loss of GAN  Loss of our network in detail Deep Raindrop Removal
  146. 146. STRUCT Group146 Objective Evaluation Compared Methods  CNN Rain Drop[Eigen13]  Pix2pix-cGan[Isola16] Configurations of Ours Metric PSNR SSIM Eigen13 28.59 0.6726 Pix2pix 30.14 0.8299 A 29.25 0.7853 A + D 30.88 0.8670 A + AD 30.60 0.8710 Proposed 31.57 0.9023 Deep Raindrop Removal  A (Autoencoder)  A+D (A+Discriminator)  A+AD (A+Attentive D)  AA+AD (Our Method)
  147. 147. STRUCT Group147 Subjective Evaluation Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Deep Raindrop Removal
  148. 148. STRUCT Group148 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  149. 149. STRUCT Group149 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  150. 150. STRUCT Group150 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  151. 151. STRUCT Group151 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  152. 152. STRUCT Group152 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  153. 153. STRUCT Group153 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  154. 154. STRUCT Group154 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  155. 155. STRUCT Group155 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  156. 156. STRUCT Group156 Raindrop Image PS[PAMI13] Pix2pix[CVPR17] Ours Eigen[ICCV13] Subjective Evaluation Deep Raindrop Removal
  157. 157. STRUCT Group157 Conclusion  Image Enhancement  Super-Resolution / Raindrop Removal / Rain Streak Removal  Re-Thinking  Combine MAP and DL-Based Prior  Multi-Task Learning  Side Prior  Adversarial Learning  Learned Prior  Experimental Results  Better performance in quantitative and qualitative evaluation Conclusion
  158. 158. STRUCT Group liujiaying@pku.edu.cn yangwenhan@pku.edu.cn

×