Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 3: prior embedding deep super resolution

13 views

Published on

ICME2019 Tutorial: Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 3: prior embedding deep super resolution

Published in: Internet
  • Be the first to comment

  • Be the first to like this

Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 3: prior embedding deep super resolution

  1. 1. IEEE ICME-2019 8:30 – 12:00, July 08 Shanghai, China Intelligent Image/Video Editing Tutorial
  2. 2. STRUCT Group Jiaying Liu Wenhan Yang Prior Embedding Deep Super-Resolution Part 3
  3. 3. Outline Background and Related Work / 005 Deep Band-Based Image Super-Resolution / 014 STR-ResNet for Video Super-Resolution / 037 Prior Embedding Deep Super-Resolution
  4. 4. Outline Background and Related Work / 005 Deep Band-Based Image Super-Resolution / 014 STR-ResNet for Video Super-Resolution / 037 Prior Embedding Deep Super-Resolution
  5. 5. STRUCT Group Challenges and Solutions Prior Embedding Deep Super-Resolution5  Ill-Posedness  Solution • One-to-Infinity • Additional Constraints • Model architectures • Loss functions • Signal structure embedding • Domain knowledge [Haefner18] [Haefner18] Bjoern Haefner et al., "Fight Ill-Posedness With Ill-Posedness: Single-Shot Variational Depth Super-Resolution From Shading", CVPR, 2018.
  6. 6. STRUCT Group DL SR Route Deep Image Super-Resolution6 RDBSRGAN DBPN SRCNN (TPAMI16) VDSR (CVPR16) FSRCNN (ECCV16) DRCN (CVPR16) LapSRN (CVPR17) SRGAN (CVPR17) DBPN (CVPR18) RDB (CVPR18)
  7. 7. STRUCT Group07 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network
  8. 8. STRUCT Group08 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization LapSRN LISTA DEGREE
  9. 9. STRUCT Group09 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization 2017-2018 Perceptual SR DEGREE Christian Ledig et al., Photo-realistic single image super-resolution using a generative adversarial network, CVPR, 2017.
  10. 10. STRUCT Group010 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization 2017-2018 Perceptual SR SFTGAN Xintao Wang et al., Recovering realistic texture in image super-resolution by deep spatial feature transform, CVPR, 2018.
  11. 11. STRUCT Group011 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization 2017-2018 Perceptual SR 2017-2018 DenseNet + ResNet RDB RDN Yulun Zhang et al., Residual dense network for image super-resolution, CVPR, 2018.
  12. 12. STRUCT Group012 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization 2017-2018 Perceptual SR 2017-2018 DenseNet + ResNet Residual Channel Attention Network 2018 RCAN Yulun Zhang et al., Image Super-Resolution Using Very Deep Residual Channel Attention Networks, ECCV, 2018.
  13. 13. Outline Background and Related Work / 005 Deep Band-Based Image Super-Resolution / 014 STR-ResNet for Video Super-Resolution / 037 Prior Embedding Deep Super-Resolution
  14. 14. STRUCT Group14 Deep Band-Based Image Super-Resolution  Deep Band-Based Image Super-Resolution Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution Wenhan Yang, Jiashi Feng, Jianchao Yang, Fang Zhao, Jiaying Liu, Zongming Guo, and Shuicheng Yan, IEEE TIP 2017
  15. 15. STRUCT Group Previous Works Deep Band-Based Image Super-Resolution15  Issues • Traditional Regularizer in MAP framework • Limited capacity to describe complex features • Deep learning-Based Methods • Network  Black box, How it works ? • How to embed priors ?
  16. 16. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution16  Limitations • The accuracy of reconstruction for each band[Singh14] • High energy bands  macrostructures • low energy sub-bands  suffer from attenuation [Singh14] A. Singh and N. Ahuja, "Sub-Band Energy Constraints for Self-Similarity based Super-Resolution", ICPR, 2014.
  17. 17. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution17  Reconstruction for each band • Entire signal  separate signals[Freeman91, Taubman94] • Pay attention to the signal details of each band      2 2 ˆ argmin , 1,2,..., i i i i p i n x x DHx y x ˆx  1 ˆ n i i i w x     2 2 ˆ argmin p x x DHx y x [Freeman91] WT. Freeman et al., "The Design and Use of Steerable Filters", TPAMI, 1991. [Taubman94] D. Taubman et al., "Multirate Multirate 3-D Subband Coding of Video", TIP, 1994.
  18. 18. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution18  Gradual reconstruction • F : band recovery • G : aggregation function • F and G  linear transformation[Singh14, Song16] • Supervised learning[Chatterjee07, Singh14]       1 1 ˆ, ,i i i i i i s s x sˆi x [Song16] S. Song et al., "Joint Sub-Band based Neighbor Embedding for Image Super-Resolution", ICASSP, 2016. [Chatterjee07] D. Taubman et al., "Super-Resolution Using Sub-Band Constrained Total Variation", SSVM, 2007. F1 G1 0 y s Lx1 1 ˆx 1 s F2 G2 Lx2 2 ˆx Fn Gn ˆn x … 1n s Lxn  ˆn s x
  19. 19. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution19  Gradual learned reconstruction • End-to-end learning • Not dependent on the specific band choice • G : summation  Concise, does not introduce other parameters     1 1i i i i s s s F1  0 y s 1 ˆx 1 s F2 2 ˆx Fn ˆn x … 1n s  ˆn s x Lx
  20. 20. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution20  Gradual learned reconstruction • F : Convs + ReLu  Nonlinear • Unsupervised sub-band learning     1 1i i i i s s s conv  0 y s 1 ˆx 1 s 2 ˆx ˆn x … 1n s  ˆn s x Lxconv conv conv conv conv
  21. 21. STRUCT Group Prior Embedding Modeling Deep Band-Based Image Super-Resolution21 • Regularization embedding: internal + auxiliary [Weston08] J. Weston et al., "Deep Learning via Semi-Supervised Embedding", ICML, 2008. Layer 1 Input Layer 2 Layer 3 Output Embedding Space Layer 1 Input Layer 2 Layer 3 Output Layer 1 Input Layer 2 Layer 3 Output Embedding Layer (a) Output (b) Internal (c) Auxiliary Embedding Space  Space Embedding Network[Weston08]
  22. 22. STRUCT Group Prior Embedding Modeling Deep Band-Based Image Super-Resolution22 • Auxiliary Constraints • Auxiliary information Layer 1 Input Layer 2 Layer 3 Output Regularization embedding  Regularization embedding Prior Map
  23. 23. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution23  Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
  24. 24. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution24  Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
  25. 25. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution25  Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
  26. 26. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution26  Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
  27. 27. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution27  Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
  28. 28. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution28  Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
  29. 29. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution29  Objective Results Dataset Set5 Set14 BSD100 Method ×2 ×3 ×4 ×2 ×3 ×4 ×2 ×3 ×4 A+ 36.56 32.60 30.30 32.14 29.07 27.28 30.78 28.18 26.77 TSE-SR 36.47 32.62 30.24 32.21 29.14 27.38 31.18 28.30 26.85 JSB-NE 36.59 32.32 30.08 32.34 28.98 27.22 31.22 28.14 26.71 CNN 36.34 32.39 30.09 32.18 29.00 27.20 31.11 28.20 26.70 CNN-L 36.66 32.75 30.49 32.45 29.30 27.50 31.36 28.41 26.90 CSCN 36.88 33.10 30.86 32.50 29.42 27.64 31.40 28.50 27.03 VDSR 37.53 33.66 31.35 33.03 29.77 28.01 31.90 28.82 27.29 DEGREE 37.54 33.72 31.43 33.01 29.78 28.02 31.76 28.69 27.14 DEGREE -MV 37.58 33.76 31.47 33.06 29.82 28.10 31.80 28.74 27.20
  30. 30. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution30  Subject Results (x3) HR A+ SRCNN JSB-NE CSCN DEGREE
  31. 31. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution31  Compare to VDSR and DRCN HR VDSR DRCN DEGREE
  32. 32. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution32  Compare to VDSR and DRCN HR VDSR DRCN DEGREE
  33. 33. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution33  Edge Prediction Zebra (3×) Butterfly (4×)
  34. 34. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution34  Sub-Band Recovery Butterfly (2×)
  35. 35. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution35  Sub-Band Recovery Butterfly (2×)
  36. 36. Outline Background and Related Work / 005 Deep Band-Based Image Super-Resolution / 014 STR-ResNet for Video Super-Resolution / 037 Prior Embedding Deep Super-Resolution
  37. 37. STRUCT Group37 STR-ResNet for Video Super-Resolution  STR-ResNet for Video Super-Resolution Video Super-Resolution Based on Spatial-Temporal Recurrent Residual Networks Wenhan Yang, Jiashi Feng, Guosen Xie, Jiaying Liu, Zongming Guo, Shuicheng Yan CVIU 2018
  38. 38. STRUCT Group038 Representative Work STR-ResNet for Video Super-Resolution 2015 BRCN Huang et al., Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution, NIPS, 2015.
  39. 39. STRUCT Group039 Representative Work STR-ResNet for Video Super-Resolution Renjie Liao et. al., Video Super-Resolution via Deep Draft-Ensemble Learning, ICCV, 2015. 2015 BRCN 2016 Draft Learning
  40. 40. STRUCT Group040 Representative Work STR-ResNet for Video Super-Resolution 2015 BRCN 2016 Draft Learning A.Kappeler, S.Yoo, Q.Dai, A.K.Katsaggelos. Video Super-Resolution with Convolutional Neural Networks, TCI, 2016. 2016 VSRNet
  41. 41. STRUCT Group041 Representative Work STR-ResNet for Video Super-Resolution 2015 BRCN 2016 Draft Learning 2016 VSRNet 2017 Sub-Pixel Network Wenzhe Shi et al., Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, CVPR, 2016.
  42. 42. STRUCT Group042 Representative Work STR-ResNet for Video Super-Resolution 2015 BRCN 2016 Draft Learning 2016 VSRNet 2017 Sub-Pixel Network Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, Jiaya Jia, Detail-revealing Deep Video Super-resolution, ICCV, 2017. 2018 Detail Revealing SR
  43. 43. STRUCT Group Explicit vs. Implicit STR-ResNet for Video Super-Resolution43  Explicit Motion Modeling • Regions with salient geometric features: better details • Smooth regions: artifacts • Motion compensation: high complexity • Draft Learning[Liao15]: 50×50  625 s  Implicit Motion Modeling • Robust, highly efficient • Automatic network learning • Deficiency: motion modeling Bicubic Draft BRCN Draft
  44. 44. STRUCT Group Spatial Temporal Recurrent ResNet STR-ResNet for Video Super-Resolution44  Implicit Motion Modeling • There is no motion compensation and alignment  Spatial Temporal ResNet • Modeling intra-frame redundancy and inter-frame correspondence jointly • Motion embedding • Inter-frame residue
  45. 45. STRUCT Group Spatial and Temporal Joint Modeling (1/2) STR-ResNet for Video Super-Resolution45  Spatial Domain
  46. 46. STRUCT Group Spatial and Temporal Joint Modeling (2/2) STR-ResNet for Video Super-Resolution46  Spatial Temporal Domain
  47. 47. STRUCT Group Implicit Motion Embedding (1/3) STR-ResNet for Video Super-Resolution47  Temporal Redundancy in Different Temporal Domain
  48. 48. STRUCT Group Implicit Motion Embedding (2/3) STR-ResNet for Video Super-Resolution48  Spatial Residue vs. Temporal Residue Temporal Residue Spatial Residue Spatial and temporal residue
  49. 49. STRUCT Group Implicit Motion Embedding (3/3) STR-ResNet for Video Super-Resolution49  Input and Predict Inter-Frame Residue
  50. 50. STRUCT Group Network Architecture (1/2) STR-ResNet for Video Super-Resolution50  Single Image Super-Resolution Network
  51. 51. STRUCT Group Network Architecture (2/2) STR-ResNet for Video Super-Resolution51  Multi-Frame Video Super-Resolution Network
  52. 52. STRUCT Group Experimental Comparison STR-ResNet for Video Super-Resolution52  Experimental Setting (1/2) • Dataset • 20 videos from Xiph.org Video Test Media1 • 75,000 33×33 overlapped patches2 • Comparison methods • A+[Timofte14], SRCNN[Dong15], VE2, 3DSKR[Takeda09], Draft SR[Liao15], and BRCN[Huang15] • Kernel: 3×3, Channel: 64 • Degradation: 9×9 • Blur level 1.6, SF 4 1https://media.xiph.org/video/derf/ 2http://www.infognition.com/videoenhancer/
  53. 53. STRUCT Group Experimental Comparison STR-ResNet for Video Super-Resolution53  Experimental Setting (2/2) • Loss function • Training • Before 250,000 iterations: 0.0001 • After 250,000 iterations: “Focus on” fine-tuning, 0.00001
  54. 54. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution54  Testing Set Tractor Sunflower Blue Sky Station Pedestrian Rush Hour
  55. 55. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution55  Objective Evaluation • 4× enlargement
  56. 56. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution56  Subjective results (1/4)
  57. 57. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution57  Subjective results (2/4)
  58. 58. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution58  Subjective results (3/4)
  59. 59. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution59  Subjective results (4/4)
  60. 60. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution60  Trade-off between Complexity and Performance
  61. 61. STRUCT Group Summary STR-ResNet for Video Super-Resolution61  Joint Spatial and Temporal Modeling • Spatial redundancy  spatial and temporal redundancy • Implicit motion modeling  Spatial Temporal Recurrent ResNet • Recurrent network • Inter-frame connection • Implicit motion modeling • Inter-frame residue embedding
  62. 62. STRUCT Group liujiaying@pku.edu.cn yangwenhan@pku.edu.cn

×