Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 3: prior embedding deep super resolution
•
1 like•146 views
This document summarizes a presentation on prior embedding deep super-resolution. It discusses challenges like ill-posedness and proposes solutions like additional constraints and embedding signal structure. It reviews representative works in deep image super-resolution from 2014-2018. It also summarizes research on deep band-based image super-resolution and STR-ResNet for video super-resolution, discussing network architectures, experimental results and comparisons to other methods.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 3: prior embedding deep super resolution
Similar to Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 3: prior embedding deep super resolution (20)
More from Wanjin Yu
More from Wanjin Yu (14)
Recently uploaded
Recently uploaded (20)
Intelligent Image Enhancement and Restoration - From Prior Driven Model to Advanced Deep Learning Part 3: prior embedding deep super resolution
- 1. IEEE ICME-2019 8:30 – 12:00, July 08 Shanghai, China Intelligent Image/Video Editing Tutorial
- 2. STRUCT Group Jiaying Liu Wenhan Yang Prior Embedding Deep Super-Resolution Part 3
- 3. Outline Background and Related Work / 005 Deep Band-Based Image Super-Resolution / 014 STR-ResNet for Video Super-Resolution / 037 Prior Embedding Deep Super-Resolution
- 4. Outline Background and Related Work / 005 Deep Band-Based Image Super-Resolution / 014 STR-ResNet for Video Super-Resolution / 037 Prior Embedding Deep Super-Resolution
- 5. STRUCT Group Challenges and Solutions Prior Embedding Deep Super-Resolution5 Ill-Posedness Solution • One-to-Infinity • Additional Constraints • Model architectures • Loss functions • Signal structure embedding • Domain knowledge [Haefner18] [Haefner18] Bjoern Haefner et al., "Fight Ill-Posedness With Ill-Posedness: Single-Shot Variational Depth Super-Resolution From Shading", CVPR, 2018.
- 6. STRUCT Group DL SR Route Deep Image Super-Resolution6 RDBSRGAN DBPN SRCNN (TPAMI16) VDSR (CVPR16) FSRCNN (ECCV16) DRCN (CVPR16) LapSRN (CVPR17) SRGAN (CVPR17) DBPN (CVPR18) RDB (CVPR18)
- 7. STRUCT Group07 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network
- 8. STRUCT Group08 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization LapSRN LISTA DEGREE
- 9. STRUCT Group09 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization 2017-2018 Perceptual SR DEGREE Christian Ledig et al., Photo-realistic single image super-resolution using a generative adversarial network, CVPR, 2017.
- 10. STRUCT Group010 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization 2017-2018 Perceptual SR SFTGAN Xintao Wang et al., Recovering realistic texture in image super-resolution by deep spatial feature transform, CVPR, 2018.
- 11. STRUCT Group011 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization 2017-2018 Perceptual SR 2017-2018 DenseNet + ResNet RDB RDN Yulun Zhang et al., Residual dense network for image super-resolution, CVPR, 2018.
- 12. STRUCT Group012 Representative Work Deep Image Super-Resolution 2014-2016 Deep Network 2015-2017 Signal Regularization 2017-2018 Perceptual SR 2017-2018 DenseNet + ResNet Residual Channel Attention Network 2018 RCAN Yulun Zhang et al., Image Super-Resolution Using Very Deep Residual Channel Attention Networks, ECCV, 2018.
- 13. Outline Background and Related Work / 005 Deep Band-Based Image Super-Resolution / 014 STR-ResNet for Video Super-Resolution / 037 Prior Embedding Deep Super-Resolution
- 14. STRUCT Group14 Deep Band-Based Image Super-Resolution Deep Band-Based Image Super-Resolution Deep Edge Guided Recurrent Residual Learning for Image Super-Resolution Wenhan Yang, Jiashi Feng, Jianchao Yang, Fang Zhao, Jiaying Liu, Zongming Guo, and Shuicheng Yan, IEEE TIP 2017
- 15. STRUCT Group Previous Works Deep Band-Based Image Super-Resolution15 Issues • Traditional Regularizer in MAP framework • Limited capacity to describe complex features • Deep learning-Based Methods • Network Black box, How it works ? • How to embed priors ?
- 16. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution16 Limitations • The accuracy of reconstruction for each band[Singh14] • High energy bands macrostructures • low energy sub-bands suffer from attenuation [Singh14] A. Singh and N. Ahuja, "Sub-Band Energy Constraints for Self-Similarity based Super-Resolution", ICPR, 2014.
- 17. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution17 Reconstruction for each band • Entire signal separate signals[Freeman91, Taubman94] • Pay attention to the signal details of each band 2 2 ˆ argmin , 1,2,..., i i i i p i n x x DHx y x ˆx 1 ˆ n i i i w x 2 2 ˆ argmin p x x DHx y x [Freeman91] WT. Freeman et al., "The Design and Use of Steerable Filters", TPAMI, 1991. [Taubman94] D. Taubman et al., "Multirate Multirate 3-D Subband Coding of Video", TIP, 1994.
- 18. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution18 Gradual reconstruction • F : band recovery • G : aggregation function • F and G linear transformation[Singh14, Song16] • Supervised learning[Chatterjee07, Singh14] 1 1 ˆ, ,i i i i i i s s x sˆi x [Song16] S. Song et al., "Joint Sub-Band based Neighbor Embedding for Image Super-Resolution", ICASSP, 2016. [Chatterjee07] D. Taubman et al., "Super-Resolution Using Sub-Band Constrained Total Variation", SSVM, 2007. F1 G1 0 y s Lx1 1 ˆx 1 s F2 G2 Lx2 2 ˆx Fn Gn ˆn x … 1n s Lxn ˆn s x
- 19. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution19 Gradual learned reconstruction • End-to-end learning • Not dependent on the specific band choice • G : summation Concise, does not introduce other parameters 1 1i i i i s s s F1 0 y s 1 ˆx 1 s F2 2 ˆx Fn ˆn x … 1n s ˆn s x Lx
- 20. STRUCT Group Sub-Bands Super-Resolution Deep Band-Based Image Super-Resolution20 Gradual learned reconstruction • F : Convs + ReLu Nonlinear • Unsupervised sub-band learning 1 1i i i i s s s conv 0 y s 1 ˆx 1 s 2 ˆx ˆn x … 1n s ˆn s x Lxconv conv conv conv conv
- 21. STRUCT Group Prior Embedding Modeling Deep Band-Based Image Super-Resolution21 • Regularization embedding: internal + auxiliary [Weston08] J. Weston et al., "Deep Learning via Semi-Supervised Embedding", ICML, 2008. Layer 1 Input Layer 2 Layer 3 Output Embedding Space Layer 1 Input Layer 2 Layer 3 Output Layer 1 Input Layer 2 Layer 3 Output Embedding Layer (a) Output (b) Internal (c) Auxiliary Embedding Space Space Embedding Network[Weston08]
- 22. STRUCT Group Prior Embedding Modeling Deep Band-Based Image Super-Resolution22 • Auxiliary Constraints • Auxiliary information Layer 1 Input Layer 2 Layer 3 Output Regularization embedding Regularization embedding Prior Map
- 23. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution23 Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
- 24. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution24 Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
- 25. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution25 Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
- 26. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution26 Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
- 27. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution27 Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
- 28. STRUCT Group Network Architecture Deep Band-Based Image Super-Resolution28 Edge-Guide Sub-Band Reconstruction Sub-band reconstruction Prior embedding Feature extraction Residual learning Loss function
- 29. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution29 Objective Results Dataset Set5 Set14 BSD100 Method ×2 ×3 ×4 ×2 ×3 ×4 ×2 ×3 ×4 A+ 36.56 32.60 30.30 32.14 29.07 27.28 30.78 28.18 26.77 TSE-SR 36.47 32.62 30.24 32.21 29.14 27.38 31.18 28.30 26.85 JSB-NE 36.59 32.32 30.08 32.34 28.98 27.22 31.22 28.14 26.71 CNN 36.34 32.39 30.09 32.18 29.00 27.20 31.11 28.20 26.70 CNN-L 36.66 32.75 30.49 32.45 29.30 27.50 31.36 28.41 26.90 CSCN 36.88 33.10 30.86 32.50 29.42 27.64 31.40 28.50 27.03 VDSR 37.53 33.66 31.35 33.03 29.77 28.01 31.90 28.82 27.29 DEGREE 37.54 33.72 31.43 33.01 29.78 28.02 31.76 28.69 27.14 DEGREE -MV 37.58 33.76 31.47 33.06 29.82 28.10 31.80 28.74 27.20
- 30. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution30 Subject Results (x3) HR A+ SRCNN JSB-NE CSCN DEGREE
- 31. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution31 Compare to VDSR and DRCN HR VDSR DRCN DEGREE
- 32. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution32 Compare to VDSR and DRCN HR VDSR DRCN DEGREE
- 33. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution33 Edge Prediction Zebra (3×) Butterfly (4×)
- 34. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution34 Sub-Band Recovery Butterfly (2×)
- 35. STRUCT Group Experimental Results Deep Band-Based Image Super-Resolution35 Sub-Band Recovery Butterfly (2×)
- 36. Outline Background and Related Work / 005 Deep Band-Based Image Super-Resolution / 014 STR-ResNet for Video Super-Resolution / 037 Prior Embedding Deep Super-Resolution
- 37. STRUCT Group37 STR-ResNet for Video Super-Resolution STR-ResNet for Video Super-Resolution Video Super-Resolution Based on Spatial-Temporal Recurrent Residual Networks Wenhan Yang, Jiashi Feng, Guosen Xie, Jiaying Liu, Zongming Guo, Shuicheng Yan CVIU 2018
- 38. STRUCT Group038 Representative Work STR-ResNet for Video Super-Resolution 2015 BRCN Huang et al., Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution, NIPS, 2015.
- 39. STRUCT Group039 Representative Work STR-ResNet for Video Super-Resolution Renjie Liao et. al., Video Super-Resolution via Deep Draft-Ensemble Learning, ICCV, 2015. 2015 BRCN 2016 Draft Learning
- 40. STRUCT Group040 Representative Work STR-ResNet for Video Super-Resolution 2015 BRCN 2016 Draft Learning A.Kappeler, S.Yoo, Q.Dai, A.K.Katsaggelos. Video Super-Resolution with Convolutional Neural Networks, TCI, 2016. 2016 VSRNet
- 41. STRUCT Group041 Representative Work STR-ResNet for Video Super-Resolution 2015 BRCN 2016 Draft Learning 2016 VSRNet 2017 Sub-Pixel Network Wenzhe Shi et al., Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, CVPR, 2016.
- 42. STRUCT Group042 Representative Work STR-ResNet for Video Super-Resolution 2015 BRCN 2016 Draft Learning 2016 VSRNet 2017 Sub-Pixel Network Xin Tao, Hongyun Gao, Renjie Liao, Jue Wang, Jiaya Jia, Detail-revealing Deep Video Super-resolution, ICCV, 2017. 2018 Detail Revealing SR
- 43. STRUCT Group Explicit vs. Implicit STR-ResNet for Video Super-Resolution43 Explicit Motion Modeling • Regions with salient geometric features: better details • Smooth regions: artifacts • Motion compensation: high complexity • Draft Learning[Liao15]: 50×50 625 s Implicit Motion Modeling • Robust, highly efficient • Automatic network learning • Deficiency: motion modeling Bicubic Draft BRCN Draft
- 44. STRUCT Group Spatial Temporal Recurrent ResNet STR-ResNet for Video Super-Resolution44 Implicit Motion Modeling • There is no motion compensation and alignment Spatial Temporal ResNet • Modeling intra-frame redundancy and inter-frame correspondence jointly • Motion embedding • Inter-frame residue
- 45. STRUCT Group Spatial and Temporal Joint Modeling (1/2) STR-ResNet for Video Super-Resolution45 Spatial Domain
- 46. STRUCT Group Spatial and Temporal Joint Modeling (2/2) STR-ResNet for Video Super-Resolution46 Spatial Temporal Domain
- 47. STRUCT Group Implicit Motion Embedding (1/3) STR-ResNet for Video Super-Resolution47 Temporal Redundancy in Different Temporal Domain
- 48. STRUCT Group Implicit Motion Embedding (2/3) STR-ResNet for Video Super-Resolution48 Spatial Residue vs. Temporal Residue Temporal Residue Spatial Residue Spatial and temporal residue
- 49. STRUCT Group Implicit Motion Embedding (3/3) STR-ResNet for Video Super-Resolution49 Input and Predict Inter-Frame Residue
- 50. STRUCT Group Network Architecture (1/2) STR-ResNet for Video Super-Resolution50 Single Image Super-Resolution Network
- 51. STRUCT Group Network Architecture (2/2) STR-ResNet for Video Super-Resolution51 Multi-Frame Video Super-Resolution Network
- 52. STRUCT Group Experimental Comparison STR-ResNet for Video Super-Resolution52 Experimental Setting (1/2) • Dataset • 20 videos from Xiph.org Video Test Media1 • 75,000 33×33 overlapped patches2 • Comparison methods • A+[Timofte14], SRCNN[Dong15], VE2, 3DSKR[Takeda09], Draft SR[Liao15], and BRCN[Huang15] • Kernel: 3×3, Channel: 64 • Degradation: 9×9 • Blur level 1.6, SF 4 1https://media.xiph.org/video/derf/ 2http://www.infognition.com/videoenhancer/
- 53. STRUCT Group Experimental Comparison STR-ResNet for Video Super-Resolution53 Experimental Setting (2/2) • Loss function • Training • Before 250,000 iterations: 0.0001 • After 250,000 iterations: “Focus on” fine-tuning, 0.00001
- 54. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution54 Testing Set Tractor Sunflower Blue Sky Station Pedestrian Rush Hour
- 55. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution55 Objective Evaluation • 4× enlargement
- 56. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution56 Subjective results (1/4)
- 57. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution57 Subjective results (2/4)
- 58. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution58 Subjective results (3/4)
- 59. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution59 Subjective results (4/4)
- 60. STRUCT Group Experimental Results STR-ResNet for Video Super-Resolution60 Trade-off between Complexity and Performance
- 61. STRUCT Group Summary STR-ResNet for Video Super-Resolution61 Joint Spatial and Temporal Modeling • Spatial redundancy spatial and temporal redundancy • Implicit motion modeling Spatial Temporal Recurrent ResNet • Recurrent network • Inter-frame connection • Implicit motion modeling • Inter-frame residue embedding