SlideShare a Scribd company logo
1 of 32
Download to read offline
Meta-Transfer Learning
for Zero-Shot Super-Resolution
Image Processing Team
Dawoon Heo Jeongho Shin Seonok Kim(🙋)
Presented in CVPR 2020
Authors - Jae Woong Soh, Sunwoo Cho, Nam Ik Cho (Seoul National University)
Introduction
Previous Methods
Proposed Methods
Experiments
Discussion
Conclusion
Introduction
MZSR
MZSR (Meta Transfer Learning for Zero Shot Super Resolution)
Zero Shot Super Resolution
Recent CNN based methods
Trained on large-scale ‘external’
dataset
Applicable only to the specific
condition of data that they are
supervised
(e.g., “bicubic” downsampled
noise-free image)
Flexibility
Few gradient updates
🤔 👍
💡
It can be applied to
real-world scenes.
One single gradient
update can yield quite
considerable results.
Meta-Learning
MAML1) scheme for fast
adaptation of ZSSR
Proposed for flexible
internal learning.
1) Model-Agnostic Meta-Learning (https://arxiv.org/abs/1703.03400)
Transfer-Learning
Employed a pre-trained
network with meta-learning
method
4
Previous Methods
MZSR
Background
Bicubic Down-sampling
Why Down-sample images
2) Comparison of bicubic interpolation with some 1- and 2-dimensional interpolations (Wikipedia).
2)
A neighborhood of 4X4 pixel is
considered to perform the interpolation
Down-
sample
LR
HR
Super-
Resolution
HR
LR- HR pairs is used
in the training phase
The trained network
used for SR
in the test phase
6
CNN-based Methods
Most of the recent CNN-based
methods are applicable only to
the specific condition of data
that they are supervised.
(e.g., “bicubic” downsampling.)
They are rained on large-scale
external dataset and the number
of parameters are large to boost
performance
Limitations
Convolutional neural networks (CNNs) have shown dramatic
improvements in single image super-resolution (SISR) by using large-
scale external samples.
MZSR
Bicubic
3) ECCV ’18 Image Super-Resolution Using Very Deep Residual Channel Attention Networks (https://arxiv.org/abs/1807.02758)
Gaussian
RCAN 3)
RCAN
34.28dB
👍
👍
27.15dB
7
SISR (Single Image Super-Resolution)
SISR is to find a plausible high-resolution image from its counterpart low-
resolution image.
Downsample
LR
HR
MZSR
LR
HR
Low Resolution
Image
Convolution
Decimation
Gaussian noise
Blur kernel
Low Resolution
Image
8
ZSSR (Zero-Shot Super-Resolution)
Requires thousands of gradient
updates, which require a large
amount of time.
Cannot fully exploit large-scale
external dataset.
Limitations
Train on HR-LR pairs from the test
image itself.
Downsample
LR
HR
LR
HR
Learns image specific internal
information
Totally unsupervised or self-
supervised
MZSR
9
Question 🤔
Proposed Methods
MZSR (Meta Transfer Learning for Zero Shot Super Resolution)
The overall scheme of our proposed MZSR
MZSR
12
Large-scale Training
MZSR
Pixel-wise L1 loss
Bicubic
Prediction
Ground-truth
Synthesized large number of paired dataset.
Trained the network to learn super-resolution of
“bicubic” degradation model by minimizing the loss.
Paired dataset
13
Meta-Transfer Learning
MZSR
Adopt optimization-based meta-training step. Mostly
follow MAML with some modifications.
Synthesize paired dataset with Gaussian kernels.
Covariance matrix
MAML(Model-Agnostic Meta-Learning)
14
Task-level training, test dataset
Dataset for meta-transfer learning
A kernel distribution is p(k) and each kernel is
determined by a covariance matrix ∑.
Meta-Transfer Learning
MZSR
The model parameters 𝜃 are optimized to achieve
minimal test error of 𝜃 with respect to 𝜃!.
Meta-transfer optimization
Task-level gradient updates with the respect to model
parameter 𝜃.
15
Parameter update rule
Parameter update
Meta-learning rate
Kernel distribution
Task-level learning rate
Meta-Test
MZSR
Self-supervised ZSSR(Zero-Shot Super Resolution)
Learn internal information within a single image.
With a given LR image, we downsample it with corresponding downsampling
kernel to generate 𝚰"#$(Image son)
Perform a few gradient updates with respect to the model parameter using a
single pair of “LR son” and a given image.
Downsample
LR
HR
LR
HR 16
Algorithm
MZSR
Large-scaling
Training
Meta-transfer
learning
Optimization
17
Question 🤔
18
Experiments
Measurement and Kernels
d: direct subsampling
b: bicubic subsampling
MZSR
PSNR (dB): peak-signal-to-noise ratio
SSIM: structural index similarity
PSNR and SSIM Gaussian blur kernels
anisotropic
isotropic isotropic isotropic
20
The average PSNR/SSIM results on “bicubic” downsampling scenario with ×2 on benchmarks
Evaluations on “Bicubic” Downsampling
MZSR
21
Results on various kernel environments (X2)
The average PSNR/SSIM results on various kernels with ×2 on benchmarks
MZSR
22
Visualized Results
Visualized comparisons of super-resolution results (×2)
MZSR
23
Visualized Results
MZSR
Visualized comparisons of super-resolution results (×2)
24
Discussion
Number of Gradient Updates
MZSR
The initial point of MZSR shows the worst performance, but in one iteration, our method quickly adapts to the im-
age condition and shows the best performance among the compared methods.
Visualization of the initial point and after one iteration of each method
MZSR
pre-trained network
on “bicubic” degradation
26
Multi-scale Models
MZSR
Average PSNR results of multi-scale model on Set5
With multiple scaling factors, the task distribution p(𝑇) becomes more complex, in which the meta-learner struggles
to capture such regions that are suitable for fast adaptation.
As MZSR learns internal information of CNN, such images with multi-scale recurrent patterns show plausible
results even with large scaling factors.
The number in parenthesis is PSNR loss compared to the
single-scale model.
MZSR results on scaling factor × 4 with blur kernel 𝑔!.#
$
27
Complexity
MZSR
MZSR with a single gradient update requires the shortest time among comparisons.
ZSSR requires thousands of forward and backward pass to get a super-resolved image.
Comparisons of the number of parameters
and time complexity
28
Conclusion
Conclusion
MZSR
MZSR is a fast, flexible, and lightweight self-supervised super-resolution method by exploiting both external and
internal samples.
MZSR adopts an optimization-based meta-learning method with transfer learning to seek an initial point that is
sensitive to different conditions of blur kernels.
MZSR can quickly adapt to specific image conditions within a few gradient updates.
MZSR outperforms other methods, including ZSSR, which requires thousands of gradient descent iterations.
Super-resolved results (X2)
30
Question 🤔
Sources
MZSR Paper (https://arxiv.org/pdf/2002.12213.pdf)
MZSR GitHub (https://github.com/JWSoh/MZSR)
ComputerVisionFoundation Videos (https://www.youtube.com/watch?v=ZEgZtvxuT4U)
Deep Learning Paper Review and Practice (https://www.youtube.com/watch?v=PUtFz4vqXHQ)
Seonok Kim (sokim0991@korea.ac.kr)
Presenter

More Related Content

What's hot

Jonathan Ronen - Variational Autoencoders tutorial
Jonathan Ronen - Variational Autoencoders tutorialJonathan Ronen - Variational Autoencoders tutorial
Jonathan Ronen - Variational Autoencoders tutorialJonathan Ronen
 
Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...
Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...
Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...Sujit Pal
 
PR-231: A Simple Framework for Contrastive Learning of Visual Representations
PR-231: A Simple Framework for Contrastive Learning of Visual RepresentationsPR-231: A Simple Framework for Contrastive Learning of Visual Representations
PR-231: A Simple Framework for Contrastive Learning of Visual RepresentationsJinwon Lee
 
Single Image Super Resolution Overview
Single Image Super Resolution OverviewSingle Image Super Resolution Overview
Single Image Super Resolution OverviewLEE HOSEONG
 
Emerging Properties in Self-Supervised Vision Transformers
Emerging Properties in Self-Supervised Vision TransformersEmerging Properties in Self-Supervised Vision Transformers
Emerging Properties in Self-Supervised Vision TransformersSungchul Kim
 
Enhanced Deep Residual Networks for Single Image Super-Resolution
Enhanced Deep Residual Networks for Single Image Super-ResolutionEnhanced Deep Residual Networks for Single Image Super-Resolution
Enhanced Deep Residual Networks for Single Image Super-ResolutionNAVER Engineering
 
MLIP - Chapter 5 - Detection, Segmentation, Captioning
MLIP - Chapter 5 - Detection, Segmentation, CaptioningMLIP - Chapter 5 - Detection, Segmentation, Captioning
MLIP - Chapter 5 - Detection, Segmentation, CaptioningCharles Deledalle
 
ViT (Vision Transformer) Review [CDM]
ViT (Vision Transformer) Review [CDM]ViT (Vision Transformer) Review [CDM]
ViT (Vision Transformer) Review [CDM]Dongmin Choi
 
Few shot learning/ one shot learning/ machine learning
Few shot learning/ one shot learning/ machine learningFew shot learning/ one shot learning/ machine learning
Few shot learning/ one shot learning/ machine learningﺁﺻﻒ ﻋﻠﯽ ﻣﯿﺮ
 
Super resolution in deep learning era - Jaejun Yoo
Super resolution in deep learning era - Jaejun YooSuper resolution in deep learning era - Jaejun Yoo
Super resolution in deep learning era - Jaejun YooJaeJun Yoo
 
PR-366: A ConvNet for 2020s
PR-366: A ConvNet for 2020sPR-366: A ConvNet for 2020s
PR-366: A ConvNet for 2020sJinwon Lee
 
Support vector machines (svm)
Support vector machines (svm)Support vector machines (svm)
Support vector machines (svm)Sharayu Patil
 
【論文読み会】Deep Reinforcement Learning at the Edge of the Statistical Precipice
【論文読み会】Deep Reinforcement Learning at the Edge of the Statistical Precipice【論文読み会】Deep Reinforcement Learning at the Edge of the Statistical Precipice
【論文読み会】Deep Reinforcement Learning at the Edge of the Statistical PrecipiceARISE analytics
 
Optimization in Deep Learning
Optimization in Deep LearningOptimization in Deep Learning
Optimization in Deep LearningYan Xu
 
(SURVEY) Semi Supervised Learning
(SURVEY) Semi Supervised Learning(SURVEY) Semi Supervised Learning
(SURVEY) Semi Supervised LearningYamato OKAMOTO
 
Ensemble learning Techniques
Ensemble learning TechniquesEnsemble learning Techniques
Ensemble learning TechniquesBabu Priyavrat
 
Segment Anything
Segment AnythingSegment Anything
Segment Anythingfake can
 
Support vector machine
Support vector machineSupport vector machine
Support vector machineMusa Hawamdah
 

What's hot (20)

Jonathan Ronen - Variational Autoencoders tutorial
Jonathan Ronen - Variational Autoencoders tutorialJonathan Ronen - Variational Autoencoders tutorial
Jonathan Ronen - Variational Autoencoders tutorial
 
Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...
Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...
Transfer Learning and Fine Tuning for Cross Domain Image Classification with ...
 
PR-231: A Simple Framework for Contrastive Learning of Visual Representations
PR-231: A Simple Framework for Contrastive Learning of Visual RepresentationsPR-231: A Simple Framework for Contrastive Learning of Visual Representations
PR-231: A Simple Framework for Contrastive Learning of Visual Representations
 
Tuning learning rate
Tuning learning rateTuning learning rate
Tuning learning rate
 
Single Image Super Resolution Overview
Single Image Super Resolution OverviewSingle Image Super Resolution Overview
Single Image Super Resolution Overview
 
Emerging Properties in Self-Supervised Vision Transformers
Emerging Properties in Self-Supervised Vision TransformersEmerging Properties in Self-Supervised Vision Transformers
Emerging Properties in Self-Supervised Vision Transformers
 
Enhanced Deep Residual Networks for Single Image Super-Resolution
Enhanced Deep Residual Networks for Single Image Super-ResolutionEnhanced Deep Residual Networks for Single Image Super-Resolution
Enhanced Deep Residual Networks for Single Image Super-Resolution
 
MLIP - Chapter 5 - Detection, Segmentation, Captioning
MLIP - Chapter 5 - Detection, Segmentation, CaptioningMLIP - Chapter 5 - Detection, Segmentation, Captioning
MLIP - Chapter 5 - Detection, Segmentation, Captioning
 
ViT (Vision Transformer) Review [CDM]
ViT (Vision Transformer) Review [CDM]ViT (Vision Transformer) Review [CDM]
ViT (Vision Transformer) Review [CDM]
 
Few shot learning/ one shot learning/ machine learning
Few shot learning/ one shot learning/ machine learningFew shot learning/ one shot learning/ machine learning
Few shot learning/ one shot learning/ machine learning
 
Super resolution in deep learning era - Jaejun Yoo
Super resolution in deep learning era - Jaejun YooSuper resolution in deep learning era - Jaejun Yoo
Super resolution in deep learning era - Jaejun Yoo
 
PR-366: A ConvNet for 2020s
PR-366: A ConvNet for 2020sPR-366: A ConvNet for 2020s
PR-366: A ConvNet for 2020s
 
Support vector machines (svm)
Support vector machines (svm)Support vector machines (svm)
Support vector machines (svm)
 
【論文読み会】Deep Reinforcement Learning at the Edge of the Statistical Precipice
【論文読み会】Deep Reinforcement Learning at the Edge of the Statistical Precipice【論文読み会】Deep Reinforcement Learning at the Edge of the Statistical Precipice
【論文読み会】Deep Reinforcement Learning at the Edge of the Statistical Precipice
 
Recurrent Neural Networks
Recurrent Neural NetworksRecurrent Neural Networks
Recurrent Neural Networks
 
Optimization in Deep Learning
Optimization in Deep LearningOptimization in Deep Learning
Optimization in Deep Learning
 
(SURVEY) Semi Supervised Learning
(SURVEY) Semi Supervised Learning(SURVEY) Semi Supervised Learning
(SURVEY) Semi Supervised Learning
 
Ensemble learning Techniques
Ensemble learning TechniquesEnsemble learning Techniques
Ensemble learning Techniques
 
Segment Anything
Segment AnythingSegment Anything
Segment Anything
 
Support vector machine
Support vector machineSupport vector machine
Support vector machine
 

Similar to [딥논읽] Meta-Transfer Learning for Zero-Shot Super-Resolution paper review

IEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsIEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsVijay Karan
 
IEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsIEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsVijay Karan
 
Face alignment by deep convolutional network with adaptive learning rate
Face alignment by deep convolutional network with adaptive learning rateFace alignment by deep convolutional network with adaptive learning rate
Face alignment by deep convolutional network with adaptive learning rateZhiwen Shao
 
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...IOSR Journals
 
Algorithm for Lossy Image Compression using FPGA
Algorithm for Lossy Image Compression using FPGAAlgorithm for Lossy Image Compression using FPGA
Algorithm for Lossy Image Compression using FPGAMistral Solutions
 
Deep vs diverse architectures for classification problems
Deep vs diverse architectures for classification problemsDeep vs diverse architectures for classification problems
Deep vs diverse architectures for classification problemsColleen Farrelly
 
AI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
AI Class Topic 6: Easy Way to Learn Deep Learning AI TechnologiesAI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
AI Class Topic 6: Easy Way to Learn Deep Learning AI TechnologiesValue Amplify Consulting
 
Machine learning in science and industry — day 4
Machine learning in science and industry — day 4Machine learning in science and industry — day 4
Machine learning in science and industry — day 4arogozhnikov
 
Asilomar09 compressive superres
Asilomar09 compressive superresAsilomar09 compressive superres
Asilomar09 compressive superresHoàng Sơn
 
ABayesianApproachToLocalizedMultiKernelLearningUsingTheRelevanceVectorMachine...
ABayesianApproachToLocalizedMultiKernelLearningUsingTheRelevanceVectorMachine...ABayesianApproachToLocalizedMultiKernelLearningUsingTheRelevanceVectorMachine...
ABayesianApproachToLocalizedMultiKernelLearningUsingTheRelevanceVectorMachine...grssieee
 
Image Processing IEEE 2015 Projects
Image Processing IEEE 2015 ProjectsImage Processing IEEE 2015 Projects
Image Processing IEEE 2015 ProjectsVijay Karan
 
M.E Computer Science Remote Sensing Projects
M.E Computer Science Remote Sensing ProjectsM.E Computer Science Remote Sensing Projects
M.E Computer Science Remote Sensing ProjectsVijay Karan
 
Comparison of Neural Network Training Functions for Hematoma Classification i...
Comparison of Neural Network Training Functions for Hematoma Classification i...Comparison of Neural Network Training Functions for Hematoma Classification i...
Comparison of Neural Network Training Functions for Hematoma Classification i...IOSR Journals
 
PSO-based Training, Pruning, and Ensembling of Extreme Learning Machine RBF N...
PSO-based Training, Pruning, and Ensembling of Extreme Learning Machine RBF N...PSO-based Training, Pruning, and Ensembling of Extreme Learning Machine RBF N...
PSO-based Training, Pruning, and Ensembling of Extreme Learning Machine RBF N...ijceronline
 
Deep learning based recommender systems (lab seminar paper review)
Deep learning based recommender systems (lab seminar paper review)Deep learning based recommender systems (lab seminar paper review)
Deep learning based recommender systems (lab seminar paper review)hyunsung lee
 
Image Processing IEEE 2015 Projects
Image Processing IEEE 2015 ProjectsImage Processing IEEE 2015 Projects
Image Processing IEEE 2015 ProjectsVijay Karan
 
Remote Sensing IEEE 2015 Projects
Remote Sensing IEEE 2015 ProjectsRemote Sensing IEEE 2015 Projects
Remote Sensing IEEE 2015 ProjectsVijay Karan
 
M phil-computer-science-remote-sensing-projects
M phil-computer-science-remote-sensing-projectsM phil-computer-science-remote-sensing-projects
M phil-computer-science-remote-sensing-projectsVijay Karan
 
M phil-computer-science-remote-sensing-projects
M phil-computer-science-remote-sensing-projectsM phil-computer-science-remote-sensing-projects
M phil-computer-science-remote-sensing-projectsVijay Karan
 

Similar to [딥논읽] Meta-Transfer Learning for Zero-Shot Super-Resolution paper review (20)

IEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsIEEE 2015 Matlab Projects
IEEE 2015 Matlab Projects
 
IEEE 2015 Matlab Projects
IEEE 2015 Matlab ProjectsIEEE 2015 Matlab Projects
IEEE 2015 Matlab Projects
 
Face alignment by deep convolutional network with adaptive learning rate
Face alignment by deep convolutional network with adaptive learning rateFace alignment by deep convolutional network with adaptive learning rate
Face alignment by deep convolutional network with adaptive learning rate
 
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
Using Multi-layered Feed-forward Neural Network (MLFNN) Architecture as Bidir...
 
deep CNN vs conventional ML
deep CNN vs conventional MLdeep CNN vs conventional ML
deep CNN vs conventional ML
 
Algorithm for Lossy Image Compression using FPGA
Algorithm for Lossy Image Compression using FPGAAlgorithm for Lossy Image Compression using FPGA
Algorithm for Lossy Image Compression using FPGA
 
Deep vs diverse architectures for classification problems
Deep vs diverse architectures for classification problemsDeep vs diverse architectures for classification problems
Deep vs diverse architectures for classification problems
 
AI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
AI Class Topic 6: Easy Way to Learn Deep Learning AI TechnologiesAI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
AI Class Topic 6: Easy Way to Learn Deep Learning AI Technologies
 
Machine learning in science and industry — day 4
Machine learning in science and industry — day 4Machine learning in science and industry — day 4
Machine learning in science and industry — day 4
 
Asilomar09 compressive superres
Asilomar09 compressive superresAsilomar09 compressive superres
Asilomar09 compressive superres
 
ABayesianApproachToLocalizedMultiKernelLearningUsingTheRelevanceVectorMachine...
ABayesianApproachToLocalizedMultiKernelLearningUsingTheRelevanceVectorMachine...ABayesianApproachToLocalizedMultiKernelLearningUsingTheRelevanceVectorMachine...
ABayesianApproachToLocalizedMultiKernelLearningUsingTheRelevanceVectorMachine...
 
Image Processing IEEE 2015 Projects
Image Processing IEEE 2015 ProjectsImage Processing IEEE 2015 Projects
Image Processing IEEE 2015 Projects
 
M.E Computer Science Remote Sensing Projects
M.E Computer Science Remote Sensing ProjectsM.E Computer Science Remote Sensing Projects
M.E Computer Science Remote Sensing Projects
 
Comparison of Neural Network Training Functions for Hematoma Classification i...
Comparison of Neural Network Training Functions for Hematoma Classification i...Comparison of Neural Network Training Functions for Hematoma Classification i...
Comparison of Neural Network Training Functions for Hematoma Classification i...
 
PSO-based Training, Pruning, and Ensembling of Extreme Learning Machine RBF N...
PSO-based Training, Pruning, and Ensembling of Extreme Learning Machine RBF N...PSO-based Training, Pruning, and Ensembling of Extreme Learning Machine RBF N...
PSO-based Training, Pruning, and Ensembling of Extreme Learning Machine RBF N...
 
Deep learning based recommender systems (lab seminar paper review)
Deep learning based recommender systems (lab seminar paper review)Deep learning based recommender systems (lab seminar paper review)
Deep learning based recommender systems (lab seminar paper review)
 
Image Processing IEEE 2015 Projects
Image Processing IEEE 2015 ProjectsImage Processing IEEE 2015 Projects
Image Processing IEEE 2015 Projects
 
Remote Sensing IEEE 2015 Projects
Remote Sensing IEEE 2015 ProjectsRemote Sensing IEEE 2015 Projects
Remote Sensing IEEE 2015 Projects
 
M phil-computer-science-remote-sensing-projects
M phil-computer-science-remote-sensing-projectsM phil-computer-science-remote-sensing-projects
M phil-computer-science-remote-sensing-projects
 
M phil-computer-science-remote-sensing-projects
M phil-computer-science-remote-sensing-projectsM phil-computer-science-remote-sensing-projects
M phil-computer-science-remote-sensing-projects
 

More from taeseon ryu

OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...taeseon ryu
 
3D Gaussian Splatting
3D Gaussian Splatting3D Gaussian Splatting
3D Gaussian Splattingtaeseon ryu
 
Hyperbolic Image Embedding.pptx
Hyperbolic  Image Embedding.pptxHyperbolic  Image Embedding.pptx
Hyperbolic Image Embedding.pptxtaeseon ryu
 
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정taeseon ryu
 
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdfLLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdftaeseon ryu
 
Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories taeseon ryu
 
Packed Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation ExtractionPacked Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation Extractiontaeseon ryu
 
MOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement LearningMOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement Learningtaeseon ryu
 
Scaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language ModelsScaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language Modelstaeseon ryu
 
Visual prompt tuning
Visual prompt tuningVisual prompt tuning
Visual prompt tuningtaeseon ryu
 
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdfvariBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdftaeseon ryu
 
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdfReinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdftaeseon ryu
 
The Forward-Forward Algorithm
The Forward-Forward AlgorithmThe Forward-Forward Algorithm
The Forward-Forward Algorithmtaeseon ryu
 
Towards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural NetworksTowards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural Networkstaeseon ryu
 
BRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive SummarizationBRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive Summarizationtaeseon ryu
 

More from taeseon ryu (20)

VoxelNet
VoxelNetVoxelNet
VoxelNet
 
OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...OpineSum Entailment-based self-training for abstractive opinion summarization...
OpineSum Entailment-based self-training for abstractive opinion summarization...
 
3D Gaussian Splatting
3D Gaussian Splatting3D Gaussian Splatting
3D Gaussian Splatting
 
JetsonTX2 Python
 JetsonTX2 Python  JetsonTX2 Python
JetsonTX2 Python
 
Hyperbolic Image Embedding.pptx
Hyperbolic  Image Embedding.pptxHyperbolic  Image Embedding.pptx
Hyperbolic Image Embedding.pptx
 
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
MCSE_Multimodal Contrastive Learning of Sentence Embeddings_변현정
 
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdfLLaMA Open and Efficient Foundation Language Models - 230528.pdf
LLaMA Open and Efficient Foundation Language Models - 230528.pdf
 
YOLO V6
YOLO V6YOLO V6
YOLO V6
 
Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories Dataset Distillation by Matching Training Trajectories
Dataset Distillation by Matching Training Trajectories
 
RL_UpsideDown
RL_UpsideDownRL_UpsideDown
RL_UpsideDown
 
Packed Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation ExtractionPacked Levitated Marker for Entity and Relation Extraction
Packed Levitated Marker for Entity and Relation Extraction
 
MOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement LearningMOReL: Model-Based Offline Reinforcement Learning
MOReL: Model-Based Offline Reinforcement Learning
 
Scaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language ModelsScaling Instruction-Finetuned Language Models
Scaling Instruction-Finetuned Language Models
 
Visual prompt tuning
Visual prompt tuningVisual prompt tuning
Visual prompt tuning
 
mPLUG
mPLUGmPLUG
mPLUG
 
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdfvariBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
variBAD, A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning.pdf
 
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdfReinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
Reinforced Genetic Algorithm Learning For Optimizing Computation Graphs.pdf
 
The Forward-Forward Algorithm
The Forward-Forward AlgorithmThe Forward-Forward Algorithm
The Forward-Forward Algorithm
 
Towards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural NetworksTowards Robust and Reproducible Active Learning using Neural Networks
Towards Robust and Reproducible Active Learning using Neural Networks
 
BRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive SummarizationBRIO: Bringing Order to Abstractive Summarization
BRIO: Bringing Order to Abstractive Summarization
 

Recently uploaded

Abnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxAbnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxzeus70441
 
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary MicrobiologyLAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary MicrobiologyChayanika Das
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaDr.Mahmoud Abbas
 
Observational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsObservational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsSérgio Sacani
 
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasBACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasChayanika Das
 
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...Sérgio Sacani
 
Pests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPirithiRaju
 
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxGENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxRitchAndruAgustin
 
cybrids.pptx production_advanges_limitation
cybrids.pptx production_advanges_limitationcybrids.pptx production_advanges_limitation
cybrids.pptx production_advanges_limitationSanghamitraMohapatra5
 
The Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionThe Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionJadeNovelo1
 
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests GlycosidesGLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests GlycosidesNandakishor Bhaurao Deshmukh
 
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...Chayanika Das
 
linear Regression, multiple Regression and Annova
linear Regression, multiple Regression and Annovalinear Regression, multiple Regression and Annova
linear Regression, multiple Regression and AnnovaMansi Rastogi
 
Gas-ExchangeS-in-Plants-and-Animals.pptx
Gas-ExchangeS-in-Plants-and-Animals.pptxGas-ExchangeS-in-Plants-and-Animals.pptx
Gas-ExchangeS-in-Plants-and-Animals.pptxGiovaniTrinidad
 
BACTERIAL DEFENSE SYSTEM by Dr. Chayanika Das
BACTERIAL DEFENSE SYSTEM by Dr. Chayanika DasBACTERIAL DEFENSE SYSTEM by Dr. Chayanika Das
BACTERIAL DEFENSE SYSTEM by Dr. Chayanika DasChayanika Das
 
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxQ4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxtuking87
 
Oxo-Acids of Halogens and their Salts.pptx
Oxo-Acids of Halogens and their Salts.pptxOxo-Acids of Halogens and their Salts.pptx
Oxo-Acids of Halogens and their Salts.pptxfarhanvvdk
 

Recently uploaded (20)

Abnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptxAbnormal LFTs rate of deco and NAFLD.pptx
Abnormal LFTs rate of deco and NAFLD.pptx
 
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary MicrobiologyLAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
LAMP PCR.pptx by Dr. Chayanika Das, Ph.D, Veterinary Microbiology
 
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer ZahanaEGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
EGYPTIAN IMPRINT IN SPAIN Lecture by Dr Abeer Zahana
 
Observational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive starsObservational constraints on mergers creating magnetism in massive stars
Observational constraints on mergers creating magnetism in massive stars
 
Ultrastructure and functions of Chloroplast.pptx
Ultrastructure and functions of Chloroplast.pptxUltrastructure and functions of Chloroplast.pptx
Ultrastructure and functions of Chloroplast.pptx
 
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika DasBACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
BACTERIAL SECRETION SYSTEM by Dr. Chayanika Das
 
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
Observation of Gravitational Waves from the Coalescence of a 2.5–4.5 M⊙ Compa...
 
Pests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPRPests of Sunflower_Binomics_Identification_Dr.UPR
Pests of Sunflower_Binomics_Identification_Dr.UPR
 
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptxGENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
GENERAL PHYSICS 2 REFRACTION OF LIGHT SENIOR HIGH SCHOOL GENPHYS2.pptx
 
cybrids.pptx production_advanges_limitation
cybrids.pptx production_advanges_limitationcybrids.pptx production_advanges_limitation
cybrids.pptx production_advanges_limitation
 
AZOTOBACTER AS BIOFERILIZER.PPTX
AZOTOBACTER AS BIOFERILIZER.PPTXAZOTOBACTER AS BIOFERILIZER.PPTX
AZOTOBACTER AS BIOFERILIZER.PPTX
 
The Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and FunctionThe Sensory Organs, Anatomy and Function
The Sensory Organs, Anatomy and Function
 
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests GlycosidesGLYCOSIDES Classification Of GLYCOSIDES  Chemical Tests Glycosides
GLYCOSIDES Classification Of GLYCOSIDES Chemical Tests Glycosides
 
Introduction Classification Of Alkaloids
Introduction Classification Of AlkaloidsIntroduction Classification Of Alkaloids
Introduction Classification Of Alkaloids
 
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
ESSENTIAL FEATURES REQUIRED FOR ESTABLISHING FOUR TYPES OF BIOSAFETY LABORATO...
 
linear Regression, multiple Regression and Annova
linear Regression, multiple Regression and Annovalinear Regression, multiple Regression and Annova
linear Regression, multiple Regression and Annova
 
Gas-ExchangeS-in-Plants-and-Animals.pptx
Gas-ExchangeS-in-Plants-and-Animals.pptxGas-ExchangeS-in-Plants-and-Animals.pptx
Gas-ExchangeS-in-Plants-and-Animals.pptx
 
BACTERIAL DEFENSE SYSTEM by Dr. Chayanika Das
BACTERIAL DEFENSE SYSTEM by Dr. Chayanika DasBACTERIAL DEFENSE SYSTEM by Dr. Chayanika Das
BACTERIAL DEFENSE SYSTEM by Dr. Chayanika Das
 
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptxQ4-Mod-1c-Quiz-Projectile-333344444.pptx
Q4-Mod-1c-Quiz-Projectile-333344444.pptx
 
Oxo-Acids of Halogens and their Salts.pptx
Oxo-Acids of Halogens and their Salts.pptxOxo-Acids of Halogens and their Salts.pptx
Oxo-Acids of Halogens and their Salts.pptx
 

[딥논읽] Meta-Transfer Learning for Zero-Shot Super-Resolution paper review

  • 1. Meta-Transfer Learning for Zero-Shot Super-Resolution Image Processing Team Dawoon Heo Jeongho Shin Seonok Kim(🙋) Presented in CVPR 2020 Authors - Jae Woong Soh, Sunwoo Cho, Nam Ik Cho (Seoul National University)
  • 4. MZSR MZSR (Meta Transfer Learning for Zero Shot Super Resolution) Zero Shot Super Resolution Recent CNN based methods Trained on large-scale ‘external’ dataset Applicable only to the specific condition of data that they are supervised (e.g., “bicubic” downsampled noise-free image) Flexibility Few gradient updates 🤔 👍 💡 It can be applied to real-world scenes. One single gradient update can yield quite considerable results. Meta-Learning MAML1) scheme for fast adaptation of ZSSR Proposed for flexible internal learning. 1) Model-Agnostic Meta-Learning (https://arxiv.org/abs/1703.03400) Transfer-Learning Employed a pre-trained network with meta-learning method 4
  • 6. MZSR Background Bicubic Down-sampling Why Down-sample images 2) Comparison of bicubic interpolation with some 1- and 2-dimensional interpolations (Wikipedia). 2) A neighborhood of 4X4 pixel is considered to perform the interpolation Down- sample LR HR Super- Resolution HR LR- HR pairs is used in the training phase The trained network used for SR in the test phase 6
  • 7. CNN-based Methods Most of the recent CNN-based methods are applicable only to the specific condition of data that they are supervised. (e.g., “bicubic” downsampling.) They are rained on large-scale external dataset and the number of parameters are large to boost performance Limitations Convolutional neural networks (CNNs) have shown dramatic improvements in single image super-resolution (SISR) by using large- scale external samples. MZSR Bicubic 3) ECCV ’18 Image Super-Resolution Using Very Deep Residual Channel Attention Networks (https://arxiv.org/abs/1807.02758) Gaussian RCAN 3) RCAN 34.28dB 👍 👍 27.15dB 7
  • 8. SISR (Single Image Super-Resolution) SISR is to find a plausible high-resolution image from its counterpart low- resolution image. Downsample LR HR MZSR LR HR Low Resolution Image Convolution Decimation Gaussian noise Blur kernel Low Resolution Image 8
  • 9. ZSSR (Zero-Shot Super-Resolution) Requires thousands of gradient updates, which require a large amount of time. Cannot fully exploit large-scale external dataset. Limitations Train on HR-LR pairs from the test image itself. Downsample LR HR LR HR Learns image specific internal information Totally unsupervised or self- supervised MZSR 9
  • 12. MZSR (Meta Transfer Learning for Zero Shot Super Resolution) The overall scheme of our proposed MZSR MZSR 12
  • 13. Large-scale Training MZSR Pixel-wise L1 loss Bicubic Prediction Ground-truth Synthesized large number of paired dataset. Trained the network to learn super-resolution of “bicubic” degradation model by minimizing the loss. Paired dataset 13
  • 14. Meta-Transfer Learning MZSR Adopt optimization-based meta-training step. Mostly follow MAML with some modifications. Synthesize paired dataset with Gaussian kernels. Covariance matrix MAML(Model-Agnostic Meta-Learning) 14 Task-level training, test dataset Dataset for meta-transfer learning A kernel distribution is p(k) and each kernel is determined by a covariance matrix ∑.
  • 15. Meta-Transfer Learning MZSR The model parameters 𝜃 are optimized to achieve minimal test error of 𝜃 with respect to 𝜃!. Meta-transfer optimization Task-level gradient updates with the respect to model parameter 𝜃. 15 Parameter update rule Parameter update Meta-learning rate Kernel distribution Task-level learning rate
  • 16. Meta-Test MZSR Self-supervised ZSSR(Zero-Shot Super Resolution) Learn internal information within a single image. With a given LR image, we downsample it with corresponding downsampling kernel to generate 𝚰"#$(Image son) Perform a few gradient updates with respect to the model parameter using a single pair of “LR son” and a given image. Downsample LR HR LR HR 16
  • 20. Measurement and Kernels d: direct subsampling b: bicubic subsampling MZSR PSNR (dB): peak-signal-to-noise ratio SSIM: structural index similarity PSNR and SSIM Gaussian blur kernels anisotropic isotropic isotropic isotropic 20
  • 21. The average PSNR/SSIM results on “bicubic” downsampling scenario with ×2 on benchmarks Evaluations on “Bicubic” Downsampling MZSR 21
  • 22. Results on various kernel environments (X2) The average PSNR/SSIM results on various kernels with ×2 on benchmarks MZSR 22
  • 23. Visualized Results Visualized comparisons of super-resolution results (×2) MZSR 23
  • 24. Visualized Results MZSR Visualized comparisons of super-resolution results (×2) 24
  • 26. Number of Gradient Updates MZSR The initial point of MZSR shows the worst performance, but in one iteration, our method quickly adapts to the im- age condition and shows the best performance among the compared methods. Visualization of the initial point and after one iteration of each method MZSR pre-trained network on “bicubic” degradation 26
  • 27. Multi-scale Models MZSR Average PSNR results of multi-scale model on Set5 With multiple scaling factors, the task distribution p(𝑇) becomes more complex, in which the meta-learner struggles to capture such regions that are suitable for fast adaptation. As MZSR learns internal information of CNN, such images with multi-scale recurrent patterns show plausible results even with large scaling factors. The number in parenthesis is PSNR loss compared to the single-scale model. MZSR results on scaling factor × 4 with blur kernel 𝑔!.# $ 27
  • 28. Complexity MZSR MZSR with a single gradient update requires the shortest time among comparisons. ZSSR requires thousands of forward and backward pass to get a super-resolved image. Comparisons of the number of parameters and time complexity 28
  • 30. Conclusion MZSR MZSR is a fast, flexible, and lightweight self-supervised super-resolution method by exploiting both external and internal samples. MZSR adopts an optimization-based meta-learning method with transfer learning to seek an initial point that is sensitive to different conditions of blur kernels. MZSR can quickly adapt to specific image conditions within a few gradient updates. MZSR outperforms other methods, including ZSSR, which requires thousands of gradient descent iterations. Super-resolved results (X2) 30
  • 32. Sources MZSR Paper (https://arxiv.org/pdf/2002.12213.pdf) MZSR GitHub (https://github.com/JWSoh/MZSR) ComputerVisionFoundation Videos (https://www.youtube.com/watch?v=ZEgZtvxuT4U) Deep Learning Paper Review and Practice (https://www.youtube.com/watch?v=PUtFz4vqXHQ) Seonok Kim (sokim0991@korea.ac.kr) Presenter