Submit Search
Upload
Deep Learningによる超解像の進歩
•
44 likes
•
29,958 views
H
Hiroto Honda
Follow
deep learningベースの超解像手法についてのまとめ
Read less
Read more
Technology
Report
Share
Report
Share
1 of 36
Download now
Download to read offline
Recommended
実装レベルで学ぶVQVAE
実装レベルで学ぶVQVAE
ぱんいち すみもと
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
Deep Learning JP
3D CNNによる人物行動認識の動向
3D CNNによる人物行動認識の動向
Kensho Hara
【メタサーベイ】Vision and Language のトップ研究室/研究者
【メタサーベイ】Vision and Language のトップ研究室/研究者
cvpaper. challenge
モデルアーキテクチャ観点からのDeep Neural Network高速化
モデルアーキテクチャ観点からのDeep Neural Network高速化
Yusuke Uchida
【DL輪読会】言語以外でのTransformerのまとめ (ViT, Perceiver, Frozen Pretrained Transformer etc)
【DL輪読会】言語以外でのTransformerのまとめ (ViT, Perceiver, Frozen Pretrained Transformer etc)
Deep Learning JP
SSII2021 [OS2-01] 転移学習の基礎:異なるタスクの知識を利用するための機械学習の方法
SSII2021 [OS2-01] 転移学習の基礎:異なるタスクの知識を利用するための機械学習の方法
SSII
Transformer メタサーベイ
Transformer メタサーベイ
cvpaper. challenge
Recommended
実装レベルで学ぶVQVAE
実装レベルで学ぶVQVAE
ぱんいち すみもと
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
Deep Learning JP
3D CNNによる人物行動認識の動向
3D CNNによる人物行動認識の動向
Kensho Hara
【メタサーベイ】Vision and Language のトップ研究室/研究者
【メタサーベイ】Vision and Language のトップ研究室/研究者
cvpaper. challenge
モデルアーキテクチャ観点からのDeep Neural Network高速化
モデルアーキテクチャ観点からのDeep Neural Network高速化
Yusuke Uchida
【DL輪読会】言語以外でのTransformerのまとめ (ViT, Perceiver, Frozen Pretrained Transformer etc)
【DL輪読会】言語以外でのTransformerのまとめ (ViT, Perceiver, Frozen Pretrained Transformer etc)
Deep Learning JP
SSII2021 [OS2-01] 転移学習の基礎:異なるタスクの知識を利用するための機械学習の方法
SSII2021 [OS2-01] 転移学習の基礎:異なるタスクの知識を利用するための機械学習の方法
SSII
Transformer メタサーベイ
Transformer メタサーベイ
cvpaper. challenge
画像生成・生成モデル メタサーベイ
画像生成・生成モデル メタサーベイ
cvpaper. challenge
Anomaly detection 系の論文を一言でまとめた
Anomaly detection 系の論文を一言でまとめた
ぱんいち すみもと
動画認識サーベイv1(メタサーベイ )
動画認識サーベイv1(メタサーベイ )
cvpaper. challenge
全力解説!Transformer
全力解説!Transformer
Arithmer Inc.
近年のHierarchical Vision Transformer
近年のHierarchical Vision Transformer
Yusuke Uchida
【DL輪読会】ViT + Self Supervised Learningまとめ
【DL輪読会】ViT + Self Supervised Learningまとめ
Deep Learning JP
[DL輪読会]相互情報量最大化による表現学習
[DL輪読会]相互情報量最大化による表現学習
Deep Learning JP
畳み込みニューラルネットワークの高精度化と高速化
畳み込みニューラルネットワークの高精度化と高速化
Yusuke Uchida
[DL輪読会]画像を使ったSim2Realの現況
[DL輪読会]画像を使ったSim2Realの現況
Deep Learning JP
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII
[DL輪読会]Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
[DL輪読会]Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Deep Learning JP
Skip Connection まとめ(Neural Network)
Skip Connection まとめ(Neural Network)
Yamato OKAMOTO
モデルアーキテクチャ観点からの高速化2019
モデルアーキテクチャ観点からの高速化2019
Yusuke Uchida
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Yusuke Uchida
【メタサーベイ】数式ドリブン教師あり学習
【メタサーベイ】数式ドリブン教師あり学習
cvpaper. challenge
[DL輪読会]ICLR2020の分布外検知速報
[DL輪読会]ICLR2020の分布外検知速報
Deep Learning JP
【論文読み会】Deep Clustering for Unsupervised Learning of Visual Features
【論文読み会】Deep Clustering for Unsupervised Learning of Visual Features
ARISE analytics
深層学習の数理
深層学習の数理
Taiji Suzuki
12. Diffusion Model の数学的基礎.pdf
12. Diffusion Model の数学的基礎.pdf
幸太朗 岩澤
「世界モデル」と関連研究について
「世界モデル」と関連研究について
Masahiro Suzuki
Recent Progress on Single-Image Super-Resolution
Recent Progress on Single-Image Super-Resolution
Hiroto Honda
SeRanet introduction
SeRanet introduction
Kosuke Nakago
More Related Content
What's hot
画像生成・生成モデル メタサーベイ
画像生成・生成モデル メタサーベイ
cvpaper. challenge
Anomaly detection 系の論文を一言でまとめた
Anomaly detection 系の論文を一言でまとめた
ぱんいち すみもと
動画認識サーベイv1(メタサーベイ )
動画認識サーベイv1(メタサーベイ )
cvpaper. challenge
全力解説!Transformer
全力解説!Transformer
Arithmer Inc.
近年のHierarchical Vision Transformer
近年のHierarchical Vision Transformer
Yusuke Uchida
【DL輪読会】ViT + Self Supervised Learningまとめ
【DL輪読会】ViT + Self Supervised Learningまとめ
Deep Learning JP
[DL輪読会]相互情報量最大化による表現学習
[DL輪読会]相互情報量最大化による表現学習
Deep Learning JP
畳み込みニューラルネットワークの高精度化と高速化
畳み込みニューラルネットワークの高精度化と高速化
Yusuke Uchida
[DL輪読会]画像を使ったSim2Realの現況
[DL輪読会]画像を使ったSim2Realの現況
Deep Learning JP
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII
[DL輪読会]Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
[DL輪読会]Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Deep Learning JP
Skip Connection まとめ(Neural Network)
Skip Connection まとめ(Neural Network)
Yamato OKAMOTO
モデルアーキテクチャ観点からの高速化2019
モデルアーキテクチャ観点からの高速化2019
Yusuke Uchida
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Yusuke Uchida
【メタサーベイ】数式ドリブン教師あり学習
【メタサーベイ】数式ドリブン教師あり学習
cvpaper. challenge
[DL輪読会]ICLR2020の分布外検知速報
[DL輪読会]ICLR2020の分布外検知速報
Deep Learning JP
【論文読み会】Deep Clustering for Unsupervised Learning of Visual Features
【論文読み会】Deep Clustering for Unsupervised Learning of Visual Features
ARISE analytics
深層学習の数理
深層学習の数理
Taiji Suzuki
12. Diffusion Model の数学的基礎.pdf
12. Diffusion Model の数学的基礎.pdf
幸太朗 岩澤
「世界モデル」と関連研究について
「世界モデル」と関連研究について
Masahiro Suzuki
What's hot
(20)
画像生成・生成モデル メタサーベイ
画像生成・生成モデル メタサーベイ
Anomaly detection 系の論文を一言でまとめた
Anomaly detection 系の論文を一言でまとめた
動画認識サーベイv1(メタサーベイ )
動画認識サーベイv1(メタサーベイ )
全力解説!Transformer
全力解説!Transformer
近年のHierarchical Vision Transformer
近年のHierarchical Vision Transformer
【DL輪読会】ViT + Self Supervised Learningまとめ
【DL輪読会】ViT + Self Supervised Learningまとめ
[DL輪読会]相互情報量最大化による表現学習
[DL輪読会]相互情報量最大化による表現学習
畳み込みニューラルネットワークの高精度化と高速化
畳み込みニューラルネットワークの高精度化と高速化
[DL輪読会]画像を使ったSim2Realの現況
[DL輪読会]画像を使ったSim2Realの現況
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
SSII2022 [SS1] ニューラル3D表現の最新動向〜 ニューラルネットでなんでも表せる?? 〜
[DL輪読会]Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
[DL輪読会]Swin Transformer: Hierarchical Vision Transformer using Shifted Windows
Skip Connection まとめ(Neural Network)
Skip Connection まとめ(Neural Network)
モデルアーキテクチャ観点からの高速化2019
モデルアーキテクチャ観点からの高速化2019
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
Swin Transformer (ICCV'21 Best Paper) を完璧に理解する資料
【メタサーベイ】数式ドリブン教師あり学習
【メタサーベイ】数式ドリブン教師あり学習
[DL輪読会]ICLR2020の分布外検知速報
[DL輪読会]ICLR2020の分布外検知速報
【論文読み会】Deep Clustering for Unsupervised Learning of Visual Features
【論文読み会】Deep Clustering for Unsupervised Learning of Visual Features
深層学習の数理
深層学習の数理
12. Diffusion Model の数学的基礎.pdf
12. Diffusion Model の数学的基礎.pdf
「世界モデル」と関連研究について
「世界モデル」と関連研究について
Similar to Deep Learningによる超解像の進歩
Recent Progress on Single-Image Super-Resolution
Recent Progress on Single-Image Super-Resolution
Hiroto Honda
SeRanet introduction
SeRanet introduction
Kosuke Nakago
Small Deep-Neural-Networks: Their Advantages and Their Design
Small Deep-Neural-Networks: Their Advantages and Their Design
Forrest Iandola
小數據如何實現電腦視覺,微軟AI研究首席剖析關鍵
小數據如何實現電腦視覺,微軟AI研究首席剖析關鍵
CHENHuiMei
Urs Köster Presenting at RE-Work DL Summit in Boston
Urs Köster Presenting at RE-Work DL Summit in Boston
Intel Nervana
Scaling Up AI Research to Production with PyTorch and MLFlow
Scaling Up AI Research to Production with PyTorch and MLFlow
Databricks
Operationalizing SDN
Operationalizing SDN
ADVA
Software Defined Visualization (SDVis): Get the Most Out of ParaView* with OS...
Software Defined Visualization (SDVis): Get the Most Out of ParaView* with OS...
Intel® Software
Transformer 動向調査 in 画像認識
Transformer 動向調査 in 画像認識
Kazuki Maeno
Deep Learning Hardware: Past, Present, & Future
Deep Learning Hardware: Past, Present, & Future
Rouyun Pan
PointNet
PointNet
PetteriTeikariPhD
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Vijay Srinivas Agneeswaran, Ph.D
"Designing CNN Algorithms for Real-time Applications," a Presentation from Al...
"Designing CNN Algorithms for Real-time Applications," a Presentation from Al...
Edge AI and Vision Alliance
Synthetic dialogue generation with Deep Learning
Synthetic dialogue generation with Deep Learning
S N
Hao hsiang ma resume
Hao hsiang ma resume
Eliot Ma
MCL303-Deep Learning with Apache MXNet and Gluon
MCL303-Deep Learning with Apache MXNet and Gluon
Amazon Web Services
Introduction to Deep Learning and neon at Galvanize
Introduction to Deep Learning and neon at Galvanize
Intel Nervana
GTC Europe 2017 Keynote
GTC Europe 2017 Keynote
NVIDIA
(Research Note) Delving deeper into convolutional neural networks for camera ...
(Research Note) Delving deeper into convolutional neural networks for camera ...
Jacky Liu
Convolutional neural network
Convolutional neural network
Yan Xu
Similar to Deep Learningによる超解像の進歩
(20)
Recent Progress on Single-Image Super-Resolution
Recent Progress on Single-Image Super-Resolution
SeRanet introduction
SeRanet introduction
Small Deep-Neural-Networks: Their Advantages and Their Design
Small Deep-Neural-Networks: Their Advantages and Their Design
小數據如何實現電腦視覺,微軟AI研究首席剖析關鍵
小數據如何實現電腦視覺,微軟AI研究首席剖析關鍵
Urs Köster Presenting at RE-Work DL Summit in Boston
Urs Köster Presenting at RE-Work DL Summit in Boston
Scaling Up AI Research to Production with PyTorch and MLFlow
Scaling Up AI Research to Production with PyTorch and MLFlow
Operationalizing SDN
Operationalizing SDN
Software Defined Visualization (SDVis): Get the Most Out of ParaView* with OS...
Software Defined Visualization (SDVis): Get the Most Out of ParaView* with OS...
Transformer 動向調査 in 画像認識
Transformer 動向調査 in 画像認識
Deep Learning Hardware: Past, Present, & Future
Deep Learning Hardware: Past, Present, & Future
PointNet
PointNet
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
Distributed deep learning_over_spark_20_nov_2014_ver_2.8
"Designing CNN Algorithms for Real-time Applications," a Presentation from Al...
"Designing CNN Algorithms for Real-time Applications," a Presentation from Al...
Synthetic dialogue generation with Deep Learning
Synthetic dialogue generation with Deep Learning
Hao hsiang ma resume
Hao hsiang ma resume
MCL303-Deep Learning with Apache MXNet and Gluon
MCL303-Deep Learning with Apache MXNet and Gluon
Introduction to Deep Learning and neon at Galvanize
Introduction to Deep Learning and neon at Galvanize
GTC Europe 2017 Keynote
GTC Europe 2017 Keynote
(Research Note) Delving deeper into convolutional neural networks for camera ...
(Research Note) Delving deeper into convolutional neural networks for camera ...
Convolutional neural network
Convolutional neural network
Recently uploaded
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Juan lago vázquez
Architecting Cloud Native Applications
Architecting Cloud Native Applications
WSO2
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
danishmna97
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Victor Rentea
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
Sandro Moreira
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
apidays
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Victor Rentea
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
rafiqahmad00786416
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
Andrey Devyatkin
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
Remote DBA Services
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
WSO2
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
ThousandEyes
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Deepika Singh
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
apidays
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Orbitshub
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
sammart93
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Angeliki Cooney
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
Rustici Software
Recently uploaded
(20)
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Polkadot JAM Slides - Token2049 - By Dr. Gavin Wood
Architecting Cloud Native Applications
Architecting Cloud Native Applications
CNIC Information System with Pakdata Cf In Pakistan
CNIC Information System with Pakdata Cf In Pakistan
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
[BuildWithAI] Introduction to Gemini.pdf
[BuildWithAI] Introduction to Gemini.pdf
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
Modular Monolith - a Practical Alternative to Microservices @ Devoxx UK 2024
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
How to Troubleshoot Apps for the Modern Connected Worker
How to Troubleshoot Apps for the Modern Connected Worker
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Mcleodganj Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Biography Of Angeliki Cooney | Senior Vice President Life Sciences | Albany, ...
Corporate and higher education May webinar.pptx
Corporate and higher education May webinar.pptx
Deep Learningによる超解像の進歩
1.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Deep Learningによる 超解像の進歩
2.
Copyright © DeNA Co.,Ltd. All Rights Reserved. ⾃⼰紹介 2 Hiroto Honda @hirotomusiker n メーカー研究所
→ 2017/1 DeNA n ETH Zurich CVLにて客員(2013-2014) n CVPR NTIRE Workshop Program Committee n DeNA AI研究開発エンジニア n 現職:Object Detection (OSS: https://github.com/DeNA/Chainer_Mask_R-CNN ) n 前職:Low-Level Vision, Computational, Sensor LSI
3.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Contents n 超解像は試しやすい n 初期のSISRネットワーク ⁃
SRCNN, ESPCN, VDSR ⁃ Upsampling⼿法– deconv or pixelshuffle n ベースライン⼿法:SRResNet ⁃ SRResNet, SRGAN, and EDSR n 超解像とperception ⁃ 復元結果とロス関数の関係 ⁃ Perception – Distortion Tradeoff n まとめ 3
4.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像とは n 低解像度画像 n ⾼解像度画像 4 復元
5.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像は試しやすい! 5 original(HR) LR resize train アノテーションが不要な Self-supervised learningの⼀種
6.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像の進歩 6 https://github.com/jbhuang0604/SelfExSRPSNR* [dB] (over bicubic) on Set5 dataset, x4 +1.86 +2.93 +2.06 +3.63 A+0.0 bicubic 2015 20172014 2016 +4.20 +2.48 PSNR data from:5) SRCNN
VDSR SRResNet EDSRESPCN 超解像の精度は年々向上している * PSNR = 10 log10 (2552 / MSE ) when max value is 255
7.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像ネットワークの学習 n 正解画像からpatchをcropする HR n
patchをダウンサンプルする LR = g(HR) n バッチを編成する {LR}, {HR} n ネットワークfを学習する ロス関数は: MSE(HR, f(LR)) n ...以上! 7 LR=g(HR) f(LR) HR f MSE e.g. bicubic down-sampling
8.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Non-deep⼿法: 辞書ベースのアルゴリズム 8 = 係数を最適化する 8 ベースライン: A+ (2014) http://www.vision.ee.ethz.ch/~timofter/publications/Timofte-ACCV-2014.pdf = 学習済みの辞書 x 0 + x 0 + x 0.8 + x 0.8 + x 0.05 + x 0.05 + LR patch HR patch
9.
Copyright © DeNA Co.,Ltd. All Rights Reserved. n 初期のSISR networks ⁃
SRCNN, ESPCN, VDSR ⁃ Upsampling⼿法 – deconv or pixelshuffle 9
10.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 最初のDeep超解像– SRCNN 10 Kernel size: 9 – 1 –
5 or 9 – 3 – 5 or 9 – 5 – 5 from:1) ⾮常にシンプルで計算量も少ない bicubic x2
11.
Copyright © DeNA Co.,Ltd. All Rights Reserved. VDSR: ディープなSRCNN 11 from:3) 3x3, 64 ch D= 5 to 20
12.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Efficient sub-pixel CNN (ESPCN) 12 SRCNNと違い、LR画像をconvするので効率的 Kernel size 5 – 3 – 3 from:2)
13.
Copyright © DeNA Co.,Ltd. All Rights Reserved. SRCNN / VDSR とESPCNの違い n Post-upsamplingのほうが効率的だが、1.6倍 といった⾮整数の upsamplingができない 13 SRCNN, VDSR ESPCN bicubic x2
output input Pixel shuffle x2 ch h w
14.
Copyright © DeNA Co.,Ltd. All Rights Reserved. CNNによるアップスケール - Deconvolution or PixelShuffle? n
Deconvolution 14 https://distill.pub/2016/deconv-checkerboard/ 位置ごとに関与する画素数が均⼀ではないため 格⼦パターンが出てしまう
15.
Copyright © DeNA Co.,Ltd. All Rights Reserved. CNNによるアップスケール - Deconvolution or PixelShuffle? n
resize – convolutionしては? 15 格⼦パターンはなくなる Resize(low-pass)により情報が失われる可能性があるので、 Nearest neighborで埋める⽅法も
16.
Copyright © DeNA Co.,Ltd. All Rights Reserved. CNNによるアップスケール - Deconvolution or PixelShuffle? n
Sub-pixel convolution (aka. PixelShuffle) 16 各位置でチャネルの情報をタイルする e.g. 9 channels -> 3x3 サブピクセル 格⼦ノイズフリーではない from:2)
17.
Copyright © DeNA Co.,Ltd. All Rights Reserved. n ベースライン⼿法:SRResNet ⁃ SRResNet,
SRGAN, and EDSR 17
18.
Copyright © DeNA Co.,Ltd. All Rights Reserved. SRResnet and SRGAN – twitter CVPR’17 18 Skip
connection pixel shuffle x2 MSE MSE Discriminator Trained VGG Perceptual Loss Discriminator Loss MSE Loss from:4) pixel shuffle x2 ch h w ・3種類のロス関数 ・MSEのみを使⽤する場合SRResNetと呼ぶ 24 residual blocks, 64 ch
19.
Copyright © DeNA Co.,Ltd. All Rights Reserved. SRResnet* and SRGAN – ネットワーク詳細 19 ・resblockとskip connection ・pixel shuffle upsampling from:4)
20.
Copyright © DeNA Co.,Ltd. All Rights Reserved. さらに⾼精度に特化したEnhanced Deep Super Resolution (EDSR) ソウル⼤ 20 32 residual blocks,
256 ch Skip connection x2 x2 l1 l1 Loss from:5)
21.
Copyright © DeNA Co.,Ltd. All Rights Reserved. PSNRと⾒た⽬ 21 from:5) 20dB台で1dB違うと明らかに⾒た⽬が変わる
22.
Copyright © DeNA Co.,Ltd. All Rights Reserved. n 超解像とPerception ⁃ 復元結果とロス関数の関係 ⁃
Perception – Distortion Tradeoff 22
23.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 主観評価とPSNR 23 Original SRResNet 25.53dB SRGAN 21.15dB bicubic 21.59dB Method→ PSNR → from: 4)
24.
Copyright © DeNA Co.,Ltd. All Rights Reserved. SRResnet and SRGAN – lossでこんなに違う 24 MSE loss
● ● Perceptual loss using VGG ● Discriminator loss ● ● from:4) PSNRが 最も⾼い
25.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 3タイプのロス関数 ①l1/l2 loss ②perceptual loss ③GAN
loss 25 generated image real / fake ground truth multi-scale feature matching VGG discrimi- nator generated image ground truth generated image ground truth Low Distortion Good Perception
26.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Perception-Distortion Tradeoff どの⼿法も、low distortionとgood perceptual qualityを 同時に満たせない → tradeoff把握が⼤事 26 from:8)
27.
Copyright © DeNA Co.,Ltd. All Rights Reserved. 超解像の⽬的はなにか? 27 Accurate Plausible 正確な復元 ⾃然な復元 どちらを選ぶかは、⽤途次第!! 引⽤元:4)
28.
Copyright © DeNA Co.,Ltd. All Rights Reserved. n まとめ 28
29.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Progress on SISR – 精度と速度 29 PSNR [dB] (over bicubic) on Set5 dataset, x4 +1.86 +2.93 +2.06 +3.63 A+ SRCNN
VDSR SRResNet EDSR0.0 bicubic 2015 20172014 2016 +4.20 ESPCN +2.48 0.44 0.04 0.74 1.33 40.7 ・CNNを通る画像サイズ ・中間レイヤのチャネル数 で計算量が⼤きく変化する PSNRデータ引⽤元:5) Mega-Multiplication per one input pixel for x2 restoration
30.
Copyright © DeNA Co.,Ltd. All Rights Reserved. NTIRE 2017 超解像コンペでのベンチマーク詳細 30 EDSR SRResNet VDSR ESPCN SRCNN A+ from: 9)
31.
Copyright © DeNA Co.,Ltd. All Rights Reserved. まとめ n 超解像はdeepが主流、⾼精度だが計算量が⼤きい n resblock連結
+ skip connectionや、pixel shuffle upsamplingが重要 n SRResNetベースの⼿法がベースライン n ʻAccurateʼ か ʻPlausibleʼ かは⽤途次第。 31
32.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Appendix: Residual Dense Network for Super-Resolution 32 DenseNetベースのSRResNet from: 6)
33.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Appendix: Deep Back-Projection Networks For Super-Resolution (best PSNR in NTIRE ʼ18 x8 bicubic downsampling track) 33 from: 7)
34.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Datasets n DIV2K dataset
(train, val) https://data.vision.ee.ethz.ch/cvl/DIV2K/ n Set5 dataset (test) http://people.rennes.inria.fr/Aline.Roumy/results/SR_BMVC12.html n B100 dataset (test) https://www2.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/ n Urban100 dataset (test) https://sites.google.com/site/jbhuang0604/publications/struct_sr 34
35.
Copyright © DeNA Co.,Ltd. All Rights Reserved. Competitions n NTIRE2017: New Trends
in Image Restoration and Enhancement workshop and challenge on image super- resolution in conjunction with CVPR 2017 http://www.vision.ee.ethz.ch/ntire17/ report: http://www.vision.ee.ethz.ch/~timofter/publications/Timofte-CVPRW-2017.pdf n NTIRE2018: New Trends in Image Restoration and Enhancement workshop and challenge on super-resolution, dehazing, and spectral reconstructionin conjunction with CVPR 2018 http://www.vision.ee.ethz.ch/ntire18/ report: http://openaccess.thecvf.com/content_cvpr_2018_workshops/papers/w13/Timofte_NTIRE_2018 _Challenge_CVPR_2018_paper.pdf n PIRM2018: Workshop and Challenge on Perceptual Image Restoration and Manipulation in conjunction with ECCV 2018 https://www.pirm2018.org/ 35
36.
Copyright © DeNA Co.,Ltd. All Rights Reserved. References 1) Dong et
al., Image Super-Resolution Using Deep Convolutional Networks, https://arxiv.org/abs/1501.00092 2) Shi et al., Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network, https://arxiv.org/abs/1609.05158 3) Kim et al., Accurate Image Super-Resolution Using Very Deep Convolutional Networks, https://arxiv.org/pdf/1511.04587 4) Ledig et al., Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , https://arxiv.org/abs/1609.04802 5) Lim et al., Enhanced Deep Residual Networks for Single Image Super-Resolution, https://arxiv.org/abs/1707.02921 6) Zhang et al., Residual Dense Network for Image Super-Resolution, https://arxiv.org/abs/1802.08797 7) Haris et al., Deep Back-Projection Networks For Super-Resolution, https://arxiv.org/pdf/1803.02735.pdf 8) Blau et al., Perception Distortion Tradeoff, https://arxiv.org/abs/1711.06077 9) Timofte et al., NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results , http://www.vision.ee.ethz.ch/~timofter/publications/Timofte-CVPRW-2017.pdf
Download now