Paper: https://arxiv.org/abs/2004.00448 Github: https://github.com/clovaai/cutblur Youtube: https://youtu.be/MQR5mnsCvbM

1. Rethinking Data Augmentation
for Image Super-resolution
: A Comprehensive Analysis and a New Strategy; CutBlur & MoA
Jaejun Yoo*, Namhyuk Ahn*, and Kyung-Ah Sohn

2. Motivation
* Various levels of vision tasks
High-level: Semantic recognition
(e.g., classification, object detection)
Mid-level: Super-pixel
(e.g., segmentation)
Low-level: Pixels, Edges, Colors
(e.g., super-resolution, denoising)
?????
MixUp, Cutout, CutMix …

3. Analysis on existing DA methods
“Sharp transitions, mixed image contents or losing the relationships of pixels can
degrade SR performance.”
e.g., Cutout fails (discarding pixels) and every feature method fails (manipulation).
Training curves when applied feature DA’s

4. Analysis on existing DA methods
• DA methods in pixel space bring
some improvements when applied
very carefully.

5. Analysis on existing DA methods
• DA methods in pixel space bring
some improvements when applied
very carefully.
• Cutout:
Original setting (drop 25% of of pixels in a
rectangular shape) significantly degrades the
performance because it erases spatial information
too much. However, erasing tiny amount of pixels
(0.1% random pixels) boosts the performance (2~3
pixels of 48x48 input patch)
Cutout

6. Analysis on existing DA methods
• DA methods in pixel space bring
some improvements when applied
very carefully.
• Mixup & CutMix:
Improvements of using CutMix are marginal. We
suspect this happens because CutMix generates a
drastic sharp transition between two different
images.
Improvements of using Mixup is better than
CutMix but it still generates unrealistic image and
affects to the image structure.
Mixup CutMix

7. Analysis on existing DA methods
• DA methods in pixel space bring
some improvements when applied
very carefully.
• CutMixup:
To verify our hypothesis, we combine benefits of
Mixup and CutMix; CutMixup. CutMixup
provides various boundary cases while minimizes
the sharp transition by retaining partial cues as
Mixup does.
CutMixup

8. Analysis on existing DA methods
• DA methods in pixel space bring
some improvements when applied
very carefully.
• Blend & RGB permutation:
To push further, we tried a constant blending and
RGB channel permutation, which turn out to be
very simple but effective strategies showing big
performance enhancement (dB).
Note that both methods do not incur any structure
modification to an image.
BlendRGB perm.

9. CutBlur

10. CutBlur
• What does the model learn from CutBlur?
• CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the
necessary region.
Super-resolution results of a model (EDSR) trained without CutBlur and its error residual (Δ)
Error residual (Δ)Output

11. CutBlur
• What does the model learn from CutBlur?
• CutBlur prevents the SR model from over-sharpening an image and helps it to super-resolve only the
necessary region.
Super-resolution results of a model (EDSR) trained CutBlur and its error residual (Δ)
Error residual (Δ)Output
with

12. Improved generalization: over-sharpening
• Super-resolution (SR)
• Trained on ×4 scale factor dataset and tested on different scale factor (×2)

13. Improved generalization: over-smoothing
• Denoising
• Trained on severe noise (! = 70) & tested on mild noise (! = 30).

14. Improved generalization: over-removal
• JPEG artifact removal
• Trained on a mild compression rate & tested on a severe compression rate

15. Mixture of Augmentation (MoA)
• During the training phase …
• Randomly select single augmentation at
every step. (among the curated DA list)
• Apply it!

16. Comparison on diverse benchmark models and datasets
• SRCNN (0.07M) – ECCV’14, CARN (1.14M) – ECCV’18, RCAN (15.6M) – ECCV’18, EDSR (43.1M) – CVPRW’17
• DIV2K (synthetic), RealSR (real-world)
• Our method shows consistent improvement for different models (parameters) and
datasets (different environments and size):

17. Code: https://github.com/clovaai/cutblur
Paper: https://arxiv.org/abs/2004.00448
QR code for
code & paper
For more details,
please visit our website: