Saliency Grafting: Innocuous Attribution-Guided Mixup with Calibrated Label Mixing

1. Saliency Grafting
Innocuous Attribution-Guided Mixup
with Calibrated Label Mixing
Joonhyung Park, June Yong Yang, Jinwoo Shin, Sung Ju Hwang, Eunho Yang
AAAI 2022
Machine Learning & Intelligence Laboratory

2. • Deep Neural Networks (DNNs) are always data-hungry
• DNNs require an immense amount of training data to improve generalization performance
• Preparing a sufficient amount of data is not always possible as it consumes manpower and
budget
• What happens if there is insufficient data?
• DNNs are predisposed to memorize the training data and exhibit degraded performance on
the unseen data – Overfitting
Data Scarcity

3. • Data Augmentation
• A technique to generate virtual data from training dataset to alleviate data scarcity issue
• It aims to prevent the network from overfitting and improve the generalization
performance for unseen data
• Data Augmentation
• Mixup-based Augmentation
• Mixup (ICLR `18): Interpolate two images linearly
• CutMix (ICCV `19): Randomly cropping a rectangular
region from an image and paste it to another image
• High diversity of augmented data (Stochastic)
• Alleviation of information loss due to regional dropout
Motivation
Mixup-based - Generating new virtual data by mixing two images
1) Mixup 2) CutMix 3) Manifold Mixup
Artificial Noise - Injecting perturbation to one image
1) Horizontal Flip 2) Color distortion 3) Regional dropout(Cutout)
√
CutMix
Mixup
Fish Bird
0.5 0.5
Fish Bird
𝟔
𝟒𝟗
𝟒𝟑
𝟒𝟗
Mixup-based augmentations

4. • Context-agnostic nature of existing methods
• Existing methods perform random mixing without considering the context of the image
• Mismatch between label and augmented data occurs frequently
• Maximum saliency* strategy
• A mixing method to reveal the most informative region of an object
Ex) PuzzleMix (ICML`20), AttentiveCutMix (ICASSP`20)
• The problem of mixing senseless regions has been solved, but there are limitations
• Deterministic - Diversity of augmented data decreases
• Overfitting - Mixing only highly discriminative parts adversely affects generalization performance
• Naï
ve label mixing - Label mixing method entirely depends on the mixing ratio
Motivation
* Saliency: Importance of features that the neural network attends to classify objects
Senseless Occlusion
Fish Bird
𝟔
𝟒𝟗
𝟒𝟑
𝟒𝟗
Mismatch

5. • Saliency Grafting aims to maintain high sample diversity while generating the
innocuous data to the neural networks
• Three core components of Saliency Grafting
• Stochastic patch selection
• Calibrated label mixing
• Cost-effective algorithm
Saliency Grafting
Stochastic
Label
Mismatch
Comparison of Mixup-based augmentations
Saliency Grafting
CutMix
Attentive CutMix
PuzzleMix
Mixup
Deterministic
Label
Match

6. • Stochastic salient patch sampling
• Identify senseless region of data through a saliency map-based algorithm
• All parts of object except the background can be stochastically mixed (Highly informative)
• Do NOT generate senseless data while keeping high diversity of augmented data
Saliency Grafting: Stochastic patch selection
∗ 𝑺𝒔𝒕: Saliency for a region (𝑠, 𝑡) of image
Temperature
Threshold

7. • Calibrated label mixing
• Label mixing ratio is adaptively determined based on the relative importance of images
• Compensate the class according to the importance of the pasting region - Figure (i) (ii)
• Penalize the class according to the occlusion of the pasted image - Figure (iii)
Saliency Grafting: Calibrated label mixing
Label mixing function 𝝍
Label mixing ratio λ
∗ 𝑰 𝑺𝒊, 𝑴𝒊 =
𝑆𝑖 ⨀𝑀𝑖
𝑆𝑖
: Importance function of the image
given the mixing matrix 𝑀𝑖 and saliency map 𝑆𝑖

8. • Cost-effective algorithm
• Related works incur additional computation/memory cost to exploit saliency information
- Additional pretrained network
- Solving optimization problems
• Saliency Grafting is efficient as the existing random augmentations
- Generate saliency map efficiently (Low computation cost)
- Using self-augmentation scheme (No additional pretrained network)
Cost-effective

9. • Saliency Grafting consistently achieves the superior performances for image
classification benchmark datasets
• Saliency Grafting also exhibits best performances on extreme data scarcity condition
Image classification results
Tiny-ImageNet dataset (ResNet-18)
CIFAR-100 dataset (PyramidNet-200)
Stressed condition – Data scarcity

10. • We design experiment to compare Saliency Grafting and other augmentations in
terms of sample diversity
• Each method trains the network by generating additional augmented data 𝒌-times
from the mini-batch; each method tries to diversify the mini-batch by producing k
independent augmented batches with its own randomness
Generating augmented data 𝒌-times
Saliency Grafting consistently improves as k increases
→ Sample diversity can be guaranteed innocuity

11. • Our contribution is threefold:
• We discuss the potential weaknesses of current Mixup-based augmentation strategies and
present a novel data augmentation strategy that can generate diverse yet meaningful data
through saliency-based sampling
• We present a novel label mixing method to calibrate the generated label to match the
information contained in the newly generated data.
• Through extensive experiments, we show that models trained with Saliency Grafting
outperform others - even under data corruption or data scarcity
Conclusion