Fighting Fire with Fire: Contrastive Debiasing without Bias-free Data via Generative Bias-transformation

1. Fighting Fire with Fire: Contrastive Debiasing
without Bias-free Data
via Generative Bias-transformation
Yeonsung Jung, Hajin Shim, June Yong Yang, Eunho Yang
Korea Advanced Institute of Science and Technology (KAIST)
ICML 2023

2. • Deep neural networks easily rely on the malignant biases as shortcuts.
• Malignant bias
• Colored MNIST: Digits (task) and Color (malignant bias)
• Addressing bias problem is crucial in real-world applications.
• Significant consequences for domains such as healthcare or autonomous driving.
What is Debiasing?
2
[1] Lee et al., Learning Debiased Representation via Disentangled Feature Augmentation, NeurIPS’2021
Colored MNIST [1]
Biased samples
(Bias-aligned samples)
Bias-free samples
(Bias-conflicting samples)

3. • Existing works focus on identifying and emphasizing handful bias-free samples.
• Even though bias-aligned samples are the majority and also contain valuable task-
related information, they are often underutilized or even discarded.
• Also, previous methods easily break down in regimes where bias-free samples are
scarce or absent.
Motivation
3
Bias-aligned samples Bias-free samples
≪
(Minority)
(Majority)
< Previous >

4. • Existing works focus on identifying and emphasizing handful bias-free samples.
• Even though bias-aligned samples are the majority and also contain valuable task-
related information, they are often underutilized or even discarded.
• Also, previous methods easily break down in regimes where bias-free samples are
scarce or absent.
• Let’s focus on utilizing abundant bias-aligned samples.
Motivation
4
Bias-aligned samples Bias-free samples
< Ours >
≪
(Minority)
(Majority)

5. • Image translation models (e.g. CycleGAN[1], StarGAN[2]) render an image 𝑥 from a
source domain 𝑦 to a target domain 𝑦!.
• We hypothesize that in the case of biased dataset, the attributes representing the
the target domain 𝒚!
likely to be biased.
Synthesizing Bias-frees Samples via Image Translation Models
5
[1] Zhu at al., Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks, ICCV’2017
[2] Choi et al., StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation, CVPR’2018
Image 𝑥
Translation
model
Output
Bias-aligned samples
Generated
Bias-free samples
Target domain 𝑦!

6. • We examine the behavior of the GAN-based images translation model, StarGAN, on
biased datasets.
• We observed that the domain discriminator of image translation models perceives
the malignant bias as the representative trait of the class 𝑦.
Image Translation Models Favor Malignant Bias.
6
Domain classification
Real/Fake
1) Lead the discriminator to be biased
Bias-transformed
Biased sample
Target class
<Training StarGAN with biased dataset>
Discriminator !
Generator "
#!
~%

7. • We examine the behavior of the GAN-based images translation model, StarGAN, on
biased datasets.
• We observed that the domain discriminator of image translation models perceives
the malignant bias as the representative trait of the class 𝑦.
• The accuracy of bias-aligned samples increases to nearly 100%.
Image Translation Models Favor Malignant Bias.
7

8. • We also observed that the biased domain classifier 𝐷"#$ induces a biased generator 𝐺
by optimizing the domain classification loss:
Image Translation Models Favor Malignant Bias.
8
ℒ"#$
%
= 𝔼(',))~𝒟,)!~𝒴[− log 𝐷"#$(𝑦!
|𝐺 𝑥, 𝑦!
)]
Domain classification
Real/Fake
1) Lead the discriminator to be biased
Bias-transformed
Biased sample
Target class
2) Lead the generator to translate bias features
<Training StarGAN with biased dataset>
Discriminator !
Generator "
#!~%

9. • We also observed that the biased domain classifier 𝐷"#$ induces a biased generator 𝐺
by optimizing the domain classification loss:
• Bias-translated images on various biased datasets.
Image Translation Models Favor Malignant Bias.
9
ℒ"#$
%
= 𝔼(',))~𝒟,)!~𝒴[− log 𝐷"#$(𝑦!
|𝐺 𝑥, 𝑦!
)]

10. • Fighting Fire with Fire: Pitting one bias type against another via contrastive
learning[1].
• Employing the bias-translation model as an augmentation method of contrastive learning.
• By contrasting the bias-aligned sample and the bias-transformed view against each
other, we can attenuate the bias while learning the task-related attributes.
Contrastive Debiasing via Generative Bias-transformation (CDvG)
10
<Whole framework>
!
"!
#"~% &′
&
(′
(
Maximize agreement by
Contrastive loss
Bias-transformed
View Generation
##~%
ℒ!"
ℒ!# +
Bias-transformed
!′
Biased sample
!
)!
~*
Encoder +
Pretrained ,
[1] Chen at al., A Simple Framework for Contrastive Learning of Visual Representations, ICML’2020

11. • Quantitative results
• Integration with the previous works which focus on bias-free samples (e.g. LfF).
Experiments
11

12. • Qualitative results
• Visualization of learned representations using T-SNE on Colored MNIST
• Visualization of Grad-CAM heatmaps on BFFHQ
Experiments
12

13. Thank you !
Any Questions ?