Semi-Supervised Semantic Segmentation with Cross Pseudo Supervision, CVPR 2021.

1. PR-343
Sungchul Kim

2. Contents
1. Semi-supervised learning
2. Cross Pseudo Supervision
3. Experiments
2

3. Semi-supervised learning
3
https://steemit.com/kr/@jiwoopapa/realistic-evaluation-of-semi-supervised-learning-algorithms

4. Semi-supervised learning
4
• Mean Teacher (NIPS 2017)
https://arxiv.org/abs/1703.01780

5. Semi-supervised learning
5
• Unsupervised Data Augmentation (2019, PR-189)
https://arxiv.org/abs/1904.12848
PR-189: Unsupervised Data Augmentation for Consistency Training

6. Semi-supervised learning
6
• MixMatch (NeurIPS 2019, PR-195)
https://arxiv.org/abs/1905.02249
PR-195: MixMatch: A Holistic Approach to Semi-Supervised Learning
http://dsba.korea.ac.kr/seminar/?mod=document&uid=68

7. Semi-supervised learning
7
• FixMatch (NeurIPS 2020, PR-237)
https://arxiv.org/abs/2001.07685
PR-237: FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence

8. Semi-supervised learning
8
• Why semi-supervised semantic segmentation is challenging?
• Dense outputs require pixel-wise prediction confidences, which are difficult to estimate.
• Pixel-wise tasks : Pixel-wise classification, Pixel-wise regression
• The maximum classification probability for pixel-wise classification is unavailable in pixel-wise regression.
• Existing perturbations designed for SSL are not suitable for dense outputs.
• Strong perturbations in the input will change the input image and its label.
• The perturbed inputs from the same original image have different labels.
https://arxiv.org/abs/2008.05258

9. Semi-supervised learning
9
• CutMix-Seg (BMVC 2020)
https://arxiv.org/abs/1906.01916

10. Semi-supervised learning
10
• Cross-Consistency Training (CVPR 2020)
https://openaccess.thecvf.com/content_CVPR_2020/papers/Ouali_Semi-Supervised_Semantic_Segmentation_With_Cross-Consistency_Training_CVPR_2020_paper.pdf

11. Semi-supervised learning
11
• Guided Collaborative Training (ECCV 2020)
https://arxiv.org/abs/2008.05258

12. Semi-supervised learning
12
• PseudoSeg (ICLR 2021)
https://arxiv.org/abs/2010.09713

13. Cross Pseudo Supervision
13
CPS GCT
CutMix-Seg PseudoSeg

14. Cross Pseudo Supervision
14
• Structure
• 𝑃1 𝑃2 : the segmentation conficence map
• 𝑌1 𝑌2 : the predicted one-hot label map
• Loss
• ℒ𝑠 : supervision loss
• ℒ𝑐𝑝𝑠 : cross pseudo supervision loss
• ℒ𝑐𝑝𝑠 = ℒ𝑐𝑝𝑠
𝑙
+ ℒ𝑐𝑝𝑠
𝑢
• ℒ𝑐𝑝𝑠
𝑙
: the cross pseudo supervision loss on the labeled data
• ℒ𝑐𝑝𝑠
𝑢
: the cross pseudo supervision loss on the unlabeled data
• 𝜆 : the trade-off weight

15. Cross Pseudo Supervision
15
• Incorporation with the CutMix augmentation
• Input the CutMixed image into the two networks 𝑓 θ1 and 𝑓 θ2
• Mix the two pseudo segmentation maps as the supervision of the other segmentation network

16. Experiments
16
• Datasets
• PASCAL VOC 2012
• Cityscapes
• Divide the whole training set to two groups
• The labeled set : randomly sub-sampling 1/2, 1/4, 1/8 and 1/16 of the whole set
• The unlabeled set : the remaining images
• Evaluation
• Mean Intersection-over-Union (mIoU) metric
• Implementation details
• The weights of two backbones with the same weights pre-trained on ImageNet
• The weights of two segmentation heads (of DeepLabv3+) randomly

17. Experiments
17
• Improvements over baselines
• All the methods are based on DeepLabv3+ with (a) ResNet-50 or (b) ResNet-101.

18. Experiments
18
• Comparison with SOTA
• MT : Mean-Teacher / CCT : Cross-Consistency Training / GCT : Guided Collaborative Training
• Table 1 : PASCAL VOC 2012

19. Experiments
19
• Comparison with SOTA
• MT : Mean-Teacher / CCT : Cross-Consistency Training / GCT : Guided Collaborative Training
• Table 2 : Cityscapes

20. Experiments
20
• Empirical Study
• Cross pseudo supervision (ℒ𝑐𝑝𝑠) & Comparison with cross probability consistency (ℒ𝑐𝑝𝑐)
• The influence of applying the proposed cross pseudo supervision loss

21. Experiments
21
• Empirical Study
• The trade-off weight 𝝀
• PASCAL VOC 2012 : 𝜆 = 1.5
• Cityscapes : 𝜆 = 6
• Single-network pseudo supervision (SPS) vs. cross pseudo supervision (CPS)
• PASCAL VOC 2012
CPS
SPS
PseudoSeg

22. Experiments
22
• Empirical Study
• Combination/comparison with self-training (Noisy Student, CVPR 2020)
• Train over labeled set  predict pseudo labels for unlabeled set
 retrain over labeled and unlabeled set with pseudo labels
https://arxiv.org/abs/1911.04252

23. Experiments
23
• Empirical Study
• Qualitative Results
• (c) : supervised only
• mis-classifies many pixels due to limited labeled samples
• (d) : CPS w/o CutMix
• mislabel some dog pixels as horse pixels
• (e) : CPS w/ CutMix
• well done!

24. Thank you!
24