Slides presented at International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2021, Strasbourg, France. DOI: 10.1007/978-3-030-87193-2_28

1. MT-UDA: Towards Unsupervised Cross-modality Medical Image
Segmentation with Limited Source Labels
Ziyuan Zhao1,2, Kaixin Xu2, Shumeng Li1,2, Zeng Zeng2, Cuntai Guan1
Presenter: Zhao Ziyuan (G2104205L)
1 Nanyang Technological University 2 Institute for Infocomm Research, A*STAR
Th-S2: Image Segmentation + Domain Adaptation # 678

2. Introduction – Medical Image Segmentation
- Segmentation in medical physics plays a crucial role in medical image analysis (MedIA)
- For instance, left atrium (LA) segmentation can provide a pre-operative assessment of its anatomy,
which is essential for treating various cardiovascular diseases, such as atrial fibrillation
- Deep learning has been widely used for medical image segmentation
Left Atrial Cavity 3D LA Visualization
2018 Atrial Segmentation Challenge. https://atriaseg2018.cardiacatlas.org/

3. Challenge (1) – Label Scarcity
- DCNNs are data-hungry and require large amounts of well-annotated data.
- Annotating medical images is laborious, expensive, and requires human expertise → Label Scarcity
Time & money
consuming
Knowledge-driven
Labor-intensive
Image DCNNs Segmentation map

4. Challenge (2) – Domain Shift
- In real-world clinical scenarios, medical images are acquired with different physical principles and
modalities, e.g., MRI & CT→ different visual appearance & distribution (Domain Gap)
- DCNNs suffer from severe performance degradation when domain shift (e.g., CT → MRI)
Cardiac Label
Cardiac CT
Source Domain
Annotate
DCNNs
Transfer
Cardiac MR
Feed
Domain Gap
Target Domain

5. Existing Work – Unsupervised Domain Adaptation
- Image adaptation – Cycle GAN (2017)
- Feature adaptation – ADDA (2017)
- Sequentially combine two adaptive strategies – CyCADA (2018)
- Synergistic Image and Feature Adaptation – SIFA (2019, 2020)
Cardiac CT Cardiac MR
Domain Gap
CycleGAN [1]
ADDA [2]
Cycada [3]
SIFA [4]
[1] Zhu, Jun-Yan, et al. "Unpaired image-to-image translation using cycle-consistent adversarial networks." ICCV 2017
[2] Tzeng, Eric, et al. "Adversarial discriminative domain adaptation." CVPR 2017.
[3] Hoffman, Judy, et al. "Cycada: Cycle-consistent adversarial domain adaptation.“ ICML 2018
[4] Chen, Cheng, et al. "Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation." TMI 2020
Source Domain Target Domain

6. Problem – Source Label Scarcity
- Despite the success of adversarial learning in UDA, these methods heavily rely on abundant source
labels.
- Become sub-optimal when only limited source labels are available in clinical deployment.
- Motivates us to study a challenging UDA scenario – source label scarcity
Source Label Scarcity
Cardiac CT Cardiac MR
Domain Gap
Source Domain Target Domain
Less
Annotations
Lower
Performance

7. Motivation – SSL + UDA
- Image-level adaptation → generates a lot of synthetic images with abundant information, which can be
leveraged for semi-supervised learning
- Appearance consistency → synthetic and real images from the same domain maintain a similar visual
appearance
- Structural consistency → transformed images should have the same structural information as the
original ones
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021
SSL
Labeled data
Performance
Number of labeled data
DCNNs
Source Domain
Target Domain
Style Transfer

8. Method – MT-UDA
- Investigate the feasibility of integrating SSL into UDA under source label scarcity
- Develop a label-efficient UDA framework based on mean teacher (MT) to explore the knowledge from
both domains
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021

9. Method (1)– Dual Cycle Alignment Module
- Generate synthetic samples for two domains using generative adversarial networks
- Synthesize source-like domain images and target-like domain images via adversarial learning
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021

10. Method (2)– Semantic Knowledge Transfer
- Appearance consistency → synthetic and real images from the same domain maintain a similar visual
appearance
- Employ the mean teacher (MT) model to distill the intra-domain semantic knowledge by forcing the
prediction consistency
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021

11. Method (3)– Structural Knowledge Transfer
- Structural consistency → transformed images should have the same structural information as the
original ones
- Propose a teacher model for keeping structural consistency between predictions of source images and
corresponding synthetic target images
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021

12. Experiments
- Dataset
- Multi-Modality Whole Heart Segmentation (MM-WHS) 2017 dataset
- Unpaired 20 MR and 20 CT volumes with ground truth masks
- Data preprocessing
- MR → source domain, CT → target domain
- SIFA setting (-16) → 16: 4 random split for train / val
- Our setting (-4)→ 4 MR volumes are labelled for UDA under source label scarcity
- Images were cropped into the size of 256 x 256
- Implementation details
- Test on fake MR images generated from CT
- Backbone: U-Net
- Supervised loss: Dice + Cross-entropy
- Total loss:
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021

13. Results (1) – Quantitative Comparison
- Degraded performance on target domain when using 4 labeled source domain scans
- MT and UA-MT can help improve the segmentation performance on target domain
- Demonstrate the feasibility of integrating SSL into UDA for label-efficient UDA
suffix −4 or −16
# labelled source scans
used for training
[1] Dou, Qi, et al. "Unsupervised cross-modality domain adaptation of convnets for biomedical image segmentations with adversarial loss." IJCAI 2018
[2] Chen, Cheng, et al. "Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation." AAAI 2019
[3] Chen, Cheng, et al. "Unsupervised bidirectional cross-modality adaptation via deeply synergistic image and feature alignment for medical image segmentation." TMI 2020
[4] Tarvainen, Antti, and Harri Valpola. "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.“NIPS 2017
[5] Yu, Lequan, et al. "Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation.“ MICCAI 2019

14. Results (2) – Qualitative Comparison
- It is observed that our method can generate more reliable masks with fewer false positives
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021

15. Results (3) – Ablation Study
- Remove one of the teacher models, separately
- W/o semantic knowledge transfer (MT-UDA-NS)
- W/o structural knowledge transfer (MTUDA-NT)
- Replace structural consistency loss with MSE loss (MT-UDA-NS-MSE)
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021

16. Conclusions
- Study a practical, challenging, and different UDA setting from the past, where only limited source labels
are accessible → Source Label Scarcity
- Investigate the feasibility of integrating SSL into UDA under source label scarcity
- Propose a label-efficient UDA framework for cross-modality medical image segmentation
- Leverage intra-domain semantic knowledge and exploit inter-domain structural knowledge
concurrently, thereby mitigating both the domain discrepancy and source label scarcity.
Zhao, Ziyuan, et al. "MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels." MICCAI 2021

17. Thanks
Zhao_Ziyuan@i2r.a-star.edu.sg