[MICCAI 2022] Meta-hallucinator: Towards Few-Shot Cross-Modality Cardiac Image Segmentation
•Download as PPTX, PDF•
1 like•135 views
Slides presented at International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2022, Singapore. DOI: 10.1007/978-3-031-16443-9_13
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to [MICCAI 2022] Meta-hallucinator: Towards Few-Shot Cross-Modality Cardiac Image Segmentation
Similar to [MICCAI 2022] Meta-hallucinator: Towards Few-Shot Cross-Modality Cardiac Image Segmentation (20)
More from Ziyuan Zhao
More from Ziyuan Zhao (8)
Recently uploaded
Recently uploaded (20)
[MICCAI 2022] Meta-hallucinator: Towards Few-Shot Cross-Modality Cardiac Image Segmentation
- 1. Paper ID: 422 Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation Ziyuan Zhao1,2,3, Fangcheng Zhou2,4, Zeng Zeng2,3, Cuntai Guan1, and S. Kevin Zhou5,6 Presenter: Zhao Ziyuan 1 NTU, Singapore 2 I2R, A*STAR, Singapore 3 AI3, A*STAR, Singapore 4 NUS, Singapore 5 MIRACLE, USTC, Suzhou, China 6 ICT, CAS, Beijing, China
- 2. Introduction – Cardiac Image Segmentation - Accurate segmentation of cardiac substructure in multi-modality heart images is important for the diagnosis and treatment of cardiovascular diseases. - Deep learning has been widely used for cardiac image segmentation in recent years. Chen et al. Deep Learning for Cardiac Image Segmentation: A Review. Front Cardiovasc Med.
- 3. Challenge – Source Label Scarcity - While deep learning methods hitherto have achieved considerable success in medical image segmentation, they are still hampered by domain shift and label scarcity. - The source domain not only exhibits domain shift w.r.t. the target domain but also suffers from label scarcity. Annotate Transfer Source Domain Distribution Target Domain Distribution Train Domain Shift Source Label Scarcity Expert (Costly and Laborious) Limited Labels No Labels
- 4. Method – Transformation-consistent Meta-hallucination - Propose a novel transformation-consistent meta-hallucination framework for label-efficient UDA. - Meta-learning: Introduce a meta-learning episodic training strategy to optimize hallucination and segmentation. - Self-ensembling learning: Develop a hallucination-consistent self-ensembling model for data hallucination and cross-domain knowledge transfer. Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022
- 5. Method (1)– Gradient-based meta-hallucination learning - We introduce a “hallucinator” to augment the training set to narrow the domain gap at the image level and generate useful samples for boosting the segmentation performance - We advance the hallucinator into the meta-learning process to learn how to hallucinate useful samples for the segmentation model. - In a meta-train step, the parameters of hallucinator 𝒢𝛹 and segmenter ℱ𝜃 are updated with the meta- train set via an inner-loop update, and the total meta-train objective is defined as: Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022 Meta-train Meta-test 𝓓𝒔 𝓓𝒕 𝓓𝒔 𝓓𝒕 Hallucinator Inner-loop update Segmenter Student Hallucinator 𝓛𝒔𝒆𝒈 𝓛𝒔𝒆𝒈 𝝍 𝝍′ 𝜵𝝍𝓛𝒔𝒆𝒈 𝜵𝝍𝓛𝒕𝒓𝒂𝒏𝒔 𝜽 𝜽 ′ 𝜵𝜽 𝓛𝒔𝒆𝒈 𝜵𝜽 𝓛𝒔𝒆𝒈 𝓛𝒕𝒓𝒂𝒏𝒔 𝓛𝒕𝒓𝒂𝒏𝒔
- 6. Method (2)– Hallucination-consistent Self-ensembling Learning - We apply the same spatial transformations produced by the hallucinator to the student inputs and the teacher outputs and enable the alignment between their final outputs for self-ensembling learning. - We impose the hallucination-consistent loss in the meta-test step since we expect such regularization on unseen data for robust adaptation. Then the meta-test loss is defined as: - Finally, the total objective of meta-learning is defined as: Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022 Student Teacher Hallucinator Hallucinator 𝓛𝒔𝒆𝒈 𝓛𝒄𝒐𝒏 Meta-test 𝓓𝒔 𝓓𝒕
- 7. Experimental Results - Dataset - Multi-Modality Whole Heart Segmentation (MM-WHS) 2017 dataset - For UDA, MR (20) → Source Domain, CT (20) → Target Domain - Metric: Dice score (Dice) and Average Surface Distance (ASD) - Implementation details - Segmenter: U-Net - Hallucinator: image-and-spatial transformer networks (ISTNs) - style translation → CycleGAN - image registration → STN Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022 [MM-WHS dataset]
- 8. Experimental Results (1) – 25% Source Annotations - Both self-ensemble methods and augmentation methods can help unsupervised domain adaptation under source label scarcity - Our method achieves better performance than other methods by a large margin. Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022
- 9. Experimental Results (2) – One-shot UDA - Degraded performance on target domain when using 4 labeled source domain scans - MT and UA-MT can help improve the segmentation performance on target domain - Demonstrate the feasibility of integrating SSL into UDA for label-efficient UDA Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022
- 10. Experimental Results (3) – Qualitative Comparison Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022
- 11. Experimental Results (4) – Ablation Analysis - Meta-Seg: advance mean teacher (MT) into meta-learning. - Meta-Hal: incorporate the hallucination module into meta-learning for data augmentation Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022
- 12. Conclusions - We propose a novel transformation-consistent meta-hallucination framework for label-efficient UDA. - Extensive experiments demonstrate the effectiveness of the proposed meta-hallucinator. - Our meta-hallucinator can be easily extended to various segmentation tasks suffering from domain shifts and label scarcity. Zhao, Ziyuan, et al. "Meta-hallucinator: Towards few-shot cross-modality cardiac image segmentation." MICCAI 2022
Editor's Notes
- Welcome to my presentation. My name is Zhao Ziyuan Here I am presenting our paper titled Meta-hallucinator /həˌluː.sɪˈneɪ: te/: for few-shot cross-modality cardiac image segmentation.
- Heart diseases are the leading cause of death globally. Therefore, cardiac image segmentation is critical in many clinical applications related to heart diseases. Recently, deep convolutional neural networks have achieved markable progress in cardiac image segmentation.
- However, deep convolutional neural networks are very data-hungry. Training such networks requires a large amount of data. In the medical field, annotations for medical image segmentation are difficult to be collected, since they are costly and laborious. On the other hand, given the various imaging modalities with different distributions, models trained on one domain always obtain reduced performance when applying to another domain. Current UDA methods mainly focus on leveraging source labeled and target unlabeled data for domain alignment. Source annotations, however, are also not so easy to access due to expert requirements and privacy problems. Therefore, it is essential to develop a UDA model to address both source label scarcity and domain shift.
- In this work, we propose a novel transformation-consistent meta-hallucination scheme for unsupervised domain adaptation under source label scarcity. We introduce a meta-learning training strategy to optimize both the hallucination and segmentation models by simulating structural variances and domain shifts in the training process. To further facilitate data hallucination and cross-domain knowledge transfer, we develop a hallucination-consistent self-ensembling model. The hallucination model generates helpful samples for segmentation, while the segmentation model leverages transformation-consistent constraints and segmentation objectives to facilitate the hallucination process.
- In label-scarce domain shift scenarios, we are encouraged to hallucinate useful samples for diversifying training distributions to deal with label scarcity and domain shift. To this end, we introduce a “hallucinator” module to augment the training set. The objective of the hallucinator is to narrow the domain gap at the image level and generate useful samples for boosting the segmentation performance. We advance the hallucinator into the meta-learning process to learn how to hallucinate useful samples for segmentation. In each iteration of meta-learning, the training data is randomly split into two subsets, meta-train set and meta-test set. Each meta-learning step includes a meta-train step and a meta-test step. in a meta-train step, the parameters of the hallucinator and the segmenter, are updated with the meta-train set in an inner-loop update.
- To effectively leverage the rich knowledge in the unlabeled data, we take advantage of the mean-teacher model based on self-ensembling. we construct a teacher with the same architecture as the segmenter. In light of that segmentation is desired to be transformation equivariant at the spatial level, we introduce a hallucination-consistent self-ensembling scheme to further promote unsupervised regularization. We apply the same spatial transformations produced by the hallucinator to the student inputs and the teacher outputs, and enable the alignment between their final outputs by minimizing the consistency loss. Then, the meta-test loss can be defined as the combination of segmentation loss, consistency loss and transformation loss. Finally, the total objective of meta-learning is to minimize the losses of the two steps.
- We evaluated our method on MM-WHS dataset, which includes 20 MR and 20 CT volumes. We employed MR as source domain and CT as target domain. Dice score and ASD are employed for evaluation. We employ U-Net as the segmentation model and ISTN as the hallucination model.
- With 25% source annotations, various UDA methods show unsatisfactory adaptation performance. It is observed that self-ensembling methods, such as MT can help relax the dependence on source labels by leveraging unlabeled data, while transformation methods such as ISTN can also improve the segmentation performance by generating augmented samples. Our method achieves better performance than others by a lot, showing the effectiveness of the proposed method for few-shot UDA.
- With fewer source labels (1-shot), our method shows larger improvements than other methods, demonstrating that meta-hallucinator is beneficial in label-scarce adaptation scenarios.
- We present the qualitative results of different methods trained on four source labels. It is observed that our method produces fewer false positives and segments cardiac substructures with smoother boundaries.
- we conduct an ablation analysis on key components of the proposed method. Consistent performance improvements are obtained with our method.
- Thanks for listening to my presentation.