[ICIP 2022] MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation
•Download as PPTX, PDF•
0 likes•74 views
Slides presented at IEEE International Conference on Image Processing (ICIP), 2022, Bordeaux, France. DOI: 10.1109/ICIP46576.2022.9897591
![Introduction – Cardiac Image Segmentation
- Cardiovascular disease is one of the most serious threats to human health globally, and automatic
cardiac substructure segmentation can help disease diagnosis and treatment planning.
- Deep learning has been widely used for medical image segmentation
Cardiac Substructure segmentation[1]
[1] Hu, Xinrong , et al. "Semi-supervised Contrastive Learning for Label-efficient Medical Image Segmentation." MICCAI, 2021.](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
Recommended
More Related Content
What's hot
What's hot (20)
Similar to [ICIP 2022] MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation
Similar to [ICIP 2022] MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation (20)
More from Ziyuan Zhao
More from Ziyuan Zhao (11)
Recently uploaded
Recently uploaded (20)
[ICIP 2022] MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation
- 1. MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation Ziyuan Zhao, Jinxuan Hu, Zeng Zeng, Xulei Yang, Peisheng Qian, Bharadwaj Veeravalli, Cuntai Guan I2R, A*STAR, Singapore Nanyang Technological University, Singapore National University of Singapore, Singapore ID: 1311
- 2. Introduction – Cardiac Image Segmentation - Cardiovascular disease is one of the most serious threats to human health globally, and automatic cardiac substructure segmentation can help disease diagnosis and treatment planning. - Deep learning has been widely used for medical image segmentation Cardiac Substructure segmentation[1] [1] Hu, Xinrong , et al. "Semi-supervised Contrastive Learning for Label-efficient Medical Image Segmentation." MICCAI, 2021.
- 3. Challenge – Annotation Scarcity - DCNNs are data-hungry and require large amounts of well-annotated data. - Annotation process is expensive, laborious, time-consuming, and requires expert knowledge, leading to label scarcity. - In this regard, many not-so-supervised methods like contrastive learning have been proposed to reduce annotation efforts.
- 4. Solution - Contrastive Learning - Intuition: Push original and augmented images (positive pairs) closer and push original and negative images (negative pairs) away by minimizing contrastive loss. - Limitation: Mostly used for image-level global representations. Basic intuition behind contrastive learning paradigm[2] [2] Jaiswal A , Babu A R , Zadeh M Z , et al. A Survey on Contrastive Self-supervised Learning[J]. 2020.
- 5. Related work - Semi-Supervised contrastive learning Pixel-level local representation: - use a decoder to extract local representations from the feature map - use labels to form more accurate contrastive pairs (positive pairs & negative pairs) [1] Hu, Xinrong , et al. "Semi-supervised Contrastive Learning for Label-efficient Medical Image Segmentation." MICCAI, 2021. Existing contrastive learning methods for cardiac image segmentation[1]
- 6. Motivation - To realize the 2D segmentation of cardiac substructures via contrastive learning methodology. - Existing work only applied contrastive learning on the last layer of encoder or decoder, ignoring the rich multi-scale information for robust representation learning. - Volumetric information is still under-explored for contrastive learning due to computational complexity.
- 7. Method - Overview - Model Architecture: 4 components - Multi-scale Multi-view Global-Local contrastive learning framework (MMGL)
- 8. Method (1) - Multi-view Co-training - Different three views from the transaxial plane(x-axis), sagittal plane(y-axis) and coronal plane(z-axis) have related spatial information (Volumetric information). - Main view: transaxial view - Auxiliary views: sagittal view and coronal view. The auxiliary views can add more information.
- 9. Method (2) - Multi-scale Global Unsupervised Contrastive Learning - To extract image-level representation: - Global unsupervised contrastive loss of the e-th layer: - Multi-scale global unsupervised contrastive loss:
- 10. Method (3) - Multi-scale Local Supervised Contrastive Learning - To extract pixel-level representation: - Local supervised contrastive loss for a feature map 𝑓𝑙 : - Multi-scale local supervised contrastive loss:
- 11. Method (4) - Multi-scale Deeply Supervised Learning - To utilize multi-scale information in fine-tune stage: - Deeply supervised loss:
- 12. Dataset - MM-WHS(Multi-Modality Whole Heart Segmentation) - The expert labels for MM-WHS consist of 7 cardiac substructures: left ventricle (LV), left atrium (LA), right ventricle (RV), right atrium (RA), myocardium (MYO), ascending aorta (AA), and pulmonary artery (PA). - 20 CT 3D volumes from multiple sites. Training set, validation set, and testing set : (2 : 1 : 1)
- 13. Experimental Results - Comparison with existing methods: - Random: randomly initialize the model without any pretraining method - Global: only apply the global unsupervised contrastive learning - Global+Local: apply unsupervised global and local contrastive learning - SemiContrast: apply global unsupervised learning and local supervised learning - MMGL has better performance than other methods and demonstrates the effectiveness of multi-scale features for contrastive learning with multi-view images.
- 14. Ablation Studies - Ablation experiments demonstrate the effectiveness of the four components since every component has increased the performance.
- 15. Conclusions - We propose a multi-scale multi-view global and local contrastive learning strategy to solve the problem of label scarcity in medical image segmentation tasks. - Add multi-view information in the pre-training stage as auxiliary information without additional annotations. - Employed a multi-scale method in the pre-training stage to explicitly leverage the information in the different hidden layers from both the encoder and decoder parts of the network. - Injected deep supervision in the fine-tuning stage.
Editor's Notes
- Welcome to my presentation. My name is Zhao Ziyuan. Here I am presenting Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation. MMGL for short.
- According to World Health Organization, heart diseases are the leading cause of death. Cardiac image segmentation is a crucial step in many clinical applications related to heart diseases. So, automatic segmentation of cardiac substructures is very meaningful, can help release the workload of doctors and facilitate diagnosis and treatment planning. Recently, deep convolutional neural networks have achieved markable progress in medical image segmentation.
- However, deep learning methods are usually data-hungry. In practice, when we conduct medical image segmentation tasks, the main challenge we met is the scarcity of annotation. Therefore, we need to use some not-so-supervised label-efficient methods to reduce annotation efforts and tackle this challenge.
- Contrastive learning is one of those promising and popular methods. The intuition behind contrastive learning is minimizing defined contrastive loss to pull similar representations closer while pushing the dissimilar ones further away. So following this intuition, when we design our model, we mainly focus on the design of the contrastive loss and focus on how to define the similar pairs and the dissimilar pairs. It is worth noting that contrastive learning as a variant of self-supervised learning is just a pre-training strategy. That means though we do not need labels in the pre-training stage, we still need some labels in the downstream task in the fine-tuning stage to complete the segmentation. Contrastive learning also has one limitation, it is usually used for extracting image-level global representations for classification tasks. While some works have been working on extracting pixel-level local representations. The most recent one is semi-supervised contrastive learning.
- For global unsupervised contrastive learning, they only used the encoder to extract image-level features in the pretraining stage without any labels. However, for semi-supervised contrastive learning, they added the decoder to extract pixel-level local features and they used a small portion of labeled data to decide the positive and negative contrastive pairs and thus extracted more accurate local features.
- However, the above-mentioned related work has some limitations. Firstly, all of them only applied contrastive learning on the last layer of the encoder or decoder, ignoring the rich multi-scale information that can be extracted from different layers of the network. Secondly, the volumetric information between these 2D slices is ignored due to computational complexity and can be further explored.
- So in our work, we propose a Multi-scale Multi-view Global-Local semi-supervised contrastive learning framework. We fully take advantage of multi-view co-training and multi-scale learning in both the pre-train and fine-tune stages of the segmentation workflow. For the pretraining stage, we first design a multi-scale global unsupervised contrastive learning module to extract global features from different views and encoder layers. Next, we develop a multi-scale local supervised contrastive learning scheme by adding the decoder layers on top of the encoder to extract multi-scale local features from different views. In the fine-tuning stage, we train the segmentation model with multi-scale deeply supervised learning with a few labels.
- There are three imaging planes of the heart. Transaxial plane, sagittal plane, and coronal plane. They have related spatial information. So, in our work, we set the transaxial plane as our main view which is the target view for segmentation and we set the other two views as the auxiliary views for more information. When we implement this module, we just simply add the auxiliary views’ images into the training set in the pre-training stage to form the multi-view images.
- For Multi-scale Global Unsupervised Contrastive Learning, we aim to extract image-level global representations from different layers of the network. The encoder part of the UNet is often used to extract global representations. To form contrastive pairs, we forward a batch of multi-view inputs through two augmentation pipelines to get an augmentation set. Then, to extract multi-scale global representations, we add projection heads after each block of the encoder, enhancing the representation ability of extracted features. Then, we calculate the global unsupervised loss at different layers. To get multi-scale global unsupervised contrastive loss, we simply sum up global contrastive losses from different levels. The λ is the balancing weight.
- Multi-scale global unsupervised contrastive learning only learns the image-level features. But, for the segmentation task, we need pixel-level information. So, to learn the dense pixel-level features, we introduce the local supervised contrastive loss. By adding label information, we can form positive and negative pairs more precisely. The local supervised contrastive loss is calculated as in the equation which is a variation of the global loss. Ω contains the total points in the feature map. P contains the positive points of the feature map that share the same annotation and N contains the negative points that have different annotations. Similar to the multi-scale global loss, to get multi-scale local supervised contrastive loss, we simply sum up local contrastive losses from different levels. The λ is the balancing weight.
- In the last fine-tune stage, we adopt the multi-scale deeply supervision strategy with a small portion of labelled data to utilize multi-scale information in different levels and we only trained with the main view images. The deeply supervised loss is formulated here.
- In this paper, the dataset we use is the public dataset provided in Multi-Modality Whole Heart Segmentation Challenge, MMWHS for short. The expert labels for MM-WHS consist of 7 cardiac substructures as listed here. For the experiment, we utilize the given 20 CT 3D volumes to train and evaluate our model The dataset is split into a training set, validation set, and testing set with a ratio of (2: 1: 1).
- Compared to the random baseline, all different contrastive learning methods can improve the performance against the label scarcity problem. The semi-Contrast method obtains better performance than others since it leverages label information for contrastive learning. We also observe that the proposed method has better performance than other methods with the same portion of labeled data, demonstrating the effectiveness of multi-scale features for contrastive learning with multi-view images. Our MMGL model achieves up to 84.9% in Dice score when training with 40% labeled data, which is the highest performance. We can also observe that all combinations of weights outperform other contrastive learning methods.
- To further analyze the effectiveness of different key components of MMGL, we also perform a comprehensive ablation analysis, we repeat 4 times to get the mean and standard deviation. We start by the random baseline and achieve 72.3% in Dice with 20% labeled data. By adding deeply supervised learning, the metric is improved to 73.8%. By adding multi-scale global learning, the metric is improved to 77.3%. When applying multi-view images, the metric is enhanced by 3.8%. Finally, MMGL increases the performance to 82.8% by adding the multi-scale local learning. In total, the proposed method improves the performance by 10.5%. These experiments demonstrate the effectiveness of the four components.
- In conclusion, we propose a multi-scale multi-view global and local contrastive learning strategy to solve the problem of label scarcity in medical image segmentation tasks. First, we added multi-view information in the pre-training stage as auxiliary information without additional annotations. Second, we employed a multi-scale method in the pre-training stage to explicitly leverage the information in the different hidden layers from both the encoder and decoder parts of the network. Thirdly, we injected deep supervision in the fine-tuning stage. In the ablation study, we prove that all our components can improve the performance of contrastive self-supervised medical image segmentation.
- Thanks for listening to my presentation.