SlideShare a Scribd company logo
Detection-aided medical image segmentation
using deep learning
8. September 2017 | Master Thesis
Míriam Bellver Bueno, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Xavier Giró-i-Nieto,
Jordi Torres, Luc Van Gool
Outline
§  Introduction
§  Related Work
§  Method
§  Experimental Validation
§  Conclusions
8. September 2017 2
Introduction
Detection-aided medical image segmentation using deep learning
8. September 2017 3
Introduction
§  Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§  Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
8. September 2017 4
Introduction
§  Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§  Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
§  Time-consuming
§  Subjective
8. September 2017 5
Introduction
§  Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§  Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
§  Time-consuming
§  Subjective
8. September 2017 6
Fully Automatic Tool ✔
Introduction
§  We will focus on liver and its lesions segmentation tasks for
afterwards segmenting several organs and other anatomical structures
8. September 2017 7
Lesion GT
Liver GT
Introduction
§  Lesion segmentation is a quite challenging task:
§  Low contrast between lesion, liver and other organs
§  Lesions are variable in terms of shape, size and texture
§  Noise in CT scans
§  Typically statistical shape and intensity distribution models
8. September 2017 8
Introduction
§  Lesion segmentation is a quite challenging task:
§  Low contrast between lesion, liver and other organs
§  Lesions are variable in terms of shape, size and texture
§  Noise in CT scans
§  Typically statistical shape and intensity distribution models
§  Deep Convolutional Neural Networks have demonstrated to be
successful on challenging tasks as lesion and liver segmentation
8. September 2017 9
Introduction
§  Our pipeline is based on the strengths of a segmentation network and
a detector:
8. September 2017 10
Segmentation
Network
Specializes on fine localization
of lesions
Detection
Network
Learns global features given a
whole liver patch
Introduction
§  Tasks:
§  Develop a method to segment the lesion and the liver from CT scans using
in the framework of the LiTS Challenge.
§  Prove generality of segmentation network with Visceral dataset.
8. September 2017 11
Related Work
Detection-aided medical image segmentation using deep learning
8. September 2017 12
Convolutional Neural Networks (CNNs)
8. September 2017 13
LeCun, Y. (2015). LeNet-5, convolutional neural networks
Segmentation
§  Fully Convolutional Networks (FCNs)
8. September 2017 14
Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation
Segmentation
§  Encoder-Decoder architecture
8. September 2017 15
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2015). Segnet: A deep convolutional encoder-decoder
architecture for image segmentation
Segmentation
§  U-net
8. September 2017 16
Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks
for biomedical image segmentation
Segmentation
§  Deep Retinal Image Understanding (DRIU)
8. September 2017 17
Maninis, K. K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016, October). Deep retinal image understanding
Input
Fine feature maps Coarse feature maps
Vessels Optic Disc
Specialized
Layers
Image
Base Network Architecture
Detection
8. September 2017 18
Segmentation Detection
Detection
8. September 2017 19
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016). Region-based convolutional networks for
accurate object detection and segmentation
Method
Detection-aided medical image segmentation using deep learning
8. September 2017 20
Architecture
Liver Segmentation
1
Lesion Segmentation
3
Bounding box sampling
2
8. September 2017 21
Architecture: Segmentation Network
Liver Segmentation
1
Lesion Segmentation
3
Bounding box sampling
2
8. September 2017 22
Architecture: Segmentation Network
§  Segmentation network based on Deep Retinal Image Understanding
(DRIU) network (*)
§  Fully Convolutional Network (FCN)
§  Base network is VGG-16 and pre-trained with Imagenet
§  Side outputs
§  Combination of multi-scale side outputs
8. September 2017 23
Liver Segmentation
1
Les
3
Bounding box sampling
2Maninis, K. K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016, October). Deep retinal image understanding. In International
Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 140-148). Springer International Publishing.
Architecture: Detection Network
Liver Segmentation
1
Lesion Segmentation
3
Bounding box sampling
2
8. September 2017 24
Bounding Box Sampling for Detection
§  Example of bounding box sampling and labeling
§  Data augmentation with flips and rotations x 8
8. September 2017 25
Detection Model
§  Pre-trained Resnet-50 without classification
layer for Imagenet
§  A single neuron determining whether it is a
healthy tissue
8. September 2017 26
Experimental Validation
Detection-aided medical image segmentation using deep learning
8. September 2017 27
Experimental Validation: LiTS dataset
•  CT scans with the liver and lesion mask
•  Dimensions
§  Training: 131 CT scans
§  80% : 105 training volumes
§  20 % : 26 validation volumes
§  Testing: 70 CT scans
8. September 2017 28
Experimental Validation: Metrics
§  The metric to assess performance will be the Dice score (F-score)
8. September 2017 29
Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§  1. Pre-processing
§  2. Weighting of Binary Cross Entropy (BCE) loss
§  3. Stacking 3 consecutive slices at the input of the network
§  4. Using liver segmentation to segment lesion
8. September 2017 30
Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§  1. Pre-processing
§  2. Weighting of Binary Cross Entropy (BCE) loss
§  3. Stacking 3 consecutive slices at the input of the network
§  4. Using liver segmentation to segment lesion
8. September 2017 31
1. Pre-processing
§  We clipped pixel intensity values by maximum and minimum values
that statistically belong to the liver and lesion class
8. September 2017 32
0.30
Lesion
Dice Score
1. Pre-processing
§  We clipped pixel intensity values by maximum and minimum values
that statistically belong to the liver and lesion class
8. September 2017 33
0.30 0.32
Lesion
Dice Score
Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§  1. Pre-processing
§  2. Weighting of Binary Cross Entropy (BCE) loss
§  3. Stacking 3 consecutive slices at the input of the network
§  4. Using liver segmentation to segment lesion
8. September 2017 34
2. Weighting of Binary Cross Entropy (BCE) loss
Loss objective
§  Binary Cross Entropy (BCE) Loss
§  Weighted Binary Cross Entropy (BCE) Loss
8. September 2017 35
Balancing term
2. Weighting of Binary Cross Entropy (BCE) loss
Balancing Schemes
§  Per-volume balancing
§  General balancing
8. September 2017 36
2. Weighting of Binary Cross Entropy (BCE) loss
Balancing Schemes
§  Per-volume balancing
§  General balancing
8. September 2017 37
2. Weighting of Binary Cross Entropy (BCE) loss
8. September 2017 38
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Per-volume balancing
General balancing
Lesion Precision – Recall Curve
0.30 0.32
Lesion
Dice Score
2. Weighting of Binary Cross Entropy (BCE) loss
8. September 2017 39
Lesion Precision – Recall Curve
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Per-volume balancing
General balancing
0.30 0.32 0.34
Lesion
Dice Score
Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§  1. Pre-processing
§  2. Weighting of Binary Cross Entropy (BCE) loss
§  3. Stacking 3 consecutive slices at the input of the network
§  4. Using liver segmentation to segment lesion
8. September 2017 40
3. Stacking 3 consecutive slices at the input of the network
§  Volumes of data that are highly redundant
§  We input a stack of slices in the network with supervision
§  Final configuration inputs 3 slices, one at each RGB channel
8. September 2017 41
Network
3. Stacking 3 consecutive slices at the input of the network
8. September 2017 42
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
1-slice
3-slices
6-slices
9-slices
Lesion Precision – Recall Curve
Lesion Results
Liver Results
0.30 0.32 0.34
Lesion
Dice Score
3. Stacking 3 consecutive slices at the input of the network
8. September 2017 43
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
1-slice
3-slices
6-slices
9-slices
Lesion Results
Liver Results
Lesion Precision – Recall Curve
Lesion
Dice Score 0.30 0.32 0.34 0.36
Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§  1. Pre-processing
§  2. Weighting of Binary Cross Entropy (BCE) loss
§  3. Stacking 3 consecutive slices at the input of the network
§  4. Using liver segmentation to segment lesion
8. September 2017 44
4. Using liver segmentation to segment lesion
§  We worked with 2 different strategies:
1.  Back-propagation (BP) through liver: Only back-propagation through liver
2.  Multitask: Segmenting the liver and the lesion simultaneously
8. September 2017 45
4. Using liver segmentation to segment lesion
§  Features of Back-propagation (BP) through liver strategy
§  The predicted liver mask is fixed
§  The liver mask multiplies to the lesion mask during training and testing
§  The weighting term in the BCE loss now just considers liver pixels
8. September 2017 46
Network 1 ∘Network 2
4. Using liver segmentation to segment lesion
§  Pros of BP through liver ✔
§  Network learns from relevant pixels
§  Positive/Negative pixels are more balanced
§  Cons of BP through liver ✖
§  If there is a mistake in the liver mask, this mistake will affect the lesion
segmentation
8. September 2017 47
4. Using liver segmentation to segment lesion
§  Features of Multi-task liver strategy
§  The output of the network is a channel for the liver and another for the
lesion.
§  The loss is the sum of the two BCE losses . Both classes can happen at the
same time. No competition among classes.
8. September 2017 48
Network 1 ∘Network 2
Network 1 Network 2
4. Using liver segmentation to segment lesion
§  Pros of Multi-task ✔
§  The gradients of the network have information of both the liver and lesion
§  Efficient and Scalable
§  Cons of Multi-task ✖
§  Difficult to weight each task properly
8. September 2017 49
4. Using liver segmentation to segment lesion
8. September 2017 50
Difference in masking during testing or testing + training
Performance of various strategies
Lesion
Dice Score 0.30 0.32 0.34 0.36
4. Using liver segmentation to segment lesion
8. September 2017 51
Difference in masking during testing or testing + training
Performance of various strategies
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39
Motivation for Detection
8. September 2017 52
Lesion GT
Lesion Pred.
Liver GT
Liver Pred.
Motivation for Detection
8. September 2017 53
Lesion GT
Lesion Pred.
Liver GT
Liver Pred.
False Positives
Detection
8. September 2017 54
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Precision-Recall for detection of lesion
Detection
§  Examples of detected positive windows
8. September 2017 55
Lesion GT
Lesion Pred.
Liver GT
Liver Pred.
Detection
§  Examples of detected positive windows
8. September 2017 56
Before Detection After Detection
Detection
8. September 2017 57
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multitask
BP through liver
Multitask + Det
BP through liver + Det
Lesion Precision – Recall Curve
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39
Detection
8. September 2017 58
Lesion Precision – Recall Curve
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multitask
BP through liver
Multitask + Det
BP through liver + Det
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41
Post- processing: 3D – Conditional Random Fields
§  It will add spatial coherence in the 3 dimensions, based on space and
appearance
8. September 2017 59
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41
Post- processing: 3D – Conditional Random Fields
§  It will add spatial coherence in the 3 dimensions, based on space and
appearance
8. September 2017 60
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41 0.43
Post- processing: 3D – Conditional Random Fields
8. September 2017 61
Summary of Lesion Segmentation
8. September 2017 62
Lesion
Dice Score
(validation set)
26 CT scans 62
0.30 0.32 0.34 0.36 0.39 0.41 0.43
62
Summary of Lesion Segmentation
8. September 2017 63
Lesion
Dice Score
(validation set)
26 CT scans 63
0.30 0.32 0.34 0.36 0.39 0.41 0.43
63
~ 43 % improvement
Summary of Lesion Segmentation
8. September 2017 64
Lesion
Dice Score
(validation set)
26 CT scans 64
0.30 0.32 0.34 0.36 0.39 0.41 0.43
64
0.41 0.54 0.57 0.59
~ 43 % improvement
Lesion
Dice Score
(Challenge
test set)
70 CT scans
Summary of Lesion Segmentation
8. September 2017 65
Lesion
Dice Score
(validation set)
26 CT scans 65
0.30 0.32 0.34 0.36 0.39 0.41 0.43
65
0.41 0.54 0.57 0.59
~ 43 % improvement
~ 44 % improvement
Lesion
Dice Score
(Challenge
test set)
70 CT scans
Summary of Lesion Segmentation
8. September 2017 66
Lesion
Dice Score
(validation set)
26 CT scans 66
0.30 0.32 0.34 0.36 0.39 0.41 0.43
66
0.41 0.54 0.57 0.59
~ 43 % improvement
~ 44 % improvement
0.68
Top
Entry
Lesion
Dice Score
(Challenge
test set)
70 CT scans
…
Experimental Validation: Visceral
•  CT scans with segmentation for 20 organs and anatomical structures
•  Dimensions
§  Training: 20 CT scans
§  90% : 18 training volumes
§  10 % : 2 validation volumes
8. September 2017 67
Visceral Experiments
8. September 2017 68
Conclusions
Detection-aided medical image segmentation using deep learning
8. September 2017 69
Conclusions
§  We improved the baseline we worked on developing a set of strategies
applicable to other pipelines.
§  Detection + Segmentation improved over only Segmentation.
§  Using the liver for the lesion segmentation boosted the performance.
Learning from relevant samples seems a good practice.
§  Segmentation network has a lot of potential to segment different kinds of
structures in a single forward pass.
8. September 2017 70
Publication
8. September 2017 71
Acknowledgements
8. September 2017 72
Thank you for your attention!
8. September 2017 73
31. July 2017 74
Publication
8. July 2017 75
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multi-task
BP through liver
Multi-task + BP through liver
3. Using liver segmentation to segment lesion
1. August 2012 76
0.32 0.34 0.36
Lesion
Dice Score
Detection
1. August 2012 77
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multitask
BP through liver
Multitask + Det
BP through liver + Det
0.32 0.34 0.36 0.38 0.41
Lesion
Dice Score
Post- processing: 3D – Conditional Random Fields
1. August 2012 78
0.32 0.34 0.36 0.39 0.41
Lesion
Dice Score
Post- processing: 3D – Conditional Random Fields
1. August 2012 79
0.32 0.34 0.36 0.39 0.41 0.43
Lesion
Dice Score
LiTS Challenge
1. August 2012 80
Visceral Experiments
1. August 2012 81
Visualizations – Seg. + Det. (2/2)
1. August 2012 82
Visualizations – Seg. + Det. (1/2)
1. August 2012 83
Visualizations – Seg. + Det. (2/2)
1. August 2012 84
Bounding Box Sampling for Detection
§  Single-scale 2D windows of 50x50 with stride 50
§  A window is placed if it overlaps > 25% with the liver
§  A windows is considered as positive if inside it there are >50 lesion
pixels
§  A margin of 15 pixels is added to all windows to have more context
31. July 2017 85
50
50 80
80
Image Classification
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Precision-Recall for classification of lesion
Image Classification with VGG-16
Image Classification with ResNet-50
31. July 2017 86
•  As proof of concept we first
trained a classifier that
distinguished between healthy
image / unhealthy image
•  Resnet – 50 layers really
improved compared to VGG-16
architecture

More Related Content

More from Universitat Politècnica de Catalunya

Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoTowards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Universitat Politècnica de Catalunya
 
The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021
Universitat Politècnica de Catalunya
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Universitat Politècnica de Catalunya
 
Open challenges in sign language translation and production
Open challenges in sign language translation and productionOpen challenges in sign language translation and production
Open challenges in sign language translation and production
Universitat Politècnica de Catalunya
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosGeneration of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Universitat Politècnica de Catalunya
 
Discovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in MinecraftDiscovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in Minecraft
Universitat Politècnica de Catalunya
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...
Universitat Politècnica de Catalunya
 
Intepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural NetworksIntepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural Networks
Universitat Politècnica de Catalunya
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Universitat Politècnica de Catalunya
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Universitat Politècnica de Catalunya
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Universitat Politècnica de Catalunya
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Universitat Politècnica de Catalunya
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Universitat Politècnica de Catalunya
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Universitat Politècnica de Catalunya
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Universitat Politècnica de Catalunya
 
Curriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object SegmentationCurriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object Segmentation
Universitat Politècnica de Catalunya
 
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Universitat Politècnica de Catalunya
 
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Universitat Politècnica de Catalunya
 
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
Universitat Politècnica de Catalunya
 
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
Universitat Politècnica de Catalunya
 

More from Universitat Politècnica de Catalunya (20)

Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-NietoTowards Sign Language Translation & Production | Xavier Giro-i-Nieto
Towards Sign Language Translation & Production | Xavier Giro-i-Nieto
 
The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021The Transformer - Xavier Giró - UPC Barcelona 2021
The Transformer - Xavier Giró - UPC Barcelona 2021
 
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
Learning Representations for Sign Language Videos - Xavier Giro - NIST TRECVI...
 
Open challenges in sign language translation and production
Open challenges in sign language translation and productionOpen challenges in sign language translation and production
Open challenges in sign language translation and production
 
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in VideosGeneration of Synthetic Referring Expressions for Object Segmentation in Videos
Generation of Synthetic Referring Expressions for Object Segmentation in Videos
 
Discovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in MinecraftDiscovery and Learning of Navigation Goals from Pixels in Minecraft
Discovery and Learning of Navigation Goals from Pixels in Minecraft
 
Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...Learn2Sign : Sign language recognition and translation using human keypoint e...
Learn2Sign : Sign language recognition and translation using human keypoint e...
 
Intepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural NetworksIntepretability / Explainable AI for Deep Neural Networks
Intepretability / Explainable AI for Deep Neural Networks
 
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
Convolutional Neural Networks - Xavier Giro - UPC TelecomBCN Barcelona 2020
 
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
Self-Supervised Audio-Visual Learning - Xavier Giro - UPC TelecomBCN Barcelon...
 
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
Attention for Deep Learning - Xavier Giro - UPC TelecomBCN Barcelona 2020
 
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
Generative Adversarial Networks GAN - Xavier Giro - UPC TelecomBCN Barcelona ...
 
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
Q-Learning with a Neural Network - Xavier Giró - UPC Barcelona 2020
 
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
Language and Vision with Deep Learning - Xavier Giró - ACM ICMR 2020 (Tutorial)
 
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
Image Segmentation with Deep Learning - Xavier Giro & Carles Ventura - ISSonD...
 
Curriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object SegmentationCurriculum Learning for Recurrent Video Object Segmentation
Curriculum Learning for Recurrent Video Object Segmentation
 
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
Deep Self-supervised Learning for All - Xavier Giro - X-Europe 2020
 
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
Deep Learning Representations for All - Xavier Giro-i-Nieto - IRI Barcelona 2020
 
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
Transcription-Enriched Joint Embeddings for Spoken Descriptions of Images and...
 
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
Object Detection with Deep Learning - Xavier Giro-i-Nieto - UPC School Barcel...
 

Recently uploaded

Research proposal seminar ,Research Methodology
Research proposal seminar ,Research MethodologyResearch proposal seminar ,Research Methodology
Research proposal seminar ,Research Methodology
doctorzlife786
 
Amul goes international: Desi dairy giant to launch fresh ...
Amul goes international: Desi dairy giant to launch fresh ...Amul goes international: Desi dairy giant to launch fresh ...
Amul goes international: Desi dairy giant to launch fresh ...
chetankumar9855
 
M44.pdf dairy management farm report of an
M44.pdf dairy management farm report of anM44.pdf dairy management farm report of an
M44.pdf dairy management farm report of an
ManjuBv2
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
saadkhan1485265
 
Welcome back to Instagram. Sign in to check out what your
Welcome back to Instagram. Sign in to check out what yourWelcome back to Instagram. Sign in to check out what your
Welcome back to Instagram. Sign in to check out what your
Virni Arrora
 
Nipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma TranscriptNipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma Transcript
zyqedad
 
CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
girewiy968
 
Potential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriatePotential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriate
huseindihon
 
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).docbai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
PhngThLmHnh
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
uapta
 
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
vrvipin164
 
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
44annissa
 
the potential of the development of the Ford–Fulkerson algorithm to solve the...
the potential of the development of the Ford–Fulkerson algorithm to solve the...the potential of the development of the Ford–Fulkerson algorithm to solve the...
the potential of the development of the Ford–Fulkerson algorithm to solve the...
huseindihon
 
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptxArtificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
vaishnavisharma877623
 
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
janvikumar4133
 
Universidad Camilo José Cela degree offer diploma Transcript
Universidad Camilo José Cela  degree offer diploma TranscriptUniversidad Camilo José Cela  degree offer diploma Transcript
Universidad Camilo José Cela degree offer diploma Transcript
taqyea
 
Data analytics and Access Program Recommendations
Data analytics and Access Program RecommendationsData analytics and Access Program Recommendations
Data analytics and Access Program Recommendations
hemantsharmaus
 
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
kuldeepsharmaks8120
 
🚂🚘 Premium Girls Call Bangalore 🛵🚡000XX00000 💃 Choose Best And Top Girl Serv...
🚂🚘 Premium Girls Call Bangalore  🛵🚡000XX00000 💃 Choose Best And Top Girl Serv...🚂🚘 Premium Girls Call Bangalore  🛵🚡000XX00000 💃 Choose Best And Top Girl Serv...
🚂🚘 Premium Girls Call Bangalore 🛵🚡000XX00000 💃 Choose Best And Top Girl Serv...
bhupeshkumar0889
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
huseindihon
 

Recently uploaded (20)

Research proposal seminar ,Research Methodology
Research proposal seminar ,Research MethodologyResearch proposal seminar ,Research Methodology
Research proposal seminar ,Research Methodology
 
Amul goes international: Desi dairy giant to launch fresh ...
Amul goes international: Desi dairy giant to launch fresh ...Amul goes international: Desi dairy giant to launch fresh ...
Amul goes international: Desi dairy giant to launch fresh ...
 
M44.pdf dairy management farm report of an
M44.pdf dairy management farm report of anM44.pdf dairy management farm report of an
M44.pdf dairy management farm report of an
 
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
High Girls Call Nagpur 000XX00000 Provide Best And Top Girl Service And No1 i...
 
Welcome back to Instagram. Sign in to check out what your
Welcome back to Instagram. Sign in to check out what yourWelcome back to Instagram. Sign in to check out what your
Welcome back to Instagram. Sign in to check out what your
 
Nipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma TranscriptNipissing University degree offer Nipissing diploma Transcript
Nipissing University degree offer Nipissing diploma Transcript
 
CHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptxCHAPTER-1-Introduction-to-Marketing.pptx
CHAPTER-1-Introduction-to-Marketing.pptx
 
Potential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriatePotential Uses of the Floyd-Warshall Algorithm as appropriate
Potential Uses of the Floyd-Warshall Algorithm as appropriate
 
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).docbai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
bai-tap-tieng-anh-lop-12-unit-4-the-mass-media (1).doc
 
DU degree offer diploma Transcript
DU degree offer diploma TranscriptDU degree offer diploma Transcript
DU degree offer diploma Transcript
 
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
Coimbatore Girls call Service 000XX00000 Provide Best And Top Girl Service An...
 
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
VIP Girls Call Mumbai 9910780858 Provide Best And Top Girl Service And No1 in...
 
the potential of the development of the Ford–Fulkerson algorithm to solve the...
the potential of the development of the Ford–Fulkerson algorithm to solve the...the potential of the development of the Ford–Fulkerson algorithm to solve the...
the potential of the development of the Ford–Fulkerson algorithm to solve the...
 
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptxArtificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
Artificial Intelligence (AI) Technology Project Proposal _ by Slidesgo.pptx
 
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
Beautiful Girls Call 9711199171 9711199171 Provide Best And Top Girl Service ...
 
Universidad Camilo José Cela degree offer diploma Transcript
Universidad Camilo José Cela  degree offer diploma TranscriptUniversidad Camilo José Cela  degree offer diploma Transcript
Universidad Camilo José Cela degree offer diploma Transcript
 
Data analytics and Access Program Recommendations
Data analytics and Access Program RecommendationsData analytics and Access Program Recommendations
Data analytics and Access Program Recommendations
 
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...🚂🚘 Premium Girls Call Nashik  🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
🚂🚘 Premium Girls Call Nashik 🛵🚡000XX00000 💃 Choose Best And Top Girl Service...
 
🚂🚘 Premium Girls Call Bangalore 🛵🚡000XX00000 💃 Choose Best And Top Girl Serv...
🚂🚘 Premium Girls Call Bangalore  🛵🚡000XX00000 💃 Choose Best And Top Girl Serv...🚂🚘 Premium Girls Call Bangalore  🛵🚡000XX00000 💃 Choose Best And Top Girl Serv...
🚂🚘 Premium Girls Call Bangalore 🛵🚡000XX00000 💃 Choose Best And Top Girl Serv...
 
potential development of the A* search algorithm specifically
potential development of the A* search algorithm specificallypotential development of the A* search algorithm specifically
potential development of the A* search algorithm specifically
 

Detection-aided liver lesion segmentation using deep learning

  • 1. Detection-aided medical image segmentation using deep learning 8. September 2017 | Master Thesis Míriam Bellver Bueno, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Xavier Giró-i-Nieto, Jordi Torres, Luc Van Gool
  • 2. Outline §  Introduction §  Related Work §  Method §  Experimental Validation §  Conclusions 8. September 2017 2
  • 3. Introduction Detection-aided medical image segmentation using deep learning 8. September 2017 3
  • 4. Introduction §  Contract Tomography (CT) images are an important tool in medical imaging to diagnose several diseases or assess treatments §  Doctors typically rely on manual or semi-automatic techniques to study anomalies in the shape and texture of organs of CT scans 8. September 2017 4
  • 5. Introduction §  Contract Tomography (CT) images are an important tool in medical imaging to diagnose several diseases or assess treatments §  Doctors typically rely on manual or semi-automatic techniques to study anomalies in the shape and texture of organs of CT scans §  Time-consuming §  Subjective 8. September 2017 5
  • 6. Introduction §  Contract Tomography (CT) images are an important tool in medical imaging to diagnose several diseases or assess treatments §  Doctors typically rely on manual or semi-automatic techniques to study anomalies in the shape and texture of organs of CT scans §  Time-consuming §  Subjective 8. September 2017 6 Fully Automatic Tool ✔
  • 7. Introduction §  We will focus on liver and its lesions segmentation tasks for afterwards segmenting several organs and other anatomical structures 8. September 2017 7 Lesion GT Liver GT
  • 8. Introduction §  Lesion segmentation is a quite challenging task: §  Low contrast between lesion, liver and other organs §  Lesions are variable in terms of shape, size and texture §  Noise in CT scans §  Typically statistical shape and intensity distribution models 8. September 2017 8
  • 9. Introduction §  Lesion segmentation is a quite challenging task: §  Low contrast between lesion, liver and other organs §  Lesions are variable in terms of shape, size and texture §  Noise in CT scans §  Typically statistical shape and intensity distribution models §  Deep Convolutional Neural Networks have demonstrated to be successful on challenging tasks as lesion and liver segmentation 8. September 2017 9
  • 10. Introduction §  Our pipeline is based on the strengths of a segmentation network and a detector: 8. September 2017 10 Segmentation Network Specializes on fine localization of lesions Detection Network Learns global features given a whole liver patch
  • 11. Introduction §  Tasks: §  Develop a method to segment the lesion and the liver from CT scans using in the framework of the LiTS Challenge. §  Prove generality of segmentation network with Visceral dataset. 8. September 2017 11
  • 12. Related Work Detection-aided medical image segmentation using deep learning 8. September 2017 12
  • 13. Convolutional Neural Networks (CNNs) 8. September 2017 13 LeCun, Y. (2015). LeNet-5, convolutional neural networks
  • 14. Segmentation §  Fully Convolutional Networks (FCNs) 8. September 2017 14 Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation
  • 15. Segmentation §  Encoder-Decoder architecture 8. September 2017 15 Badrinarayanan, V., Kendall, A., & Cipolla, R. (2015). Segnet: A deep convolutional encoder-decoder architecture for image segmentation
  • 16. Segmentation §  U-net 8. September 2017 16 Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks for biomedical image segmentation
  • 17. Segmentation §  Deep Retinal Image Understanding (DRIU) 8. September 2017 17 Maninis, K. K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016, October). Deep retinal image understanding Input Fine feature maps Coarse feature maps Vessels Optic Disc Specialized Layers Image Base Network Architecture
  • 18. Detection 8. September 2017 18 Segmentation Detection
  • 19. Detection 8. September 2017 19 Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016). Region-based convolutional networks for accurate object detection and segmentation
  • 20. Method Detection-aided medical image segmentation using deep learning 8. September 2017 20
  • 22. Architecture: Segmentation Network Liver Segmentation 1 Lesion Segmentation 3 Bounding box sampling 2 8. September 2017 22
  • 23. Architecture: Segmentation Network §  Segmentation network based on Deep Retinal Image Understanding (DRIU) network (*) §  Fully Convolutional Network (FCN) §  Base network is VGG-16 and pre-trained with Imagenet §  Side outputs §  Combination of multi-scale side outputs 8. September 2017 23 Liver Segmentation 1 Les 3 Bounding box sampling 2Maninis, K. K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016, October). Deep retinal image understanding. In International Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 140-148). Springer International Publishing.
  • 24. Architecture: Detection Network Liver Segmentation 1 Lesion Segmentation 3 Bounding box sampling 2 8. September 2017 24
  • 25. Bounding Box Sampling for Detection §  Example of bounding box sampling and labeling §  Data augmentation with flips and rotations x 8 8. September 2017 25
  • 26. Detection Model §  Pre-trained Resnet-50 without classification layer for Imagenet §  A single neuron determining whether it is a healthy tissue 8. September 2017 26
  • 27. Experimental Validation Detection-aided medical image segmentation using deep learning 8. September 2017 27
  • 28. Experimental Validation: LiTS dataset •  CT scans with the liver and lesion mask •  Dimensions §  Training: 131 CT scans §  80% : 105 training volumes §  20 % : 26 validation volumes §  Testing: 70 CT scans 8. September 2017 28
  • 29. Experimental Validation: Metrics §  The metric to assess performance will be the Dice score (F-score) 8. September 2017 29
  • 30. Experimental Validation: Ablation study for segmentation network Key features of our segmentation network §  1. Pre-processing §  2. Weighting of Binary Cross Entropy (BCE) loss §  3. Stacking 3 consecutive slices at the input of the network §  4. Using liver segmentation to segment lesion 8. September 2017 30
  • 31. Experimental Validation: Ablation study for segmentation network Key features of our segmentation network §  1. Pre-processing §  2. Weighting of Binary Cross Entropy (BCE) loss §  3. Stacking 3 consecutive slices at the input of the network §  4. Using liver segmentation to segment lesion 8. September 2017 31
  • 32. 1. Pre-processing §  We clipped pixel intensity values by maximum and minimum values that statistically belong to the liver and lesion class 8. September 2017 32 0.30 Lesion Dice Score
  • 33. 1. Pre-processing §  We clipped pixel intensity values by maximum and minimum values that statistically belong to the liver and lesion class 8. September 2017 33 0.30 0.32 Lesion Dice Score
  • 34. Experimental Validation: Ablation study for segmentation network Key features of our segmentation network §  1. Pre-processing §  2. Weighting of Binary Cross Entropy (BCE) loss §  3. Stacking 3 consecutive slices at the input of the network §  4. Using liver segmentation to segment lesion 8. September 2017 34
  • 35. 2. Weighting of Binary Cross Entropy (BCE) loss Loss objective §  Binary Cross Entropy (BCE) Loss §  Weighted Binary Cross Entropy (BCE) Loss 8. September 2017 35 Balancing term
  • 36. 2. Weighting of Binary Cross Entropy (BCE) loss Balancing Schemes §  Per-volume balancing §  General balancing 8. September 2017 36
  • 37. 2. Weighting of Binary Cross Entropy (BCE) loss Balancing Schemes §  Per-volume balancing §  General balancing 8. September 2017 37
  • 38. 2. Weighting of Binary Cross Entropy (BCE) loss 8. September 2017 38 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Per-volume balancing General balancing Lesion Precision – Recall Curve 0.30 0.32 Lesion Dice Score
  • 39. 2. Weighting of Binary Cross Entropy (BCE) loss 8. September 2017 39 Lesion Precision – Recall Curve 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Per-volume balancing General balancing 0.30 0.32 0.34 Lesion Dice Score
  • 40. Experimental Validation: Ablation study for segmentation network Key features of our segmentation network §  1. Pre-processing §  2. Weighting of Binary Cross Entropy (BCE) loss §  3. Stacking 3 consecutive slices at the input of the network §  4. Using liver segmentation to segment lesion 8. September 2017 40
  • 41. 3. Stacking 3 consecutive slices at the input of the network §  Volumes of data that are highly redundant §  We input a stack of slices in the network with supervision §  Final configuration inputs 3 slices, one at each RGB channel 8. September 2017 41 Network
  • 42. 3. Stacking 3 consecutive slices at the input of the network 8. September 2017 42 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision 1-slice 3-slices 6-slices 9-slices Lesion Precision – Recall Curve Lesion Results Liver Results 0.30 0.32 0.34 Lesion Dice Score
  • 43. 3. Stacking 3 consecutive slices at the input of the network 8. September 2017 43 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision 1-slice 3-slices 6-slices 9-slices Lesion Results Liver Results Lesion Precision – Recall Curve Lesion Dice Score 0.30 0.32 0.34 0.36
  • 44. Experimental Validation: Ablation study for segmentation network Key features of our segmentation network §  1. Pre-processing §  2. Weighting of Binary Cross Entropy (BCE) loss §  3. Stacking 3 consecutive slices at the input of the network §  4. Using liver segmentation to segment lesion 8. September 2017 44
  • 45. 4. Using liver segmentation to segment lesion §  We worked with 2 different strategies: 1.  Back-propagation (BP) through liver: Only back-propagation through liver 2.  Multitask: Segmenting the liver and the lesion simultaneously 8. September 2017 45
  • 46. 4. Using liver segmentation to segment lesion §  Features of Back-propagation (BP) through liver strategy §  The predicted liver mask is fixed §  The liver mask multiplies to the lesion mask during training and testing §  The weighting term in the BCE loss now just considers liver pixels 8. September 2017 46 Network 1 ∘Network 2
  • 47. 4. Using liver segmentation to segment lesion §  Pros of BP through liver ✔ §  Network learns from relevant pixels §  Positive/Negative pixels are more balanced §  Cons of BP through liver ✖ §  If there is a mistake in the liver mask, this mistake will affect the lesion segmentation 8. September 2017 47
  • 48. 4. Using liver segmentation to segment lesion §  Features of Multi-task liver strategy §  The output of the network is a channel for the liver and another for the lesion. §  The loss is the sum of the two BCE losses . Both classes can happen at the same time. No competition among classes. 8. September 2017 48 Network 1 ∘Network 2 Network 1 Network 2
  • 49. 4. Using liver segmentation to segment lesion §  Pros of Multi-task ✔ §  The gradients of the network have information of both the liver and lesion §  Efficient and Scalable §  Cons of Multi-task ✖ §  Difficult to weight each task properly 8. September 2017 49
  • 50. 4. Using liver segmentation to segment lesion 8. September 2017 50 Difference in masking during testing or testing + training Performance of various strategies Lesion Dice Score 0.30 0.32 0.34 0.36
  • 51. 4. Using liver segmentation to segment lesion 8. September 2017 51 Difference in masking during testing or testing + training Performance of various strategies Lesion Dice Score 0.30 0.32 0.34 0.36 0.39
  • 52. Motivation for Detection 8. September 2017 52 Lesion GT Lesion Pred. Liver GT Liver Pred.
  • 53. Motivation for Detection 8. September 2017 53 Lesion GT Lesion Pred. Liver GT Liver Pred. False Positives
  • 54. Detection 8. September 2017 54 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Precision-Recall for detection of lesion
  • 55. Detection §  Examples of detected positive windows 8. September 2017 55 Lesion GT Lesion Pred. Liver GT Liver Pred.
  • 56. Detection §  Examples of detected positive windows 8. September 2017 56 Before Detection After Detection
  • 57. Detection 8. September 2017 57 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Multitask BP through liver Multitask + Det BP through liver + Det Lesion Precision – Recall Curve Lesion Dice Score 0.30 0.32 0.34 0.36 0.39
  • 58. Detection 8. September 2017 58 Lesion Precision – Recall Curve 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Multitask BP through liver Multitask + Det BP through liver + Det Lesion Dice Score 0.30 0.32 0.34 0.36 0.39 0.41
  • 59. Post- processing: 3D – Conditional Random Fields §  It will add spatial coherence in the 3 dimensions, based on space and appearance 8. September 2017 59 Lesion Dice Score 0.30 0.32 0.34 0.36 0.39 0.41
  • 60. Post- processing: 3D – Conditional Random Fields §  It will add spatial coherence in the 3 dimensions, based on space and appearance 8. September 2017 60 Lesion Dice Score 0.30 0.32 0.34 0.36 0.39 0.41 0.43
  • 61. Post- processing: 3D – Conditional Random Fields 8. September 2017 61
  • 62. Summary of Lesion Segmentation 8. September 2017 62 Lesion Dice Score (validation set) 26 CT scans 62 0.30 0.32 0.34 0.36 0.39 0.41 0.43 62
  • 63. Summary of Lesion Segmentation 8. September 2017 63 Lesion Dice Score (validation set) 26 CT scans 63 0.30 0.32 0.34 0.36 0.39 0.41 0.43 63 ~ 43 % improvement
  • 64. Summary of Lesion Segmentation 8. September 2017 64 Lesion Dice Score (validation set) 26 CT scans 64 0.30 0.32 0.34 0.36 0.39 0.41 0.43 64 0.41 0.54 0.57 0.59 ~ 43 % improvement Lesion Dice Score (Challenge test set) 70 CT scans
  • 65. Summary of Lesion Segmentation 8. September 2017 65 Lesion Dice Score (validation set) 26 CT scans 65 0.30 0.32 0.34 0.36 0.39 0.41 0.43 65 0.41 0.54 0.57 0.59 ~ 43 % improvement ~ 44 % improvement Lesion Dice Score (Challenge test set) 70 CT scans
  • 66. Summary of Lesion Segmentation 8. September 2017 66 Lesion Dice Score (validation set) 26 CT scans 66 0.30 0.32 0.34 0.36 0.39 0.41 0.43 66 0.41 0.54 0.57 0.59 ~ 43 % improvement ~ 44 % improvement 0.68 Top Entry Lesion Dice Score (Challenge test set) 70 CT scans …
  • 67. Experimental Validation: Visceral •  CT scans with segmentation for 20 organs and anatomical structures •  Dimensions §  Training: 20 CT scans §  90% : 18 training volumes §  10 % : 2 validation volumes 8. September 2017 67
  • 69. Conclusions Detection-aided medical image segmentation using deep learning 8. September 2017 69
  • 70. Conclusions §  We improved the baseline we worked on developing a set of strategies applicable to other pipelines. §  Detection + Segmentation improved over only Segmentation. §  Using the liver for the lesion segmentation boosted the performance. Learning from relevant samples seems a good practice. §  Segmentation network has a lot of potential to segment different kinds of structures in a single forward pass. 8. September 2017 70
  • 73. Thank you for your attention! 8. September 2017 73
  • 76. 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Multi-task BP through liver Multi-task + BP through liver 3. Using liver segmentation to segment lesion 1. August 2012 76 0.32 0.34 0.36 Lesion Dice Score
  • 77. Detection 1. August 2012 77 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Multitask BP through liver Multitask + Det BP through liver + Det 0.32 0.34 0.36 0.38 0.41 Lesion Dice Score
  • 78. Post- processing: 3D – Conditional Random Fields 1. August 2012 78 0.32 0.34 0.36 0.39 0.41 Lesion Dice Score
  • 79. Post- processing: 3D – Conditional Random Fields 1. August 2012 79 0.32 0.34 0.36 0.39 0.41 0.43 Lesion Dice Score
  • 82. Visualizations – Seg. + Det. (2/2) 1. August 2012 82
  • 83. Visualizations – Seg. + Det. (1/2) 1. August 2012 83
  • 84. Visualizations – Seg. + Det. (2/2) 1. August 2012 84
  • 85. Bounding Box Sampling for Detection §  Single-scale 2D windows of 50x50 with stride 50 §  A window is placed if it overlaps > 25% with the liver §  A windows is considered as positive if inside it there are >50 lesion pixels §  A margin of 15 pixels is added to all windows to have more context 31. July 2017 85 50 50 80 80
  • 86. Image Classification 0.2 0.4 0.6 0.8 1 Recall 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Precision Precision-Recall for classification of lesion Image Classification with VGG-16 Image Classification with ResNet-50 31. July 2017 86 •  As proof of concept we first trained a classifier that distinguished between healthy image / unhealthy image •  Resnet – 50 layers really improved compared to VGG-16 architecture