https://imatge.upc.edu/web/publications/detection-aided-liver-lesion-segmentation-using-deep-learning A fully automatic technique for segmenting the liver and localizing its unhealthy tissues is a convenient tool in order to diagnose hepatic diseases and assess the response to the according treatments. In this work we propose a method to segment the liver and its lesions from Computed Tomography (CT) scans using Convolutional Neural Networks (CNNs), that have proven good results in a variety of computer vision tasks, including medical imaging. The network that segments the lesions consists of a cascaded architecture, which first focuses on the region of the liver in order to segment the lesions on it. Moreover, we train a detector to localize the lesions, and mask the results of the segmentation network with the positive detections. The segmentation architecture is based on DRIU, a Fully Convolutional Network (FCN) with side outputs that work on feature maps of different resolutions, to finally benefit from the multi-scale information learned by different stages of the network. The main contribution of this work is the use of a detector to localize the lesions, which we show to be beneficial to remove false positives triggered by the segmentation network.

1. Detection-aided medical image segmentation
using deep learning
8. September 2017 | Master Thesis
Míriam Bellver Bueno, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Xavier Giró-i-Nieto,
Jordi Torres, Luc Van Gool

2. Outline
§ Introduction
§ Related Work
§ Method
§ Experimental Validation
§ Conclusions
8. September 2017 2

3. Introduction
Detection-aided medical image segmentation using deep learning
8. September 2017 3

4. Introduction
§ Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§ Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
8. September 2017 4

5. Introduction
§ Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§ Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
§ Time-consuming
§ Subjective
8. September 2017 5

6. Introduction
§ Contract Tomography (CT) images are an important tool in medical
imaging to diagnose several diseases or assess treatments
§ Doctors typically rely on manual or semi-automatic techniques to
study anomalies in the shape and texture of organs of CT scans
§ Time-consuming
§ Subjective
8. September 2017 6
Fully Automatic Tool ✔

7. Introduction
§ We will focus on liver and its lesions segmentation tasks for
afterwards segmenting several organs and other anatomical structures
8. September 2017 7
Lesion GT
Liver GT

8. Introduction
§ Lesion segmentation is a quite challenging task:
§ Low contrast between lesion, liver and other organs
§ Lesions are variable in terms of shape, size and texture
§ Noise in CT scans
§ Typically statistical shape and intensity distribution models
8. September 2017 8

9. Introduction
§ Lesion segmentation is a quite challenging task:
§ Low contrast between lesion, liver and other organs
§ Lesions are variable in terms of shape, size and texture
§ Noise in CT scans
§ Typically statistical shape and intensity distribution models
§ Deep Convolutional Neural Networks have demonstrated to be
successful on challenging tasks as lesion and liver segmentation
8. September 2017 9

10. Introduction
§ Our pipeline is based on the strengths of a segmentation network and
a detector:
8. September 2017 10
Segmentation
Network
Specializes on fine localization
of lesions
Detection
Network
Learns global features given a
whole liver patch

11. Introduction
§ Tasks:
§ Develop a method to segment the lesion and the liver from CT scans using
in the framework of the LiTS Challenge.
§ Prove generality of segmentation network with Visceral dataset.
8. September 2017 11

12. Related Work
Detection-aided medical image segmentation using deep learning
8. September 2017 12

13. Convolutional Neural Networks (CNNs)
8. September 2017 13
LeCun, Y. (2015). LeNet-5, convolutional neural networks

14. Segmentation
§ Fully Convolutional Networks (FCNs)
8. September 2017 14
Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation

15. Segmentation
§ Encoder-Decoder architecture
8. September 2017 15
Badrinarayanan, V., Kendall, A., & Cipolla, R. (2015). Segnet: A deep convolutional encoder-decoder
architecture for image segmentation

16. Segmentation
§ U-net
8. September 2017 16
Ronneberger, O., Fischer, P., & Brox, T. (2015, October). U-net: Convolutional networks
for biomedical image segmentation

17. Segmentation
§ Deep Retinal Image Understanding (DRIU)
8. September 2017 17
Maninis, K. K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016, October). Deep retinal image understanding
Input
Fine feature maps Coarse feature maps
Vessels Optic Disc
Specialized
Layers
Image
Base Network Architecture

18. Detection
8. September 2017 18
Segmentation Detection

19. Detection
8. September 2017 19
Girshick, R., Donahue, J., Darrell, T., & Malik, J. (2016). Region-based convolutional networks for
accurate object detection and segmentation

20. Method
Detection-aided medical image segmentation using deep learning
8. September 2017 20

21. Architecture
Liver Segmentation
1
Lesion Segmentation
3
Bounding box sampling
2
8. September 2017 21

22. Architecture: Segmentation Network
Liver Segmentation
1
Lesion Segmentation
3
Bounding box sampling
2
8. September 2017 22

23. Architecture: Segmentation Network
§ Segmentation network based on Deep Retinal Image Understanding
(DRIU) network (*)
§ Fully Convolutional Network (FCN)
§ Base network is VGG-16 and pre-trained with Imagenet
§ Side outputs
§ Combination of multi-scale side outputs
8. September 2017 23
Liver Segmentation
1
Les
3
Bounding box sampling
2Maninis, K. K., Pont-Tuset, J., Arbeláez, P., & Van Gool, L. (2016, October). Deep retinal image understanding. In International
Conference on Medical Image Computing and Computer-Assisted Intervention (pp. 140-148). Springer International Publishing.

24. Architecture: Detection Network
Liver Segmentation
1
Lesion Segmentation
3
Bounding box sampling
2
8. September 2017 24

25. Bounding Box Sampling for Detection
§ Example of bounding box sampling and labeling
§ Data augmentation with flips and rotations x 8
8. September 2017 25

26. Detection Model
§ Pre-trained Resnet-50 without classification
layer for Imagenet
§ A single neuron determining whether it is a
healthy tissue
8. September 2017 26

27. Experimental Validation
Detection-aided medical image segmentation using deep learning
8. September 2017 27

28. Experimental Validation: LiTS dataset
• CT scans with the liver and lesion mask
• Dimensions
§ Training: 131 CT scans
§ 80% : 105 training volumes
§ 20 % : 26 validation volumes
§ Testing: 70 CT scans
8. September 2017 28

29. Experimental Validation: Metrics
§ The metric to assess performance will be the Dice score (F-score)
8. September 2017 29

30. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 30

31. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 31

32. 1. Pre-processing
§ We clipped pixel intensity values by maximum and minimum values
that statistically belong to the liver and lesion class
8. September 2017 32
0.30
Lesion
Dice Score

33. 1. Pre-processing
§ We clipped pixel intensity values by maximum and minimum values
that statistically belong to the liver and lesion class
8. September 2017 33
0.30 0.32
Lesion
Dice Score

34. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 34

35. 2. Weighting of Binary Cross Entropy (BCE) loss
Loss objective
§ Binary Cross Entropy (BCE) Loss
§ Weighted Binary Cross Entropy (BCE) Loss
8. September 2017 35
Balancing term

36. 2. Weighting of Binary Cross Entropy (BCE) loss
Balancing Schemes
§ Per-volume balancing
§ General balancing
8. September 2017 36

37. 2. Weighting of Binary Cross Entropy (BCE) loss
Balancing Schemes
§ Per-volume balancing
§ General balancing
8. September 2017 37

38. 2. Weighting of Binary Cross Entropy (BCE) loss
8. September 2017 38
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Per-volume balancing
General balancing
Lesion Precision – Recall Curve
0.30 0.32
Lesion
Dice Score

39. 2. Weighting of Binary Cross Entropy (BCE) loss
8. September 2017 39
Lesion Precision – Recall Curve
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Per-volume balancing
General balancing
0.30 0.32 0.34
Lesion
Dice Score

40. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 40

41. 3. Stacking 3 consecutive slices at the input of the network
§ Volumes of data that are highly redundant
§ We input a stack of slices in the network with supervision
§ Final configuration inputs 3 slices, one at each RGB channel
8. September 2017 41
Network

42. 3. Stacking 3 consecutive slices at the input of the network
8. September 2017 42
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
1-slice
3-slices
6-slices
9-slices
Lesion Precision – Recall Curve
Lesion Results
Liver Results
0.30 0.32 0.34
Lesion
Dice Score

43. 3. Stacking 3 consecutive slices at the input of the network
8. September 2017 43
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
1-slice
3-slices
6-slices
9-slices
Lesion Results
Liver Results
Lesion Precision – Recall Curve
Lesion
Dice Score 0.30 0.32 0.34 0.36

44. Experimental Validation: Ablation study for segmentation
network
Key features of our segmentation network
§ 1. Pre-processing
§ 2. Weighting of Binary Cross Entropy (BCE) loss
§ 3. Stacking 3 consecutive slices at the input of the network
§ 4. Using liver segmentation to segment lesion
8. September 2017 44

45. 4. Using liver segmentation to segment lesion
§ We worked with 2 different strategies:
1. Back-propagation (BP) through liver: Only back-propagation through liver
2. Multitask: Segmenting the liver and the lesion simultaneously
8. September 2017 45

46. 4. Using liver segmentation to segment lesion
§ Features of Back-propagation (BP) through liver strategy
§ The predicted liver mask is fixed
§ The liver mask multiplies to the lesion mask during training and testing
§ The weighting term in the BCE loss now just considers liver pixels
8. September 2017 46
Network 1 ∘Network 2

47. 4. Using liver segmentation to segment lesion
§ Pros of BP through liver ✔
§ Network learns from relevant pixels
§ Positive/Negative pixels are more balanced
§ Cons of BP through liver ✖
§ If there is a mistake in the liver mask, this mistake will affect the lesion
segmentation
8. September 2017 47

48. 4. Using liver segmentation to segment lesion
§ Features of Multi-task liver strategy
§ The output of the network is a channel for the liver and another for the
lesion.
§ The loss is the sum of the two BCE losses . Both classes can happen at the
same time. No competition among classes.
8. September 2017 48
Network 1 ∘Network 2
Network 1 Network 2

49. 4. Using liver segmentation to segment lesion
§ Pros of Multi-task ✔
§ The gradients of the network have information of both the liver and lesion
§ Efficient and Scalable
§ Cons of Multi-task ✖
§ Difficult to weight each task properly
8. September 2017 49

50. 4. Using liver segmentation to segment lesion
8. September 2017 50
Difference in masking during testing or testing + training
Performance of various strategies
Lesion
Dice Score 0.30 0.32 0.34 0.36

51. 4. Using liver segmentation to segment lesion
8. September 2017 51
Difference in masking during testing or testing + training
Performance of various strategies
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39

52. Motivation for Detection
8. September 2017 52
Lesion GT
Lesion Pred.
Liver GT
Liver Pred.

53. Motivation for Detection
8. September 2017 53
Lesion GT
Lesion Pred.
Liver GT
Liver Pred.
False Positives

54. Detection
8. September 2017 54
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Precision-Recall for detection of lesion

55. Detection
§ Examples of detected positive windows
8. September 2017 55
Lesion GT
Lesion Pred.
Liver GT
Liver Pred.

56. Detection
§ Examples of detected positive windows
8. September 2017 56
Before Detection After Detection

57. Detection
8. September 2017 57
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multitask
BP through liver
Multitask + Det
BP through liver + Det
Lesion Precision – Recall Curve
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39

58. Detection
8. September 2017 58
Lesion Precision – Recall Curve
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multitask
BP through liver
Multitask + Det
BP through liver + Det
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41

59. Post- processing: 3D – Conditional Random Fields
§ It will add spatial coherence in the 3 dimensions, based on space and
appearance
8. September 2017 59
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41

60. Post- processing: 3D – Conditional Random Fields
§ It will add spatial coherence in the 3 dimensions, based on space and
appearance
8. September 2017 60
Lesion
Dice Score 0.30 0.32 0.34 0.36 0.39 0.41 0.43

61. Post- processing: 3D – Conditional Random Fields
8. September 2017 61

62. Summary of Lesion Segmentation
8. September 2017 62
Lesion
Dice Score
(validation set)
26 CT scans 62
0.30 0.32 0.34 0.36 0.39 0.41 0.43
62

63. Summary of Lesion Segmentation
8. September 2017 63
Lesion
Dice Score
(validation set)
26 CT scans 63
0.30 0.32 0.34 0.36 0.39 0.41 0.43
63
~ 43 % improvement

64. Summary of Lesion Segmentation
8. September 2017 64
Lesion
Dice Score
(validation set)
26 CT scans 64
0.30 0.32 0.34 0.36 0.39 0.41 0.43
64
0.41 0.54 0.57 0.59
~ 43 % improvement
Lesion
Dice Score
(Challenge
test set)
70 CT scans

65. Summary of Lesion Segmentation
8. September 2017 65
Lesion
Dice Score
(validation set)
26 CT scans 65
0.30 0.32 0.34 0.36 0.39 0.41 0.43
65
0.41 0.54 0.57 0.59
~ 43 % improvement
~ 44 % improvement
Lesion
Dice Score
(Challenge
test set)
70 CT scans

66. Summary of Lesion Segmentation
8. September 2017 66
Lesion
Dice Score
(validation set)
26 CT scans 66
0.30 0.32 0.34 0.36 0.39 0.41 0.43
66
0.41 0.54 0.57 0.59
~ 43 % improvement
~ 44 % improvement
0.68
Top
Entry
Lesion
Dice Score
(Challenge
test set)
70 CT scans
…

67. Experimental Validation: Visceral
• CT scans with segmentation for 20 organs and anatomical structures
• Dimensions
§ Training: 20 CT scans
§ 90% : 18 training volumes
§ 10 % : 2 validation volumes
8. September 2017 67

68. Visceral Experiments
8. September 2017 68

69. Conclusions
Detection-aided medical image segmentation using deep learning
8. September 2017 69

70. Conclusions
§ We improved the baseline we worked on developing a set of strategies
applicable to other pipelines.
§ Detection + Segmentation improved over only Segmentation.
§ Using the liver for the lesion segmentation boosted the performance.
Learning from relevant samples seems a good practice.
§ Segmentation network has a lot of potential to segment different kinds of
structures in a single forward pass.
8. September 2017 70

71. Publication
8. September 2017 71

72. Acknowledgements
8. September 2017 72

73. Thank you for your attention!
8. September 2017 73

74. 31. July 2017 74

75. Publication
8. July 2017 75

76. 0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multi-task
BP through liver
Multi-task + BP through liver
3. Using liver segmentation to segment lesion
1. August 2012 76
0.32 0.34 0.36
Lesion
Dice Score

77. Detection
1. August 2012 77
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Multitask
BP through liver
Multitask + Det
BP through liver + Det
0.32 0.34 0.36 0.38 0.41
Lesion
Dice Score

78. Post- processing: 3D – Conditional Random Fields
1. August 2012 78
0.32 0.34 0.36 0.39 0.41
Lesion
Dice Score

79. Post- processing: 3D – Conditional Random Fields
1. August 2012 79
0.32 0.34 0.36 0.39 0.41 0.43
Lesion
Dice Score

80. LiTS Challenge
1. August 2012 80

81. Visceral Experiments
1. August 2012 81

82. Visualizations – Seg. + Det. (2/2)
1. August 2012 82

83. Visualizations – Seg. + Det. (1/2)
1. August 2012 83

84. Visualizations – Seg. + Det. (2/2)
1. August 2012 84

85. Bounding Box Sampling for Detection
§ Single-scale 2D windows of 50x50 with stride 50
§ A window is placed if it overlaps > 25% with the liver
§ A windows is considered as positive if inside it there are >50 lesion
pixels
§ A margin of 15 pixels is added to all windows to have more context
31. July 2017 85
50
50 80
80

86. Image Classification
0.2 0.4 0.6 0.8 1
Recall
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Precision
Precision-Recall for classification of lesion
Image Classification with VGG-16
Image Classification with ResNet-50
31. July 2017 86
• As proof of concept we first
trained a classifier that
distinguished between healthy
image / unhealthy image
• Resnet – 50 layers really
improved compared to VGG-16
architecture