2018 KOSRO-Oral presentation-Wonjoong Cheon

1. Deep learning application to
Patient specific organ-at-risk auto segmentation
Wonjoong Cheon1, Sang Hee Ahn2), Moonhee Lee1), Jinhyeop Lee1) , Seyjoon Park3), Dae Hyun Kim3),
Kwanghyun Cho3), Youngyih Han4)*, Do Hoon Lim4)
1 Department of Health Sciences and Technology, SAIHST, Sungkyunkwan University, Seoul, 06351, Korea
2 Proton therapy center, National Cancer Center, Department of Radiation Oncology, National Medical Center, Gyeonggi-do, 10408, Korea
3 Department of Radiation Oncology, Samsung Medical Center, Seoul, 06351, Korea
4 Department of Radiation Oncology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, 06351, Korea
Oct 12, 2018 / 36th The Korea Society for Radiation Oncology (KOSRO)

2. Introduction

3. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Jan-04
May-04
Sep-04
Jan-05
May-05
Sep-05
Jan-06
May-06
Sep-06
Jan-07
May-07
Sep-07
Jan-08
May-08
Sep-08
Jan-09
May-09
Sep-09
Jan-10
May-10
Sep-10
Jan-11
May-11
Sep-11
Jan-12
May-12
Sep-12
Jan-13
May-13
Sep-13
Jan-14
May-14
Sep-14
Jan-15
May-15
Sep-15
Jan-16
May-16
Sep-16
Jan-17
May-17
Sep-17
Jan-18
May-18
Sep-18
1) Lee sedol vs AlphaGo
2) Papers published for detecting
diabetic retinopathy using deep-
learning @ Google
Google trends: “deep-learning”
100%
80%
60%
40%
20%
Gulshan, Varun, et al. "Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs." Jama 316.22 (2016): 2402-2410.

4. Jan-04
May-04
Sep-04
Jan-05
May-05
Sep-05
Jan-06
May-06
Sep-06
Jan-07
May-07
Sep-07
Jan-08
May-08
Sep-08
Jan-09
May-09
Sep-09
Jan-10
May-10
Sep-10
Jan-11
May-11
Sep-11
Jan-12
May-12
Sep-12
Jan-13
May-13
Sep-13
Jan-14
May-14
Sep-14
Jan-15
May-15
Sep-15
Jan-16
May-16
Sep-16
Jan-17
May-17
Sep-17
Jan-18
May-18
Sep-18
Adaptive radiation therapy (ART)
• Adaptive radiation therapy is a closed-loop radiation treatment process
where the treatment can be modified using a systematic feedback of
measurements or imaging.
• Adaptive radiation therapy intends to improve radiation treatment
by systematically monitoring treatment variations and incorporating them
through re-optimize the treatment plan early on during the course of
treatment.
• Yan, Di, et al. "Adaptive radiation therapy.
"Physics in Medicine & Biology 42.1 (1997): 123.
1.Introductuion
2.M.M
3.Results
4.Conclustion
1) Lee sedol vs AlphaGo
2) Papers published for detecting
diabetic retinopathy using deep-
learning @ Google
Google trends: “deep-learning”

5. Going toward online ART with deep-learning1.Introductuion
2.M.M
3.Results
4.Conclustion
With offline ART
Without ART

6. 1.Introductuion
2.M.M
3.Results
4.Conclustion
With offline ART
Without ART
Online
ART
Going toward online ART with deep-learning

7. 1.Introductuion
2.M.M
3.Results
4.Conclustion
With offline ART
Without ART
Going toward online ART with deep-learning
Key technologies for making online ART efficient
① Delineation  Auto delineation (Support)
(Deep-learning: Supervised learning)
② Planning  Auto planning (Support)
(Deep-learning: Reinforcement learning)
③ Pre-treatment QA  Auto QA
(Monte-carlo simulation)
④ Simulation CT  CBCT to Simulation CT
(Deep-learning: Supervised learning)

8. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Company Product Method TPS integration
Accuray
MultiPlan
Auto-segmentation
Atalas-based/ model-basd Yes
BrainLab iPlan Atalas-based/ model-basd Yes
Dosisoft IMAgo Atalas-based/ model-basd Yes
Elekta ABAS Atalas-based/ model-basd No
MIM MIM Maestro Atalas-based/ model-basd No
Mirada RTx Atalas-based/ model-basd No
Philips SPICE Atalas-based/ model-basd Yes
RaySearch RayStation Atalas-based/ model-basd Yes
Varian Smart Segmentation Atalas-based/ model-basd Yes
Velocity Velocity AI Atalas-based/ model-basd No
Commercial product atlas-based auto-segmentation
Research papers for auto-segmentation with deep-learning
de Bel, Thomas, et al. "Automatic segmentation of histopathological slides of renal tissue using deep learning."Medical Imaging 2018: Digital Pathology. Vol. 10581. [15 patients]
Fechter, Tobias, et al. "Esophagus segmentation in CT via 3D fully convolutional neural network and random walk."Medical physics 44.12 (2017): 6341-6352. [50 patients]
Valindria, Vanya V., et al. "Multi-Modal Learning from Unpaired Images: Application to Multi-Organ Segmentation in CT and MRI." Applications of Computer Vision, 2018 IEEE Winter Conference on. [34 patients]
Wang, Yan, et al. "Abdominal multi-organ segmentation with organ-attention networks and statistical fusion." arXiv preprint arXiv:1804.08414 (2018) [13 patients]
Comparison of deep learning-based techniques for organ segmentation in abdominal CT images [70 patients]
Kazemifar, Samaneh, et al. "Segmentation of the prostate and organs at risk in male pelvic CT images using deep learning." arXiv preprint arXiv:1802.09587 (2018). [85 patients]

9. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Company Product Method TPS integration
Accuray
MultiPlan
Auto-segmentation
Atalas-based/ model-basd Yes
BrainLab iPlan Atalas-based/ model-basd Yes
Dosisoft IMAgo Atalas-based/ model-basd Yes
Elekta ABAS Atalas-based/ model-basd No
MIM MIM Maestro Atalas-based/ model-basd No
Mirada RTx Atalas-based/ model-basd No
Philips SPICE Atalas-based/ model-basd Yes
RaySearch RayStation Atalas-based/ model-basd Yes
Varian Smart Segmentation Atalas-based/ model-basd Yes
Velocity Velocity AI Atalas-based/ model-basd No
Products of commercial auto-segmentation
Papers of commercial auto-segmentation
de Bel, Thomas, et al. "Automatic segmentation of histopathological slides of renal tissue using deep learning." Medical Imaging 2018: Digital Pathology. Vol. 10581.
International Society for Optics and Photonics, 2018.
Luo, Zengbo, et al. "Multi-Organ Segmentation Based on Convolutional Neural Network and Random Walk for CT Image." Journal of Medical Imaging and Health Informatics 8.5 (2018): 1057-1063.
Valindria, Vanya V., et al. "Multi-Modal Learning from Unpaired Images: Application to Multi-Organ Segmentation in CT and MRI." Applications of Computer Vision (WACV), 2018 IEEE Winter Conference on. IEEE, 2018.
Wang, Yan, et al. "Abdominal multi-organ segmentation with organ-attention networks and statistical fusion." arXiv preprint arXiv:1804.08414 (2018)
Comparison of deep learning-based techniques for organ segmentation in abdominal CT images
Kazemifar, Samaneh, et al. "Segmentation of the prostate and organs at risk in male pelvic CT images using deep learning." arXiv preprint arXiv:1802.09587 (2018).

10. Population Sample A.I (Deep-learning)
Data what we got
Training
1.Introductuion
2.M.M
3.Results
4.Conclustion
Sample data
This is the reason everybody say that “big data” is necessary
New patient

11. Training strategy

12. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Automatic delineation of RF-CTs with deep-learning
network that learned through their initial simulation CT
Going toward online ART with deep-learning
Initial sim. CT Initial RT-ST
Reduced field CT (RF-CT): CT images were taken for re-simulate
Training strategy

13. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Automatic delineation of RF-CTs with deep-learning
network that learned through their initial simulation CT
Going toward online ART with deep-learning
Initial sim. CT Initial RT-ST
RF-CT
Reduced field CT (RF-CT): CT images were taken for re-simulate
Reduced field CT (RF-CT)
Training strategy

14. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Automatic delineation of RF-CTs with deep-learning
network that learned through their initial simulation CT
Going toward online ART with deep-learning
Initial sim. CT Initial RT-ST
RF-CT
Reduced field CT (RF-CT): CT images were taken for re-simulate
Variation of initial sim. CT
Reduced field CT (RF-CT)
Training strategy

15. Material and Method

16. Architecture
On the class of Convolutional neural network for visual recognition @ Stanford 2018 (CS231n)
• U-Net
• Convolutional Networks for Biomedical Image Segmentation
• Encoder-decoder architecture.
• When desired output should include localization, i.e., a class label is supposed to be
assigned to each pixel
• Training in patches helps with lack of data
Ref. Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image
segmentation." International Conference on Medical image computing and computer-assisted intervention. Springer, Cham, 2015.
•
• High Performance
• Atrous Convolution (Convolutions with upsampled filters)
• Allows user to explicitly control the resolution at which feature responses
are computed.
Ref. Chen, Liang-Chieh, et al. "Rethinking atrous convolution for semantic image segmentation." arXiv preprint
arXiv:1706.05587 (2017).
1.Introductuion
2.M.M
3.Results
4.Conclustion
DeepLab

17. Dataset1.Introductuion
2.M.M
3.Results
4.Conclustion
Lung
Initial CT,
RT-Structure
RF-CT (input)
RT-Structure:
(compare with
auto-segmentation result)
Train Test
Prostate
Initial CT,
RT-Structure
RF-CT (input)
RT-Structure:
(compare with
auto-segmentation result)
Train Test
CSI
Initial CT,
RT-Structure
RF-CT (input)
RT-Structure:
(compare with
auto-segmentation result)
Train Test

18. Results

19. OARs of RF-CT (2nd CT)1.Introductuion
2.M.M
3.Results
4.Conclustion
OARs contoured by Experts OARs contoured by deep-network
Dice-coefficient:
교집합 영역의 비율

20. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Lung
'Auto-segmentation
: volume‘
[cm3
]
'Expert
: volume‘
[cm3
]
Volume difference
[cm3
]
'Auto-segmentation
: surface area‘
[cm2
]
'Expert
: surface area‘
[cm2
]
Surface difference
[cm2
]
'Dice-coefficient'
Heart 185.474 201.220 15.746 73.323 77.416 4.093 0.929
Trachea 6.382 6.938 0.556 7.757 8.201 0.444 0.959
Esophagus 12.389 13.050 0.661 12.071 12.497 0.425 0.936
Spinal code 11.894 14.436 2.542 11.747 13.367 1.619 0.853
Both lung 594.885 597.161 2.276 159.468 159.874 0.407 0.973
Average 4.366 1.396 0.931

21. 1.Introductuion
2.M.M
3.Results
4.Conclustion
OARs of RF-CT (2nd CT)
OARs contoured by Experts OARs contoured by deep-network

22. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Prostate
'Auto-segmentation
: volume‘
[cm3
]
'Expert
: volume‘
[cm3
]
Volume difference
[cm3
]
'Auto-segmentation
: surface area‘
[cm2
]
'Expert
: surface area‘
[cm2
]
Surface difference
[cm2
]
'Dice-coefficient'
LT-RT femoral
head
29.457 30.472 1.015 21.503 21.995 0.491 0.959
Rectum (in) 17.148 17.982 0.834 14.992 15.474 0.482 0.938
Bladder 86.835 86.645 -0.189 44.209 44.145 -0.064 0.983
Penile bulb 1.023 1.193 0.169 2.290 2.535 0.246 0.885
Prostate 5.685 5.673 -0.012 7.181 7.171 -0.010 0.984
Average 0.363 0.229 0.949

23. 1.Introductuion
2.M.M
3.Results
4.Conclustion
OARs of RF-CT (2nd CT)
OARs contoured by Experts OARs contoured by deep-network

24. 1.Introductuion
2.M.M
3.Results
4.Conclustion
Prostate
'Auto-segmentation
: volume‘
[cm3
]
'Expert
: volume‘
[cm3
]
Volume difference
[cm3
]
'Auto-segmentation
: surface area‘
[cm2
]
'Expert
: surface area‘
[cm2
]
Surface difference
[cm2
]
'Dice-coefficient'
Brain 1334.881 1351.626 16.745 273.325 275.606 2.281 0.990
LT-RT Eye ball 10.524 11.491 0.967 10.827 11.480 0.653 0.937
Brain stem 17.583 19.211 1.628 15.244 16.171 0.927 0.952
Average 6.447 1.287 0.960

25. Conclusion and Summary

26. Conclusion and Summary
• The new approach for auto-contouring on RF-CT was proposed
• The proposed method only need an their initial CT and RT-ST for auto-contouring on their RF-CT
• The difference of volume, The difference of surface area, and dice-coefficient compared proposed
auto-segmentation with expert’s manual segmentation results were 3.333 cc, 0.943 cm2, and 0.944,
respectively.
• This means that 94 percent of the respondents on average match the results of experts.
• This method is a part technique for daily online-ART in the future.
• The time required for training is about 4 hours with 680 US dollar of 1 GPU.
• The time required for auto-contouring on RF-CT is just about 3.0 seconds.
• In order to advance the online ART, our research team is working on key technologies.
(Auto-planning-support, Auto QA with monte-carlo, Super-resolution technique CBCT to sim. CT)