This document discusses a deep learning approach for predicting prostate cancer Gleason Grade Group from multiparametric MRI data. The authors developed a model called SummerNet that uses a multi-level approach, combining deep learning and conventional machine learning. SummerNet achieved a Cohen's kappa of 0.26 on the test data of the PROSTATEx-2 challenge, outperforming models that only used deep learning or conventional machine learning. The authors conclude that combining deep and conventional machine learning can help reduce overfitting, and suggest directions for future improvement including using multi-channel 3D convolutional layers.

1. Prostate Cancer Diagnosis using Deep Learning
with 3D Multiparametric MRI:
Predicting the Gleason Grade Group
Saifeng Liu1, Huaixiu Zheng2, Yesu Feng3
1 The MRI Institute for Biomedical Research, Detroit, MI, USA
2 Uber Technologies Inc., San Francisco, CA, USA
3 LinkedIn, San Francisco, CA, USA
PROSTATEx-2 Challenge, AAPM 2017
Aug. 1st, 2017, Denver

2. Prostate Cancer
Estimated New Cancer Cases in Men in US (2017)2
Prostate Cancer
Other Cancers
19%
The second most common type of cancer in men1
Risk of prostate cancer: 1 in 7

3. Prostate Cancer
The second most common type of cancer in men1
Deaths caused by prostate cancer: 1 in 39
Estimated New Cancer Cases in Men in US (2017)2
Prostate Cancer
Other Cancers
19%

4. Prostate Cancer
• Limited diagnostic accuracy
• Prostate-specific antigen (PSA) test, digital rectal exam (DRE)
• Transrectal ultrasound (TRUS)/MRI guided biopsy
• Multiparametric MRI (mpMRI) with PI-RADS v2 (sensitivity & specificity ~ 0.8)3
• Few studies on the prediction of Gleason grade group using mpMRI4,5

5. PROSTATEx-2 Challenge
• Data: 162 MRI cases
• 182 lesions, 40% are used as test set.
• Multiple types of MRI images (DWI, ADC, Ktrans, T2WI).
• Each case contains at least one prostate lesion with provided location.
• Goal: Prediction of Gleason Grade Group (1 to 5)
• Evaluation metric: Quadratic-weighted Cohen’s kappa

6. • Challenges
• Ordinal classification
• Heterogenous data
• Very limited and unbalanced samples
PROSTATEx-2 Challenge
36
41
20
8 7

7. Data Preprocessing
Registration
• Interpolation
• Rigid-body registration
Region Growing
• Region growing using
the provided lesion
location
• Cropping the ROI
Train/Validation
Sample Preparation
• Train (85%)/Validation
(15%)
• Channel composition
DWI

8. Data Preprocessing
Registration
• Interpolation
• Rigid-body registration
Region Growing
• Region growing using
the provided lesion
location
• Cropping the ROI
Train/Validation
Sample Preparation
• Train (85%)/Validation
(15%)
• Channel composition
DWI
ADC

9. Data Preprocessing
Registration
• Interpolation
• Rigid-body registration
Region Growing
• Region growing using
the provided lesion
location
• Cropping the ROI
Train/Validation
Sample Preparation
• Train (85%)/Validation
(15%)
• Channel composition
DWI
ADC
Ktrans

10. Data Preprocessing
Registration
• Interpolation
• Rigid-body registration
Region Growing
• Region growing using
the provided lesion
location
• Cropping the ROI
Train/Validation
Sample Preparation
• Train (85%)/Validation
(15%)
• Channel composition
DWI
ADC
Ktrans
T2WI

11. Data Preprocessing
Registration
• Interpolation
• Rigid-body registration
Region Growing
• Region growing using
the provided lesion
location
• Cropping the ROI
Train/Validation
Sample Preparation
• Train (85%)/Validation
(15%)
• Channel composition
DWI
ADC
Ktrans
T2WI

12. Data Preprocessing
Registration
• Interpolation
• Rigid-body registration
Region Growing
• Region growing using
the provided lesion
location
• Cropping the ROI
Train/Validation
Sample Preparation
• Train (85%)/Validation
(15%)
• Channel composition
DWI
ADC
Ktrans
T2WI

13. Data Preprocessing
Registration
• Interpolation
• Rigid-body registration
Region Growing
• Region growing using
the provided lesion
location
• Cropping the ROI
Train/Validation
Sample Preparation
• Train (85%)/Validation
(15%)
• Channel composition
DWI
ADC
Ktrans
T2WI

14. Data Preprocessing
Registration
• Interpolation
• Rigid-body registration
Region Growing
• Region growing using
the provided lesion
location
• Cropping the ROI
Train/Validation
Sample Preparation
• Train (85%)/Validation
(15%)
• Channel composition
DWI
ADC
Ktrans
T2WI
DWI(D), ADC(A), Ktrans(K), T2 (T)
AKTDAK
DAT DKT

15. Multi-view Slicing: data augmentation
3D slicing

16. Multi-view Slicing: data augmentation
3D slicing

17. Multi-view Slicing: data augmentation
3D slicing

18. Multi-view Slicing: data augmentation
3D slicing

19. Multi-view Slicing: data augmentation
Rotation, shearing and translation3D slicing

20. Multi-view Slicing: data augmentation
Rotation, shearing and translation3D slicing

21. Conventional Machine Learning Method
20
20
20
20
16
16
13
13
10
9
7
6
6
5
5
Ktrans_correlation
Ktrans_min
Ktrans_variance
ADC_min
ADC_variance
Ktrans_symmetry
DWI_mean
T2_Entropy
T2_sphericity
T2_Energy
DWI_Energy
ADC_25%
ADC_SumAverage
Ktrans_Contrast
T2_25%
Level 1 Model Feature Importance
Feature
Extraction
Training
XGBoost Model
Feature
Selection
Sensitivity: 0.87
Specificity: 0.77

22. Deep Learning Architecture: XmasNet
FC1 ReLU
FC2 ReLU Softmax

23. PROSTATEx-1:
Deep Learning vs. Conventional Machine Learning
Train Data ROC Curve
0.80
0.84
XGBoost XmasNet
Test Data AUC

24. SummerNet:
Deep Learning + Conventional Machine Learning
Level 1
Model
Level 2
Model
Level 3
Model
Level 4
Model
1
2
3
4
5
Trained using conventional
machine learning
Trained using deep learning
P(ggg>1)
P(ggg>2)
P(ggg>3)
P(ggg>4)
Multi-class
Model

25. Dealing with Unbalanced Data
Training level-3 model: train sample selection
ggg=1
ggg=2
ggg=3
ggg=4
ggg=5

26. Dealing with Unbalanced Data
Training level-3 model: train sample selection ggg>3
ggg=1
ggg=2
ggg=3
ggg=4
ggg=5

27. Dealing with Unbalanced Data
Training level-3 model: train sample selection ggg≤3ggg>3
ggg=1
ggg=2
ggg=3
ggg=4
ggg=5

28. Deep Learning + Conventional Machine Learning:
Model Importance

29. Deep Learning + Conventional Machine Learning:
Model Importance
Trained using conventional
machine learning

30. Deep Learning + Conventional Machine Learning:
Model Importance
Trained using conventional
machine learning
Trained using deep learning

31. Results
0.80
0.84
XGBoost XmasNet
PROSTATEx-1
Test Data AUC
PROSTATEx-2
Test Data Cohen’s kappa
0.19
0.05
0.26
XGBoost Levels 1&2
XmasNet
SummerNet

32. Conclusions and Future Directions
• Multi-level model for predicting Gleason Grade Group with mpMRI data
• Data augmentation with 3D rotation and slicing
• Deep learning + conventional machine learning reduced over-fitting
• Single model with multiple outputs
• Multi-channel input with 3D convolution layers

33. Our Team
Huaixiu Zheng Yesu Feng Saifeng Liu

34. References
• 1. Key Statistics for Prostate Cancer. Retrieved February 10, 2017, from
https://www.cancer.org/cancer/prostate-cancer/about/key-statistics.html
• 2. American Cancer Society. Cancer facts and figures 2017.
• 3. Kasel-Seibert M et al. European journal of radiology. 2016 ;85(4):726-31.
• 4. Fehr et al. PNAS. 2015;112(46):E6265-73.
• 5. Gondo et al. European Radiology. 2014 Dec;24(12):3161-70.
• 6. Ehrenberg HR et al. In SPIE Medical Imaging 2016 (pp. 97851J-97851J).
• 7. Chen T, Guestrin C. InProceedings of the 22Nd ACM SIGKDD International Conference on Knowledge
Discovery and Data Mining 2016 (pp. 785-794). ACM.
• 8. Simonyan, K.., Zisserman, A., ArXiv14091556 Cs (2014).
• 9. Vallières, M., et al. Phys. Med. Biol. 60(14), 5471 (2015).
• 10. Saifeng Liu, Huaixiu Zheng, Yesu Feng and Wei Li. Proc. SPIE 10134, Medical Imaging 2017:
Computer-Aided Diagnosis, 1013428; doi:10.1117/12.2277121;