Delineation Variability - Human Delineation Variability Quantification - Realistic Variability Simulation - Probabilistic U-Net Radiomics - Lung Cancer Screening - Spiculation Quantification - Aggressive Lung ADC subtype prediction

1. Nov 1, 2019 @ MSKCC
Quantitative Cancer Image Analysis
Wookjin Choi, PhD
Assistant Professor of Computer Science
Virginia State University
wchoi@vsu.edu

2. Acknowledgements
Memorial Sloan Kettering Cancer Center
• Wei Lu PhD
• Sadegh Riyahi, PhD
• Jung Hun Oh, PhD
• Saad Nadeem, PhD
• Joseph O. Deasy, PhD
• Andreas Rimner, MD
• Prasad Adusumilli, MD
• Chia-ju Liu, MD
• Wolfgang Weber, MD
Stony Brook University
• Allen Tannenbaum, PhD
University of Virginia School of Medicine
• Jeffrey Siebers, PhD
• Hamidreza Nourzadeh, PhD
• Victor Gabriel Leandro Alves, PhD
• Eric Aliotta, PhD
University of Maryland School of Medicine
• Howard Zhang, PhD
• Wengen Chen, MD, PhD
• Charles White, MD
• Seth Kligerman, MD
• Shan Tan, PhD
• Jiahui Wang, PhD
2
NIH/NCI Grant R01 CA222216, R01 CA172638 and NIH/NCI Cancer Center Support Grant P30 CA008748
The ESTRO Falcon project team, Scott Kaylor of EduCase, Benjamin Nelms of Proknow for the multi-
delineator contour data presented in this work

3. Outline
1. Delineation Variability
1. Human Delineation Variability Quantification
2. Realistic Variability Simulation
3. Probabilistic U-Net
2. Radiomics
1. Lung Cancer Screening
2. Spiculation Quantification
3. Aggressive Lung ADC subtype prediction
3

4. Delineation Variability
• Over half a million patients are diagnosed with HNC each year world wide.
• RT is an important treatment
- but it requires manually intensive delineation of radiosensitive OARs.
• The OAR delineation variability is hypothesized to impact clinical outcomes.
4DVHs from various alternative OARs
1

5. Outline
1. Delineation Variability
1. Human Delineation Variability Quantification
2. Realistic Variability Simulation
3. Probabilistic U-Net
2. Radiomics
1. Lung Cancer Screening
2. Spiculation Quantification
3. Aggressive Lung ADC subtype prediction
5

6. Human Delineation Variability Quantification
and Simulation
• ESTRO Falcon contour workshop (EduCase)
• A HNC case, Larynx, 70 Gy and 35 fractions
• 14 independent manually delineated (MD) OAR structure sets (SS)
• BrainStem, Esophagus, OralCavity, Parotid_L, Parotid_R, SpinalCord, and Thyroid
• Consensus MD SS
• The simultaneous truth and performance level estimation (STAPLE)
6
1.1

7. Human Delineation Variability Quantification
and Simulation
A framework for radiation therapy variability analysis
7
RT plan
Structure Set
CT Image
Dose Distribution
Structure Sets
DV simulation
ASSD
GrowCut
RW
Other delineators
SV analysis
DV analysis
Geometric
Dosimetric
Variability analysis
Human DV
Simulated
DV
Consensus SS
1.1

8. Human Delineation Variability Simulation
(Methods)
• DV Simulation using auto-delineation (AD) methods (σ=2, 5, 10 mm)
• Average surface of standard deviation (ASSD): random perturbation
• GrowCut: cellular automata region growing
• Random walker (RW): probabilistic segmentation
8
1.1
Background 0
Foreground 1
Initial Binary Mask
Gaussian-Smoothed Mask
Gaussian Noises-Added Mask
Intensity
Spatial location
InsideOutside
σ"##$ = 2mm σ"##$ = 5mm σ"##$ = 8mm σ"##$ = 10mm
Choi et al., AAPM, 2019.

9. Human Delineation Variability Quantification
and Simulation (Methods)
•Geometric analysis
- Similarity: Dice coefficient (Volumetric, Surface)
- Distance: Hausdorff distance (HD), Actual Average Surface
Distance (AASD)
- Reference: STAPLE SS
•Dosimetric analysis
- Single dose distribution planned from a human SS
- DVH confidence bands (90%tile)
- !"#$%, !"$', !"(%, !)*
9
1.1

10. Human Delineation Variability Quantification and Simulation
(Results)
• DVH variability not predicted by geometric measures
• Large human variability
10
1.1
100%
50%
0%
100%
50%
0%
Human ASSD GrowCut RW
RightParotidLeftParotid

11. Dose Volume Coverage Map
11
Human ASSD
GrowCut RW
1.1

12. Outline
1. Delineation Variability
1. Human Delineation Variability Quantification (not realistic)
2. Realistic Variability Simulation
3. Probabilistic U-Net
2. Radiomics
1. Lung Cancer Screening
2. Spiculation Quantification
3. Aggressive Lung ADC subtype prediction
12

13. Human ASSD GrowCut RW
100%
50%
0%
Education
100%
50%
0%
Clinic1.2
13

14. ProKnow DataSet
14
1.2
Head_and_Neck_01 Total Available Jaccard
1 BrachialPlexus_R 354 206 0.52
2 Brainstem 512 512 0.87
4 BrachialPlexus_L 306 136 0.43
6 OpticChiasm 449 301 0.66
7 OpticNerve_R 238 238 0.76
8 OpticNerve_L 298 298 0.75
9 Cochlea_L 250 126 0.64
10 Cochlea_R 246 165 0.82
Head_and_Neck_02 Total Available Jaccard
3 Parotid_R 366 366 0.88
5 Parotid_L 935 654 0.84

15. • A HNC case, Nasopharynx , 70 Gy
• 5 manually delineated (MD) SSs
• BrainStem, Esophagus, OralCavity, Parotid_L, Parotid_R, and SpinalCord
• 409 plans for IMRT, VMAT, and Tomotherapy on various TPS
Eclipse, Monaco , Pinnacle, RayStation, Tomotherapy
• 3x5 Simulated DVs
• Setup error simulation using Radiation Therapy Robustness Analyzer
(RTRA)
• 1000 simulations: 3mm translation and 5 degree rotation
Plan Competition DataSet
15
1.2

16. Outline
1. Delineation Variability
1. Human Delineation Variability Quantification (not realistic)
2. Realistic Variability Simulation (enough data)
3. Probabilistic U-Net
2. Radiomics
1. Lung Cancer Screening
2. Spiculation Quantification
3. Aggressive Lung ADC subtype prediction
16

17. 17
1.3

18. A Probabilistic U-Net for Segmentation of Ambiguous Images
1.3
Kohl et al. NeurIPS 2018
18

19. Segmentations on different Variability levels (middle 5) and their occupancy map
• Implemented the model using PyTorch
• Model trained using TCIA LIDC dataset (about 2000 nodules with up to 4 radiologists delineations)
• Titan Xp, 12GB (1 week for training)
• Unstable to train, 2D segmentation, tumor
A Probabilistic U-Net (Results)
19
Variability
Low High
1.3

20. A Probabilistic U-Net for Segmentation of Ambiguous Images
1.3
Wasserstein distance?
2D → 2.5D → 3D 20

21. Outline
1. Delineation Variability
1. Human Delineation Variability Quantification (not realistic)
2. Realistic Variability Simulation (enough data)
3. Probabilistic U-Net (realistic but need improvement)
2. Radiomics
1. Lung Cancer Screening
2. Spiculation Quantification
3. Aggressive Lung ADC subtype prediction
21

22. Radiomics
22
¨ Controllable Feature Analysis
¨ More Interpretable
Lambin, et al. Eur J Cancer 2012
Aerts et al., Nature Communications, 2014
2

23. Radiomics Framework
• Automated Workflow (Python)
• Integrate all the radiomics components
• 3D Slicer, ITK (C++), Matlab, R, and Python
• Scalable: support multicore & grid computing
23
Image
Registration
• Multi-level rigid
• Deformable
• Pre/Post-CT
• MSE, MI
Segmentation
/Delineation
• Adaptive region growing
• Level set
• Grow cut
• Morphology filter
• Multi-modality
image segmentation
Feature
Extraction
• Intensity distribution
• Spatial variations
(texture)
• Geometric properties
• Jacobian feature
from DVF
• Feature selection
Predictive
Model
• ROC analyses
• Prediction models
• Validation
• Tumor response
• Recurrence
• Survival
Source codes: https://github.com/taznux/radiomics-tools
2

24. Radiomics Framework
• Automated Workflow (Python)
• Integrate all the radiomics components
• 3D Slicer, ITK (C++), Matlab, R, and Python
• Scalable: support multicore & grid computing
24
Image
Registration
• Multi-level rigid
• Deformable
• Pre/Post-CT
• MSE, MI
Tumor
Segmentation
• Adaptive region growing
• Level set
• Grow cut
• Morphology filter
• Multi-modality
image segmentation
Feature
Extraction
• Intensity distribution
• Spatial variations
(texture)
• Geometric properties
• Jacobian feature
from DVF
• Feature selection
Predictive
Model
• ROC analyses
• Prediction models
• Validation
• Tumor response
• Recurrence
• Survival
Source codes: https://github.com/taznux/radiomics-tools
Deep Learning Model
2

25. Radiomics Framework
• Automated Workflow (Python)
• Integrate all the radiomics components
• 3D Slicer, ITK (C++), Matlab, R, and Python
• Scalable: support multicore & grid computing
25
Image
Registration
• Multi-level rigid
• Deformable
• Pre/Post-CT
• MSE, MI
Tumor
Segmentation
• Adaptive region growing
• Level set
• Grow cut
• Morphology filter
• Multi-modality
image segmentation
Feature
Extraction
• Intensity distribution
• Spatial variations
(texture)
• Geometric properties
• Jacobian feature
from DVF
• Feature selection
Predictive
Model
• ROC analyses
• Prediction models
• Validation
• Tumor response
• Recurrence
• Survival
Source codes: https://github.com/taznux/radiomics-tools
Deep Learning Model
2

26. Radiomics Framework
• Automated Workflow (Python)
• Integrate all the radiomics components
• 3D Slicer, ITK (C++), Matlab, R, and Python
• Scalable: support multicore & grid computing
26
Image
Registration
• Multi-level rigid
• Deformable
• Pre/Post-CT
• MSE, MI
Tumor
Segmentation
• Adaptive region growing
• Level set
• Grow cut
• Morphology filter
• Multi-modality
image segmentation
Feature
Extraction
• Intensity distribution
• Spatial variations
(texture)
• Geometric properties
• Jacobian feature
from DVF
• Feature selection
Predictive
Model
• ROC analyses
• Prediction models
• Validation
• Tumor response
• Recurrence
• Survival
Source codes: https://github.com/taznux/radiomics-tools
Deep Learning Model
2

27. Radiomics Framework
• Automated Workflow (Python)
• Integrate all the radiomics components
• 3D Slicer, ITK (C++), Matlab, R, and Python
• Scalable: support multicore & grid computing
27
Image
Registration
• Deep Learning Optical fl
ow
• Action like flow
• Differential warp
• Dynamic filtering
• …
Tumor
Segmentation
• U-Net
• Prob. U-Net
• …
Feature
Extraction
• AlexNet
• ResNet
• VGG
• LeNet
• ….
Predictive
Model
• ROC analyses
• Prediction models
• Validation
• Tumor response
• Recurrence
• Survival
Source codes: https://github.com/taznux/radiomics-tools
2

28. Outline
1. Delineation Variability
1. Human Delineation Variability Quantification (not realistic)
2. Realistic Variability Simulation (enough data)
3. Probabilistic U-Net (realistic but need improvement)
2. Radiomics
1. Lung Cancer Screening
2. Spiculation Quantification
3. Aggressive Lung ADC subtype prediction
28

29. Lung Cancer Screening
29
¨ Early detection of lung cancer by LDCT can reduce mortality
¨ Known features correlated with PN malignancy
¤ Size, growth rate (Lung-RADS)
¤ Calcification, enhancement, solidity → texture features
¤ Boundary margins (spiculation, lobulation), attachment → shape and
appearance features
Malignant nodules Benign nodules
Size Total Malignancy
< 4mm 2038 0%
4-7 mm 1034 1%
8-20 mm 268 15%
> 20 mm 16 75%
2.1

30. ACR Lung-RADS
Category Baseline Screening Malignancy
1 No PNs; PNs with calcification
Negative
<1% chance of malignancy
2
Solid/part-solid: <6 mm
GGN: <20 mm
Benign appearance
<1% chance of malignancy
3
Solid: ≥6 to <8 mm
Part-solid: ≥6 mm with solid component <6 mm
GGN: ≥20 mm
Probably benign
1-2% chance of malignancy
4A
Solid: ≥8 to <15 mm
Part-solid: ≥8 mm with solid component ≥6 and <8 mm
Suspicious
5-15% chance of malignancy
4B
Solid: ≥15 mm
Part-solid: Solid component ≥8 mm
>15% chance of malignancy
4X
Category 3 or 4 PNs with suspicious features (e.g. enlarged lymph
nodes) or suspicious imaging findings (e.g. spiculation)
>15% chance of malignancy
30
ACR: American College of Radiology
Lung-RADS: Lung CT Screening Reporting and Data System
2.1

31. Lung Cancer Screening (Methodology)
• TCIA LIDC-IDRI public data set (n=1,010)
• Multi-institutional data
• 72 cases evaluated (31 benign and 41 malignant cases)
• Consensus contour
31
GLCM GLRM
Texture features Intensity features
2D
Shape features
3D
2.1

32. Lung Cancer Screening
(Methodology: SVM-LASSO Model )
32
SVM classification
Distinctive feature identification
Malignant?
Predicted malignancy
Feature extraction
Yes
10x10-foldCV
10-foldCV
LASSO feature selection
2.1

33. Lung Cancer Screening (Results)
• Size (BB_AP) : Highly correlated with the axial longest diameter and its
perpendicular diameter (r = 0.96, larger – more malignant)
• Texture (SD_IDM) : Tumor heterogeneity (smaller – more malignant)
33
2.1

34. Lung Screening (Results: Comparison)
Sensitivity Specificity Accuracy AUC
Lung-RADS
Clinical guideline
73.3% 70.4% 72.2% 0.74
Hawkins et al. (2016)
Radiomics – 23 features
51.7 % 92.9% 80.0% 0.83
Ma et al. (2016)
Radiomics – 583 features
80.0% 85.5% 82.7%
Buty et al. (2016)
DL – 400 SH and 4096 AlexNet features
82.4%
Kumar et al. (2015)
DL: 5000 features
79.1% 76.1% 77.5%
Proposed
Radiomics: two features (Size and Texture)
87.2% 81.2% 84.6% 0.89
34
DL: Deep Learning, SH: Spherical Harmonics
Choi et al., Medical Physics, 2018.
2.1

35. Lung Cancer Screening - Deep Learning
(Motivation)
• Data Science Bowl 2017 presented by
Booz|Allen|Hamilton & Kaggle
• Many deep learning methods were
proposed
• Top 10 teams: log loss 0.39~0.44
• Detection and Classification
• Top 99th: log loss 0.60
35
2.1

36. Lung Cancer Screening - Deep Learning
(Methodology: 3D Fully Convolutional Neural Network)
36
2.1

37. Lung Cancer Screening - Deep Learning
(Results)
• 3-fold cross-validation
• Nodule Detection: Sensitivity 95.1% with 5 false positives per scan
• Nodule Classification: Accuracy 67.4%
• Deep Learning: Feasible but not interpretable
Ranked 99th out of 1972 teams (Top 6%, Bronze medal)
37
Log loss
2.1

38. Outline
1. Delineation Variability
1. Human Delineation Variability Quantification (not realistic)
2. Realistic Variability Simulation (enough data)
3. Probabilistic U-Net (realistic but need improvement)
2. Radiomics
1. Lung Cancer Screening (not interpretable)
2. Spiculation Quantification
3. Aggressive Lung ADC subtype prediction
38

39. ACR Lung-RADS
Category Baseline Screening Malignancy
1 No PNs; PNs with calcification
Negative
<1% chance of malignancy
2
Solid/part-solid: <6 mm
GGN: <20 mm
Benign appearance
<1% chance of malignancy
3
Solid: ≥6 to <8 mm
Part-solid: ≥6 mm with solid component <6 mm
GGN: ≥20 mm
Probably benign
1-2% chance of malignancy
4A
Solid: ≥8 to <15 mm
Part-solid: ≥8 mm with solid component ≥6 and <8 mm
Suspicious
5-15% chance of malignancy
4B
Solid: ≥15 mm
Part-solid: Solid component ≥8 mm
>15% chance of malignancy
4X
Category 3 or 4 PNs with suspicious features (e.g. enlarged lymph
nodes) or suspicious imaging findings (e.g. spiculation)
>15% chance of malignancy
39
ACR: American College of Radiology
Lung-RADS: Lung CT Screening Reporting and Data System
2.2

40. Spiculation Quantification (Motivation)
• Blind Radiomics
• Semantic Features
• Semi-automatic Segmentation
- GrowCut and LevelSet
40
2.2
Radiologists spiculation score (RS) for different pulmonary nodules
1 2 3 4 5

41. Spiculation Quantification (Methodology)
41
!": = log
∑),+ , ([/(0"), /(0)), /(0+)])
∑),+ , ([0", 0), 0+])
Area Distortion Map
Nadeem et al., IEEE TVCG, 2017
Spherical Mapping
Eigenfunction
2.2

42. Spiculation Quantification: Height, Area Distortion
• Area distortion better than solid angle for non-conic spicules
• Height is medial axis length, can be perpendicular distance
• Width is measured at base, can be FWHM
!" =
∑% mean *+ % ∗ ℎ+ %
∑% ℎ+ %
!. =
∑% min *+ % ∗ ℎ+ %
∑% ℎ+ %
2.2
42Choi et al. in ShapeMI @ MICCAI 2018

43. Spiculation Quantification (Results)
- Higher correlation with the radiologist’s score (!": 0.67 AUC, P=4.29E-08, #=-0.33)
- Mean curvature method (!$: 0.59 AUC, P=0.068, #=0.22 and !%: 0.66 AUC, P=1.47E-07, #=0.29)
43
RS: radiologist's spiculation score, !": proposed spiculation score
RS 1 2 3 4 5
s1 -0.2 -0.7 -0.6 -1.1 -1.8
s2 0.1 0.2 0.4 0.6 1.6
sa 5.3 54.4 41.9 68.7 73.3
sb 0.3 0.6 0.7 0.9 0.8
2.2

44. Spiculation Quantification (Model Validation)
44
2.2
LIDC (N=883)
LUNGx (N=83)
PM subset
(N=72)
RM subset
(N=811)
Semi-auto segmentation
Spiculation Quantification
Model2Model1
Model evaluation
Model calibration
Model1’
10x10 fold
cross-validation
Independent
validation
ModelBuilding
Model2’
Model evaluation
Testing subset
(N=73)
ExternalValidation
Calibration
subset (N=10)
FeatureExtraction
Feature
extraction Strong-labeled Weak-labeled

45. Spiculation Quantification (Results: Comparison)
45
2.2

46. Outline
1. Delineation Variability
1. Human Delineation Variability Quantification (not realistic)
2. Realistic Variability Simulation (enough data)
3. Probabilistic U-Net (realistic but need improvement)
2. Radiomics
1. Lung Cancer Screening (not interpretable)
2. Spiculation Quantification (interpretable and weak-label data)
3. Aggressive Lung ADC subtype prediction
46

47. Aggressive Lung ADC Subtype Prediction (Motivation)
47
2.3
CT
MIP
PET/CT
Soild
CT PET/CT
Five classifications of lung ADC Travis et
al. JTO 2011
¨ Solid and MIP components: poor surgery/SBRT prognosis factor
¤ Benefit from lobectomy rather than limited resection
¨ Core biopsy (Leeman et al. IJROBP 2017)
¤ Minimally invasive, not routinely performed, sampling error (about 60%
agreement with pathology)
¨ Preoperative diagnostic CT and FDG PET/CT radiomics
¤ Non-invasive and routinely performed

48. Aggressive Lung ADC Subtype Prediction (Method & Results)
• Retrospectively enrolled 120 patients
• Stage I lung ADC, ≤2cm
• Preoperative diagnostic CT and FDG PET/CT
• Histopathologic endpoint
• Aggressiveness (Solid : 18 cases, MIP : 5 cases)
• 206 radiomic features & 14 clinical parameters
• SVM-LASSO model
48
Type Feature Sensitivity Specificity Accuracy AUC P
1 PET Mean of Cluster Shade 67.0±4.3% 84.7±0.9% 81.3±1.2% 0.76±0.02 0.00083
2 Clinical Age 18.1±6.6% 82.6±3.0% 70.0±3.1% 0.50±0.04 0.27
Performance of the SVM-LASSO model to predict aggressive lung ADC
Choi et al. Manuscript under review
Box plots of SUVmax (FDR q=0.0042) and PET Mean of
Cluster Shade (q=0.0021)
2.3

49. Summary
1. Delineation Variability
1. Human Delineation Variability Quantification (not realistic)
2. Realistic Variability Simulation (enough data)
3. Probabilistic U-Net (realistic but need improvement)
2. Radiomics
1. Lung Cancer Screening (not interpretable)
2. Spiculation Quantification (interpretable and weak-label data)
3. Aggressive Lung ADC subtype prediction (helpful for surgeons)
49

50. Short-term Future Works
• Human-Variability aware auto-delineation
• Variability quantification and simulation using generative models
• Develop interpretable radiomic features
• Improve spiculation quantification
• Multi-institution validation
• Integrate the radiomics framework into TPS
• Eclipse (C#), MIM (Python), Raystation (Python)
50

51. Long-term Future Works
• Comprehensive Framework for Cancer Imaging
• Multi-modal imaging
• Response prediction (Pre, Post)
• Longitudinal analysis of tumor change
• Shape analysis (e.g. Spiculation)
• Deep learning models
• Automation of Clinical Workflow
• Big Data Analytics: EMR, PACS, ROIS, Genomics, etc.
• Provide an informatics platform for comprehensive cancer therapy
51

52. Selected Publications
1. Wookjin Choi et al., “Radiomics analysis of pulmonary nodules in low-dose CT for early detection of lung cancer”, Medical
Physics, 2018
2. Wookjin Choi et al. “Technical Note: Identification of Normal Lung CT Texture Features Robust to Tumor Size for the
Prediction of Radiation-Induced Lung Disease”, International Journal of Medical Physics, Clinical Engineering and
Radiation Oncology, 2018
3. Sadegh Riyahi, Wookjin Choi, et al., “Quantifying local tumor morphological changes with Jacobian map for prediction of
pathologic tumor response to chemo-radiotherapy in locally advanced esophageal cancer”, Physics in Medicine and
Biology, 2018
4. Shan Tan, Laquan Li, Wookjin Choi, et al., “Adaptive region-growing with maximum curvature strategy for tumor
segmentation in 18F-FDG PET”, Physics in Medicine and Biology, 2017
5. Wookjin Choi et al., “Individually Optimized Contrast-Enhanced 4D-CT for Radiotherapy Simulation in Pancreatic Ductal
Adenocarcinoma”, Medical Physics, 2016
6. Wookjin Choi, Tae-Sun Choi, “Automated Pulmonary Nodule Detection based on Three-dimensional Shape-based Feature
Descriptor”, Computer Methods and Programs in Biomedicine, 2014
7. Wookjin Choi, Tae-Sun Choi, “Automated Pulmonary Nodule Detection System in Computed Tomography Images: A
Hierarchical Block Classification Approach”, Entropy, Vol. 15, No. 2, pp. 507-523, February 2013
8. Wookjin Choi, Tae-Sun Choi, “Genetic Programming-based feature transform and classification for the automatic detection
of pulmonary nodules on computed tomography images”, Information Sciences, 2012
52
Complete list of publications: https://scholar.google.com/citations?user=iHgsGLUAAAAJ

53. 53
Thank You!
Q & A
https://qradiomics.wordpress.com
E-mail: wchoi@vsu.edu