This document discusses artificial intelligence applications in radiation oncology. It begins with acknowledgements and then outlines several AI applications including radiomics tools for lung cancer screening, tumor response prediction, and predicting aggressive lung adenocarcinoma subtypes. It also discusses using AI for automatic tumor delineation and quantification of delineation variability as well as local tumor morphological changes prediction and metabolic tumor volume changes. The document provides details on methods and results for several of these AI applications in radiation oncology.
Call Girls Service Chennai Jiya 7001305949 Independent Escort Service Chennai
Ā
Artificial Intelligence in Radiation Oncology
1. Artificial Intelligence
in Radiation Oncology
Wookjin Choi, PhD
Assistant Professor of Radiation Oncology
Sidney Kimmel Medical College at Thomas Jefferson University
Wookjin.Choi@Jefferson.edu
Mar 11, 2022 @ Mayo Clinic
2. Acknowledgements
Memorial Sloan Kettering Cancer Center
ā¢ Wei Lu PhD
ā¢ Sadegh Riyahi, PhD
ā¢ Jung Hun Oh, PhD
ā¢ Saad Nadeem, PhD
ā¢ Eric Aliotta, PhD
ā¢ Joseph O. Deasy, PhD
ā¢ Andreas Rimner, MD
ā¢ Prasad Adusumilli, MD
Stony Brook University
ā¢ Allen Tannenbaum, PhD
University of Virginia School of Medicine
ā¢ Jeffrey Siebers, PhD
ā¢ Victor Gabriel Leandro Alves, PhD
University of Maryland School of Medicine
ā¢ Howard Zhang, PhD
ā¢ Wengen Chen, MD, PhD
ā¢ Charles White, MD
Thomas Jefferson University
ā¢ Yevgeniy Vinogradskiy, PhD
ā¢ Hamidreza Nourzadeh, PhD
ā¢ Adam P. Dicker, MD
2
NIH/NCI Grant R01 CA222216, R01 CA172638 and
NIH/NCI Cancer Center Support Grant P30 CA008748 and 5P30 CA056036
The ESTRO Falcon project team, Scott Kaylor of EduCase, Benjamin Nelms of Proknow for the multi-
delineator contour data presented in this work
3. AI in Radiation Oncology
3
Huynh et al. Nat Rev Clin Oncol 2020
4. 4
Netherton et al. Oncology 2021
Hype cycle for three major innovations
in radiation oncology
Automatable tasks in radiation oncology
for the modern clinic
5. Outline
ā¢ Radiomics - Decision Support Tools
- Lung Cancer Screening
- Tumor Response Prediction and Evaluation
- Aggressive Lung ADC subtype prediction
- Multimodal data: Pathology, Multiomics, etc.
ā¢ Auto Delineation and Variability Analysis
- Delineation Variability Quantification
- Dosimetric Consequences of Variabilities
- OARNet, Voxel2Mesh
5
6. Radiomics
6
ĀØ Controllable Feature Analysis
ĀØ More Interpretable
Lambin, et al. Eur J Cancer 2012
Aerts et al., Nature Communications, 2014
7. Radiomics Framework
7
Image
Registration
ā¢ Multi-level rigid
ā¢ Deformable
ā¢ Pre/Post-CT
ā¢ MSE, MI
Tumor
Segmentation
ā¢ Adaptive region growing
ā¢ Level set
ā¢ Grow cut
ā¢ Morphology filter
ā¢ Multi-modality
image segmentation
Feature
Extraction
ā¢ Intensity distribution
ā¢ Spatial variations
(texture)
ā¢ Geometric properties
ā¢ Jacobian feature
from DVF
ā¢ Feature selection
Predictive
Model
ā¢ ROC analyses
ā¢ Prediction models
ā¢ Validation
ā¢ Tumor response
ā¢ Recurrence
ā¢ Survival
Source codes: https://github.com/taznux/radiomics-tools
ā¢ Automated Workflow
- Integrate all the radiomics components
- 3D Slicer, ITK (C++), Matlab, R, and Python
- Scalable: support multicore & GPU computing
8. Radiomics Framework
8
Image
Registration
ā¢ Multi-level rigid
ā¢ Deformable
ā¢ Pre/Post-CT
ā¢ MSE, MI
Tumor
Segmentation
ā¢ Adaptive region growing
ā¢ Level set
ā¢ Grow cut
ā¢ Morphology filter
ā¢ Multi-modality
image segmentation
Feature
Extraction
ā¢ Intensity distribution
ā¢ Spatial variations
(texture)
ā¢ Geometric properties
ā¢ Jacobian feature
from DVF
ā¢ Feature selection
Predictive
Model
ā¢ ROC analyses
ā¢ Prediction models
ā¢ Validation
ā¢ Tumor response
ā¢ Recurrence
ā¢ Survival
Source codes: https://github.com/taznux/radiomics-tools
Deep Learning Model
ā¢ Automated Workflow
- Integrate all the radiomics components
- 3D Slicer, ITK (C++), Matlab, R, and Python
- Scalable: support multicore & GPU computing
9. Radiomics Framework
9
Image
Registration
ā¢ Multi-level rigid
ā¢ Deformable
ā¢ Pre/Post-CT
ā¢ MSE, MI
Tumor
Segmentation
ā¢ Adaptive region growing
ā¢ Level set
ā¢ Grow cut
ā¢ Morphology filter
ā¢ Multi-modality
image segmentation
Feature
Extraction
ā¢ Intensity distribution
ā¢ Spatial variations
(texture)
ā¢ Geometric properties
ā¢ Jacobian feature
from DVF
ā¢ Feature selection
Predictive
Model
ā¢ ROC analyses
ā¢ Prediction models
ā¢ Validation
ā¢ Tumor response
ā¢ Recurrence
ā¢ Survival
Source codes: https://github.com/taznux/radiomics-tools
Deep Learning Model
ā¢ Automated Workflow
- Integrate all the radiomics components
- 3D Slicer, ITK (C++), Matlab, R, and Python
- Scalable: support multicore & GPU computing
10. Radiomics Framework
10
Image
Registration
ā¢ Multi-level rigid
ā¢ Deformable
ā¢ Pre/Post-CT
ā¢ MSE, MI
Tumor
Segmentation
ā¢ Adaptive region growing
ā¢ Level set
ā¢ Grow cut
ā¢ Morphology filter
ā¢ Multi-modality
image segmentation
Feature
Extraction
ā¢ Intensity distribution
ā¢ Spatial variations
(texture)
ā¢ Geometric properties
ā¢ Jacobian feature
from DVF
ā¢ Feature selection
Predictive
Model
ā¢ ROC analyses
ā¢ Prediction models
ā¢ Validation
ā¢ Tumor response
ā¢ Recurrence
ā¢ Survival
Source codes: https://github.com/taznux/radiomics-tools
Deep Learning Model
ā¢ Automated Workflow
- Integrate all the radiomics components
- 3D Slicer, ITK (C++), Matlab, R, and Python
- Scalable: support multicore & GPU computing
11. Radiomics Framework
11
Image
Registration
ā¢ Multi-level rigid
ā¢ Deformable
ā¢ Pre/Post-CT
ā¢ MSE, MI
Tumor
Segmentation
ā¢ Adaptive region growing
ā¢ Level set
ā¢ Grow cut
ā¢ Morphology filter
ā¢ Multi-modality
image segmentation
Feature
Extraction
ā¢ Intensity distribution
ā¢ Spatial variations
(texture)
ā¢ Geometric properties
ā¢ Jacobian feature
from DVF
ā¢ Feature selection
Predictive
Model
ā¢ ROC analyses
ā¢ Prediction models
ā¢ Validation
ā¢ Tumor response
ā¢ Recurrence
ā¢ Survival
Source codes: https://github.com/taznux/radiomics-tools
Deep Learning Model
ā¢ Automated Workflow
- Integrate all the radiomics components
- 3D Slicer, ITK (C++), Matlab, R, and Python
- Scalable: support multicore & GPU computing
13. Lung Cancer Screening
13
ĀØ Early detection of lung cancer by LDCT can reduce mortality
ĀØ Known features correlated with PN malignancy
Ā¤ Size, growth rate (Lung-RADS)
Ā¤ Calcification, enhancement, solidity ā texture features
Ā¤ Boundary margins (spiculation, lobulation), attachment ā shape and
appearance features
Malignant nodules Benign nodules
Size Total Malignancy
< 4mm 2038 0%
4-7 mm 1034 1%
8-20 mm 268 15%
> 20 mm 16 75%
14. ACR Lung-RADS 1.0
Category Baseline Screening Malignancy
1 No PNs; PNs with calcification
Negative
<1% chance of malignancy
2
Solid/part-solid: <6 mm
GGN: <20 mm
Benign appearance
<1% chance of malignancy
3
Solid: ā„6 to <8 mm
Part-solid: ā„6 mm with solid component <6 mm
GGN: ā„20 mm
Probably benign
1-2% chance of malignancy
4A
Solid: ā„8 to <15 mm
Part-solid: ā„8 mm with solid component ā„6 and <8 mm
Suspicious
5-15% chance of malignancy
4B
Solid: ā„15 mm
Part-solid: Solid component ā„8 mm
>15% chance of malignancy
4X
Category 3 or 4 PNs with suspicious features (e.g., enlarged lymph
nodes) or suspicious imaging findings (e.g., spiculation)
>15% chance of malignancy
14
ACR: American College of Radiology
Lung-RADS: Lung CT Screening Reporting and Data System
15. Lung Cancer Screening (Methodology)
ā¢ TCIA LIDC-IDRI public data set (n=1,010)
- Multi-institutional data
- 72 cases evaluated (31 benign and 41 malignant cases)
ā¢ Consensus contour
15
GLCM GLRM
Texture features Intensity features
2D
Shape features
3D
16. Lung Cancer Screening (SVM-LASSO Model )
16
SVM classification
Distinctive feature identification
Malignant?
Predicted malignancy
Feature extraction
Yes
10x10-fold
CV
10-fold
CV
LASSO feature selection
ā¢ Size (BB_AP) : Highly correlated with the axial longest diameter
and its perpendicular diameter (r = 0.96, larger ā more
malignant)
ā¢ Texture (SD_IDM) : Tumor heterogeneity (smaller ā more
malignant)
17. Lung Screening (Results: Comparison)
Sensitivity Specificity Accuracy AUC
Lung-RADS
Clinical guideline
73.3% 70.4% 72.2% 0.74
Hawkins et al. (2016)
Radiomics ā 23 features
51.7 % 92.9% 80.0% 0.83
Ma et al. (2016)
Radiomics ā 583 features
80.0% 85.5% 82.7%
Buty et al. (2016)
DL ā 400 SH and 4096 AlexNet features
82.4%
Kumar et al. (2015)
DL: 5000 features
79.1% 76.1% 77.5%
Proposed
Radiomics: two features (Size and Texture)
87.2% 81.2% 84.6% 0.89
17
DL: Deep Learning, SH: Spherical Harmonics
Choi et al., Medical Physics, 2018.
18. ACR Lung-RADS 1.0
Category Baseline Screening Malignancy
1 No PNs; PNs with calcification
Negative
<1% chance of malignancy
2
Solid/part-solid: <6 mm
GGN: <20 mm
Benign appearance
<1% chance of malignancy
3
Solid: ā„6 to <8 mm
Part-solid: ā„6 mm with solid component <6 mm
GGN: ā„20 mm
Probably benign
1-2% chance of malignancy
4A
Solid: ā„8 to <15 mm
Part-solid: ā„8 mm with solid component ā„6 and <8 mm
Suspicious
5-15% chance of malignancy
4B
Solid: ā„15 mm
Part-solid: Solid component ā„8 mm
>15% chance of malignancy
4X
Category 3 or 4 PNs with suspicious features (e.g., enlarged lymph
nodes) or suspicious imaging findings (e.g., spiculation)
>15% chance of malignancy
18
ACR: American College of Radiology
Lung-RADS: Lung CT Screening Reporting and Data System
19. Spiculation Quantification (Motivation)
ā¢ Semantic Features
ā¢ Semi-automatic Segmentation
- GrowCut and LevelSet
19
Radiologists spiculation score (RS) for different pulmonary nodules
1 2 3 4 5
Choi et al. in CMPB 2021
24. Progression-free survival Prediction
after SBRT for early-stage NSCLC
24
Thor, Choi et al. ASTRO 2020
ā¢ 412 patients treated between 2006 and 2017
ā¢ PETs and CTs within three months prior to SBRT start.
ā¢ The median prescription dose was 50Gy in 5 fractions.
25. Progression-free survival Prediction (Results)
ā¢ PET entropy, CT number of peaks,
CT major axis, and gender.
ā¢ The most frequently selected model
included PET entropy and CT
number of peaks
- The c-index in the validation
subset was 0.77
- The prediction-stratified survival
indicated a clear separation
between the observed HR and
LR
- e.g. a PFS of 60% was observed
at 12 months in HR vs. 22
months in LR.
25
Thor, Choi et al. ASTRO 2020
26. Local tumor morphological changes
26
Jacobian Map
- Jacobian matrix: calculates rate of displacement change in each direction.
- Determinant indicates volumetric ratio of shrinkage/expansion.
012 3 = 4
012 3 > 1 volume expansion
012 3 = 1 no volume change
012 3 < 1 volume shrinkage
012 3 = 1.2 = 20% expansion
012 3 = 0.8 = 20% shrinkage (-20%)
Riyahi, Choi et al., PMB 2018
28. Local tumor morphological changes (Results)
Features P-value AUC Correlation to responders
Minimum Jacobian 0.009 0.98 -0.79
Median Jacobian 0.046 0.95 -0.72
The P-value, AUC and correlation to responders for all significant features in univariate analysis
28
Riyahi, Choi et al., PMB 2018
SVM-LASSO: AUC 0.91
29. Local Metabolic Tumor Volume Changes
29
Riyahi, Choi et al., DATRA@MICCAI 2018 AUC=0.81
30. Aggressive Lung ADC Subtype Prediction (Motivation)
30
CT
MIP
PET/CT
Soild
CT PET/CT
Five classifications of lung ADC Travis et
al. JTO 2011
ĀØ Solid and MIP components: poor surgery/SBRT prognosis factor
Ā¤ Benefit from lobectomy rather than limited resection
ĀØ Core biopsy (Leeman et al. IJROBP 2017)
Ā¤ Minimally invasive, not routinely performed, sampling error (about 60%
agreement with pathology)
ĀØ Preoperative diagnostic CT and FDG PET/CT radiomics
Ā¤ Non-invasive and routinely performed
31. Aggressive Lung ADC Subtype Prediction (Method & Results)
ā¢ Retrospectively enrolled 120 patients
- Stage I lung ADC, ā¤2cm
- Preoperative diagnostic CT and FDG PET/CT
ā¢ Histopathologic endpoint
- Aggressiveness (Solid : 18 cases, MIP : 5 cases)
ā¢ 206 radiomic features & 14 clinical parameters
ā¢ SVM-LASSO model
31
Performance of the SVM-LASSO model to predict aggressive lung ADC
Choi et al. Manuscript under review
Box plots of SUVmax (FDR q=0.004) and PET Mean of
Cluster Shade (q=0.002)
Feature Sensitivity Specificity PPV NPV Accuracy AUC
Conventional SUVmax 57.8Ā±4.6% 78.5Ā±1.4% 39.2Ā±2.3% 88.6Ā±1.1% 74.5Ā±1.4% 0.64Ā±0.01
SVM-LASSO PET Mean of Cluster Shade 67.4Ā±3.1% 86.0Ā±1.1% 53.7Ā±2.1% 91.7Ā±1.0% 82.4Ā±1.0% 0.78Ā±0.01
p-value SUVmax vs. SVM-LASSO 0.002 1e-5 7e-8 3e-5 5e-8 0.03
32. Unsupervised Learning of Deep Learned Features
from Breast Cancer Images
32
Lee, Choi et al. IEEE BIBE 2020
34. PathCNN: interpretable convolutional neural networks
for survival prediction and pathway analysis applied to glioblastoma
34
ā¢ CNNs have achieved great success
ā¢ A lack of interpretability remains a
key barrier
ā¢ Moreover, because biological array
data are generally represented in a
non-grid structured format
ā¢ PathCNN
An interpretable CNN model on
integrated multi-omics data using a
newly defined pathway image.
Oh, Choi et al. Bioinformatics 2021, joint first author Source code: https://github.com/mskspi/PathCNN.
35. PathCNN: interpretable CNNs (results)
35
Cancer PathCNN Logistic
regression SVM with RBF Neural network MiNet
GBM 0.755āĀ±ā0.009 0.668āĀ±ā0.039 0.685āĀ±ā0.037 0.692āĀ±ā0.030 0.690āĀ±ā0.032
LGG 0.877āĀ±ā0.007 0.816āĀ±ā0.036 0.884āĀ±ā0.017 0.791āĀ±ā0.031 0.854āĀ±ā0.027
LUAD 0.637āĀ±ā0.014 0.581āĀ±ā0.028 0.624āĀ±ā0.034 0.573āĀ±ā0.031 0.597āĀ±ā0.042
KIRC 0.709āĀ±ā0.009 0.654āĀ±ā0.034 0.684āĀ±ā0.027 0.702āĀ±ā0.028 0.659āĀ±ā0.030
Comparison of predictive performance with benchmark methods in terms of the area
under the curve (AUC: mean Ā± standard deviation) over 30 iterations of the 5-fold
cross validation
Note: AUCs for PathCNN were obtained with three principal components. Bold = Highest AUC for each dataset.
SVM, support vector machine; RBF, radial basis function; MiNet, Multi-omics Integrative Net; GBM, glioblastoma
multiforme; LGG, low-grade glioma; LUAD, lung adenocarcinoma; KIRC, kidney cancer.
Oh, Choi et al. Bioinformatics 2021, joint first author Source code: https://github.com/mskspi/PathCNN.
36. 36
A matrix of adjusted P-values. The row represents the 146 KEGG pathways ordered on pathway images.
The columns represent the first two principal components of each omics type. The red color indicates
key pathways with adjusted P-values < 0.001
37. Outline
ā¢ Radiomics - Decision Support Tools
- Lung Cancer Screening
- Tumor Response Prediction and Evaluation
- Aggressive Lung ADC subtype prediction
- Multimodal data: Pathology, Multiomics, etc.
ā¢ Auto Delineation and Variability Analysis
- Delineation Variability Quantification
- Dosimetric Consequences of Variabilities
- OARNet, Voxel2Mesh
37
38. Delineation Variability Quantification and Simulation
A framework for radiation therapy variability analysis
38
RT plan
Structure Set
CT Image
Dose Distribution
Structure Sets
DV simulation
ASSD
GrowCut
RW
Other delineators
SV analysis
DV analysis
Geometric
Dosimetric
Variability analysis
Human DV
Simulated
DV
Consensus SS
OARNet
Choi et al., AAPM, 2019.
39. Delineation Variability Quantification and
Simulation
ā¢ ESTRO Falcon contour workshop (EduCase)
- A HNC case, Larynx, 70 Gy and 35 fractions
- 14 independent manually delineated (MD) OAR structure sets (SS)
- BrainStem, Esophagus, OralCavity, Parotid_L, Parotid_R, SpinalCord,
and Thyroid
ā¢ Consensus MD SS
- The simultaneous truth and performance level estimation (STAPLE)
39
Choi, Nourzadeh et al., AAPM, 2019.
40. Delineation Variability Quantification and
Simulation (Methods)
ā¢ Geometric analysis
- Similarity: Dice coefficient (Volumetric, Surface)
- Distance: Hausdorff distance (HD), Actual Average Surface
Distance (AASD)
- Reference: STAPLE SS
ā¢ Dosimetric analysis
- Single dose distribution planned from a human SS
- DVH confidence bands (90%tile)
- !!"#$, !!#%, !!&$, !'(
40
Choi, Nourzadeh et al., AAPM, 2019.
41. Delineation Variability Quantification and Simulation (Results)
ā¢ DVH variability not predicted by geometric measures
ā¢ Large human variability
41
100%
50%
0%
100%
50%
0%
Human ASSD GrowCut RW
Right
Parotid
Left
Parotid
Choi, Nourzadeh et al., AAPM, 2019.
49. Interpretable Radiomics Toolkit
End-to-End Deep Learning Model for Malignancy Prediction
49
Input Ground Truth Voxel2Mesh
Choi et al., Manuscript in Preparation
Network AUC Accuracy Sensitivity Specificity F1
LIDC-PM Mesh Only 0.937 83.33 77.78 88.89 82.35
Mesh+Encoder 0.903 88.89 91.67 86.11 89.19
LUNGx Mesh Only 0.711 63.33 73.33 53.33 66.67
Mesh+Encoder 0.687 53.3 83.3 23.33 64.11
50. Summary
ā¢ Radiomics - Decision Support Tools
- Lung Cancer Screening
- Tumor Response Prediction and Evaluation
- Aggressive Lung ADC subtype prediction
- Multimodal data: Pathology, Multiomics, etc.
ā¢ Auto Delineation and Variability Analysis
- Delineation Variability Quantification
- Dosimetric Consequences of Variabilities
50
51. Short-term Future Works
ā¢ Develop interpretable radiomic features
- Improve spiculation quantification and multi-institution validation
- Multimodal data integration
ā¢ Human-Variability aware auto-delineation
- Variability quantification and simulation using generative models
- AI-guided interactive delineation editing
ā¢ Integrate the radiomics framework into TPS
- Eclipse (C#) and MIM (Python)
51
52. Long-term Future Works
ā¢ Comprehensive Framework for Cancer Imaging
- Multi-modal imaging
- Response prediction and evaluation (Pre, Mid, and Post)
- Longitudinal analysis of tumor change during treatment (MRgRT)
- Shape analysis (e.g., Spiculation)
- Deep learning models
ā¢ Automation of Clinical Workflow
- Big Data Analytics: EMR, PACS, ROIS, Genomics, etc.
- Provide an informatics platform for comprehensive cancer therapy
53
53. Selected Publications
1. Jung Hun Oh*, Wookjin Choi* et al., āPathCNN: interpretable convolutional neural networks for survival
prediction and pathway analysis applied to glioblastomaā, Bioinformatics, 2021, *joint first author
2. Wookjin Choi et al., ā Reproducible and Interpretable Spiculation Quantification for Lung Cancer Screeningā,
Computer Methods and Programs in Biomedicineā, 2021
3. Noemi Garau, Wookjin Choi, et al., ā External validation of radiomics-based predictive models in low-dose CT
screening for early lung cancer diagnosisā, Medical Physics, 2020
4. Jiahui Wang, Wookjin Choi et al., āPrediction of anal cancer recurrence after chemoradiotherapy using
quantitative image features extracted from serial 18F-FDG PET/CTā, Frontiers in oncology, 2019
5. Wookjin Choi et al., āRadiomics analysis of pulmonary nodules in low-dose CT for early detection of lung cancerā,
Medical Physics, 2018
6. Sadegh Riyahi, Wookjin Choi, et al., āQuantifying local tumor morphological changes with Jacobian map for
prediction of pathologic tumor response to chemo-radiotherapy in locally advanced esophageal cancerā, Physics
in Medicine and Biology, 2018
7. Shan Tan, Laquan Li, Wookjin Choi, et al., āAdaptive region-growing with maximum curvature strategy for tumor
segmentation in 18F-FDG PETā, Physics in Medicine and Biology, 2017
8. Wookjin Choi et al., āIndividually Optimized Contrast-Enhanced 4D-CT for Radiotherapy Simulation in Pancreatic
Ductal Adenocarcinomaā, Medical Physics, 2016
9. Wookjin Choi, Tae-Sun Choi, āAutomated Pulmonary Nodule Detection based on Three-dimensional Shape-based
Feature Descriptorā, Computer Methods and Programs in Biomedicine, 2014
54
Complete list of publications: https://scholar.google.com/citations?user=iHgsGLUAAAAJ
54. Post-Doctoral Research Fellow
Developing Interpretable Predictive Models for Radiation
Therapy
ā¢ PI: Wookjin Choi, PhD - Wookjin.Choi@Jefferson.edu
ā¢ 2 Years
ā¢ Machine Learning/Deep Learning: Radiomics (PET/CT & MR) and Bioinformatics
ā¢ Computational Medical Physics: Development of Predictive Models and
Automated Workflows, and Improve Clinical Workflow
ā¢ Internal or Extramural Research Funding Opportunities
Qualifications
ā¢ Ph.D. in Computer Science, Electrical Engineering, Medical Physics, or related
field required