Artificial Intelligence in Radiation Oncology
•
2 likes•1,766 views
Invited talk Dec 30, 2021 @ YUHS
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Artificial Intelligence in Radiation Oncology
Similar to Artificial Intelligence in Radiation Oncology (20)
More from Wookjin Choi
More from Wookjin Choi (19)
Recently uploaded
Recently uploaded (20)
Artificial Intelligence in Radiation Oncology
- 1. Dec 30, 2021 @ Yonsei University College of Medicine Artificial Intelligence in Radiation Oncology Wookjin Choi, PhD Assistant Professor of Radiation Oncology Sidney Kimmel Medical College at Thomas Jefferson University Wookjin.Choi@Jefferson.edu
- 2. Acknowledgements Memorial Sloan Kettering Cancer Center • Wei Lu PhD • Sadegh Riyahi, PhD • Jung Hun Oh, PhD • Saad Nadeem, PhD • Joseph O. Deasy, PhD • Andreas Rimner, MD • Prasad Adusumilli, MD • Chia-ju Liu, MD • Wolfgang Weber, MD Stony Brook University • Allen Tannenbaum, PhD University of Virginia School of Medicine • Jeffrey Siebers, PhD • Victor Gabriel Leandro Alves, PhD • Hamidreza Nourzadeh, PhD • Eric Aliotta, PhD University of Maryland School of Medicine • Howard Zhang, PhD • Wengen Chen, MD, PhD • Charles White, MD 2 NIH/NCI Grant R01 CA222216, R01 CA172638 and NIH/NCI Cancer Center Support Grant P30 CA008748 The ESTRO Falcon project team, Scott Kaylor of EduCase, Benjamin Nelms of Proknow for the multi- delineator contour data presented in this work
- 3. Acknowledgements Memorial Sloan Kettering Cancer Center • Wei Lu PhD • Sadegh Riyahi, PhD • Jung Hun Oh, PhD • Saad Nadeem, PhD • Eric Aliotta, PhD • Joseph O. Deasy, PhD • Andreas Rimner, MD • Prasad Adusumilli, MD Stony Brook University • Allen Tannenbaum, PhD University of Virginia School of Medicine • Jeffrey Siebers, PhD • Victor Gabriel Leandro Alves, PhD University of Maryland School of Medicine • Howard Zhang, PhD • Wengen Chen, MD, PhD • Charles White, MD Thomas Jefferson University • Hamidreza Nourzadeh, PhD 3 NIH/NCI Grant R01 CA222216, R01 CA172638 and NIH/NCI Cancer Center Support Grant P30 CA008748 The ESTRO Falcon project team, Scott Kaylor of EduCase, Benjamin Nelms of Proknow for the multi- delineator contour data presented in this work
- 4. AI in Radiation Oncology 4 Huynh et al. Nat Rev Clin Oncol 2020
- 5. 5 Netherton et al. Oncology 2021 Hype cycle for three major innovations in radiation oncology Automatable tasks in radiation oncology for the modern clinic
- 6. Outline Radiomics - Decision Support Tools • Lung Cancer Screening • Tumor Response Prediction and Evaluation • Aggressive Lung ADC subtype prediction • Multimodal data: Pathology, Multiomics, etc. Auto Delineation and Variability Analysis • Delineation Variability Quantification • Dosimetric Consequences of Variabilities • OARNet, Probabilistic U-Net 6
- 7. Radiomics 7 Controllable Feature Analysis More Interpretable Lambin, et al. Eur J Cancer 2012 Aerts et al., Nature Communications, 2014
- 8. Radiomics Framework 8 Image Registration • Multi-level rigid • Deformable • Pre/Post-CT • MSE, MI Tumor Segmentation • Adaptive region growing • Level set • Grow cut • Morphology filter • Multi-modality image segmentation Feature Extraction • Intensity distribution • Spatial variations (texture) • Geometric properties • Jacobian feature from DVF • Feature selection Predictive Model • ROC analyses • Prediction models • Validation • Tumor response • Recurrence • Survival Source codes: https://github.com/taznux/radiomics-tools • Automated Workflow (Python) - Integrate all the radiomics components - 3D Slicer, ITK (C++), Matlab, R, and Python - Scalable: support multicore & GPU computing
- 9. Radiomics Framework 9 Image Registration • Multi-level rigid • Deformable • Pre/Post-CT • MSE, MI Tumor Segmentation • Adaptive region growing • Level set • Grow cut • Morphology filter • Multi-modality image segmentation Feature Extraction • Intensity distribution • Spatial variations (texture) • Geometric properties • Jacobian feature from DVF • Feature selection Predictive Model • ROC analyses • Prediction models • Validation • Tumor response • Recurrence • Survival Source codes: https://github.com/taznux/radiomics-tools Deep Learning Model • Automated Workflow (Python) • Integrate all the radiomics components • 3D Slicer, ITK (C++), Matlab, R, and Python • Scalable: support multicore & GPU computing
- 10. Radiomics Framework 10 Image Registration • Multi-level rigid • Deformable • Pre/Post-CT • MSE, MI Tumor Segmentation • Adaptive region growing • Level set • Grow cut • Morphology filter • Multi-modality image segmentation Feature Extraction • Intensity distribution • Spatial variations (texture) • Geometric properties • Jacobian feature from DVF • Feature selection Predictive Model • ROC analyses • Prediction models • Validation • Tumor response • Recurrence • Survival Source codes: https://github.com/taznux/radiomics-tools Deep Learning Model • Automated Workflow (Python) • Integrate all the radiomics components • 3D Slicer, ITK (C++), Matlab, R, and Python • Scalable: support multicore & GPU computing
- 11. Radiomics Framework 11 Image Registration • Multi-level rigid • Deformable • Pre/Post-CT • MSE, MI Tumor Segmentation • Adaptive region growing • Level set • Grow cut • Morphology filter • Multi-modality image segmentation Feature Extraction • Intensity distribution • Spatial variations (texture) • Geometric properties • Jacobian feature from DVF • Feature selection Predictive Model • ROC analyses • Prediction models • Validation • Tumor response • Recurrence • Survival Source codes: https://github.com/taznux/radiomics-tools Deep Learning Model • Automated Workflow (Python) • Integrate all the radiomics components • 3D Slicer, ITK (C++), Matlab, R, and Python • Scalable: support multicore & GPU computing
- 12. Radiomics Framework 12 Image Registration • Multi-level rigid • Deformable • Pre/Post-CT • MSE, MI Tumor Segmentation • Adaptive region growing • Level set • Grow cut • Morphology filter • Multi-modality image segmentation Feature Extraction • Intensity distribution • Spatial variations (texture) • Geometric properties • Jacobian feature from DVF • Feature selection Predictive Model • ROC analyses • Prediction models • Validation • Tumor response • Recurrence • Survival Source codes: https://github.com/taznux/radiomics-tools Deep Learning Model • Automated Workflow (Python) • Integrate all the radiomics components • 3D Slicer, ITK (C++), Matlab, R, and Python • Scalable: support multicore & GPU computing
- 13. Radiomics Framework 13 Image Registration • Deep Learning Optical fl ow • Action like flow • Differential warp • Dynamic filtering • … Tumor Segmentation • U-Net • Prob. U-Net • UANet • … Feature Extraction • AlexNet • ResNet • VGG • LeNet • …. Predictive Model • ROC analyses • Prediction models • Validation • Tumor response • Recurrence • Survival • Automated Workflow (Python) • Integrate all the radiomics components • 3D Slicer, ITK (C++), Matlab, R, and Python • Scalable: support multicore & GPU computing
- 14. Lung Cancer Screening 14 Early detection of lung cancer by LDCT can reduce mortality Known features correlated with PN malignancy Size, growth rate (Lung-RADS) Calcification, enhancement, solidity → texture features Boundary margins (spiculation, lobulation), attachment → shape and appearance features Malignant nodules Benign nodules Size Total Malignancy < 4mm 2038 0% 4-7 mm 1034 1% 8-20 mm 268 15% > 20 mm 16 75%
- 15. ACR Lung-RADS Category Baseline Screening Malignancy 1 No PNs; PNs with calcification Negative <1% chance of malignancy 2 Solid/part-solid: <6 mm GGN: <20 mm Benign appearance <1% chance of malignancy 3 Solid: ≥6 to <8 mm Part-solid: ≥6 mm with solid component <6 mm GGN: ≥20 mm Probably benign 1-2% chance of malignancy 4A Solid: ≥8 to <15 mm Part-solid: ≥8 mm with solid component ≥6 and <8 mm Suspicious 5-15% chance of malignancy 4B Solid: ≥15 mm Part-solid: Solid component ≥8 mm >15% chance of malignancy 4X Category 3 or 4 PNs with suspicious features (e.g. enlarged lymph nodes) or suspicious imaging findings (e.g. spiculation) >15% chance of malignancy 15 ACR: American College of Radiology Lung-RADS: Lung CT Screening Reporting and Data System
- 16. Lung Cancer Screening (Methodology) • TCIA LIDC-IDRI public data set (n=1,010) • Multi-institutional data • 72 cases evaluated (31 benign and 41 malignant cases) • Consensus contour 16 GLCM GLRM Texture features Intensity features 2D Shape features 3D
- 17. Lung Cancer Screening (SVM-LASSO Model ) 17 SVM classification Distinctive feature identification Malignant? Predicted malignancy Feature extraction Yes 10x10-fold CV 10-fold CV LASSO feature selection • Size (BB_AP) : Highly correlated with the axial longest diameter and its perpendicular diameter (r = 0.96, larger – more malignant) • Texture (SD_IDM) : Tumor heterogeneity (smaller – more malignant)
- 18. Lung Screening (Results: Comparison) Sensitivity Specificity Accuracy AUC Lung-RADS Clinical guideline 73.3% 70.4% 72.2% 0.74 Hawkins et al. (2016) Radiomics – 23 features 51.7 % 92.9% 80.0% 0.83 Ma et al. (2016) Radiomics – 583 features 80.0% 85.5% 82.7% Buty et al. (2016) DL – 400 SH and 4096 AlexNet features 82.4% Kumar et al. (2015) DL: 5000 features 79.1% 76.1% 77.5% Proposed Radiomics: two features (Size and Texture) 87.2% 81.2% 84.6% 0.89 18 DL: Deep Learning, SH: Spherical Harmonics Choi et al., Medical Physics, 2018.
- 19. ACR Lung-RADS Category Baseline Screening Malignancy 1 No PNs; PNs with calcification Negative <1% chance of malignancy 2 Solid/part-solid: <6 mm GGN: <20 mm Benign appearance <1% chance of malignancy 3 Solid: ≥6 to <8 mm Part-solid: ≥6 mm with solid component <6 mm GGN: ≥20 mm Probably benign 1-2% chance of malignancy 4A Solid: ≥8 to <15 mm Part-solid: ≥8 mm with solid component ≥6 and <8 mm Suspicious 5-15% chance of malignancy 4B Solid: ≥15 mm Part-solid: Solid component ≥8 mm >15% chance of malignancy 4X Category 3 or 4 PNs with suspicious features (e.g. enlarged lymph nodes) or suspicious imaging findings (e.g. spiculation) >15% chance of malignancy 19 ACR: American College of Radiology Lung-RADS: Lung CT Screening Reporting and Data System
- 20. Spiculation Quantification (Motivation) • Blind Radiomics • Semantic Features • Semi-automatic Segmentation - GrowCut and LevelSet 20 Radiologists spiculation score (RS) for different pulmonary nodules 1 2 3 4 5 Choi et al. in CMPB 2021
- 21. Spiculation Quantification (Methodology) 21 𝜖𝑖: = log 𝑗,𝑘 𝐴 ([𝜙(𝑣𝑖), 𝜙(𝑣𝑗), 𝜙(𝑣𝑘)]) 𝑗,𝑘 𝐴 ([𝑣𝑖, 𝑣𝑗, 𝑣𝑘]) Area Distortion Map Spherical Mapping Eigenfunction
- 22. Spiculation Quantification (Results) 22 RS 1 2 3 4 5 s1 -0.2 -0.7 -0.6 -1.1 -1.8 s2 0.1 0.2 0.4 0.6 1.6 Ns 0 1 4 8 14 Nl 2 0 3 1 1 Na 1 14 4 0 1 Spiculation quantification and attachment detection via area distortion metric. The results of spiculation detection (red line: baseline of peak, black line: medial axis of peak, red X: spiculation, black X: lobulation, yellow X: attached peaks). Radiologist’s spiculation score (RS), and the proposed interpretable spiculation features ( s1 : spiculation score, s2 : spiculation score, Ns : no. of spiculation, Nl : no. of lobulation and Na : no. of attached peaks) are shown below.
- 23. Spiculation Quantification (Results: Comparison) 23 Choi et al. in CMPB 2021
- 24. Progression-free survival Prediction after SBRT for early-stage NSCLC 24 Thor, Choi et al. ASTRO 2020 • 412 patients treated between 2006 and 2017 • PETs and CTs within three months prior to SBRT start. • The median prescription dose was 50Gy in 5 fractions.
- 25. Progression-free survival Prediction (Results) • PET entropy, CT number of peaks, CT major axis, and gender. • The most frequently selected model included PET entropy and CT number of peaks - The c-index in the validation subset was 0.77 - The prediction-stratified survival indicated a clear separation between the observed HR and LR - e.g. a PFS of 60% was observed at 12 months in HR vs. 22 months in LR. 25 Thor, Choi et al. ASTRO 2020
- 26. Local tumor morphological changes 26 Jacobian Map - Jacobian matrix: calculates rate of displacement change in each direction. - Determinant indicates volumetric ratio of shrinkage/expansion. 𝐷𝑒𝑡 𝐽 = 𝐷𝑒𝑡 𝐽 > 1 volume expansion 𝐷𝑒𝑡 𝐽 = 1 no volume change 𝐷𝑒𝑡 𝐽 < 1 volume shrinkage 𝐷𝑒𝑡 𝐽 = 1.2 = 20% expansion 𝐷𝑒𝑡 𝐽 = 0.8 = 20% shrinkage (-20%) Riyahi, Choi et al., PMB 2018
- 27. Local tumor morphological changes (Results) 27
- 28. Local tumor morphological changes (Results) Features P-value AUC Correlation to responders Minimum Jacobian 0.009 0.98 -0.79 Median Jacobian 0.046 0.95 -0.72 The P-value, AUC and correlation to responders for all significant features in univariate analysis 28 Riyahi, Choi et al., PMB 2018 SVM-LASSO: AUC 0.91
- 29. Aggressive Lung ADC Subtype Prediction (Motivation) 29 CT MIP PET/CT Soild CT PET/CT Five classifications of lung ADC Travis et al. JTO 2011 Solid and MIP components: poor surgery/SBRT prognosis factor Benefit from lobectomy rather than limited resection Core biopsy (Leeman et al. IJROBP 2017) Minimally invasive, not routinely performed, sampling error (about 60% agreement with pathology) Preoperative diagnostic CT and FDG PET/CT radiomics Non-invasive and routinely performed
- 30. Aggressive Lung ADC Subtype Prediction (Method & Results) • Retrospectively enrolled 120 patients - Stage I lung ADC, ≤2cm - Preoperative diagnostic CT and FDG PET/CT • Histopathologic endpoint - Aggressiveness (Solid : 18 cases, MIP : 5 cases) • 206 radiomic features & 14 clinical parameters • SVM-LASSO model 30 Performance of the SVM-LASSO model to predict aggressive lung ADC Choi et al. Manuscript under review Box plots of SUVmax (FDR q=0.004) and PET Mean of Cluster Shade (q=0.002) Feature Sensitivity Specificity PPV NPV Accuracy AUC Conventional SUVmax 57.8±4.6% 78.5±1.4% 39.2±2.3% 88.6±1.1% 74.5±1.4% 0.64±0.01 SVM-LASSO PET Mean of Cluster Shade 67.4±3.1% 86.0±1.1% 53.7±2.1% 91.7±1.0% 82.4±1.0% 0.78±0.01 p-value SUVmax vs. SVM-LASSO 0.002 1e-5 7e-8 3e-5 5e-8 0.03
- 31. Unsupervised Learning of Deep Learned Features from Breast Cancer Images 31 Lee, Choi et al. IEEE BIBE 2020
- 32. Unsupervised Learning of Deep Learned Features 32 Slide name Silhouette optimal number Cluster set Accuracy F1-score TCGA-A7-A0DA 29 [25, 22, 6, 2, 24, 14, 0, 20, 10] 0.8829 0.8929 TCGA-A2-A0YM 20 [7, 6, 1, 5] 0.8360 0.8863 TCGA-A2-A3XT 19 [13, 10, 5, 1, 2] 0.9316 0.9514 TCGA-BH-A0BG 8 [1, 5] 0.7828 0.6857 TCGA-E2-A1LS 26 [18, 5] 0.8495 0.8680 TCGA-OL-A66I 25 [7, 18, 12, 20, 1, 0] 0.7761 0.7091 TCGA-C8-A26Y 21 [9, 18, 12, 16] 0.8594 0.8122 Lee, Choi et al. IEEE BIBE 2020
- 33. PathCNN: interpretable convolutional neural networks for survival prediction and pathway analysis applied to glioblastoma 33 Oh, Choi et al. Bioinformatics 2021, joint first author Source code: https://github.com/mskspi/PathCNN. • CNNs have achieved great success • A lack of interpretability remains a key barrier • Moreover, because biological array data are generally represented in a non-grid structured format • PathCNN An interpretable CNN model on integrated multi-omics data using a newly defined pathway image.
- 34. PathCNN: interpretable CNNs (results) 34 Cancer PathCNN Logistic regression SVM with RBF Neural network MiNet GBM 0.755 ± 0.009 0.668 ± 0.039 0.685 ± 0.037 0.692 ± 0.030 0.690 ± 0.032 LGG 0.877 ± 0.007 0.816 ± 0.036 0.884 ± 0.017 0.791 ± 0.031 0.854 ± 0.027 LUAD 0.637 ± 0.014 0.581 ± 0.028 0.624 ± 0.034 0.573 ± 0.031 0.597 ± 0.042 KIRC 0.709 ± 0.009 0.654 ± 0.034 0.684 ± 0.027 0.702 ± 0.028 0.659 ± 0.030 Comparison of predictive performance with benchmark methods in terms of the area under the curve (AUC: mean ± standard deviation) over 30 iterations of the 5-fold cross validation Note: AUCs for PathCNN were obtained with three principal components. Bold = Highest AUC for each dataset. SVM, support vector machine; RBF, radial basis function; MiNet, Multi-omics Integrative Net; GBM, glioblastoma multiforme; LGG, low-grade glioma; LUAD, lung adenocarcinoma; KIRC, kidney cancer. Oh, Choi et al. Bioinformatics 2021, joint first author Source code: https://github.com/mskspi/PathCNN.
- 35. 35 A matrix of adjusted P-values.The row represents the 146 KEGG pathways ordered on pathway images. The columns represent the first two principal components of each omics type.The red color indicates key pathways with adjusted P-values < 0.001
- 36. Outline Radiomics - Decision Support Tools • Lung Cancer Screening • Tumor Response Prediction and Evaluation • Aggressive Lung ADC subtype prediction • Multimodal data: Pathology, Multiomics, etc. Auto Delineation and Variability Analysis • Delineation Variability Quantification • Dosimetric Consequences of Variabilities • OARNet, Probabilistic U-Net 36
- 37. Delineation Variability Quantification and Simulation A framework for radiation therapy variability analysis 37 RT plan Structure Set CT Image Dose Distribution Structure Sets DV simulation ASSD GrowCut RW Other delineators SV analysis DV analysis Geometric Dosimetric Variability analysis Human DV Simulated DV Consensus SS OARNet Choi et al., AAPM, 2019.
- 38. Delineation Variability Quantification and Simulation • ESTRO Falcon contour workshop (EduCase) - A HNC case, Larynx, 70 Gy and 35 fractions - 14 independent manually delineated (MD) OAR structure sets (SS) - BrainStem, Esophagus, OralCavity, Parotid_L, Parotid_R, SpinalCord, and Thyroid • Consensus MD SS - The simultaneous truth and performance level estimation (STAPLE) 38 Choi, Nourzadeh et al., AAPM, 2019.
- 39. Delineation Variability Quantification and Simulation (Methods) •Geometric analysis - Similarity: Dice coefficient (Volumetric, Surface) - Distance: Hausdorff distance (HD), Actual Average Surface Distance (AASD) - Reference: STAPLE SS •Dosimetric analysis - Single dose distribution planned from a human SS - DVH confidence bands (90%tile) - 𝐷mean, 𝐷max, 𝐷min, 𝐷50 39 Choi, Nourzadeh et al., AAPM, 2019.
- 40. Delineation Variability Quantification and Simulation (Results) • DVH variability not predicted by geometric measures • Large human variability 40 100% 50% 0% 100% 50% 0% Human ASSD GrowCut RW Right Parotid Left Parotid Choi, Nourzadeh et al., AAPM, 2019.
- 41. Human ASSD GrowCut RW 100% 50% 0% Education 100% 50% 0% Clinic 41
- 42. Dose Volume Coverage Map (DVCM) 42 Human ASSD GrowCut RW
- 43. • Plan Competition Data Set: A HNC case, Nasopharynx , 70 Gy • PV: 409 plans for IMRT, VMAT, and Tomotherapy on various TPS Eclipse, Monaco , Pinnacle, RayStation, Tomotherapy • SV: Setup error simulation using Radiation Therapy Robustness Analyzer (RTRA) • 1000 simulations: 3mm translation and 5-degree rotation • DV: 5 manually delineated (MD) SSs • BrainStem, Esophagus, OralCavity, Parotid_L, Parotid_R, and SpinalCord • 3x5 Simulated DVs using Radiation Therapy Variability Analyzer (RTVA) Variability Analysis Plan Variability, Setup Variability, Delineation Variability 43 Choi, Nourzadeh et al., AAPM, 2020.
- 44. Dose Volume Coverage Map (DVCM) 44 Plan Variability Setup Variability Delineation Variability Choi, Nourzadeh et al., AAPM, 2020.
- 45. Plan Competition DVCM Analysis 45 Fractional Volume Dose OAR Dose Constraint Prob Threshold 5%
- 46. Plan Competition DVCM Analysis 46 TV SV×PV SV×DV PV×DV SV PV DV Average BrainStem 0.15 0.11 0.10 0.07 0.03 0.01 0.02 0.07 >0.95 Chiasm 0.68 0.00 0.65 0.56 0.00 0.00 0.49 0.34 Eye_L 0.07 0.05 0.04 0.01 0.02 0.01 0.01 0.03 Eye_R 0.02 0.01 0.01 0.01 0.00 0.00 0.01 0.01 >0.5 Lens_L 1.00 1.00 1.00 1.00 0.99 1.00 0.01 0.86 Lens_R 1.00 1.00 1.00 1.00 1.00 1.00 0.01 0.86 >0.3 Mandible 0.04 0.04 0.00 0.07 0.00 0.08 0.01 0.03 >0.2 OpticNerve_L 0.51 0.42 0.43 0.36 0.17 0.00 0.34 0.32 >0.05 OpticNerve_R 0.56 0.38 0.61 0.38 0.20 0.00 0.39 0.36 ≤0.05 SpinalCord 0.13 0.12 0.05 0.05 0.04 0.01 0.02 0.06 =0.00 Average 0.42 0.31 0.39 0.35 0.25 0.21 0.13 0.29 Fractional volume affected by different variations when constraint failing probability > 5% (worst-case scenario) Choi, Nourzadeh et al., AAPM, 2020.
- 47. OARNet: auto-delineate organs-at-risk (OARs) in head and neck (H&N) CT image 47 Soomro, Nourzadeh, Choi et al., AAPM, 2020.
- 49. A Probabilistic U-Net for Segmentation of Ambiguous Images Kohl et al. NeurIPS 2018 49
- 50. Segmentations on different Variability levels (middle 5) and their occupancy map • Implemented the model using PyTorch • Model trained using TCIA LIDC dataset (about 2000 nodules with up to 4 radiologists delineations) • Titan Xp, 12GB (1 week for training) • Unstable to train, 2D segmentation, tumor A Probabilistic U-Net (Results) 50 Variability Low High
- 51. Summary Radiomics - Decision Support Tools • Lung Cancer Screening • Tumor Response Prediction and Evaluation • Aggressive Lung ADC subtype prediction • Multimodal data: Pathology, Multiomics, etc. Auto Delineation and Variability Analysis • Delineation Variability Quantification • Dosimetric Consequences of Variabilities 51
- 52. Short-term Future Works • Develop interpretable radiomic features • Improve spiculation quantification • Multi-institution validation • Human-Variability aware auto-delineation • Variability quantification and simulation using generative models • OARNet + Probabilistic U-Net → Probabilistic OARNet • Integrate the radiomics framework into TPS • Eclipse (C#), MIM (Python), RayStation (Python) 52
- 53. Long-term Future Works • Comprehensive Framework for Cancer Imaging • Multi-modal imaging • Response prediction and evaluation (Pre, Mid, and Post) • Longitudinal analysis of tumor change • Shape analysis (e.g., Spiculation) • Deep learning models • Automation of Clinical Workflow • Big Data Analytics: EMR, PACS, ROIS, Genomics, etc. • Provide an informatics platform for comprehensive cancer therapy 53
- 54. Selected Publications 1. Jung Hun Oh*, Wookjin Choi* et al., “PathCNN: interpretable convolutional neural networks for survival prediction and pathway analysis applied to glioblastoma”, Bioinformatics, 2021, *joint first author 2. Wookjin Choi et al., “ Reproducible and Interpretable Spiculation Quantification for Lung Cancer Screening”, Computer Methods and Programs in Biomedicine”, 2021 3. Noemi Garau, Wookjin Choi, et al., “ External validation of radiomics‐based predictive models in low‐dose CT screening for early lung cancer diagnosis”, Medical Physics, 2020 4. Jiahui Wang, Wookjin Choi et al., “Prediction of anal cancer recurrence after chemoradiotherapy using quantitative image features extracted from serial 18F-FDG PET/CT”, Frontiers in oncology, 2019 5. Wookjin Choi et al., “Radiomics analysis of pulmonary nodules in low-dose CT for early detection of lung cancer”, Medical Physics, 2018 6. Sadegh Riyahi, Wookjin Choi, et al., “Quantifying local tumor morphological changes with Jacobian map for prediction of pathologic tumor response to chemo-radiotherapy in locally advanced esophageal cancer”, Physics in Medicine and Biology, 2018 7. Shan Tan, Laquan Li, Wookjin Choi, et al., “Adaptive region-growing with maximum curvature strategy for tumor segmentation in 18F-FDG PET”, Physics in Medicine and Biology, 2017 8. Wookjin Choi et al., “Individually Optimized Contrast-Enhanced 4D-CT for Radiotherapy Simulation in Pancreatic Ductal Adenocarcinoma”, Medical Physics, 2016 9. Wookjin Choi, Tae-Sun Choi, “Automated Pulmonary Nodule Detection based on Three-dimensional Shape-based Feature Descriptor”, Computer Methods and Programs in Biomedicine, 2014 54 Complete list of publications: https://scholar.google.com/citations?user=iHgsGLUAAAAJ
- 55. Hiring a Post-Doctoral Fellow Developing Interpretable Predictive Models for Radiation Therapy • PI: Wookjin Choi, PhD - Wookjin.Choi@Jefferson.edu • 2 Years • Machine Learning/Deep Learning: Radiomics (PET/CT & MR) and Bioinformatics • Computational Medical Physics: Development of Predictive Models and Automated Workflows, and Improve Clinical Workflow • Internal or Extramural Research Funding Opportunities Qualifications • Ph.D. in Computer Science, Electrical Engineering, Medical Physics, or related field required 55
- 56. 56 Thank You! Q & A https://qradiomics.com E-mail: Wookjin.Choi@Jefferson.edu
- 57. Spiculation Quantification: Height, Area Distortion • Area distortion better than solid angle for non-conic spicules • Height is medial axis length, can be perpendicular distance • Width is measured at base, can be FWHM 𝑠1 = 𝑖 mean 𝜖𝑝 𝑖 ∗ ℎ𝑝 𝑖 𝑖 ℎ𝑝 𝑖 𝑠2 = 𝑖 min 𝜖𝑝 𝑖 ∗ ℎ𝑝 𝑖 𝑖 ℎ𝑝 𝑖 57
- 58. Spiculation Quantification (Model Validation) 58
- 59. Local tumor morphological changes (Motivation) 59
- 60. Delineation Variability Simulation (Methods) • DV Simulation using auto-delineation (AD) methods (σ=2, 5, 10 mm) • Average surface of standard deviation (ASSD): random perturbation • GrowCut: cellular automata region growing • Random walker (RW): probabilistic segmentation 60 Background 0 Foreground 1 Initial Binary Mask Gaussian-Smoothed Mask Gaussian Noises-Added Mask Intensity Spatial location Inside Outside σ = 2mm σ = 5mm σ = 8mm σ = 10mm Choi et al., AAPM, 2019.
- 61. OARNet comparisons 61 (a) Dice similarity coefficient and (b) Hausdorff comparison for the alternative delineation methods. The points in the graphs are mean values and bars show the 95% confidence intervals.
Editor's Notes
- I would like to thank everyone who has helped me in the projects
- I would like to thank everyone who has helped me in the projects
- a general overview of the radiation therapy workflow with brief descriptions of expected applications of artificial intelligence (AI) at each step. The workflow begins with the decision to treat the patient with radiation therapy, followed by a simulation appointment during which medical images are acquired for treatment planning. Subsequently, the patient-specific treatment plan is created, and then the plan is subjected to approval, review and quality assurance (QA) measures prior to delivery of radiation to the patient. The patient then receives follow-up care. AI has the potential to improve radiation therapy for patients with cancer by increasing efficiency for the staff involved, improving the quality of treatments, and providing additional clinical information and predictions of treatment response to assist and improve clinical decision-making.
- (triangle: Monte Carlo; square: Inverse optimization/IMRT; circle: deep learning-based contouring). The curve depicts expectations by the target audience (those in radiation oncology and medical physics) as a function of time. Yellow, magenta, cyan, green, and blue portions of the curve denote “innovation trigger,” “peak of inflated expectations,” “trough of disillusionment,” “slope of enlightenment,” and “productivity plateau” regions, respectively. Automatable tasks in radiation oncology for the modern clinic. The extent to which each skill set is used or task is performed in this figure is not indicated and may be dependent on each clinical practice. In order to group essential tasks performed during the treatment planning process, “Physical,” “Knowledge,” and “Social” skill domains were created and are indicated by green, magenta, and blue ellipses, respectively. Skills or tasks are indicated by circles within each colored domain and may be shared between domains. Based on works cited in this review, tasks which may be automated are within the “Automatable” domain.
- How to generalize it
- A large number of image features from medical images additional information that has prognostic value
- I open sample automated workflow and essential components to public
- I open sample automated workflow and essential components to public
- I open sample automated workflow and essential components to public
- I open sample automated workflow and essential components to public
- I open sample automated workflow and essential components to public
- I open sample automated workflow and essential components to public
- Frequent use of LDCT increased number of indeterminate PNs Prediction of PN malignancy is important
- The Lung Imaging Reporting and Data System (Lung-RADS) was developed by the American College of Radiology (ACR) to standardize the screening of lung cancer on CT images. As shown in the Table, the Lung-RADS categorization is mainly based on PN size (the average of the longest and shortest diameters on axial slice) with some consideration to calcification, appearance type (solid, part-solid, and non-solid or ground glass nodule/GGN), and additional suspicious features. We also performed Lung-RADS categorization based on the PN contour and the physician’s annotations. To match the original LIDC-IDRI diagnosis, categories 3 and lower are labeled as benign and category 4 (4A, 4B, and 4X) as malignant.
- 79 LDCT scans: 36 benign and 43 malignant cases, 7 missing contours We performed Lung-RADS categorization based on the PN contour and the physician’s annotations. Having diagnosis data 157 Primary cancer 43 -> 41 biopsy-proven, progression Benign 36 -> 31 biopsy-proven, 2yrs of stable PN, progression Metastatic cancer or unknown 78
- To increase interpretability, need concise model with minimum number of features
- The proposed method showed comparable or better accuracy than others, Better than deep learning with two features
- The Lung Imaging Reporting and Data System (Lung-RADS) was developed by the American College of Radiology (ACR) to standardize the screening of lung cancer on CT images. As shown in the Table, the Lung-RADS categorization is mainly based on PN size (the average of the longest and shortest diameters on axial slice) with some consideration to calcification, appearance type (solid, part-solid, and non-solid or ground glass nodule/GGN), and additional suspicious features. We also performed Lung-RADS categorization based on the PN contour and the physician’s annotations. To match the original LIDC-IDRI diagnosis, categories 3 and lower are labeled as benign and category 4 (4A, 4B, and 4X) as malignant.
- Spiculations are spikes on the surface of PN and are important predictors of malignancy in lung cancer
- Why spherical mapping because nodule has a spherical topology and we want to simplify its representation First non-trivial eigenfunction of the Laplace-Beltrami operator Conformal mapping from surface 𝑆 to unit sphere 𝒮 2 : 𝜙:𝑆→ 𝒮 2 Compute area distortion 𝜖 𝑖 to detect base ( 𝜖 𝑖 =0) and apex (max. negative 𝜖 𝑖 )
- N s – rho with RS 0.44 S 1 – -0.36 S 2 – 0.34
- Our spiculation measures improved the radiomics model for malignancy prediction Model 9 is also mine
- 412 patients treated between 2006 and 2017 were included. Patients had to have PETs and CTs available within three months prior to SBRT start. The median prescription dose was 50Gy in 5 fractions. The planning-CT gross tumor volumes (GTVs) were propagated onto the pre-treatment PETs and CTs using b-spline deformable image registration. PET intensity features (90th percentile, entropy, maximum, mean, peak, robust mean absolute deviation, SD, valley) and CT shape features (compactness, diameter, elliptic axes, elongation, flatness, number of lobules/peaks/spicules, sphericity, surface area, surface to volume ratio, volume) were extracted. Data were split into training and hold-out validation subsets (n = 283, 123; 70%, 30%). In the training subset, the imaging features and six patient characteristics (age, gender, histology, performance status, prior surgery, tumor location) were tested for association with PFS using Cox Proportional Hazards regression with re-sampling (bootstrapping with 1000 samples). Significance was denoted at p≤0.0019 (corrected for 26 tests). A bootstrapped forward-stepwise multivariate analysis was undertaken including only non-strongly correlated predictors (Spearman’s rank, |Rs|<0.70). The most frequently selected model was explored in the validation subset in which model performance was assessed using the c-index and the prediction-stratified high and low risk tertiles (HR, LR) of the observed PFS were compared.
- Results Nineteen of the 20 identified candidate predictors were either PET or CT features (p-value range: 3E-9, 1.2E-3). The intra-imaging modality correlation between features was strong (median |Rs|: PET: 0.93; CT: 0.76) and only four features were passed on to multivariate analysis: PET entropy, CT number of peaks, CT major axis, and gender. The most frequently selected model included PET entropy and CT number of peaks; the c-index in the validation subset was 0.77 and the prediction-stratified survival indicated a clear separation between the observed HR and LR: e.g. a PFS of 60% was observed at 12 months in HR vs. 22 months in LR. Conclusion This PET and CT-based model identified the SUV distribution randomness (entropy) and spiculated tumor pattern on CTs as the most important features in predicting PFS in early stage NSCLC. The associated performance on the hold-out validation subset was good and its use has the potential to further improve the prediction of response to SBRT for this patient population. This model will be used to identify high-risk patients based on the predicted PFS in an upcoming phase II study on adjuvant immunotherapy.
- Jacobian matrix: First derivative of DVF. Stretch=displacement+scale. J matrix can be decomposed into strain and rotation matrix. J matrix shows rate of displacement change in each direction. Volumetric ratio before and after the transformation. 20 patients: 9 responders, 11 non-responders Quantitatively evaluated Multi-resolution BSpline registration Bending energy of transformation as regularization Shape, texture, intensity, ratio and clinical features (n=98)
- Minimum Jacobian – largest shrinkage, Median Jacobian – approximation of global change SVM-LASSO AUC 0.91
- NSCLC IASLC/ATS/ERS lung ADC classification: (A, B) , Acinar (C, D), Papillary (E), MIP (F), Solid (G, H) wedge resection or segmentectomy lobectory Higher rate of recurrence, vascular invasion, pleural invasion, lymph node and distant metastasis was reported in solid or micropapillary subtype Introduction The solid and micropapillary (MIP) histopathologic patterns are more aggressive subtypes of lung adenocarcinoma (ADC). This study aims to predict these aggressive histopathologic subtypes by using radiomic analysis with 18F-fluorodeoxyglucose (FDG) positron emission tomography (PET) and computed tomography (CT).
- Materials and Methods This study retrospectively examined 120 patients with stage I lung ADC with tumor size smaller than 2 cm. 206 radiomic features were extracted from preoperative CT and 18F-FDG PET/CT, characterizing the intensity, shape, and texture features of a tumor. Partial volume effects (PVE) of tumor in PET was corrected before calculating the radiomic features using recovery coefficient method. The radiomic features along with clinical parameters were used to identify those cases with aggressive histopathologic subtypes. Univariate analysis was performed to evaluate each radiomic feature using the area under the curve (AUC) of receiver operating characteristic (ROC) and p-value computed by Wilcoxon rank-sum test. False discovery rate (FDR) correction was applied to the analysis because of multiple comparisons problem (q-value). Before the model building process, distinctive features were identified using a hierarchical clustering method. We then constructed a prediction model by using a support vector machine (SVM) classifier coupled with a least absolute shrinkage and selection operator (LASSO). A 10-fold cross-validation was repeated ten times to evaluate the accuracy of the model. Results Among 119 evaluable patients, 23 patients had aggressive subtypes (18 solid, 5 MIP). In univariate analysis, ninety-eight features were significant (q<0.05, AUC: 0.65–0.79), including 18 CT features, 79 PET features and 1 clinical parameter (COPD/Emphysema). All the top 10 features were extracted from PET including 7 texture features, 2 size features, and 1 intensity-weighted size feature. The best SVM-LASSO model achieved an accuracy of 82.4% and a high negative predictive value (NPV) of 91.7% with a single texture feature - PET Mean of Cluster Shade. The more asymmetric the FDG uptake was in a tumor, the higher the Mean of Cluster Shade, and the more likely the tumor was an aggressive subtype. The model has significantly higher performance (Sensitivity: 40.4% vs. 57.8%, P=0.002; Specificity: 78.5% vs. 86.0%, P=1e-5; PPV: 53.7% vs. 39.2%, P=7e-8; NPV: 91.7% vs. 88.6%, P=3e-5, Accuracy: 82.4% vs. 74.5%, P=5e-8; AUC: 0.78 vs. 0.64, P=0.03). Conclusion We developed a radiomics model to predict aggressive subtypes of Lung ADC. We demonstrated that the model achieved significantly higher sensitivity than SUVmax, therefore it has potential to assist surgeons in selecting patients as sublobar resection candidates.
- Whole slide Detecting cancer manually in whole slide images requires significant time and effort on the laborious process. Recent advances in whole slide image analysis have stimulated the growth and development of machine learning-based approaches that improve the efficiency and effectiveness in the diagnosis of cancer diseases. In this paper, we propose an unsupervised learning approach for detecting cancer in breast invasive carcinoma (BRCA) whole slide images. The proposed method is fully automated and does not require human involvement during the unsupervised learning procedure. We demonstrate the effectiveness of the proposed approach for cancer detection in BRCA and show how the machine can choose the most appropriate clusters during the unsupervised learning procedure. Moreover, we present a prototype application that enables users to select relevant groups mapping all regions related to the groups in whole slide images.
- Prediction results for four whole slide images on 40x magnification. The first column represents the original whole slide image, the second column represents the prediction results (cancer colored as blue while tumor infilterating lymphocytes colored as red), the third column represents a heatmap for cancer regions, and the last column represents a heatmap for tumor infilterating lymhocytes regions. The whole slide images at the first two columns are resized to 5x magnfication, while the heatmap images at the last two columns are generated in 40 x 40 grids
- To address these issues, we propose a novel method, called PathCNN, that constructs an interpretable CNN model on integrated multi-omics data using a newly defined pathway image. PathCNN showed promising predictive performance in differentiating between long-term survival (LTS) and non-LTS when applied to glioblastoma multiforme (GBM). The adoption of a visualization tool coupled with statistical analysis enabled the identification of plausible pathways associated with survival in GBM. In summary, PathCNN demonstrates that CNNs can be effectively applied to multi-omics data in an interpretable manner, resulting in promising predictive power while identifying key biological correlates of disease.
- Motivation Convolutional neural networks (CNNs) have achieved great success in the areas of image processing and computer vision, handling grid-structured inputs and efficiently capturing local dependencies through multiple levels of abstraction. However, a lack of interpretability remains a key barrier to the adoption of deep neural networks, particularly in predictive modeling of disease outcomes. Moreover, because biological array data are generally represented in a non-grid structured format, CNNs cannot be applied directly. Results To address these issues, we propose a novel method, called PathCNN, that constructs an interpretable CNN model on integrated multi-omics data using a newly defined pathway image. PathCNN showed promising predictive performance in differentiating between long-term survival (LTS) and non-LTS when applied to glioblastoma multiforme (GBM). The adoption of a visualization tool coupled with statistical analysis enabled the identification of plausible pathways associated with survival in GBM. In summary, PathCNN demonstrates that CNNs can be effectively applied to multi-omics data in an interpretable manner, resulting in promising predictive power while identifying key biological correlates of disease.
- How to generalize it
- A framework for radiation therapy variability analysis, human delineation variability (DV) and simulated DV generated by auto delineation (AD) methods were analyzed using geometric measurements and dosimetric consequence, as well as dosimetric consequences of setup variability also evaluated using RTRA. If multiple humans delineated SSs are not available, consensus SS will be generated by 10 ASSD delineated SSs (5 SSs with 2mm SD and 5 SSs with 5mm SD).
- This work describes a framework to simulate DV which can generate specific levels of DV and quantify its geometric and dosimetric variations. We evaluated our framework on ESTRO Falcon dataset Fourteen independent MD head-and-neck OAR structure sets (SS) were obtained from the ESTRO Falcon group. Seven OARs were available (BrainStem, Esophagus, OralCavity, Parotid_L, Parotid_R, SpinalCord, and Thyroid). A consensus MD SS was generated by the simultaneous truth and performance level estimation (STAPLE) method. MD DV was evaluated with respect to the STAPLE SS using the Dice coefficient and Hausdorff distance (HD) geometric similarity metrics.
- DV was evaluated with respect to the STAPLE SS Each OAR AD was repeated five times with a different seed or variability level. Dice and HD were computed for each OAR AD with respect to the STAPLE SS. Dosimetric analysis was achieved by intercomparing dose-volume histograms (DVH) from a plan developed with a reference MD SS with DVHs for each MD and AD. DVH confidence bands are reported for MD and each AD method.
- The MD Dice was 0.7±0.2 (μ±σ). AD Dice values (ASSD, GrowCut, and RW) were 0.5±0.2, 0.7±0.2, and 0.8±0.1, respectively. HDs were 35.4±45.2, 27.3±19.1, 29.3±19.9, and 14.6±10.3. The simulated DV increased with increasing the seed standard deviations or variability level. The dosimetric effect was largest for MD DVs (larger OAR DVH confidence intervals and larger HD), even though the MD Dice was greater than the ASSD and GrowCut Dice values. GrowCut DV resulted in less dosimetric variation than RW, unlike the geometric indices.
- 5 Different 400 Plan
- | | Eclipse | Monaco | Pinnacle | RayStation | Tomotherapy | |:----------|------------:|-------------:|--------------:|-----------------:|-------------------:| | patientID | 227 | 75 | 49 | 25 | 33 | | | IMRT | Tomotherapy | VMAT | |:----------|-------:|--------------:|-------:| | patientID | 142 | 33 | 234 |
- | | Eclipse | Monaco | Pinnacle | RayStation | Tomotherapy | |:----------|------------:|-------------:|--------------:|-----------------:|-------------------:| | patientID | 227 | 75 | 49 | 25 | 33 | | | IMRT | Tomotherapy | VMAT | |:----------|-------:|--------------:|-------:| | patientID | 142 | 33 | 234 |
- | | Eclipse | Monaco | Pinnacle | RayStation | Tomotherapy | |:----------|------------:|-------------:|--------------:|-----------------:|-------------------:| | patientID | 227 | 75 | 49 | 25 | 33 | | | IMRT | Tomotherapy | VMAT | |:----------|-------:|--------------:|-------:| | patientID | 142 | 33 | 234 |
- Objective: To auto-delineate organs-at-risk (OARs) in head and neck (H&N) CT image sets via a compact high-performance knowledge-based model. Approach: A 3D deep learning model (OARnet) is developed and used to delineate 28 H&N OARs on CT images. OARnet utilizes a densely connected network to detect the OAR bounding-box, then delineates the OAR within the box. It reuses information from any layer to subsequent layers and uses skip connections to combine information from different dense block levels to progressively improve delineation accuracy. Training uses up to 28 expert manual delineated (MD) OARs from 165 CTs. Dice similarity coefficient (DSC) and the 95th percentile Hausdorff distance (HD95) with respect to MD is assessed for 70 other CTs. Mean, maximum, and root-mean-square dose differences with respect to MD are assessed for 56 of the 70 CTs. OARnet is compared with UaNet, AnatomyNet, and Multi-Atlas Segmentation (MAS). Wilcoxon signed-rank tests using 95% confidence intervals are used to assess significance.
- Figure 1: The Probabilistic U-Net. (a) Sampling process. Arrows: flow of operations; blue blocks: feature maps. The heatmap represents the probability distribution in the low-dimensional latent space RN (e.g., N = 6 in our experiments). For each execution of the network, one sample z 2 RN is drawn to predict one segmentation mask. Green block: N-channel feature map from broadcasting sample z. The number of feature map blocks shown is reduced for clarity of presentation. (b) Training process illustrated for one training example. Green arrows: loss functions.
- How to generalize it
- Radiation oncology information systems
- Thank you for attention! If you have any questions, I’d be pleased to answer them
- Which color line is base?
- state of the art
- More interpretable features and small number of features
- Each OAR AD was repeated five times with a different seed or variability level. Seed points generation for GrowCut and RW Gaussian Smooth – PSF
- Results: Wilcoxon signed ranked tests show that, compared with UaNet, OARnet improves (p<0.05) the DSC (23/28 OARs) and HD95 (17/28). OARnet outperforms both AnatomyNet and MAS for DSC (28/28) and HD95 (27/28). Compared with UaNet, OARnet improves median DSC up to 0.05 and HD95 up to 1.5mm. Compared with AnatomyNet and MAS, OARnet improves median (DSC, HD95) by up to (0.08, 2.7mm) and (0.17, 6.3mm). Dosimetrically, OARnet outperforms UaNet (Dmax 7/28; Dmean 10/28), AnatomyNet (Dmax 21/28; Dmean 24/28), and MAS (Dmax 22/28; Dmean 21/28). Conclusion: A compact method for OAR auto-delineation using modern deep-learning methods is described and applied to H&N auto-delineation. The DenseNet architecture is optimized using a hybrid approach that performs OAR-specific bounding box detection followed by feature recognition. Compared with other auto-delineation methods, OARnet is better than or equal to UaNet for all but one geometric (Temporal Lobe L, HD95) and one dosimetric (Eye L, mean dose) endpoint for the 28 H&N OARs, and is better than or equal to both AnatomyNet and MAS for all OARs.