American Association for Cancer Research Annual Meeting 2022 Analysis of images of routinely acquired tissue specimens promise to provide biomarkers that can be used to predict disease outcome and steer treatment, improve diagnostic reproducibility, and reveal new insights to further advance current human understanding of disease. The advent of AI and ubiquitous high-end computing are making it possible to carry out accurate whole slide image morphological and molecular tissue analyses at cellular and subcellular resolutions. AI methods are can enable exploration and discovery of novel diagnostic biomarkers grounded in prognostically predictive spatial and molecular patterns as well as quantitative assessments of predictive value and reproducibility of traditional morphological patterns employed in anatomic pathology. AI methods may be adapted to help steer treatment through integrative analysis of clinical information along with Pathology, Radiology and molecular data.

1. AI and whole slide imaging biomarkers
Joel Saltz MD, PhD
Chair and Distinguished Professor Department of Biomedical Informatics
Director Center for Engineering Driven Medicine
Professor Department of Pathology
Cherith Endowed Chair
Stony Brook University
Advances in Diagnostics and Therapeutics: Deep Learning for Cancer
12:30-2:00 PM 4/11/2022

2. • No Conflicts to Disclose

3. Ingredients of Computational Pathology Biomarkers
• Every diagnostic WSI
contains vast amounts of
data
– Tumor, lymphocyte
infiltration
– Cells – normal, dysplastic,
lymphocytes, necrotic
– Spatial distribution and
morphology of glands,
ducts
– Collagen orientation
Predict treatment
response and outcome
Patient stratification
Treatment selection
Cancer Epidemiology

4. Computational Pathology
• Just as in flow cytometry cells
are interrogated counted
clustered and classified, using
the newfound ability to
segment and classify cells
enables a spatially and formed
type of highly granular highly
quantitative characterization
of tissue.
• In this case the challenge is not to
reproduce what a human
pathologist chooses as a
classification but instead to use a
highly detailed set of labeled
building blocks to build new
biomarkers

6. Potential Clinical Applications of AI based TILs Methods
- Incorporate TILs into diagnostic criteria
- Using TILs to predict prognosis (treatment
response and survival)
- Using spatial patterns of distribution of
TILs maps to ascertain the functional
immune status of the tumor
microenvironment
- Combination of diagnostic criteria and TILs
to stratify patients, guide clinical
management, and select therapy (e.g.
immunotherapy)

7. ● Deep learning based computational stain for TILS
●TIL patterns generated from 4,759 TCGA subjects (5,202 H&E slides),
13 cancer types
●Detected TILs correlate with pathologist eye and molecular estimates
●TIL patterns linked to tumor and immune molecular features, cancer
type, and outcome
●Now refined and extended to roughly 20 cancer types

8. Pathomics tissue analytics for
Precision Medicine
(1) Tumor-TILs analyses
(2) High-resolution detection
and classification of tumor cells,
lymphocytes, and stromal cells
in the entirety of whole slide
images
(3) Support the scoring of the
number of TILs in microscopic
regions of interest chosen by
pathologists
(4) Analysis of cell composition
and features in different tumor
niches
Legend
red = lymphocytes
blue = stromal cells
green = tumor cells

9. Collaboration with ECOG-ACRIN to bring AI to
Clinical Trials

10. Collaboration with SEER Registries to Bring AI Pathology to
Surveillance and to Create Real World Clinical Research Datasets

11. TIL Pattern Descriptions
Quantitative – Arvind Rao
• Agglomerative clustering
• Cluster indices representing cluster
number, density, cluster size, distance
between clusters
• Traditional spatial statistics measures
• R package clusterCrit by Bernard
Desgraupes - Ball-Hall, Banfield-
Raftery, C Index, and Determinant
Ratio indices
Qualitative (Alex Lazar, Raj Gupta)
‘‘Brisk, diffuse’’ diffusely
infiltrative TILs scattered
throughout at least 30%
of the area of the tumor
(1,856 cases);
‘‘Brisk, band-like’’ - band-
like boundaries bordering
the tumor at its
periphery (1,185);
‘‘Nonbrisk, multi-focal’’
loosely scattered TILs
present in less
than 30% but more than
5% of the area of the
tumor (1,083);
‘‘Non-brisk, focal’’ for
TILs scattered throughout
less than 5% but greater
than 1% of the area of
the tumor (874);
‘‘None’’ < 1% TILS - in 143
cases

12. TIL Pattern Characterization
Cutaneous Malignant Melanoma,TIL map with “Brisk, Band-like” structural pattern
Cutaneous Malignant Melanoma,TIL map with “Brisk, Band-
like” structural pattern
”
Squamous Cell Carcinoma of the Lung, TIL map with “Brisk,
Diffuse” structural pattern

13. Our Second Generation Spatial TIL Patterns
Began with approximately 10 putative patterns,
carried out analyses on TCGA data to identify five
patterns that appeared to be predictive using a
subset of TCGA data
Carried out through analysis on both TCGA Breast
Cancer and Carolina Breast Cancer Study
cohorts
Collaboration between UNC group led by Melissa
Troester and our Stony Brook Group
Unpublished results - only presenting TCGA
results here – CBCS results will be in publication
Patterns of TIL Infiltration

16. TCGA Risky Features
Univariate and
multivariate
cox
proportional
hazards model
of PFI
comparing
cases with 2+
risky features
to those with 0
or 1

17. Community Challenge – Breast Cancer Tumor/TIL AI
Methods

18. AI Pathology Algorithm Validation FDA Led Collaborations
Dataset of pathologist annotations for algorithms that process whole slide images
• Researchers from FDA, academic colleagues in
collaboration with the Alliance for Digital
Pathology, are collecting pathologist
annotations as data for AI/ML algorithm
validation
• Application - tumor infiltrating lymphocyte
(TIL) detection and quantitation..
• Training materials and workflows to
crowdsource pathologist image annotations on
two modes: an optical microscope and two
digital platforms.
• The microscope platform allows the same ROIs
to be evaluated in both modes.
• Pursue an FDA Medical Device Development
Tool Qualification

19. Validating Artificial Intelligence Methods for Clinical Use
A Pathologist-Annotated Dataset for Validating
Artificial Intelligence: A Description and Pilot
Study

20. Federated Training – FeTS
Collaborative Effort led by Spyridon Bakas
NCI ITCR Program
• Radiology
• Concluded the largest real world
federation to-date
Speaker: S.Bakas, UPenn
• Pathology
– 2-D digitized tissue sections
– Extremely Large size
– 150,000x150,000 = ~20,000 MB
• Proof of Concept concluded
successfully [1]
• Deployment to UH3 SEER network
underway
[1] U.Baid, et al. "Federated Learning for the Classification of
Tumor Infiltrating Lymphocytes", arXiv preprint arXiv:2203.16622
(2022)

21. FeTS-PRISM: Quantitative Results
Data divided vertically, according to anatomy.
Each site's data divided in independent training/validation
Unseen Validation Data :
Not all tumor sites present TILs
 St.6: Train: 2%, Val: 0
 St.7: Train: 24% Val: 3.2% Speaker: S.Bakas, UPenn
[1] U.Baid, et al. "Federated Learning for the Classification of Tumor
Infiltrating Lymphocytes", arXiv preprint arXiv:2203.16622 (2022)

22. Imaging, Biomarker Discovery, & Engineering
Science (IBES)
Wei Zhao, PhD

23. Aim 1: Develop and translate clinically
multi-modal cancer imaging technologies
for cancer detection, diagnosis,
prevention, and treatment
Aim 2: Advance biomarker discovery
using engineering and deep learning
approaches to characterize the cancer
phenotype at the multi-scale level
Imaging, Biomarker Discovery, & Engineering Sciences (IBES)

24. Aim 2: Deep Learning based Pathomic Biomarkers
PIs: Saltz, Kurc, Chen,
Samaras, Prasanna, Moffitt
Funding: 4UH3 CA225021-03,
5U24 CA215109-04
SR: BB-SR, TA-SR
Pubs: Amer.J.Pathology, 2020
Scientific Data, 2020
Goal: Use AI to create a
highly detailed set of labeled
building blocks for new
biomarkers
Impact: Predict patient
outcomes and therapeutic
response, leverage SEER
registries.
AI for automated
detection and
prediction of outcome
Quantitative assessment of
spatial relations between
tumor and TIL
Knowledge of Tumor
heterogeneity, phenotype
crucial for treatment
decision

25. Stony Brook Deep Learning Pathology Faculty
Dimitis Samaras, Chao Chen, Tahsin Kurc, Prateek Prasanna, Raj Gupta

26. Vu Nguyen– Graduate Student
Computer Science
Anne Zhao – Assistant
Professor, Dept Pathology
Stony Brook
Raj Gupta – Department of
BMI Stony Brook
Deep Learning and
Tissue
Characterization
Trainee Team circa
2018
Le Hou– software engineer,
google core, Mountain View
Han Le– Software Development
Engineer II, Amazon, Seattle

27. The Stony Brook Multi-Scale Student Group:
David Belinsky, Mahmudul Hasan
Prantik Howlader

28. SEER UG3 Team
• Stony Brook
– Joel Saltz MD, PhD
– Tahsin Kurc PhD
– Dimitris Samara PhD
– Erich Bremer
– Le Hou
– Shahira Abousamra
– Raj Gupta
– Han Le
– Bridge Wang
• Emory
– Ashish Sharma PhD
– Ryan Birmingham
– Nan Li
• Georgia SEER Registry
– Kevin Ward MPH, PhD
• Rutgers
– David J. Foran PhD
– Evita Sadimin, MD
– Wenjin Chen, PhD
– Doreen Loh M.S.
– Jian Ren, Ph.D
– Christine Minerowicz, MD
• Rutgers SEER/ NJ State Cancer
Registry
– Antoinette Stroup, PhD
– Adrian Botchway, CTR
– Gerald Harris, PhD
• University Kentucky; Kentucky
SEER Registry
– Eric B. Durbin, DrPH, MS
– Isaac Hands, MPH
– John Williams, MA
– Justin Levens

29. QuIP ITCR Team
Stony Brook University
Joel Saltz
Tahsin Kurc
Raj Gupta
Dimitris Samaras
Erich Bremer
Fusheng Wang
Tammy DiPrima
Le Hou
NCI / DCEG
Jonas S Almeida
Emory University
Ashish Sharma
Ryan Birmingham
Nan Li
University of Tennessee
Knoxville
Jeremy Logan
Scott Klasky
Harvard University
Rick Cummings
ITCR/IMAT Supplement
Richard Levinson
Raj Gupta
Tahsin Kurc
Joel Saltz
Han Le
Maozheng Zhao
Dimitris Samaras

30. 30
The PRISM Team
• Fred Prior, PhD
• Jonathan Bona, PhD
• Kirk Smith
• Lawrence Tarbox, PhD
• Mathias Brochhausen, PhD
• Roosevelt Dobbins
• Tracy Nolan
• William Bennett
• Ashish Sharma, PhD
• Annie Gu
• Mohanapriya Narapareddy
• Monjoy Saha, PhD
• Pradeeban Kathiravelu, PhD
• Joel Saltz, MD, PhD
• Erich Bremer
• Rajrisi Gupta MD
• Tahsin Kurc, PhD
• Tammy DiPrima
• TJ Fitzgerald, MD
• Fran Laurie

31. Thanks - Funding
• U24CA180924, U24CA215109, NCIP/Leidos 14X138 and
HHSN261200800001E, UG3CA225021-01 from the NCI;
R01LM011119-01 and R01LM009239 from the NLM, Bob
Beals and Betsy Barton, Stony Brook Mount Sinai Seed
Funding, Pancreatic Cancer Action Network
• This research used resources provided by the National
Science Foundation XSEDE Science Gateways program and
the Pittsburgh Supercomputer Center BRIDGES-AI facility