Keynote presentation at IEEE International Conference on Healthcare Informatics

1. Digital Pathology, FDA Approval and
Precision Medicine
Joel Saltz
Department of Biomedical Informatics
Stony Brook University
August 24 2017

2. Digital Pathology as Clinical Informatics
• Pathology data is employed in care guidelines and
clinical settings for virtually all cancer disease sites.
• Treatment decisions frequently hinge on subjective
assessments -- poor inter-observer reproducibility.
• Widespread clinical adoption of Digital Pathology
platforms in coming years
• Combination of Digital Pathology platforms and
maturing of machine learning and artificial intelligence
methodology will make possible adoption of image
data driven decision support systems.
• Development and adoption of such systems will have
tremendous impact on improving quality and
consistency of clinical decision making.

4. • Improve reproducibility in traditional Pathology
assessments (e.g. Gleason grade, NSCLC subtypes)
• Precise scoring of well known criteria ( tumor
infiltrating lymphocytes, mitoses and IHC staining)
• Development of novel computational methods to
employ Pathology image information to predict
response to cancer treatment and outcomes.
Pathology Image Driven Decision Support

5. Digital Pathology as Precision Medicine
• Predict treatment
outcome, select,
monitor
treatments
• Reduce inter-
observer variability
in diagnosis
• Computer assisted
exploration of new
classification
schemes
• Tumor
heterogeneity,
Immune response

6. Digital Pathology as Precision Medicine
• Statistical analyses and machine learning to link
Radiology/Pathology features to “omics” and outcome
biological phenomena
• Image analysis and deep learning methods to extract
features from images
• Support queries against ensembles of features
extracted from multiple datasets
• Identify and segment trillions of objects – nuclei,
glands, ducts, nodules, tumor niches
• Analysis of integrated spatially mapped
structural/”omic” information to gain insight into
cancer mechanism and to choose best intervention

8. Why We Need Data Analytics and AI
Or
A Story Concerning Elephants and
Pathology

10. Rolling Hills

11. Mountains

12. Aggressive Bear

13. Wilted Salad

14. Helsinki - Search iPhone photos for UK
credit card PIN

15. Successful Lunch

16. Non Small
Cell Adeno
Carcinoma
Acinar
Subtype

17. Papillary
Subtype

18. Lepidic
Subtype

19. Acinar with
foci of
Lepidic and
Papillary
Patterns
WHO 2015:
Pathologists are
supposed to report
patterns in 5%
increments!

20. Machine Learning and Generation of Human Pathology
Classifications
Hiro Shimada, Metin Gurcan, Jun Kong, Lee Cooper Joel Saltz - 2006
BISTI/NIBIB Center for Grid Enabled Image Analysis - P20 EB000591,
PI Saltz

21. Brain Tumor Classification – CVPR 2016

22. Combining Information from Patches

23. Brain Tumor Classification Results
Le Hou, Dimitris Samaras, Tahsin Kurc, Yi Gao, Liz Vanner, James
Davis, Joel Saltz

24. Moonshot group
• Stony Brook, Institute for Systems
Biology, MD Anderson, Emory group
• TCGA Pan Cancer Immune Group – led
by ISB researchers
• Deep dive into linked molecular and
image based characterization of cancer
related immune response

26. TIL Quantitation and Distribution
• The most common diagnostic tool in pathology is the
H&E tissue image
• FDA just approved use of whole slide images in primary
Pathology diagnosis
• TCGA dataset, which comprises 33 tumor types,
contains over 30,000 tissue slide images.
• Link pattern of tumor infiltrating distribution to
outcome, “omics”, treatment
• Deep Learning TIL method requires modest training and
curation – suitable for high throughput analyses

27. Importance of Immune System in Cancer Treatment and
Prognosis
• Spatial context and the nature of cellular heterogeneity of
the tumor microenvironment, in terms of the immune
infiltrate into the tumor center and invasive margin, are
important in cancer prognosis
• Prognostic factors that quantify such spatial TIL densities in
different tumor regions have been shown to have high
prognostic value – they may be superior to the standard
TNM classification
• Immune related assays employed in determining
Checkpoint Inhibitor immune therapy in several cancer
types

28. Imaging Based TIL Analysis Workflow
Deep Learning Training and Validation
• Algorithm first trained on
image patches
• Corrected heat maps used
to generate new
predictions
• Parameter optimization
using six patches chosen
from each image using
stratified sampling method
• Comparison with molecular
data to carefully check
highly discrepant
molecular/image results

29. Interactive Deep Learning Training Tool
Anne Zhao MD, Pathology Informatics Fellow

34. Stratified Sampling Used to Set Thresholds

35. Example Algorithm Assisted Markups

36. Comparison with Molecular Lymphocyte Estimates
Estimate leukocyte fraction from methylation
CIBERSORT to identify lymphocyte subset:
B.cells.naive, B.cells.memory, T.cells.CD8,
T.cells.CD4.naive, T.cells.CD4.memory.resting,
T.cells.CD4.memory.activated, T.cells.follicular.helper,
T.cells.regulatory..Tregs, T.cells.gamma.delta,
NK.cells.resting, NK.cells.activated
Compare with lymphocyte fraction obtained from TIL
image analysis

37. Spearman rank
correlation= 0.45
p value 10-29
Spearman
rank
correlation=
0.35
p value 10-32
spearman rank
correlation= 0.33
p value = 10-06
Not
significant
Comparisons with Molecular TIL Estimates:
Lung, Breast, Pancreas, Uveal Melanoma - TCGA

38. Correlation with Molecular TIL Estimates

39. Quantify Clustering
Indices Top
TIL map
Bottom
TIL map
Banfeld-Raftery 14605.01 11756.92
Det Ratio 257.16 -200.42
Scott-Symons index 26202.77 20141.75

40. Preliminary Results on Relationship Between
TIL Pattern and Outcome
• Preliminary studies based on SKCM within TCGA
suggest that these 4 indices are significantly
associated with outcome (OS) after adjusting for
clinical and molecular correlates (age at diagnosis,
gender, mutation load etc.). Further, these were
found to retain significance even after adjustment
for multiple hypothesis testing

41. Preliminary results on relationship
between TIL pattern and outcome
• Preliminary studies on TCGA
skin Melanoma -- indices are
significantly associated with
outcome (OS) after adjusting
for clinical and molecular
correlates (age at diagnosis,
gender, mutation load etc.).
• Examining relationship
between “omics”, outcome
and clustering metrics
%Survival vs
Banfeld Raferty Index

42. Integrative
Morphology/”omics”
Quantitative Feature
Analysis in Pathology:
Emory In Silico Center for
Brain Tumor Research (PI =
Dan Brat, PD= Joel Saltz)
NLM/NCI: Integrative
Analysis/Digital Pathology
R01LM011119,
R01LM009239 (Dual PIs
Joel Saltz, David Foran)
J Am Med Inform Assoc.
2012 Integrated
morphologic analysis for
the identification and
characterization of
disease subtypes.
Pathomics
Sequence of papers 2009 – 2016 with Brat, Kurc, Cooper, Kong and others

43. Integrative Brain Tumor Research Center
histology neuroimaging
clincalpathology
Integrated
Analysis
molecular
High-resolution whole-slide microscopy
Multiplex IHC

44. CANCER NUCLEI larger size range NORMAL NUCLEI smaller size range
IMAGES FROM THE SAME PATIENT, AS THE SAME MAGNIFICATION
CANCER NORMAL
Nuclear Morphology : Size, Shape, Texture of Billions of Nuclei

45. Nuclear and Cell Morphometry Features

46. Deep Learning Classification

47. Tools to Analyze Morphology and Spatially Mapped Molecular Data - U24
CA180924
• Aim 1 Analysis pipelines for multi- scale,
integrative image analysis.
• Aim 2: Database infrastructure to manage and
query Pathomics features.
• Aim 3: HPC software that targets clusters, cloud
computing, and leadership scale systems.
• Aim 4: Develop visualization middleware to relate
Pathomics feature and image data and to
integrate Pathomics image and “omic” data.

48. SEER Virtual Tissue Repository
• Lynne Penberthy MD, MPH NCI SEER
• Ed Helton PhD NCI CBIIT Clinical Imaging Program
• Ulrike Wagner CBIIT Clinical Imaging Program
• Radim Moravec NCI PhD, NCI SEER
• Ashish Sharma PhD Biomedical Informatics Emory
• Joel Saltz MD, PhD Biomedical Informatics Stony Brook
• Tahsin Kurc PhD Biomedical Informatics Stony Brook
• Georgia Tourassi, Oak Ridge National Laboratory
Vision – Enable population/epidemiological cancer
research that leverages rich cancer phenotype
information available from Pathology tissue studies
NCIP/Leidos 14X138 and HHSN261200800001E - NCI

49. SEER Virtual Tissue Repository

50. SEER Virtual Tissue Repository
• Create linked collection of de-identified clinical data
and whole slide images
• Extract features from a sample set of images
(pancreas and breast cancer).
• Enable search, analysis, epidemiological
characterization
• Pilot focus on extreme outcome Breast Cancer,
Pancreatic Cancer cases
• Display images and analyzed features

51. Nuclear Segmentation
• Clinical studies contain billions to trillions of cells
• Optimized algorithm tuning methods
• Curation to carry out optimized segmentation of nuclear
material
• Yi Gao, Allen Tannenbaum, Dimitris Samaras, Le Hou, Tahsin
Kurc
• Many heroic undergraduates

52. 52
QuIP Segmentation Curation Web Application
1. User interactively marks up regions and
selects results from best analysis run for
each region.
2. Selections are refined through review
processes supervised by an expert
Pathologist
Curation of Segmentation Results

53. Feature Explorer Suite
• Explore Relationship Between Imaging Features,
Outcome, ”omics”
• Explore relationships between features and explore
how features relate to images

54. Feature Explorer - Integrated Pathomics Features, Outcomes
and “omics” – TCGA NSCLC Adeno Carcinoma Patients

55. Feature Explorer - Integrated Pathomics Features, Outcomes
and “omics” – TCGA NSCLC Adeno Carcinoma Patients

56. Collaboration with MGH – Feature Explorer – Radiology Brain
MR/Pathology Features

57. Collaboration with SBU Radiology – TCGA NSCLC Adeno Carcinoma
Integrative Radiology, Pathology, “omics”, outcome
Mary Saltz, Mark Schweitzer SBU Radiology

58. Pathomics
Relationship Between Image and FeaturesLeveraging Visualization to Aid in Feature Management
Step 1: Choose a case from the TCGA atlas (case #20)
Step 2: Select two features of interest; X
axis (area), Y axis (perimeter)
Step 3: Zoom in on region of interest
Step 4: Pick a specific nucleus of interest.
Each dot represents a single nucleus
Step 5: Evaluate the features selected in the context of
the specific nucleus and where this nucleus is located
within the whole slide image
The tool provides visual context for feature evaluation. This technique maps both intuitive features (i.e.
size, shape, color) and non-intuitive features (i.e. wavelets, texture) to the ground truth of source
images through an interactive web-based user interface.
Selected nucleus
geolocated within
whole slide image
Detects
elongated
nucleus
Going from the whole slide data set to selected features and back to the image
Adding a visual perspective by using a live web-based interactive tool (http://sbu-bmi.github.io/featurescape/u24/Preview.html)

59. Select Feature Pair – dots correspond to nuclei

60. Subregion selected – form of gating analogous to flow
cytometry

61. Sample Nuclei from Gated Region

62. Gated Nuclei in Context

63. Dissemination
• Containers
• Containerized segmentation algorithm/FeatureDB Employed
to support TIES, MICCAI, and competitions supported through
Kalpathy-Kramer ITCR
• Full containerized implementation of
caMicroscope/FeatureDB/Segmentation algorithm/Feature
Scape - Feb 1 2017
• Cloud Pilots
• TCIA
• HPC via NSF and DOE
• TCGA – PanCanAtlas – Lymphocyte characterization
• Integrated Features/NLP joint with TIES

64. ITCR Team
Stony Brook University
Joel Saltz
Tahsin Kurc
Yi Gao
Allen Tannenbaum
Erich Bremer
Jonas Almeida
Alina Jasniewski
Fusheng Wang
Tammy DiPrima
Andrew White
Le Hou
Furqan Baig
Mary Saltz
Emory University
Ashish Sharma
Adam Marcus
Oak Ridge National Laboratory
Scott Klasky
Dave Pugmire
Jeremy Logan
Yale University
Michael Krauthammer
Harvard University
Rick Cummings

66. Funding – Thanks!
• This work was supported in part by U24CA180924-
01, NCIP/Leidos 14X138 and HHSN261200800001E
from the NCI; R01LM011119-01 and R01LM009239
from the NLM
• This research used resources provided by the
National Science Foundation XSEDE Science
Gateways program under grant TG-ASC130023 and
the Keeneland Computing Facility at the Georgia
Institute of Technology, which is supported by the
NSF under Contract OCI-0910735.

67. Thanks!