Review the basic principles of machine learning. Learn what texture analysis is and how to apply it to medical imaging. Understand how to combine texture analysis and machine learning for lesion classification tasks. Learn the how to visualize and analyze results. Understand how to avoid common mistakes like overfitting and incorrect model selection.

1. Introduction to Machine Learning and Texture
Analysis for Lesion Characterization
Barbaros Selnur Erdal, PhD
Luciano M.S Prevedello, M.D. MPH
Kevin Mader, PhD
Joshy Cyriac
Bram Stieltjes, M.D. PhD

2. Materials
a.Slides
i. http://bit.ly/2AfdoIe
b.KNIME + Workflows + Data (zip file to extract for your own computer)
i. Already installed on RSNA computers on Desktop
ii.https://www.dropbox.com/s/3fcjvr0lfxfzmgd/knime_3.4.1.zip?dl=0
c.Extras
i. Dataset on Kaggle
1.https://www.kaggle.com/kmader/lungnodemalignancy
ii.A Kaggle Kernel (Jupyter Notebook) using Keras
1.https://www.kaggle.com/kmader/lung-node-cnn

3. Learning Objectives
a.Review the basic principles of machine learning.
b.Learn what texture analysis is and how to apply it to medical imaging.
c.Understand how to combine texture analysis and machine learning for
lesion classification tasks.
d.Learn the how to visualize and analyze results.
e.Understand how to avoid common mistakes like overfitting and
incorrect model selection.

4. Objectives
• Become valued discussion partner
• Critical thinking
• Learn how to collect domain knowledge well
• Ask the right questions
• Validation data
• Don’t be too easily impressed
• Too much artificial not enough intelligence in AI
• Winning technical contests isn’t the same as solving clinical problems or
understanding images
• Don’t lose sight of the big picture
• Not objectives
• Become world-class machine learning experts

5. Outline
• Introduction / Starting KNIME (Kevin)
• Why ML and Texture Analysis are important? (Luciano)
• Framework Overview (Kevin)
• Value Prop, Decision, ML Task
• Data Sources
• Collecting Data - Preprocessing
• Features - Basic Textures
• Building Models
• Features - Textures Deep Dive (Selnur)
• From Textures to Deep Learning (Luciano)
• Conclusion

6. Python might be good, but ...
• Devices, Sessions, Graphs, Ops, Context
Managers?
• tf.Session(config = tf.ConfigProto(gpu_options =
tf.GPUOptions(per_process_gpu_memory_fraction = 0.5)))
• Code gets messy very quickly
• Poor variable names
• Minimal documentation
• Custom functions / scattered .py
• Multiple library versions

7. KNIME + Workflows
• Medical workflows are complicated involving a large number of steps
• We want transparent, reproducible pipelines for running analysis in
research and production settings

8. Should I learn KNIME?
• Supports
• Matlab, R, Python scripts
• Java code snippets
• Writing your own plugins (Eclipse)
• Natural Language Processing
• Image Processing (full ImageJ / FIJI support, ImgLib2 integration)
• Machine Learning Models (WEKA, scikit-learn, Decision Trees,
PMML)
• Deep Learning (DL4J, Keras model import, and full keras support
coming)
• JavaScript Visualization
• Report Generation
• Excel Input / Output
• Database connectivity

9. • Why ML and Texture Analysis are
important?

11. What is texture analysis
gray-level cooccurrence
matrix
Zhang Y. MRI texture analysis in multiple sclerosis. Int J Biomed Imaging. 2012;2012:762804.

12. Clinical ApplicationsGlioblastoma MGMT methylation:
Korfiatis P, Kline TL, Coufalova L, Lachance DH, Parney IF, Carter RE, Buckner JC, Erickson BJ. MRI texture
features as biomarkers to predict MGMT methylation status in glioblastomas. Med Phys. 2016 Jun;43(6):2835.
Methylated
Unmethylated

13. Clinical Applications
Glioblastoma MGMT methylation:
Korfiatis P, Kline TL, Coufalova L, Lachance DH, Parney IF, Carter RE, Buckner JC, Erickson BJ. MRI texture
features as biomarkers to predict MGMT methylation status in glioblastomas. Med Phys. 2016 Jun;43(6):2835.
Sensitivity: 80.3%
Specificity: 81.3%

14. Clinical Applications
Ability to detect presence of EGFR mutation in
patients w/ Adenocarcinoma of the Lung
Ozkan E, West A, Dedelow JA, Chu BF, Zhao W, Yildiz VO, Otterson GA, Shilo K, Ghosh S, King M, White RD, Erdal
BS. CT Gray-Level Texture Analysis as a Quantitative Imaging Biomarker of Epidermal Growth Factor Receptor
Mutation Status in Adenocarcinoma of the Lung. AJR Am J Roentgenol. 2015 Nov;205(5):1016-25.

15. Texture Analysis
Ozkan E, West A, Dedelow JA, Chu BF, Yildiz VO, Ghosh S, MD, Zhao W, MD, Shilo K, Otterson GA, White
RD, Erdal BS. CT gray level texture analysis as a quantitative imaging biomarker for epidermal growth factor
receptor mutation status in adenocarcinoma of the lung. AJR 2015

16. Clinical Applications
AUC of 0.89
Can Quantitative CT Texture Analysis be Used to Differentiate Fat-poor
Renal Angiomyolipoma from Renal Cell Carcinoma on Unenhanced CT
Images?
Hodgdon T, McInnes MD, Schieda N, Flood TA, Lamb L, Thornhill RE. Can Quantitative CT Texture
Analysis be Used to Differentiate Fat-poor Renal Angiomyolipoma from Renal Cell Carcinoma on
Unenhanced CT Images? Radiology. 2015 Sep;276(3):787-96.

17. Framework (machinelearningcanvas.com / louis@dorard.me)

18. Value Proposition
• We want to help radiologists make decisions about lung nodules
they see

19. Decision
• Is the lung nodule malignant?

20. ML Task
• Identify a given region of interest as malignant or benign
• Input:
• Image of suspect nodule
• Output
• Benign or malignant (Category)

21. Data Sources
1.This course
a. Lung Nodules (64 x 64 tiles)
i. Benign
ii. Malignant
b. Taken from LUNA16 Competition
2. Your own hospital
a. PACS
b. RIS Reports
c. Pathology Reports

22. Data Sources

23. Standard Table
Slice ImageAge
Contrast

24. Enhanced Table
Slice Image
(normalized)
Age
Contrast

25. Collecting Data
• This Course
• Data is already prepared we just have to join it
• At your hospital
• Screen PACS for Chest CTs
• NLP Processing of RIS for Lung Cancer Patients
• KIS Analysis for Pathology Followups
• Manually Labeling of Positive and Negative Nodes

26. Collecting Data
• This Course
• We turn the images
into individual tiles
• We then read in a list
(CSV Reader) with a
malignancy score for
each tile
• We combine them
together

27. Features
• Local patterns of pixels
• 64 x 64 x 1 should be a good start (2D)
• 64 x 64 x 5 would be better (3D)
• HU values
• Patient Age, Gender, Smoking History ??
• Not Referring Physician
• Not Institution
• Not Accession Number or Patient ID
• Not Patient Name
• Not Contrast
• Not Pixel Spacing

29. Adding Features
1.Geometric Features
a. Shape analysis
b. Diameter
measurements
2.Haralick Features
3.Moments
4.Patterns
5.Statistics Features
6.Histogram Features

30. Feature Exploration
1.Calculate image texture features for each image
2.Calculate statistics for features based on groups
a. benign and malignant

31. Statistics
1.Independent groups t-test
a. Groups the data by a variable (Malignancy
in our case)
b. Compares the values for selected
independent variables and shows if they
have a statistically significant difference and
the associated p-value

33. Offline Evaluations
• Predict the category as malignancy or not
• Penalize based on category
• (right or wrong)
• confidence (better be low confidence and wrong, than high confidence
and wrong)

34. Malignancy Results
1.Labels
a. Positive - Malignant
b. Negative - Benign
2. Scoring
a. True Positives - Correctly Identified as Malignant
b. True Negatives - Correctly Identified as Benign
c. False Positives - Incorrectly identified As Malignant (overdiagnosed)

35. ROC Curve
1. True Positives - Correctly Identified as
Malignant
2. True Negatives - Correctly Identified as
Benign
3. False Positives - Incorrectly identified As
Malignant (overdiagnosed)
4. False Negatives - Incorrectly Identified as
Benign (missed cancer)

36. Making Predictions
• Physician clicks on center of a nodule (x,y,z) in DICOM viewer
• We grab an ROI based on that lesion and give it to the model
• Model returns a category and confidence

37. Building a Model
1.Models
a. Partitioning
i. Training Data
ii. Testing Data
b. Model Selection
i. Model
Representation
c. Scoring
i. Confusion Matrix
ii. R^2
iii. ROC Curve

38. Classification / Malignancy
1. Positive - Malignant
2.Negative - Without Contrast

39. Decision Trees

40. Start Texture Deep Dive

41. 41

42. 42

43. Calculations
gray-level cooccurrence
matrix
Zhang Y. MRI texture analysis in multiple sclerosis. Int J Biomed Imaging. 2012;2012:762804.

44. 44

45. 45
CT Gray Level Texture Analysis as
a Quantitative Imaging Biomarker
for Epidermal Growth Factor
Receptor Mutation Status in
Adenocarcinoma of the Lung.
[Ozkan E, et al. Am J Roentgenol
2015]
Radiomics

46. 46
Normalization

47. 47
Texture changes

48. 48
Pitfalls

49. Texture Analysis Workflow Example
Filter FiltersFilter Filters

50. Start from Textures to Deep Learning

51. Image Analysis: Convolutional
Neural Network
• A subtype of Artificial Neural Network
• Biologically inspired - organization of visual cortex
51Heinz Wässle. Parallel processing in the mammalian retina
Nature Reviews Neuroscience 5, 747-757
Light Signal
0.7 to 1.5 million ganglion cells 96.6 million photoreceptors

52. Deep Learning for Images
28 x 28

53. Deep Learning for Images
0.1 0.2 0.3
0.4 0.5 0.6
0.7 0.8 0.9
0.1 0.2 0.3
0.4 0.5 0.6
0.7 0.8 0.9
28 x 28 original Conv 1

54. Deep Learning for Images
Conv 1
0.1 0.2 0.3
0.4 0.5 0.6
0.7 0.8 0.9
0.9 0.8 0.7
0.6 0.5 0.4
0.3 0.2 0.1
Conv 2

55. Deep Learning for Images
Conv 2 Max Pooling

56. Deep Learning for Images
818,743
Max Pooling Dense Weights
Dense Activation

57. Deep Learning for Images
Filter FiltersFilter Filters

58. CONVOLUTIONAL NETWORKS
4Quant | BIG IMAGE ANALYTICS
Pixels Edges Object parts Object models
→ → →

59. Deep Learning with
Convolutional Neural
Networks
Learn textures and patterns from the data

60. Building a Deep Model
1. Here we actually sketch out the
model (from convolutions to
pooling to fully-connected)

61. Entire Deep Pipeline
Model Classifying
And Scoring on the
Validation Data
Dividing into training
and validation data
https://www.kaggle.com/kmader/siim-
medical-image-analysis-
tutorial/discussion/31506

62. Training the Model
1. Training occurs by back-
propagating the different between
the prediction and reality
2.An epoch is one pass through the
entire dataset
3.Many problems require multiple
epochs of training in order to
learn all of the complicated
features

63. Validating the Model
1.Just like with the
decision tree we can
use the Predictor class
to predict the class
using the trained
Neural Network.
2.We can then use the
Scorer and the ROC
curve to show how
accurate the model

64. Visualizing the Model
1.We can use the DL4J
Feedforward Predictor
to show us what is
happening inside of the
model.
2.We combine this with a
DataRow to Image to
show the image as an
X, Y image for each
convolutional filter (T

65. Visualizing the Model
1.We can use the DL4J Feedforward
Predictor to show us what is
happening inside of the model.
2.We combine this with a DataRow to
Image to show the image as an X, Y
image for each convolutional filter (T
dimension)
3.We see the different features the CNN
learned (like smoothing and edges)
Smoothing out
Vertical Edges

66. More Complicated Models
1.LeNet - Created by
Yann LeCun
(http://yann.lecun.com/
exdb/lenet/)
2.We need to add an
additional layer at the
end to make it classify
into two categories
instead of 500

67. Improving Models
1.Tweaking
hyperparameters
2.From AUC 0.71 to 0.89
(massive improvement)

68. Shortcomings
1. Dataset Issues
○ Are all slices really nodules?
○ 2D?
○ Resolution
○ Scanner Type
2. Model Issues
○ 2D
○ 64 x 64 input
○ Negative samples

69. Tips for Success
1. Interdisciplinary teams
2. Focus on your strengths
3. Ask the right questions
What can go wrong
https://www.kaggle.com/kmader/simple-nn-with-keras
Overfitting
Catastrophic Forgetting

70. Continuing your education
1. Kaggle
○ Open Datasets
■ Segment Lungs
■ Classify Nodules
■ Find tumors
○ Competitions
2. Luke Oakden-Rayner’s Blog (MD-PhD)
https://lukeoakdenrayner.wordpress.com/

71. Every person looks a little bit different
Every scan looks a little bit different
Every scanner makes a slightly different image
Every physician marks a slightly different region