Download as PDF, PPTX
Uploaded byPradeep Redddy Raamana
Machine learning, biomarker accuracy and best practices
The document outlines the goals of a workshop focused on understanding machine learning, particularly through the lens of accuracy and the use of biomarker applications in clinical settings. It provides an overview of key concepts in machine learning, including supervised and unsupervised learning, classification techniques, support vector machines, and the importance of cross-validation for model generalizability. The session emphasizes both the clinical applications of machine learning, such as decision support and personalized medicine, and methodologies for ensuring accurate and effective model performance.
More Related Content
Similar to Machine learning, biomarker accuracy and best practices
Machine learning, biomarker accuracy and best practices
- 1. Machine learning, biomarker accuracy and best practices Pradeep Reddy Raamana crossinvalidation.com
- 2. Singular goal ofworkshop Accuracydistribution model 1 model 2 model 3 model 4 model 5 model 6 • understand
- 3. Singular goal ofworkshop Accuracydistribution model 1 model 2 model 3 model 4 model 5 model 6 • understand • machine learning
- 4. Singular goal ofworkshop Accuracydistribution model 1 model 2 model 3 model 4 model 5 model 6 • understand • machine learning • support vector machine
- 5. Singular goal ofworkshop Accuracydistribution model 1 model 2 model 3 model 4 model 5 model 6 • understand • machine learning • support vector machine • classification accuracy
- 6. Singular goal ofworkshop Accuracydistribution model 1 model 2 model 3 model 4 model 5 model 6 • understand • machine learning • support vector machine • classification accuracy • cross-validation
- 7. What is MachineLearning?
- 8. What is MachineLearning? • “giving computers the ability to learn without being explicitly programmed.”
- 9. What is MachineLearning? • “giving computers the ability to learn without being explicitly programmed.” • i.e. building algorithms to learn patterns in data
- 10. What is MachineLearning? • “giving computers the ability to learn without being explicitly programmed.” • i.e. building algorithms to learn patterns in data • automatically
- 11. Examples images from varioussites on internet
- 12. Examples images from varioussites on internet
- 13. Examples images from varioussites on internet
- 14. Examples images from varioussites on internet
- 15. Examples images from varioussites on internet
- 16. Examples images from varioussites on internet
- 17. Clinical Application ofML • ML has many clinical applications, including:
- 18. Clinical Application ofML • ML has many clinical applications, including: • computer-aided diagnosis • clinical decision support
- 19. Clinical Application ofML • ML has many clinical applications, including: • computer-aided diagnosis • clinical decision support • personalized medicine
- 20. Clinical Application ofML • ML has many clinical applications, including: • computer-aided diagnosis • clinical decision support • personalized medicine • treatment and monitoring
- 21. Clinical Application ofML • ML has many clinical applications, including: • computer-aided diagnosis • clinical decision support • personalized medicine • treatment and monitoring • better care and service delivery systems (reduce length of hospitalization, optimize resource redistribution etc)
- 22. Clinical Application ofML • ML has many clinical applications, including: • computer-aided diagnosis • clinical decision support • personalized medicine • treatment and monitoring • better care and service delivery systems (reduce length of hospitalization, optimize resource redistribution etc) • I will focus on biomarkers today!
- 23. Clinical Application ofML • ML has many clinical applications, including: • computer-aided diagnosis • clinical decision support • personalized medicine • treatment and monitoring • better care and service delivery systems (reduce length of hospitalization, optimize resource redistribution etc) • I will focus on biomarkers today! • more on how to assess their utility
- 24. Clinical Application ofML • ML has many clinical applications, including: • computer-aided diagnosis • clinical decision support • personalized medicine • treatment and monitoring • better care and service delivery systems (reduce length of hospitalization, optimize resource redistribution etc) • I will focus on biomarkers today! • more on how to assess their utility • less on how to identify, build and tune them.
- 25. Types of Machinelearning Data Labelled SupervisedUnsupervised Data Not labelled
- 26.
- 27.
- 28. Unsupervised: examples images fromwikipedia.com and gerfficient.com
- 29. Unsupervised: examples • Clustering imagesfrom wikipedia.com and gerfficient.com
- 30. Unsupervised: examples • Clustering •Blind source separation • PCA • ICA images from wikipedia.com and gerfficient.com
- 31.
- 32.
- 33.
- 34.
- 35.
- 36. Supervised: examples support vector machine linearclassifier A B
- 37. Supervised: examples support vector machine linearclassifier A B decision tree is x1 < 1.5 BA yes no
- 38.
- 39.
- 40.
- 41.
- 42. Terminology names→ counter↓ sepal width sepal length petal width petal length class 1 0.2 1.10.4 1 setosa features (variables, dimensions, columns etc) PetalSepal
- 43. Terminology names→ counter↓ sepal width sepal length petal width petal length class 1 0.2 1.10.4 1 setosa 2 0.35 0.9 0.1 2 setosa 3 0.3 … 4 0.28 versicolor 5 .. versicolor … .. .. … 0.45 virginica N 0.35 virginica samples (observations, data points etc) features (variables, dimensions, columns etc) PetalSepal
- 44. Terminology names→ counter↓ sepal width sepal length petal width petal length class 1 0.2 1.10.4 1 setosa 2 0.35 0.9 0.1 2 setosa 3 0.3 … 4 0.28 versicolor 5 .. versicolor … .. .. … 0.45 virginica N 0.35 virginica samples (observations, data points etc) features (variables, dimensions, columns etc) PetalSepal y ↓ → X
- 45.
- 46.
- 47. Classification Training data New testdata Build the classifier
- 48. Classification Training data New testdata map to known classes Build the classifier
- 49.
- 50. Support Vector Machine(SVM) • A popular classification technique
- 51. Support Vector Machine(SVM) • A popular classification technique
- 52. Support Vector Machine(SVM) • A popular classification technique • At its core, it is • binary (separate two classes)
- 53. Support Vector Machine(SVM) • A popular classification technique • At its core, it is • binary (separate two classes) • linear (boundary: line in 2d or hyperplane in n-d)
- 54. Support Vector Machine(SVM) • A popular classification technique • At its core, it is • binary (separate two classes) • linear (boundary: line in 2d or hyperplane in n-d) • Its power lies in finding the boundary between classes difficult to separate
- 55. How does SVMwork? x1 x2
- 56. How does SVMwork? x1 x2
- 57. How does SVMwork? x1 x2
- 58. How does SVMwork? L1 x1 x2
- 59. How does SVMwork? L1 L2 x1 x2
- 60. How does SVMwork? L1 L2 L3 x1 x2
- 61. How does SVMwork? L1 L2 L3 x1 x2
- 62. How does SVMwork? L1 L2 L3 x1 x2
- 63. How does SVMwork? L1 L2 L3 x1 x2 support vectors
- 64. How does SVMwork? L3 x1 x2 support vectors
- 65. How does SVMwork? L3 x1 x2 support vectors
- 66. Harder problem (classesare not linearly separable) x1 x2
- 67. Harder problem (classesare not linearly separable) x1 x2
- 68. Harder problem (classesare not linearly separable) L1 x1 x2
- 69. Harder problem (classesare not linearly separable) L1 x1 x2 L1→less errors, smaller margin
- 70. Harder problem (classesare not linearly separable) L1 L2 x1 x2 L1→less errors, smaller margin
- 71. Harder problem (classesare not linearly separable) L1 L2 x1 x2 L1→less errors, smaller margin L2→more errors, larger margin
- 72. Harder problem (classesare not linearly separable) L1 L2 x1 x2 L1→less errors, smaller margin L2→more errors, larger margin Tradeoff between error and margin!
- 73. Harder problem (classesare not linearly separable) L1 L2 x1 x2 L1→less errors, smaller margin L2→more errors, larger margin Tradeoff between error and margin! parameter C: penalty for misclassification
- 74.
- 75.
- 76.
- 77.
- 78.
- 79.
- 80.
- 81.
- 82. Transform to higher dimensions x1 x2=x1^2 Weturned the linear problem into a nonlinear problem.
- 83. Transform to higher dimensions x1 x2=x1^2 Weturned the linear problem into a nonlinear problem. This trick is achieved via kernel functions!
- 84.
- 85.
- 86.
- 87. Recap: SVM • Linearclassifier at its core
- 88. Recap: SVM • Linearclassifier at its core • Boundary with max. margin
- 89. Recap: SVM • Linearclassifier at its core • Boundary with max. margin • Input data can be transformed to higher dimensions to achieve better separation
- 90. Classifier Performance • Howdo you evaluate how well the classifier works? • input unseen data with known labels (ground truth) • make predictions with previously trained classifier • using ground truth, • compute % of when prediction matches ground truth —> classification accuracy
- 91.
- 92.
- 93.
- 94. P. Raamana What isgeneralizability? available data (sample*) 23*has a statistical definition
- 95. P. Raamana What isgeneralizability? available data (sample*) 23*has a statistical definition
- 96. P. Raamana What isgeneralizability? available data (sample*) desired: accuracy on unseen data (population*) 23*has a statistical definition
- 97. P. Raamana What isgeneralizability? available data (sample*) desired: accuracy on unseen data (population*) 23*has a statistical definition
- 98. P. Raamana What isgeneralizability? available data (sample*) desired: accuracy on unseen data (population*) out-of-sample predictions 23*has a statistical definition
- 99. P. Raamana What isgeneralizability? available data (sample*) desired: accuracy on unseen data (population*) out-of-sample predictions 23 avoid overfitting *has a statistical definition
- 100.
- 101.
- 102.
- 103.
- 104.
- 105. P. Raamana 50 shadesof overfitting 25 Decade Population © mathworks human annihilation?
- 106. P. Raamana “Clever formsof overfitting” 26from http://hunch.net/?p=22
- 107. P. Raamana “Clever formsof overfitting” 26from http://hunch.net/?p=22
- 108.
- 109. P. Raamana Cross-validation • Whatis cross-validation? Training set Test set 27
- 110. P. Raamana Cross-validation • Whatis cross-validation? • How to perform it? Training set Test set ≈ℵ≈ 27
- 111. P. Raamana Cross-validation • Whatis cross-validation? • How to perform it? • What are the effects of different CV choices? Training set Test set ≈ℵ≈ 27
- 112. P. Raamana Cross-validation • Whatis cross-validation? • How to perform it? • What are the effects of different CV choices? Training set Test set ≈ℵ≈ negative bias unbiased positive bias 27
- 113. P. Raamana CV helpsquantify generalizability 28
- 114. P. Raamana CV helpsquantify generalizability 28
- 115. P. Raamana CV helpsquantify generalizability 28
- 116. P. Raamana CV helpsquantify generalizability 28
- 117. P. Raamana Why cross-validate? Trainingset Test set 29
- 118. P. Raamana Why cross-validate? Trainingset Test set bigger training set better learning 29
- 119. P. Raamana Why cross-validate? Trainingset Test set bigger training set better learning better testing bigger test set 29
- 120. P. Raamana Why cross-validate? Trainingset Test set bigger training set better learning better testing bigger test set Key: Train & test sets must be disjoint. 29
- 121. P. Raamana Why cross-validate? Trainingset Test set bigger training set better learning better testing bigger test set Key: Train & test sets must be disjoint. And the dataset or sample size is fixed. 29
- 122. P. Raamana Why cross-validate? Trainingset Test set bigger training set better learning better testing bigger test set Key: Train & test sets must be disjoint. And the dataset or sample size is fixed. They grow at the expense of each other! 29
- 123. P. Raamana Why cross-validate? Trainingset Test set bigger training set better learning better testing bigger test set Key: Train & test sets must be disjoint. And the dataset or sample size is fixed. They grow at the expense of each other! 29
- 124. P. Raamana Why cross-validate? Trainingset Test set bigger training set better learning better testing bigger test set Key: Train & test sets must be disjoint. And the dataset or sample size is fixed. They grow at the expense of each other! cross-validate to maximize both 29
- 125.
- 126. P. Raamana Use cases •“When setting aside data for parameter estimation and validation of results can not be afforded, cross-validation (CV) is typically used” 30
- 127. P. Raamana Use cases •“When setting aside data for parameter estimation and validation of results can not be afforded, cross-validation (CV) is typically used” • Use cases: 30
- 128. P. Raamana accuracydistribution fromrepetitionofCV(%) Use cases •“When setting aside data for parameter estimation and validation of results can not be afforded, cross-validation (CV) is typically used” • Use cases: • to estimate generalizability (test accuracy) 30
- 129. P. Raamana accuracydistribution fromrepetitionofCV(%) Use cases •“When setting aside data for parameter estimation and validation of results can not be afforded, cross-validation (CV) is typically used” • Use cases: • to estimate generalizability (test accuracy) • to pick optimal parameters (model selection) 30
- 130. P. Raamana accuracydistribution fromrepetitionofCV(%) Use cases •“When setting aside data for parameter estimation and validation of results can not be afforded, cross-validation (CV) is typically used” • Use cases: • to estimate generalizability (test accuracy) • to pick optimal parameters (model selection) • to compare performance (model comparison). 30
- 131. P. Raamana Key Aspectsof CV 31
- 132. P. Raamana Key Aspectsof CV 1. How you split the dataset into train/test 31
- 133. P. Raamana Key Aspectsof CV 1. How you split the dataset into train/test •maximal independence between training and test sets is desired. 31
- 134. P. Raamana Key Aspectsof CV 1. How you split the dataset into train/test •maximal independence between training and test sets is desired. •This split could be • over samples (e.g. indiv. diagnosis) samples (rows) 31
- 135. P. Raamana Key Aspectsof CV 1. How you split the dataset into train/test •maximal independence between training and test sets is desired. •This split could be • over samples (e.g. indiv. diagnosis) samples (rows) 31 healt hy dise ase
- 136. P. Raamana Key Aspectsof CV 1. How you split the dataset into train/test •maximal independence between training and test sets is desired. •This split could be • over samples (e.g. indiv. diagnosis) • over time (for task prediction in fMRI) time (columns) samples (rows) 31 healt hy dise ase
- 137. P. Raamana Key Aspectsof CV 1. How you split the dataset into train/test •maximal independence between training and test sets is desired. •This split could be • over samples (e.g. indiv. diagnosis) • over time (for task prediction in fMRI) time (columns) samples (rows) 31 healt hy dise ase
- 138. P. Raamana Key Aspectsof CV 1. How you split the dataset into train/test •maximal independence between training and test sets is desired. •This split could be • over samples (e.g. indiv. diagnosis) • over time (for task prediction in fMRI) 2. How often you repeat randomized splits? •to expose classifier to full variability •As many as times as you can e.g. 100 ≈ℵ≈ time (columns) samples (rows) 31 healt hy dise ase
- 139. P. Raamana Validation set Trainingset 32*biased towards X —> overfit to X
- 140. P. Raamana Validation set goodnessof fit of the model Training set 32*biased towards X —> overfit to X
- 141. P. Raamana Validation set goodnessof fit of the model biased* towards the training set Training set 32*biased towards X —> overfit to X
- 142. P. Raamana Validation set goodnessof fit of the model biased* towards the training set Training set Test set 32*biased towards X —> overfit to X
- 143. P. Raamana Validation set goodnessof fit of the model biased* towards the training set Training set Test set ≈ℵ≈ 32*biased towards X —> overfit to X
- 144. P. Raamana Validation set optimize parameters goodnessof fit of the model biased* towards the training set Training set Test set ≈ℵ≈ 32*biased towards X —> overfit to X
- 145. P. Raamana Validation set optimize parameters goodnessof fit of the model biased towards the test set biased* towards the training set Training set Test set ≈ℵ≈ 32*biased towards X —> overfit to X
- 146. P. Raamana Validation set optimize parameters goodnessof fit of the model biased towards the test set biased* towards the training set Training set Test set Validation set ≈ℵ≈ 32*biased towards X —> overfit to X
- 147. P. Raamana Validation set optimize parameters goodnessof fit of the model biased towards the test set biased* towards the training set evaluate generalization independent of training or test sets Training set Test set Validation set ≈ℵ≈ 32*biased towards X —> overfit to X
- 148. P. Raamana Validation set optimize parameters goodnessof fit of the model biased towards the test set biased* towards the training set evaluate generalization independent of training or test sets Whole dataset Training set Test set Validation set ≈ℵ≈ 32*biased towards X —> overfit to X
- 149. P. Raamana Validation set optimize parameters goodnessof fit of the model biased towards the test set biased* towards the training set evaluate generalization independent of training or test sets Whole dataset Training set Test set Validation set ≈ℵ≈ inner-loop 32*biased towards X —> overfit to X
- 150. P. Raamana Validation set optimize parameters goodnessof fit of the model biased towards the test set biased* towards the training set evaluate generalization independent of training or test sets Whole dataset Training set Test set Validation set ≈ℵ≈ inner-loop outer-loop 32*biased towards X —> overfit to X
- 151.
- 152.
- 153. P. Raamana Terminology 33 Data split Training Testing Validation Purpose(Do’s) Train model to learn its core parameters Optimize hyperparameters Evaluate fully-optimized classifier to report performance
- 154. P. Raamana Terminology 33 Data split Training Testing Validation Purpose(Do’s) Train model to learn its core parameters Optimize hyperparameters Evaluate fully-optimized classifier to report performance Don’ts (Invalid use) Don’t report training error as the test error! Don’t do feature selection or anything supervised on test set to learn or optimize! Don’t use it in any way to train classifier or optimize parameters
- 155. P. Raamana Terminology 33 Data split Training Testing Validation Purpose(Do’s) Train model to learn its core parameters Optimize hyperparameters Evaluate fully-optimized classifier to report performance Don’ts (Invalid use) Don’t report training error as the test error! Don’t do feature selection or anything supervised on test set to learn or optimize! Don’t use it in any way to train classifier or optimize parameters Alternative names Training (no confusion) Validation (or tweaking, tuning, optimization set) Test set (more accurately reporting set)
- 156.
- 157.
- 158. P. Raamana K-fold CV TrainTest, 4th fold trial 1 2 … k 34
- 159. P. Raamana K-fold CV TrainTest, 4th fold trial 1 2 … k 34
- 160. P. Raamana K-fold CV TrainTest, 4th fold trial 1 2 … k 34
- 161. P. Raamana K-fold CV TrainTest, 4th fold trial 1 2 … k 34
- 162. P. Raamana K-fold CV Testsets in different trials are indeed mutually disjoint Train Test, 4th fold trial 1 2 … k 34
- 163. P. Raamana K-fold CV Testsets in different trials are indeed mutually disjoint Train Test, 4th fold trial 1 2 … k Note: different folds won’t be contiguous. 34
- 164. P. Raamana K-fold CV Testsets in different trials are indeed mutually disjoint Train Test, 4th fold trial 1 2 … k Note: different folds won’t be contiguous. 34
- 165. P. Raamana Repeated HoldoutCV Set aside an independent subsample (e.g. 30%) for testing whole dataset 35
- 166. P. Raamana Repeated HoldoutCV Train Test trial 1 2 … n Set aside an independent subsample (e.g. 30%) for testing whole dataset 35
- 167. P. Raamana Repeated HoldoutCV Train Test trial 1 2 … n Set aside an independent subsample (e.g. 30%) for testing whole dataset 35
- 168. P. Raamana Repeated HoldoutCV Train Test trial 1 2 … n Set aside an independent subsample (e.g. 30%) for testing whole dataset 35
- 169. P. Raamana Repeated HoldoutCV Train Test trial 1 2 … n Set aside an independent subsample (e.g. 30%) for testing whole dataset 35
- 170. P. Raamana Repeated HoldoutCV Train Test trial 1 2 … n Note: there could be overlap among the test sets from different trials! Hence large n is recommended. Set aside an independent subsample (e.g. 30%) for testing whole dataset 35
- 171. Typical workflow Whole dataset (randomizedsplit) Training set (with labels) feature extraction selection parameter optimization (on training data only) Trained classifier Test set: rest (no labels) Same feature extraction Select same features Evaluate on test set Pool predictions over repetitions NextCVrepetitioniofn Accuracydistribution
- 172. Software • There isa free machine learning toolbox in every major language! • Check below for the latest techniques/toolboxes: • http://www.jmlr.org/mloss/ or • http://mloss.org/software/
- 173. Confusion Matrices Feature Importance Accuracydistributions Intuitive comparison of misclassification rates neuropredict : easy and comprehensive predictive analysis input features • each feature set could be any set of numbers estimated from sample by itself (intrinsic, not group-wise) • designed to seamlessly compare many features (n>1), if they are all from same set of samples belonging to same classes • supports many input formats. • plugs directly into outputs from popular software like Freesurfer. neuropredict • performs cross-validation, in such a way to increase power of statistical comparisons later on • tracks misclassification rates: class- and subject-wise, for each feature • measures feature importance • statistical comparison of predictive performance • intuitive visualizations • stream-lined comparison for a large number of features! docs: http://neuropredict.readthedocs.io code: github.com/raamana/neuropredict twitter: @raamana_
- 174.
- 175. neuropredict features • Auto-readingof neuroimaging features
- 176. neuropredict features • Auto-readingof neuroimaging features • Auto-evaluation of predictive accuracy
- 177. neuropredict features • Auto-readingof neuroimaging features • Auto-evaluation of predictive accuracy • Auto-comparison of performance …
- 178. neuropredict features • Auto-readingof neuroimaging features • Auto-evaluation of predictive accuracy • Auto-comparison of performance … • Notice the word I am repeating?
- 179. neuropredict features • Auto-readingof neuroimaging features • Auto-evaluation of predictive accuracy • Auto-comparison of performance … • Notice the word I am repeating? • Auto • Auto • Auto
- 180. neuropredict features • Auto-readingof neuroimaging features • Auto-evaluation of predictive accuracy • Auto-comparison of performance … • Notice the word I am repeating? • Being automatic is important, without which, it becomes hard, and error-prone!! • Auto • Auto • Auto
- 181.
- 182.
- 183.
- 184. Confusion Matrices Accuracy distributionsIntuitive comparison of misclassification rates sample outputs
- 185. Confusion Matrices Feature Importance Accuracydistributions Intuitive comparison of misclassification rates sample outputs
- 186. Now, it’s timeto neuropredict! xkcd .com/raamana
- 187. Model selection Friedman, J.,Hastie, T., & Tibshirani, R. (2008). The elements of statistical learning. Springer, Berlin: Springer series in statistics.