Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Towards psychoinformatics with machine learning and brain imaging

1,042 views

Published on

Informatics in the psychological sciences brings fascinating challenges as mental processes or pathologies have fuzzy definition and are hard to quantify. Brain imaging brings rich data on the neural substrate of these concepts, yet it is a non trivial link.

The goal of this presentation is to put forward basic ideas of "psychoinformatics", using advanced processing on brain images to quantify better the elements of psychology.

It discusses how machine learning can bridge brain images to behavior: to describe better mental processes involved in brain activity, or to extract biomarkers of pathologies, individual traits, or cognition.

Published in: Science
  • Login to see the comments

Towards psychoinformatics with machine learning and brain imaging

  1. 1. Towards psychoinformatics with machine learning and brain imaging Gaël Varoquaux
  2. 2. Towards psychoinformatics with machine learning and brain imaging Gaël Varoquaux Propose and illustrate a research program
  3. 3. Accumulation of functional brain data G Varoquaux 2
  4. 4. Theories of the mind Laws of the mind for perception, decision, action, emotion... G Varoquaux 3
  5. 5. Studying mental processes 1 Craft an experimental condition that recruits it G Varoquaux 4
  6. 6. Studying mental processes 1 Craft an experimental condition that recruits it 2 Do an elementary psychological manipulation G Varoquaux 4
  7. 7. Studying mental processes - -Results in a contrast Study by oppositions G Varoquaux 4
  8. 8. Studying mental processes - -Results in a contrast Study by oppositions Results tied to a simple psychological manipulation bound to a paradigm G Varoquaux 4
  9. 9. Proposing Generalization to build broader theories bridging paradigms Paradigm 1: Seen G Varoquaux 5
  10. 10. Proposing Generalization to build broader theories bridging paradigms Paradigm 1: Seen Paradigm 2: Imagined G Varoquaux 5
  11. 11. Generalization for broader theories bridging paradigms Seen objects G Varoquaux 6
  12. 12. Brain imaging ⇒ brain-mind associations Difficulty: modeling behavior & cognition G Varoquaux 7
  13. 13. Predictive models can broaden theories by generalizing brain-mind associations to arbitrary new tasks and stimuli [Varoquaux and Poldrack 2018] 1 Beyond oppositions: encoding 2 Generalizing across tasks 3 Universal cognitive representations 4 Across subjects: biomarkers G Varoquaux 8
  14. 14. 1 Beyond oppositions: encoding [Eickenberg... 2017] G Varoquaux 9
  15. 15. 1 Decomposing psychological process To study vision: Breaking down stimuli G Varoquaux 10
  16. 16. 1 Decomposing psychological process To study vision: Breaking down stimuli [Hubel and Wiesel 1962] Neurons receptive to Gabors (edges) G Varoquaux 10
  17. 17. 1 Decomposing psychological process To study vision: Breaking down stimuli [Hubel and Wiesel 1962] Neurons receptive to Gabors (edges) [Logothetis... 1995] Shapes in inferior temporal cortex G Varoquaux 10
  18. 18. 1 Decomposing psychological process To study vision: Breaking down stimuli Image V1 cortex V2 cortex Inferior temporal cortex Fusiform face area Jack? Is there a “face” region? A “foot” region? A “left big toe” region? G Varoquaux 10
  19. 19. 1 Crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs G Varoquaux 11
  20. 20. 1 Crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs G Varoquaux 11
  21. 21. 1 Crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs - G Varoquaux 11
  22. 22. 1 Crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs - Representing only one aspect of the stimuli: too much reductionism G Varoquaux 11
  23. 23. 1 Not crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs - Representing only one aspect of the stimuli: too much reductionism Encoding Building complex representations of stimuli Predicting brain response from them G Varoquaux 11
  24. 24. 1 Decomposing visual stimuli Low-level visual cortex is tuned to natural image statistics [Olshausen et al. 1996] What drives high-level representations? The concept of “left big toe”? G Varoquaux 12
  25. 25. 1 Decomposing visual stimuli Low-level visual cortex is tuned to natural image statistics [Olshausen et al. 1996] What drives high-level representations? Convolutional Net G Varoquaux 12
  26. 26. 1 Brain mapping: encoding with conv nets StimulirepresentationBrainactivity Input Layer 1 Layer 5 Linear predictive models Convolutional Net G Varoquaux 13
  27. 27. 1 Brain mapping: encoding with conv nets StimulirepresentationBrainactivity Input Layer 1 Layer 5 Linear predictive models Convolutional Net Explains the brain activity from the stimuli much better than hand-crafted features On data from [Kay... 2008] natural images [Huth... 2012] movies [Eickenberg... 2017] G Varoquaux 13
  28. 28. 1 Brain mapping: encoding with conv nets [Eickenberg... 2017] High-level conv-net layer map to high-level visual areas G Varoquaux 14
  29. 29. 1 Brain mapping with conv nets: retinotopy Mapping response to exentricity: artifical retinotopy G Varoquaux 15
  30. 30. 1 Brain mapping with conv nets: high-level concepts Opposing face versus place in the [Haxby... 2001] stimuli G Varoquaux 16
  31. 31. [Eickenberg... 2017] Beyond oppositions: encoding Recovers the hierarchy of visual modules Generalizes from natural images to retinotopy or face vs place G Varoquaux 17
  32. 32. 2 Generalizing across tasks Characterizing the function of brain structures How much is observed activity a consequence of specificities of the paradigm? [Schwartz... 2013, Varoquaux... 2018] G Varoquaux 18
  33. 33. 2 Large-scale decoding for reverse inference Cat Dog Ouaf Miaou [Poldrack... 2009] G Varoquaux 19
  34. 34. 2 Large-scale decoding for reverse inference Cat Dog Ouaf Miaou [Poldrack... 2009] Variability: an opportunity and a challenge Technical heterogeneity (scanner, stimulus modality...) Paradigmatic isolation (cognition) [Newell 1973] G Varoquaux 19
  35. 35. 2 Generalizing to arbitrary paradigms [Schwartz... 2013, Varoquaux... 2018] What is this brain doing? Describe tasks by their cognitive components Multi-label prediction: presence or absence of each label Visual Auditory Foot Hand Calculation Reading Checkboard Face Place Object Digit Saccade ... ... ... ... ... ... G Varoquaux 20
  36. 36. 2 Generalizing to arbitrary paradigms [Schwartz... 2013, Varoquaux... 2018] What is this brain doing? Describe tasks by their cognitive components Multi-label prediction: presence or absence of each label Visual Auditory Checkboard Face ... ... ... ... Prediction across studies Describe a task never seen from the brain activity it evokes G Varoquaux 20
  37. 37. 2 Generalizing to arbitrary paradigms [Schwartz... 2013, Varoquaux... 2018] What is this brain doing? + + Regions specific to that predict facets of cognition G Varoquaux 20
  38. 38. 2 Decoding mega-analysis [Schwartz... 2013, Varoquaux... 2018] 30 studies 837 subjects 196 experimental conditions 6919 activation maps Different labs Dehaene Poldrack Wager ... Various cognitive domains Language Vision Decision making Mathematics ... All manually preprocessed, labeled, and curated G Varoquaux 21
  39. 39. 2 Decoding mega-analysis [Schwartz... 2013, Varoquaux... 2018] 30 studies 837 subjects 196 experimental conditions 6919 activation maps Different labs Dehaene Poldrack Wager ... Various cognitive domains Language Vision Decision making Mathematics ... All manually preprocessed, labeled, and curated Decoding arbitrary new paradigms visualauditoryVcheckerboard Hcheckerboardobjectsscramblefacesplacesdigits wordsrighthandlefthandrightfootleftfootlanguagehumansound non-humansound 0.2 0.4 0.6 0.8 1.0 Predictionscore (ROCAUC) ontologydecoder logisticregression naiveBayes neurosynth ­0.3 ­0.2 ­0.1 0.0 +0.1 +0.2 +0.3 Relative score G Varoquaux 21
  40. 40. 2 Mapping: regions specific to facets of cognition [Schwartz... 2013, Varoquaux... 2018] Contrasts: good rejection of confounds Decoding: reverse inference ⇒ Consensus to define regions The “place” concept Forward term effect Brain response to tasks with the “place” concept Forward ontology contrast Contrasting responses to “place” and related concepts Reverse decoder map Regions that predict a place- recognition task Consensus forward & reverse Regions selected by forward and reverse inference Replaces the careful crafting of control conditions G Varoquaux 22
  41. 41. 2 A functional atlas: central gyrus, motor networks [Schwartz... 2013, Varoquaux... 2018] Neurosynth Reverse inference (Logistic regression) Decoding in an ontology z = -15 Reverse inference -15 nference DecodDecoding in an visual faces places objects scrambled words digits horizontal - vertical checkerboard Visual regions Contrasts -based analysis Neurosynth reverse inference Decoding and ontology + Language and auditory mapping in temporal cortex + Calculation and spatial attenation in IPS + Mapping the motor system G Varoquaux 23
  42. 42. [Varoquaux... 2018] Generalizing across tasks Cat Dog Ouaf Miaou Multi-label prediction Prediction (discriminative models) more robust to heterogeneity Decoding ⇒ evidence beyond ⇒ a paradigm Atlasing cognition G Varoquaux 24
  43. 43. Need an ontology of cognition Episodic memory working memory lexical memory Syntax comprehension Reading Mental calculation Shape recognition Face recognition Memory Language Vision G Varoquaux 25
  44. 44. 3 Universal cognitive representations [Mensch... 2017] Face Shape Reading Shape Language Music Saccades Calculation G Varoquaux 26
  45. 45. 3 Universal cognitive representations [Mensch... 2017] Face Shape Reading Shape Language Music Saccades CalculationMulti-task learning Solve many related but different problems Learn commonalities intermediate representations G Varoquaux 26
  46. 46. 3 Deep architecture [Bengio 2009] ... Great for multiple output (tasks) G Varoquaux 27
  47. 47. 3 Deep architecture [Bengio 2009] ... Great for multiple output (tasks) Millions of parameters, thousands of data points G Varoquaux 27
  48. 48. 3 Deep architecture 2 2 Great for multiple output (tasks) Millions of parameters, thousands of data points Simplify G Varoquaux 27
  49. 49. 3 Shallow architecture 2 linear model Great for multiple output (tasks) Millions of parameters, thousands of data points Simplify simplify more G Varoquaux 27
  50. 50. 3 Shallow architecture 2 2 Great for multiple output (tasks) Millions of parameters, thousands of data points Simplify simplify more G Varoquaux 27
  51. 51. 3 Deep linear model – multi-task – Bayesian 4TB resting-state data HCP900 OpenfMRI HCP Camcan Brainomics 42000 task fMRI contrast maps for one model ... Many rest & task fMRI studies ... x Information flow x Learned network combination 1st layer assignment Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Joint training Bayesian deep learning Resting-state functional loadings 512 resting-state networks Task functional loadings 128 task-optimized networks Contrast maps 200,000 voxels Contrast name Probability vector FACE BODY HOUSE TOOL 512 components [Mensch... 2018] G Varoquaux 28
  52. 52. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information flow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks G Varoquaux 28
  53. 53. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information flow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks Improves decoding in each study 0% 20% 40% 60% 80% Decoding accuracy on test set Stop-signal w/ spoken & manual resp. The Human Connectome Project UCLA LA5C Simon task Plain or mirror-reversed text Stop-signal CamCan audio-visual Localizer Cross-language repetition priming Twin localizer High-level math Spatio-temporal judgement (retake) Constit. struct. of sent. & music Sentence/music complexity Word & object processing Emotion regulation Arithmetic & saccades Brainomics localizer Incidental encoding Spatio-temporal judgement False belief Visual object recognition Face recognition Stop-signal & classification Classification learning Compression Motor task & word/verb generation BART, stop-signal, emotion Rhyme judgment Auditory & Visual Oddball Weather prediction Stop-signal & classification (retake) Balloon Analog Risk-taking Mixed-gambles Classif. learning & reversal Task fMRI study Decoding from voxels Decoding from multi-study networks Chance level -5% 0% 5%10%15% Multi-study acc. gain G Varoquaux 28
  54. 54. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information flow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks Improves decoding in each study 4 16 32 100 400 Number of train subjects 0% 5% 10% 15% 20% Multistudyacc.gain Study Small studies benefit from large ones G Varoquaux 28
  55. 55. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information flow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks Enables smaller sample sizes 7 15 23 31 39 # training subjects in target study 70% 75% 80% 85% Testaccuracy Pinel et al. '07 9 18 28 37 47 70% 80% 90% Papadopoulos- Orfanos '12 60 12 60% 65% C (Sh 37 47 oulos- '12 60 121 181 242 302 60% 65% CamCan AV (Shafto et al. '14) Standard decoding Multi-study decoder 2 4 7 9 12 20% 40% 60% Cohen '09 G Varoquaux 28
  56. 56. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information flow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks G Varoquaux 28
  57. 57. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information flow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks G Varoquaux 28
  58. 58. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information flow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks G Varoquaux 28
  59. 59. [Mensch... 2017] Universal cognitive representations Multi-task decoding: - decoding the original task labels Deep linear models: - intermediate representations across studies - statistical power from big to small studies Face Shape Reading Shape Language Music Saccades Calculation G Varoquaux 29
  60. 60. [Mensch... 2017] Universal cognitive representations Multi-task decoding: - decoding the original task labels Deep linear models: - intermediate representations across studies - statistical power from big to small studies Face Shape Reading Shape Language Music Saccades Calculation G Varoquaux 29
  61. 61. 4 Across subjects: biomarkers G Varoquaux 30
  62. 62. 4 Beyond heterogeneity: predicting autism across sites Accuracy Fraction of subjects used More data is better (up to 1000 subjects) [Abraham... 2017] G Varoquaux 31
  63. 63. 4 Brain aging: a surrogate biomarker [Liem... 2017] Predicting brain aging = chronological age Multi-modal: brain connectivity and morphology Age prediction: mean absolute error of 4.3 years G Varoquaux 32
  64. 64. 4 Brain aging: a surrogate biomarker [Liem... 2017] Predicting brain aging = chronological age Multi-modal: brain connectivity and morphology Age prediction: mean absolute error of 4.3 years Discrepency with chronological age correlates with cognitive impairment 0 2 4 Brain aging discrepancy (years) -0.38 0.74 1.72 Objective Cognitive Impairment group Normal Mild Major G Varoquaux 32
  65. 65. @GaelVaroquaux Psychoinformatics with machine learning Prediction for broader theories AI to model stimuli / the world Explicit generalization across paradigms Cat Dog Ouaf Miaou
  66. 66. @GaelVaroquaux Psychoinformatics with machine learning Prediction for broader theories AI to model stimuli / the world Explicit generalization across paradigms Beyond oppositions Encoding complete descriptions of tasks Decoding multiple facets of cognitions
  67. 67. @GaelVaroquaux Psychoinformatics with machine learning Prediction for broader theories AI to model stimuli / the world Explicit generalization across paradigms Beyond oppositions Encoding complete descriptions of tasks Decoding multiple facets of cognitions Useful with imperfect labels Extracting common representations Surrogate biomarkers
  68. 68. @GaelVaroquaux Psychoinformatics with machine learning Prediction for broader theories AI to model stimuli / the world Explicit generalization across paradigms Beyond oppositions Encoding complete descriptions of tasks Decoding multiple facets of cognitions Useful with imperfect labels Extracting common representations Surrogate biomarkers Software: nilearn http://nilearn.github.io ni
  69. 69. References I A. Abraham, M. P. Milham, A. Di Martino, R. C. Craddock, D. Samaras, B. Thirion, and G. Varoquaux. Deriving reproducible biomarkers from multi-site resting-state data: An autism-based example. NeuroImage, 147:736–745, 2017. Y. Bengio. Learning deep architectures for ai. Foundations and trends in Machine Learning, 2:1–127, 2009. M. Eickenberg, A. Gramfort, G. Varoquaux, and B. Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. NeuroImage, 152:184–194, 2017. J. V. Haxby, I. M. Gobbini, M. L. Furey, ... Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293:2425, 2001. D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of physiology, 160:106, 1962.
  70. 70. References II A. G. Huth, S. Nishimoto, A. T. Vu, and J. L. Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76:1210, 2012. K. N. Kay, T. Naselaris, R. J. Prenger, and J. L. Gallant. Identifying natural images from human brain activity. Nature, 452:352, 2008. F. Liem, G. Varoquaux, J. Kynast, F. Beyer, S. K. Masouleh, J. M. Huntenburg, L. Lampe, M. Rahim, A. Abraham, R. C. Craddock, ... Predicting brain-age from multimodal imaging data captures cognitive impairment. NeuroImage, 2017. N. K. Logothetis, J. Pauls, and T. Poggio. Shape representation in the inferior temporal cortex of monkeys. Current Biology, 5:552, 1995.
  71. 71. References III A. Mensch, J. Mairal, B. Thirion, and G. Varoquaux. Learning neural representations of human cognition across many fMRI studies. In NIPS, 2017. A. Mensch, J. Mairal, B. Thirion, and G. Varoquaux. Extracting universal representations of cognition across brain-imaging studies. In prep, 2018. A. Newell. You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium. 1973. B. Olshausen ... Emergence of simple-cell remainsceptive field properties by learning a sparse code for natural images. Nature, 381:607, 1996. R. A. Poldrack, Y. O. Halchenko, and S. J. Hanson. Decoding the large-scale structure of brain function by classifying mental states across individuals. Psychological Science, 20:1364, 2009. Y. Schwartz, B. Thirion, and G. Varoquaux. Mapping cognitive ontologies to and from the brain. In NIPS, 2013.
  72. 72. References IV G. Varoquaux and R. A. Poldrack. Predictive models can overcome reductionism in cognitive neuroimaging. In rev, 2018. G. Varoquaux and B. Thirion. How machine learning is shaping cognitive neuroimaging. GigaScience, 3:28, 2014. G. Varoquaux, Y. Schwartz, R. A. Poldrack, B. Gauthier, D. Bzdok, J. Poline, and B. Thirion. Atlases of cognition with large-scale brain mapping. In rev, 2018.

×