Towards psychoinformatics with machine learning and brain imaging

1. Towards psychoinformatics with machine learning and brain imaging Gaël Varoquaux

2. Towards psychoinformatics with machine learning and brain imaging Gaël Varoquaux Propose and illustrate a research program

3. Accumulation of functional brain data G Varoquaux 2

4. Theories of the mind Laws of the mind for perception, decision, action, emotion... G Varoquaux 3

5. Studying mental processes 1 Craft an experimental condition that recruits it G Varoquaux 4

6. Studying mental processes 1 Craft an experimental condition that recruits it 2 Do an elementary psychological manipulation G Varoquaux 4

7. Studying mental processes - -Results in a contrast Study by oppositions G Varoquaux 4

8. Studying mental processes - -Results in a contrast Study by oppositions Results tied to a simple psychological manipulation bound to a paradigm G Varoquaux 4

9. Proposing Generalization to build broader theories bridging paradigms Paradigm 1: Seen G Varoquaux 5

10. Proposing Generalization to build broader theories bridging paradigms Paradigm 1: Seen Paradigm 2: Imagined G Varoquaux 5

11. Generalization for broader theories bridging paradigms Seen objects G Varoquaux 6

12. Brain imaging ⇒ brain-mind associations Diﬃculty: modeling behavior & cognition G Varoquaux 7

13. Predictive models can broaden theories by generalizing brain-mind associations to arbitrary new tasks and stimuli [Varoquaux and Poldrack 2018] 1 Beyond oppositions: encoding 2 Generalizing across tasks 3 Universal cognitive representations 4 Across subjects: biomarkers G Varoquaux 8

14. 1 Beyond oppositions: encoding [Eickenberg... 2017] G Varoquaux 9

15. 1 Decomposing psychological process To study vision: Breaking down stimuli G Varoquaux 10

16. 1 Decomposing psychological process To study vision: Breaking down stimuli [Hubel and Wiesel 1962] Neurons receptive to Gabors (edges) G Varoquaux 10

17. 1 Decomposing psychological process To study vision: Breaking down stimuli [Hubel and Wiesel 1962] Neurons receptive to Gabors (edges) [Logothetis... 1995] Shapes in inferior temporal cortex G Varoquaux 10

18. 1 Decomposing psychological process To study vision: Breaking down stimuli Image V1 cortex V2 cortex Inferior temporal cortex Fusiform face area Jack? Is there a “face” region? A “foot” region? A “left big toe” region? G Varoquaux 10

19. 1 Crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs G Varoquaux 11

20. 1 Crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs G Varoquaux 11

21. 1 Crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs - G Varoquaux 11

22. 1 Crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs - Representing only one aspect of the stimuli: too much reductionism G Varoquaux 11

23. 1 Not crafting stimuli for cognitive oppositions Is there a “face” region? A “foot” region? A “left big toe” region? vs - Representing only one aspect of the stimuli: too much reductionism Encoding Building complex representations of stimuli Predicting brain response from them G Varoquaux 11

24. 1 Decomposing visual stimuli Low-level visual cortex is tuned to natural image statistics [Olshausen et al. 1996] What drives high-level representations? The concept of “left big toe”? G Varoquaux 12

25. 1 Decomposing visual stimuli Low-level visual cortex is tuned to natural image statistics [Olshausen et al. 1996] What drives high-level representations? Convolutional Net G Varoquaux 12

26. 1 Brain mapping: encoding with conv nets StimulirepresentationBrainactivity Input Layer 1 Layer 5 Linear predictive models Convolutional Net G Varoquaux 13

27. 1 Brain mapping: encoding with conv nets StimulirepresentationBrainactivity Input Layer 1 Layer 5 Linear predictive models Convolutional Net Explains the brain activity from the stimuli much better than hand-crafted features On data from [Kay... 2008] natural images [Huth... 2012] movies [Eickenberg... 2017] G Varoquaux 13

28. 1 Brain mapping: encoding with conv nets [Eickenberg... 2017] High-level conv-net layer map to high-level visual areas G Varoquaux 14

29. 1 Brain mapping with conv nets: retinotopy Mapping response to exentricity: artiﬁcal retinotopy G Varoquaux 15

30. 1 Brain mapping with conv nets: high-level concepts Opposing face versus place in the [Haxby... 2001] stimuli G Varoquaux 16

31. [Eickenberg... 2017] Beyond oppositions: encoding Recovers the hierarchy of visual modules Generalizes from natural images to retinotopy or face vs place G Varoquaux 17

32. 2 Generalizing across tasks Characterizing the function of brain structures How much is observed activity a consequence of speciﬁcities of the paradigm? [Schwartz... 2013, Varoquaux... 2018] G Varoquaux 18

33. 2 Large-scale decoding for reverse inference Cat Dog Ouaf Miaou [Poldrack... 2009] G Varoquaux 19

34. 2 Large-scale decoding for reverse inference Cat Dog Ouaf Miaou [Poldrack... 2009] Variability: an opportunity and a challenge Technical heterogeneity (scanner, stimulus modality...) Paradigmatic isolation (cognition) [Newell 1973] G Varoquaux 19

35. 2 Generalizing to arbitrary paradigms [Schwartz... 2013, Varoquaux... 2018] What is this brain doing? Describe tasks by their cognitive components Multi-label prediction: presence or absence of each label Visual Auditory Foot Hand Calculation Reading Checkboard Face Place Object Digit Saccade ... ... ... ... ... ... G Varoquaux 20

36. 2 Generalizing to arbitrary paradigms [Schwartz... 2013, Varoquaux... 2018] What is this brain doing? Describe tasks by their cognitive components Multi-label prediction: presence or absence of each label Visual Auditory Checkboard Face ... ... ... ... Prediction across studies Describe a task never seen from the brain activity it evokes G Varoquaux 20

37. 2 Generalizing to arbitrary paradigms [Schwartz... 2013, Varoquaux... 2018] What is this brain doing? + + Regions speciﬁc to that predict facets of cognition G Varoquaux 20

38. 2 Decoding mega-analysis [Schwartz... 2013, Varoquaux... 2018] 30 studies 837 subjects 196 experimental conditions 6919 activation maps Diﬀerent labs Dehaene Poldrack Wager ... Various cognitive domains Language Vision Decision making Mathematics ... All manually preprocessed, labeled, and curated G Varoquaux 21

39. 2 Decoding mega-analysis [Schwartz... 2013, Varoquaux... 2018] 30 studies 837 subjects 196 experimental conditions 6919 activation maps Diﬀerent labs Dehaene Poldrack Wager ... Various cognitive domains Language Vision Decision making Mathematics ... All manually preprocessed, labeled, and curated Decoding arbitrary new paradigms visualauditoryVcheckerboard Hcheckerboardobjectsscramblefacesplacesdigits wordsrighthandlefthandrightfootleftfootlanguagehumansound non-humansound 0.2 0.4 0.6 0.8 1.0 Predictionscore (ROCAUC) ontologydecoder logisticregression naiveBayes neurosynth 0.3 0.2 0.1 0.0 +0.1 +0.2 +0.3 Relative score G Varoquaux 21

40. 2 Mapping: regions specific to facets of cognition [Schwartz... 2013, Varoquaux... 2018] Contrasts: good rejection of confounds Decoding: reverse inference ⇒ Consensus to define regions The “place” concept Forward term effect Brain response to tasks with the “place” concept Forward ontology contrast Contrasting responses to “place” and related concepts Reverse decoder map Regions that predict a place- recognition task Consensus forward & reverse Regions selected by forward and reverse inference Replaces the careful crafting of control conditions G Varoquaux 22

41. 2 A functional atlas: central gyrus, motor networks [Schwartz... 2013, Varoquaux... 2018] Neurosynth Reverse inference (Logistic regression) Decoding in an ontology z = -15 Reverse inference -15 nference DecodDecoding in an visual faces places objects scrambled words digits horizontal - vertical checkerboard Visual regions Contrasts -based analysis Neurosynth reverse inference Decoding and ontology + Language and auditory mapping in temporal cortex + Calculation and spatial attenation in IPS + Mapping the motor system G Varoquaux 23

42. [Varoquaux... 2018] Generalizing across tasks Cat Dog Ouaf Miaou Multi-label prediction Prediction (discriminative models) more robust to heterogeneity Decoding ⇒ evidence beyond ⇒ a paradigm Atlasing cognition G Varoquaux 24

43. Need an ontology of cognition Episodic memory working memory lexical memory Syntax comprehension Reading Mental calculation Shape recognition Face recognition Memory Language Vision G Varoquaux 25

44. 3 Universal cognitive representations [Mensch... 2017] Face Shape Reading Shape Language Music Saccades Calculation G Varoquaux 26

45. 3 Universal cognitive representations [Mensch... 2017] Face Shape Reading Shape Language Music Saccades CalculationMulti-task learning Solve many related but diﬀerent problems Learn commonalities intermediate representations G Varoquaux 26

46. 3 Deep architecture [Bengio 2009] ... Great for multiple output (tasks) G Varoquaux 27

47. 3 Deep architecture [Bengio 2009] ... Great for multiple output (tasks) Millions of parameters, thousands of data points G Varoquaux 27

48. 3 Deep architecture 2 2 Great for multiple output (tasks) Millions of parameters, thousands of data points Simplify G Varoquaux 27

49. 3 Shallow architecture 2 linear model Great for multiple output (tasks) Millions of parameters, thousands of data points Simplify simplify more G Varoquaux 27

50. 3 Shallow architecture 2 2 Great for multiple output (tasks) Millions of parameters, thousands of data points Simplify simplify more G Varoquaux 27

51. 3 Deep linear model – multi-task – Bayesian 4TB resting-state data HCP900 OpenfMRI HCP Camcan Brainomics 42000 task fMRI contrast maps for one model ... Many rest & task fMRI studies ... x Information ﬂow x Learned network combination 1st layer assignment Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Joint training Bayesian deep learning Resting-state functional loadings 512 resting-state networks Task functional loadings 128 task-optimized networks Contrast maps 200,000 voxels Contrast name Probability vector FACE BODY HOUSE TOOL 512 components [Mensch... 2018] G Varoquaux 28

52. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information ﬂow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks G Varoquaux 28

53. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information ﬂow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks Improves decoding in each study 0% 20% 40% 60% 80% Decoding accuracy on test set Stop-signal w/ spoken & manual resp. The Human Connectome Project UCLA LA5C Simon task Plain or mirror-reversed text Stop-signal CamCan audio-visual Localizer Cross-language repetition priming Twin localizer High-level math Spatio-temporal judgement (retake) Constit. struct. of sent. & music Sentence/music complexity Word & object processing Emotion regulation Arithmetic & saccades Brainomics localizer Incidental encoding Spatio-temporal judgement False belief Visual object recognition Face recognition Stop-signal & classification Classification learning Compression Motor task & word/verb generation BART, stop-signal, emotion Rhyme judgment Auditory & Visual Oddball Weather prediction Stop-signal & classification (retake) Balloon Analog Risk-taking Mixed-gambles Classif. learning & reversal Task fMRI study Decoding from voxels Decoding from multi-study networks Chance level -5% 0% 5%10%15% Multi-study acc. gain G Varoquaux 28

54. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information ﬂow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks Improves decoding in each study 4 16 32 100 400 Number of train subjects 0% 5% 10% 15% 20% Multistudyacc.gain Study Small studies beneﬁt from large ones G Varoquaux 28

55. 3 Deep linear model – multi-task – Bayesian OpenfMRI HCP Camcan Brainomics ... ... Information ﬂow Non-negative matrix factorization SUCCESSFUL GO HOUSE HORIZONTAL CHECKERBOARD Study-wise decoding REWARD BALLOON EXPLODE ............ Deep linear model Task functional loadings 128 task-optimized networks [Mensch... 2018] Intermediate representation: Task-optimized networks Enables smaller sample sizes 7 15 23 31 39 # training subjects in target study 70% 75% 80% 85% Testaccuracy Pinel et al. '07 9 18 28 37 47 70% 80% 90% Papadopoulos- Orfanos '12 60 12 60% 65% C (Sh 37 47 oulos- '12 60 121 181 242 302 60% 65% CamCan AV (Shafto et al. '14) Standard decoding Multi-study decoder 2 4 7 9 12 20% 40% 60% Cohen '09 G Varoquaux 28

59. [Mensch... 2017] Universal cognitive representations Multi-task decoding: - decoding the original task labels Deep linear models: - intermediate representations across studies - statistical power from big to small studies Face Shape Reading Shape Language Music Saccades Calculation G Varoquaux 29

60. [Mensch... 2017] Universal cognitive representations Multi-task decoding: - decoding the original task labels Deep linear models: - intermediate representations across studies - statistical power from big to small studies Face Shape Reading Shape Language Music Saccades Calculation G Varoquaux 29

61. 4 Across subjects: biomarkers G Varoquaux 30

62. 4 Beyond heterogeneity: predicting autism across sites Accuracy Fraction of subjects used More data is better (up to 1000 subjects) [Abraham... 2017] G Varoquaux 31

63. 4 Brain aging: a surrogate biomarker [Liem... 2017] Predicting brain aging = chronological age Multi-modal: brain connectivity and morphology Age prediction: mean absolute error of 4.3 years G Varoquaux 32

64. 4 Brain aging: a surrogate biomarker [Liem... 2017] Predicting brain aging = chronological age Multi-modal: brain connectivity and morphology Age prediction: mean absolute error of 4.3 years Discrepency with chronological age correlates with cognitive impairment 0 2 4 Brain aging discrepancy (years) -0.38 0.74 1.72 Objective Cognitive Impairment group Normal Mild Major G Varoquaux 32

65. @GaelVaroquaux Psychoinformatics with machine learning Prediction for broader theories AI to model stimuli / the world Explicit generalization across paradigms Cat Dog Ouaf Miaou

66. @GaelVaroquaux Psychoinformatics with machine learning Prediction for broader theories AI to model stimuli / the world Explicit generalization across paradigms Beyond oppositions Encoding complete descriptions of tasks Decoding multiple facets of cognitions

67. @GaelVaroquaux Psychoinformatics with machine learning Prediction for broader theories AI to model stimuli / the world Explicit generalization across paradigms Beyond oppositions Encoding complete descriptions of tasks Decoding multiple facets of cognitions Useful with imperfect labels Extracting common representations Surrogate biomarkers

68. @GaelVaroquaux Psychoinformatics with machine learning Prediction for broader theories AI to model stimuli / the world Explicit generalization across paradigms Beyond oppositions Encoding complete descriptions of tasks Decoding multiple facets of cognitions Useful with imperfect labels Extracting common representations Surrogate biomarkers Software: nilearn http://nilearn.github.io ni

69. References I A. Abraham, M. P. Milham, A. Di Martino, R. C. Craddock, D. Samaras, B. Thirion, and G. Varoquaux. Deriving reproducible biomarkers from multi-site resting-state data: An autism-based example. NeuroImage, 147:736–745, 2017. Y. Bengio. Learning deep architectures for ai. Foundations and trends in Machine Learning, 2:1–127, 2009. M. Eickenberg, A. Gramfort, G. Varoquaux, and B. Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. NeuroImage, 152:184–194, 2017. J. V. Haxby, I. M. Gobbini, M. L. Furey, ... Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 293:2425, 2001. D. H. Hubel and T. N. Wiesel. Receptive ﬁelds, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of physiology, 160:106, 1962.

70. References II A. G. Huth, S. Nishimoto, A. T. Vu, and J. L. Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76:1210, 2012. K. N. Kay, T. Naselaris, R. J. Prenger, and J. L. Gallant. Identifying natural images from human brain activity. Nature, 452:352, 2008. F. Liem, G. Varoquaux, J. Kynast, F. Beyer, S. K. Masouleh, J. M. Huntenburg, L. Lampe, M. Rahim, A. Abraham, R. C. Craddock, ... Predicting brain-age from multimodal imaging data captures cognitive impairment. NeuroImage, 2017. N. K. Logothetis, J. Pauls, and T. Poggio. Shape representation in the inferior temporal cortex of monkeys. Current Biology, 5:552, 1995.

71. References III A. Mensch, J. Mairal, B. Thirion, and G. Varoquaux. Learning neural representations of human cognition across many fMRI studies. In NIPS, 2017. A. Mensch, J. Mairal, B. Thirion, and G. Varoquaux. Extracting universal representations of cognition across brain-imaging studies. In prep, 2018. A. Newell. You can’t play 20 questions with nature and win: Projective comments on the papers of this symposium. 1973. B. Olshausen ... Emergence of simple-cell remainsceptive ﬁeld properties by learning a sparse code for natural images. Nature, 381:607, 1996. R. A. Poldrack, Y. O. Halchenko, and S. J. Hanson. Decoding the large-scale structure of brain function by classifying mental states across individuals. Psychological Science, 20:1364, 2009. Y. Schwartz, B. Thirion, and G. Varoquaux. Mapping cognitive ontologies to and from the brain. In NIPS, 2013.

72. References IV G. Varoquaux and R. A. Poldrack. Predictive models can overcome reductionism in cognitive neuroimaging. In rev, 2018. G. Varoquaux and B. Thirion. How machine learning is shaping cognitive neuroimaging. GigaScience, 3:28, 2014. G. Varoquaux, Y. Schwartz, R. A. Poldrack, B. Gauthier, D. Bzdok, J. Poline, and B. Thirion. Atlases of cognition with large-scale brain mapping. In rev, 2018.

Towards psychoinformatics with machine learning and brain imaging

Gael Varoquaux

Towards psychoinformatics with machine learning and brain imaging