Analysing cortical organisation and its relation to cognition: A machine learning & discrete optimization approach

1. MeTrICS Analysing cortical organisation and its relation to cognition A machine learning & discrete optimization approach Emma C. Robinson King’s College London emma.robinson@kcl.ac.uk. @emrobSci ecr05 1 https://metrics- lab.github.io/

2. Overview 2 What makes brain imaging analysis so challenging? Improving precision through cortical surface modelling Comparing images across populations using Multimodal Surface matching (MSM) • (A discrete optimization-based tool for correspondence matching) Modelling the Human Brain using Big Data and Machine Learning • The Human Connectome project’s Multimodal Brain Parcellation (Nature 2016) Looking to the future… • Can (Geodesic) Deep Learning tools offer further precision? Tutorial: Getting up and running with surface-based brain analysis

3. MeTrICS The Cerebral Cortex • Highly convoluted, Outer layer • Location of neuronal cell bodies • Responsible for higher- order cognition • E.g. sensory perception/spatial resoning/language understanding Origin of brain functional signal

4. MeTrICS The Cerebral Cortex • Highly convoluted, Outer layer • Responsible for higher-order cognition • Contains functionally specialised units • Connected in a network 4

5. MeTrICS The Cerebral Cortex • Most evolved relative to non human primates • Implicated in neurological and psychiatric disease 5DeFelipe, Javier. "The evolution of the brain, the human nature of cortical circuits, and intellectual creativity." Frontiers in neuroanatomy 5 (2011): 29. Pre-frontal

6. MeTrICS The Cerebral Cortex • Highly variable in shape • And functional organisation • Particularly within the frontal lobe • On top of this most cognitive disorders are highly heterogenous in their presentations and symptoms

7. MeTrICS A spatial normalization approach • Traditionally, medical imaging and radiology has performed population studies by spatially normalizing data to a global average template 7

8. MeTrICS A spatial normalization approach • One approach to doing this is to use Free-form Deformations (Rueckert 1999) 8 Here, the displacement at different points in the image is controlled by deformation of a coarse control-point grid i.e. the deformation of intermediate point x’ will be determined from a weighted sum of contributions of neighbouring control- points, up to 3 hops away (third order) with weights determined by distance)

9. MeTrICS A spatial normalization approach • All brains are deformed to match a global average space • Then imaging data is compared across the population at each spatial location • Which implicitly assumes that brains can be exactly matched… 9

10. Surface-based processing 10

11. MeTrICS Cortical constrained analysis Best analysed as a surface • Better captures geodesic distances along cortical sheet • Improves registration • Improves smoothing 11

12. MeTrICS Cortical constrained analysis 12 Volumetric smoothing mixes signals Surface-constrained smoothing averages only GM signals Best analysed as a surface • Better captures geodesic distances along cortical sheet • Improves registration • Improves smoothing

13. MeTrICS FreeSurfer • The first team to do this were FreeSurfer (Fischl et al) • Full suite of tools for fitting triangulated meshes to inner and outer cortical boundaries of MRI images 13 Step 1 – cortical segmentation in 3D Tissue segmentation is traditionally performed with Expectation Maximisation of a mixture of Gaussians model 0.95 0.05 0.95 0.95 0.95 0.05 WMGMCSF

14. MeTrICS FreeSurfer • The first team to do this were FreeSurfer (Fischl et al) • Full suite of tools for fitting triangulated meshes to inner and outer cortical boundaries of MRI images 14

15. MeTrICS FreeSurfer • The first team to do this were FreeSurfer (Fischl et al) • Full suite of tools for fitting triangulated meshes to inner and outer cortical boundaries 15 • Inflating to sphere • And performing surface registration

16. MeTrICS Big Data brain imaging Available pipelines/data sets: 16 *soon • Adults • Children > 2 yrs • Monkeys • Neonates (29-45 weeks GA) • Fetuses (to come) • Young healthy adults (20-30 yrs) *FreeSurfer slide: shorturl.at/acl45

17. MeTrICS The (Developing) Human Connectome Projects • Pushing the boundaries of image processing • HCP 1200 data sets young adults • High spatial and temporal resolution • sMRI (T1 and T2) • HARDI dMRI • fMRI (1 hour resting state; 7 tasks) • dHCP - 861 neonates (144 Pre-term), 266 foetuses (so far) • sMRI (T1 and T2) • HARDI dMRI (300 volumes, 3 shells) • fMRI (20 mins resting state) 17Makropoulos and Robinson et al. The Developing Human Connectome Project: a Minimal Processing Pipeline for Neonatal Cortical Surface Reconstruction. NeuroImage (2018)

18. MeTrICSThe (Developing) Human Connectome Projects • Highly rich feature sets including: • imaging • Genetics-> 4.3 million SNP array • Cognitive test scores/eye tracking • Demographics • Open access – links at end 18

19. Multimodal Surface Matching (MSM) Correspondence matching with discrete optimisation 19

20. MeTrICS Surface registration • In FreeSurfer spheres are deformed to improve alignment of coarse scale (sulcal depth) patterns • Performed on high- resolution dense meshes • Optimised with Gradient descent 20 Fischl, Bruce, et al. "High-resolution intersubject averaging and a coordinate system for the cortical surface." Human brain mapping 8.4 (1999): 272-284.

21. MeTrICS Surface registration However • cortical folding is highly variable • And may not reflect the underlying functional organization of the cortex As such folding based alignment doesn’t always do a good job of aligning brain function 21 Fischl, Bruce, et al. "Cortical folding patterns and predicting cytoarchitecture." Cerebral cortex 18.8 (2008): 1973-1980.

22. MeTrICS Surface registration However • cortical folding is highly variable • And may not reflect the underlying functional organization of the cortex As such folding based alignment doesn’t always do a good job of aligning brain function 22Nenning, Karl-Heinz, et al. "Diffeomorphic functional brain surface alignment: Functional demons." NeuroImage 156 (2017): 456-465.

23. Curvature Task/rest fMRI Myelin* Structural Connectivity Sotiropoulos et al NeuroImage 2016 Glasser, 2011. J. Neurosci, 31 11597-11616 *reflects patterns of cellular organisation Rich multimodal feature sets

24. MeTrICS Multimodal Surface Matching • Spherical framework for Multimodal cortical surface registration • Use low resolution control point grids to constrain the deformation • Optimised using discrete methods 24 x LABEL%POINT% OPTIMAL%LABEL% CONTROL%POINT% CONTROL%GRID% FIXED%REFERENCE%GRID% A# B# C# ROTATIONAL%DEFORMATION%R lp Robinson, Emma C., et al Neuroimage 100 (2014): 414-426. Robinson, Emma C., et al. NeuroImage 167 (2018): 453-465.

25. MeTrICS 𝐸 𝒍 = $ %∈' 𝑐% 𝑙% + 𝜆 $ %,- ∈. 𝑉%-(𝑙%, 𝑙-) Discrete Optimisation • More traditionally used for image segmentation • Equivalent to discrete Markov Random Field models 25 cp(lp) are called unary potentials (Data likelihood term) 𝑉%-(𝑙%, 𝑙-) are called pairwise potentials Smoothness penalty 𝑙% represents some choice of labelling

26. MeTrICS Discrete Optimisation • More traditionally used for image segmentation • Based on Markov Random field models Likelihood term 27 𝑐% 𝑙 = lnP I6 l) e.g. The data likelihood term could estimate the likelihood of seeing a data points with the intensity of x6 coming from the Gaussian distribution for label 𝑙

27. MeTrICS Discrete Optimisation • More traditionally used for image segmentation • Based on Markov Random field models Smoothness penalty • Discourage neighbouring data points from taking different labels • But not so much that it discourages discontinuities where they are suggested by the data term 28 • Penalizes discontinuities between voxels of similar intensities if |Ip − Iq| < σ . • If voxels are very different, |Ip − Iq| > σ, then the penalty is small. 𝑉%- 𝑙%, 𝑙- = 𝑒 9 :!9:" # ;<#

28. MeTrICS Discrete Optimisation • Solved as a max-flow min-cut problem • Image voxels represented as nodes in a graph • Source and Sink nodes represent foreground and background 29 Seek a ‘cut’ (segmentation) which: •Maximises the difference in intensities between classes •Minimises the total weight of edges severed in the cut

29. MeTrICS Discrete Optimisation • Solved as a max-flow min-cut problem • Image voxels represented as nodes in a graph • Source and Sink nodes represent foreground and background 30 𝐸 𝒍 = ' !∈# 𝑐! 𝑙! 𝜆 ' !,% ∈& 𝑉!%(𝑙!, 𝑙%) Summed cost of each edge ’cut’ equals MRF energy 𝒘𝒊𝒋 = 𝑽𝒊𝒋 𝒍𝒊, 𝒍𝒋 𝒘 𝒔𝒊 = 𝒄𝒊 𝒍𝒊 = 𝒔 𝒘𝒕𝒋 = 𝒄𝒋(𝒍𝒋 = 𝒕)

30. MeTrICS Discrete Optimisation • Solved as a max-flow min-cut problem • Image voxels represented as nodes in a graph • Source and Sink nodes represent foreground and background 31

31. MeTrICS Max-flow Min Cut • Shown for Toy problem • ‘distances’ between cities 32

32. MeTrICS Max-flow Min Cut • Shown for Toy problem • ‘distances’ between cities • Must assign flow so as to equalize inflow and outflow from each vertex • Flow through network is limited by lowest capacity edge 33

33. MeTrICS Max-flow Min Cut • The Ford Fulkerson Method • Start by assigning zero flow to all edges • Find example path • Identify bottleneck capacity along the path • Subtract this flow capacity from all edges in p • Repeat until no more augmenting paths can be found 34 [1] L. Ford and D. Fulkerson, Flows in Networks. Princeton Univ. Press, 1962.

34. MeTrICS Max-flow Min Cut • The Ford Fulkerson Method • Start by assigning zero flow to all edges • Find example path • Identify bottleneck capacity along the path • Subtract this flow capacity from all edges in p • Repeat until no more augmenting paths can be found 35 Max flow=8 [1] L. Ford and D. Fulkerson, Flows in Networks. Princeton Univ. Press, 1962.

37. MeTrICS 𝜶 −expansion • Traditional max-flow min-cut can be slow • The most famous extension is probably the 𝜶-expansion algorithm by Yuri Boykov • But only works for metric smoothness penalties i.e. 𝑉 𝛼, 𝛽 = 0 𝛼 = β 𝑉 𝛼, 𝛽 = 𝑉 𝛽, 𝛼 ≥ 0 𝑉 𝛼, 𝛽 ≤ 𝑉 𝛼, 𝛾 + 𝑉 𝛾, 𝛽 • The 3rd (triangle penalty) implies problem is convex and is rarely true in practice 38 Boykov, Yuri, Olga Veksler, and Ramin Zabih. "Fast approximate energy minimization via graph cuts." IEEE Transactions on pattern analysis and machine intelligence 23.11 (2001): 1222-1239.

38. MeTrICS Fast-PD • Solves a metric labeling representation of the MRF for {0,1} variables • Optimised as a primal-dual linear problem • Primal approx. of integer (labelling) problem • Dual – relaxed continuous form 39 min cp (a) a∈L ∑ xp (a)+ Vpq (a,b)xpa (a,b) a,,b∈L ∑p,q∈E ∑p∈P ∑ xp (a) =1a∑ xpq (a,b) −a∑ xq (b) = 0 xpq (a,b) −b∑ xp (a) = 0 mincT x Ax = b, x ≥ 0 maxbT y AT y ≤ c PRIMAL DUAL cT x ≤ fbT y Primal-Dual form Metric Labelling problem Komodakis et al. Approximate Labeling via Graph Cuts Based on Linear Programming, PAMI 2007

39. MeTrICS Higher-Order Clique Reduction • Proposed by Ishikawa CVPR 2009, 2014 • Allows for high order order penalty terms through polynomial approximation • e.g. 40 Glocker, Ben, et al. "Triangleflow: Optical flow with triangulation-based higher-order likelihoods." European Conference on Computer Vision. Springer Berlin Heidelberg, 2010.

40. MeTrICS MSM • Labels 𝑙" represent a finite choice of possible displacements for each control point 𝑝 • 𝑐" 𝑙" estimates the similarity of data patches around each control point e.g. using • cross correlation • Sum of square differences • Normalised Mutual Information 41 x LABEL%POINT% OPTIMAL%LABEL% CONTROL%POINT% CONTROL%GRID% FIXED%REFERENCE%GRID% A# B# C# ROTATIONAL%DEFORMATION%R lp (lp, lq, lr) = CC(l0 pqr, Jpqr) = 1 cov((l0 pqr, Jpqr) l0 pqr Jpqr

41. MeTrICS MSM regularisation MSM offers pairwise smoothness penalties: 𝑉%- 𝑙%, 𝑙- = log 𝑅D" 𝑅% E 𝑅D# 𝑅- F = 𝜃%- • Penalises differences between the proposed, cumulative rotational displacements of points 𝑝 and 𝑞 • Here 𝑅'./ 𝑅'/ is rotation of control point from its original position to new location given by label 𝑙!/ 𝑙 42 x𝑅D. 𝑝 𝑙% x (𝑅D/)E 𝑞 𝑙- 𝑅D. (𝑅D/)E~Ι Accumulated gradients

42. MeTrICS MSMstrain Control point grid faces projected onto inflated 32K surface Surface normal F Wper = µ 2 (Rk + R k 2) +  2 (Jk + J k 2) • Penalise 𝑊%-M (biomechanical strain) • Estimate 𝐹 (local affine deformation of triangles) where from eigenvalues 𝜆O, 𝜆; obtain • J= 𝜆1 𝜆2 (isotropic –area changes) • R = 𝜆1,/𝜆2 (anisotropic – shape changes) • Implemented with HOCR (Ishikawa 2014) and FastPD • but still very slow relative to pairwise form ***Improves alignment of complex multimodal data, whilst keeping distortions within a plausible biological range*** Robinson, Emma C., et al. NeuroImage 167 (2018): 453-465.

43. MeTrICS aMSM • Rather than penalizing the displacements of coordinates on the sphere instead penalize the displacements of the corresponding anatomy • Allows for biomechanical modelling of longitudinal cortical growth • Discrete displacement labels still fit to spherical mesh!! 44 SAS SSS a) e) b) x LABEL POINT OPTIMAL LABEL CONTROL POINT c) f) d) MSS SSS + G TSS DAS TAS g) F Robinson, Emma C., et al. NeuroImage 167 (2018): 453-465.

44. Applications of (MSM) The Human Connectome Project Multimodal Parcellation and beyond … 45

45. MeTrICS aMSM for modelling cortical development • 24 very preterm infants (born <30 weeks PMA, 15 male, 15 female) • scanned 2-4 times before or at term-equivalent (36-40 weeks PMA) 46 Garcia, Kara E., Robinson E.C. et al. "Dynamic patterns of cortical expansion during folding of the preterm human brain." PNAS (2018) • Regions of statistically significant differences relative to global growth were observed • Growth rates in primary motor, sensory, visual & insula, decrease over time (green). relative expansion increases in the temporal lobe over time

46. MSM cortical atlassing • Atlas = global average model of cortical shape and/or functional organisation • Bozek et al 2018 – spatio-temporal atlas of cortical growth over the early neonatal period 47 Jelena Bozek et al. NeuroImage 179 (2018): 11-29.

47. Generative Modelling of Cortical development • Surface-based features can be used to improve volumetric alignment • From which voxel-wise generative models can be built • Using Gaussian Process regression of • 3D warps • T1 and T1w intensity changes 48 Gaussian Process model of cortical Growth O'Muircheartaigh, J, Robinson EC et al. "Brain (2020).

48. Generative Modelling of Cortical development 49 O'Muircheartaigh, J, Robinson EC et al. "Brain (2020). …and how it is impacted by prematurity • Surface-based features can be used to improve volumetric alignment • From which voxel-wise generative models can be built • Using Gaussian Process regression of • 3D warps • T1 and T1w intensity changes

49. MeTrICS Modelling cortical functional organisation Independent Component Analysis (ICA) Brain activity at each point in time 𝑋 𝑡, 𝑣 reflects a mixture 𝐴 of independent spatial maps 𝑆 50 X(t, v) = LX l Al(t)Sl(v) <latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit> http://users.ics.aalto.fi/whyj/publications/thesis/thesis_node8.html

50. MeTrICS Modelling cortical functional organisation Independent Component Analysis (ICA) Brain activity at each point in time 𝑋 𝑡, 𝑣 reflects a mixture 𝐴 of independent spatial maps 𝑆 • Typically performed by pooling spatially aligned data across a group to boost SNR 51 X(t, v) = LX l Al(t)Sl(v) <latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit>

51. MeTrICS ICA modes • At high resolution these return spatially-contiguous, localized regions • At coarse-scales these reflect functionally co-ordinated modules in the brain 52 Surface ICA modes derived from dHCP neonatal data, aligned with MSM

52. MeTrICS Brain Connectivity Networks • What wires together ‘fires’ together • distant regions coordinate through synchronized signalling • Estimate networks from the partial (or full) correlation of time- series 𝑨𝒊 for each region 53Smith, Stephen M., et al. "Functional connectomics from resting-state fMRI." Trends in cognitive sciences 17.12 (2013): 666-682.

53. MeTrICS MSM improves alignment of areal features • Driving MSM alignment with resting state fMRI improves the sharpness of task fMRI group maps • Rest-fMRI is fMRI acquired with participant not doing anything in particular • Task-fMRI is acquired with participant performing a cognitive task 54Smith, Stephen M., et al. "Functional connectomics from resting-state fMRI." Trends in cognitive sciences 17.12 (2013): 666-682.

54. MeTrICS Improving alignment of areal features • MSM can be driven with multimodal feature sets including • Cortical folding • Cortical myelination • rfMRI ICA maps • Here, 𝑐" 𝑙" is estimated by summing up similarities of multivariate feature vectors for overlapping patches 55 x LABEL%POINT% OPTIMAL%LABEL% CONTROL%POINT% CONTROL%GRID FIXED%REFERENCE%GRID% A# B# C# ROTATIONAL%DEFORMAT e.g. similarities summed up for feature vectors estimated at each point on the fixed reference grid within overlapping patch (moving points interpolated using barycentric interpolation)

55. MeTrICSThe Human Connectome Project’s Multimodal Parcellation of the Human Cerebral Cortex • Expert manual annotations of 180 functionally specialised regions (per hemisphere) on (MSM-aligned) group average data • 97 entirely new areas • 83 areas previously reported by histological studies 56Glasser, Matthew F., et al. "A multi-modal parcellation of human cerebral cortex." Nature 536.7615 (2016): 171-178.

56. MeTrICS Topological variability in the human brain Evidence from folding • Van Essen, David C. "Neuroimage 28.3 (2005): 635-662. And function • Amunts, K., A. Schleicher, and K. Zilles. Neuroimage 37.4 (2007): 1061-1065. • Wang D et al Nature neuroscience.2015;18(12):1853. • Gordon EM Cerebral Cortex 2017;27(1):386-99. • Glasser, Matthew F., et al. "A multimodal parcellation of human cerebral cortex." Nature (2016). 57

57. MeTrICS Machine learning for cortical parcellation • Given the known risk of topological variation • The HCP MMPv1 propagates the group map to individuals via training of a Multi-Layer Perception (Fully connected neural network) • Binary (per region) classifications • used to train classifier ONLY where subject data closely agrees with group 58 Hacker, Carl D., et al. "Resting state network estimation in individual subjects." Neuroimage 82 (2013): 616-633.

58. MeTrICSMachine learning for cortical parcellation The HCP MMPv1: Output from Classifier for 4 example datasets 59 Group Average Glasser, Matthew F., Tim Coalson, Emma C. Robinson et al. "A multimodal parcellation of human cerebral cortex." Nature (2016).

59. MeTrICSMachine learning for cortical parcellation The HCP MMPv1: semantic segmentation implemented as a U-net

60. MeTrICS Predicting Phenotypes • We can use the HCP parcellation to start to predict cognitive phenotypes i.e. fluid intelligence • Using a random forest trained on • 110 Cortical imaging features • averaged for each of the 360 regions 61 • Cross-validated R2 = 0.347 • Feature Importance mapped back to the image space

61. MeTrICS Predicting Phenotypes • Canonical Correlation: • Explores relationship between two multivariate sets of variables • For two matrices 𝑿 and 𝒀, where columns 𝑿𝒊 and 𝒀𝒊 represent different variable sets for some example i (imaging data/demographic variables) • Wish to find 2 new sets sets of basis vectors 𝑨 and 𝑩 (one for 𝑿, one for 𝒀) that will rotate data into new basis 𝑨’, 𝑩: that maximises correlations: • 𝑨’𝑿 and 𝑩’𝒀 maximise 62 ρ = corr(A′ X, B′ Y)

62. MeTrICS Predicting Phenotypes • Smith et al. Nature NeuroScience 2015 • Canonical correlation of functional connectivity matrices vs HCP demographics 63

63. Looking to the future (Geodesic) Deep Learning.. 64

64. MeTrICS The limitations of spatial normalisation • Topology-preserving transforms are unsuited to alignment of the human brain • However, in the absence of such regularisation the problem becomes extremely ill-posed 65

65. MeTrICS The cost of spatial misalignment • Residual spatial misalignment may reduce the sensitivity of population studies • Recent studies have also suggested that comparative studies of brain networks are confounded by these errors 66 Bijsterbosch, et al. elife (2018 and 2019).

66. MeTrICS Spectral alignment • Spectral (Hyper) alignment approaches have been proposed • But have low spatial resolution and a tendency to assign correspondences between spatially distant regions (due to long range functional correlations) 67 Langs, Georg, et al. Advances in neural information processing systems. 2010. Nenning, Karl-Heinz, et al. "Diffeomorphic functional brain surface alignment: Functional demons." NeuroImage 156 (2017): 456-465.

67. MeTrICS Could Deep Learning provide and answer? • Learns to represent image through convolution with a hierarchy of increasing complex filters • Offers a degree of transformation invariance 68 Low level (early) filters – small receptive field – edge detectors Mid level (early) filters – larger receptive field – textures At later layers filters detect whole objects

68. MeTrICS Geodesic Deep Learning • Techniques for applying convolutional operations to graphs or manifolds • Either: • Spatial (e.g. Spherical U-Net, which learns hexagonal filters) • Spectral e.g. ChebNet, Caleynet 69 Seong, Si-Baek, et al Frontiers in Neuroinformatics 12 (2018): 42. Zhao, Fenqiang, et al. "Spherical U-Net on Cortical Surfaces: Methods and Applications." IPMI 2019. CayleyNets: Graph Convolutional Neural Networks with Complex Rational Spectral Filters

69. MeTrICS Geodesic Deep Learning 70 • Prototyping on prediction of gestational age at scan from neonatal cortical imaging data • DL outperforms spatial-normalisation + ML x Train mae=0.198; Test mae= 0.493 x Train mae=0.41; Test mae= 0.95 Derived through deep learning Derived through traditional machine learning

70. MeTrICS Geodesic Deep Learning • Prototyping on prediction of gestational age at scan from neonatal cortical imaging data shows promise Increasing GA 32 37 41 44 Means Features visualised using Grad CAM Selvaraju, Ramprasaath R., et al. ICCV. 2017.

71. MeTrICS Geodesic Deep Learning… On graphs • Siamese networks can be used to learn a metric by which to compare functional brain networks • To generate classifiers for gender and Autism • Ktena et al Neuroimage (2018)

72. Tutorial 73 How do I start working with cortical surface imaging data

73. MeTrICS Getting data The following ‘Big’ data cortical imaging data sets are open and subject to download (subject to user agreements) 74 (no cortical processing yet) 1. https://db.humanconnectome.org/ ~ 1100 young adult datasets 2. 558 session (505 subjs) of Neonatal data: http://www.developingconnec tome.org/second-data- release/ 3. But 40,000 images and growing https://bbams.ndph.ox.ac.uk/ ams/researcher_home.jsp

74. MeTrICS dHCP neonatal data release 2nd Data Release • 558 session (505 subjects) of Neonatal data: • T1, T2, fMRI and dMRI volumes, original, cleaned, pre-processed and mapped to volumetric template space • Cortical surface meshes and features • http://www.developingconnectome.org/data-release/information-registration-and-download/ • Released as Torrent (troubleshooting documents available) • Support via https://neurostars.org/tags/developing-hcp 75

75. MeTrICS HCP connectomedb 76 • Easiest to start with: • Highest resolution, lowest noise • Easiest/most flexible download platform • Requires IBMs aspera connect • Then you can just choose how many samples and what data per sample https://db.humanconnectome.org/ https://wiki.humanconnectome.org/ https://www.humanconnectome.org/contact-usHELP?

76. MeTrICS Downloading a test subject and target data Select: A) Single subject data 77 Pray aspera works first time I used safari and needed to restart Once you know what files you need you can download on mass using curl https://wiki.humanconnectome.org/display/PublicData/How+To+Access+Subject+Data+via+REST

77. MeTrICS Downloading test data Select: Tutorial Data set 78

78. MeTrICS Connectome Workbench • wb_command – image processing e.g. surface based smoothing, resampling etc • wb_view – excellent volume/surface viewer • Tutorials • PDF available with tutorial dataset download • http://mvpa.blogspot.com/2017/08/g etting-started-with-connectome.html https://www.humanconnectome.org/software/connectome-workbench

79. MeTrICS BALSA • https://balsa.wustl.edu/ • Database of HCP outputs • Include HCP parcellation dataset Refers to a wb_view file fomat

80. MeTrICS HCP ‘Atlas’ spaces • All data in the HCP has been aligned to an average (template) brain using either: • MSMSulc (folding based alignment) • MSMall (folding, myelin and function) • There are 2 template spaces • MNInonlinear (164k) • fsaverage32k (32k) MSMSulc MSMall

81. HCP ‘Atlas’ spaces • All data in the HCP has been aligned to an average (template) brain using either: • MSMSulc (folding based alignment) • MSMall (folding, myelin and function) • There are 2 template spaces • MNInonlinear (164k) • fsaverage32k (32k) • One Native space 82 AVERAGENATIVE

82. MeTrICS HCP data • Each subject folder presents data in Native space and resampled onto templates • Unless specifically stated (in the filename) the files will be transformed via MSMSulc File types: • Niftii (.nii.gz) volumes • Gifti surfaces surf.gii) • Gifti metrics (.func/.shape/.label.gii) • CIFITI (.nii) ls 100307/MNINonLinear/fsaverage_LR32k/ 100307.L.MyelinMap.32k_fs_LR.func.gii 100307.L.aparc.32k_fs_LR.label.gii 100307.L.aparc.a2009s.32k_fs_LR.label.gii 100307.L.atlasroi.32k_fs_LR.shape.gii 100307.L.corrThickness.32k_fs_LR.shape.gii 100307.L.curvature.32k_fs_LR.shape.gii 100307.L.inflated.32k_fs_LR.surf.gii 100307.L.midthickness.32k_fs_LR.surf.gii 100307.L.pial.32k_fs_LR.surf.gii 100307.L.rfMRI_REST.ica_d30_MSMAllDeDriftFinal_d26.SubjectsD.func.gii 100307.L.sphere.32k_fs_LR.surf.gii 100307.L.sulc.32k_fs_LR.shape.gii 100307.L.thickness.32k_fs_LR.shape.gii 100307.L.very_inflated.32k_fs_LR.surf.gii 100307.L.white.32k_fs_LR.surf.gii 100307.MyelinMap.32k_fs_LR.dscalar.nii 100307.aparc.32k_fs_LR.dlabel.nii 100307.aparc.a2009s.32k_fs_LR.dlabel.nii 100307.corrThickness.32k_fs_LR.dscalar.nii 100307.curvature.32k_fs_LR.dscalar.nii 100307.rfMRI_REST.ica_d30_MSMAllDeDriftFinal_d26.SubjectsD.dscalar.nii 100307.sulc.32k_fs_LR.dscalar.nii 100307.thickness.32k_fs_LR.dscalar.nii CIFTI

83. MeTrICS CIFTI • Left and right surface metrics • Plus deep grey matter volume • fMRI inspired

84. MeTrICS How can I process new data onto surfaces dHCP structural pipeline (minimal requirement T2w MRI) • https://github.com/BioMedIA/dhcp-structural-pipeline • Includes docker installation - contact j.cupitt@imperial.ac.uk • Pial surface extraction can be a bit buggy for different acquisition parameters • Support via https://neurostars.org/tags/developing-hcp HCP structural pipeline (minimal requirement T1 and T2w MRI) • https://www.humanconnectome.org/software/hcp-mr-pipelines • If only T1 look to the original freesurfer https://surfer.nmr.mgh.harvard.edu/fswiki/DownloadAndInstall • https://www.humanconnectome.org/contact-us dHCP Atlases • https://brain-development.org/brain-atlases/atlases-from-the-dhcp-project/ • https://github.com/ecr05/dHCP_template_alignment (Script for running MSM surface-to- template alignment for neonates) HCP Atlases • https://db.humanconnectome.org/data/projects/HCP_1200 85

85. MeTrICS Running MSM Download from • Github: https://github.com/ecr05/MSM_HOC R/releases • FSL : https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/ MSM SAS SSS a) e) b) x LABEL POINT OPTIMAL LABEL CONTROL POINT c) f) d) MSS SSS + G TSS DAS TAS g) F Option 1 sMSM Option 2 aMSM

86. MeTrICS Running MSMSulc sMSM Inputs: -o output filepath --inmesh path to input sphere --refmesh path to target sphere --indata path to input metric --refdata path to target metric --conf config --trans for chaining trans- formations • Pairwise config https://github.com/ecr05/MSM_HOCR/blob/master/ config/basic_configs/config_standard_MSMpair • HOCR config https://github.com/ecr05/MSM_HOCR/blob/master/ config/basic_configs/config_standard_MSM_strain msm -- inmesh=111716/MNINonLinear/Native/111716.L.sphere.rot.native.surf.gii -- refmesh=HCP_WB_Tutorial_1.0/Q1-Q6_R440.L.sphere.164k_fs_LR.surf.gii -- indata=111716/MNINonLinear/Native/111716.L.sulc.native.shape.gii -- refdata=HCP_WB_Tutorial_1.0/Q1-Q6_R440.L.sulc.164k_fs_LR.shape.gii -- conf=config_standard_MSMpair -o 111716_Q1-Q6_R440.L.MSMSulc -v MSMSulc: wb_command -cifti-separate HCP_WB_Tutorial_1.0/Q1- Q6_R440.sulc.164k_fs_LR.dscalar.nii COLUMN -metric CORTEX_LEFT HCP_WB_Tutorial_1.0/Q1-Q6_R440.L.sulc.164k_fs_LR.shape.gii Outputs: • 111716_Q1-Q6_R440.L.MSMSulc.sphere.reg.surf.gii • 111716_Q1-Q6_R440.L.MSMSulctransformed_and_reprojected.func.gii

87. MeTrICS Running MSMall sMSM Inputs: -o output filepath --inmesh path to input sphere --refmesh path to target sphere --indata path to input metric --refdata path to target metric --conf config --trans for chaining trans- formations • Pairwise regularisation (not recommended) • HOCR config https://github.com/ecr05/MSM_HOCR/blob/master/ config/HCP_multimodal_alignment/MSMAllStrainFi nalconf1to1_1to3_2 MSMall*: *Toy example not actual call – for exact details see HCP pipelines msm -- inmesh=111716/MNINonLinear/Native/111716.L.sphere.rot.native.surf.gii -- trans=111716/MNINonLinear/Native/ 111716.L.sphere.MSMSulc.native.surf.gii --in_register=111716/ MNINonLinear/fsaverage_LR32k/111716.L.sphere.32k_fs_LR.surf.gii --refmesh=HCP_WB_Tutorial_1.0/Q1-Q6_R440.L.sphere.32k_fs_LR.surf.gii -- indata=../HCP_PARCELLATION/TRAININGDATA/featuresets/ 111716.L.MultiModal_Features_MSMAll_2_d41_WRN_DeDrift.32k_fs_LR.func.gii --refdata=../HCP_PARCELLATION/TRAININGDATA/featuresets/ MEAN.L.MultiModal_Features_MSMAll_2_d41_WRN_DeDrift.32k_fs_LR.func.gii --conf=MSMAllStrainFinalconf1to1_1to3_2 -o 111716_Q1-Q6_R440.L.MSMall • --trans allows you to initialise with MSMSulc whilst regularizing over full deformation • fMRI data only exists on 32k template space thus –in_register allows for resampling of the initial warp to the 32K mesh space

88. MeTrICS Running aMSM aMSM Inputs: -o output filepath --inmesh path to input sphere --refmesh path to target sphere --indata path to input metric --refdata path to target metric --conf config --trans for chaining trans- formations --inanat. Input anatomy (i.e. white surface) --refanat Target anatomy • Needs many iterations - Runs for 24 hours!!! Hidden options msm --inmesh=111716/MNINonLinear/Native/ 111716.L. sphere.rot.native.surf.gii --refmesh=HCP_WB_Tutorial_1.0/Q1- Q6_R440.L.sphere.164k_fs_LR.surf.gii --indata=111716/MNINonLinear/Native/ 111716.L.sulc.native.shape.gii --refdata=HCP_WB_Tutorial_1.0/Q1- Q6_R440.L.sulc.164k_fs_LR.shape.gii -- conf=config_standard_aMSM -o 111716_Q1-Q6_R440.L.MSMSulc -v --inanat=111716/MNINonLinear/Native/ 111716.L.white.native.surf.gii --refanat=HCP_WB_Tutorial_1.0/Q1- Q6_R440.L.white.164k_fs_LR.surf.gii aMSM: Significantly smooths deformations of the anatomical mesh

89. MeTrICS A note on configs • --regoption determines mode • 1 for pairwise • 3 for sMSM strain • 5 for aMSM strain • --bulkmod ---shearmod are 𝜅 and 𝜇 in strain equation • 𝜆 controls regularisation strengths • --it = iterations • 5 ok for pairwise (with –rescaleL) • Otherwise use 10-20 • aMSM will need more

90. MeTrICS ML Tips and Tricks • You can load data into Python using nibabel https://nipy.org/nibabel/ metric=nibabel.load(‘some.func.gii’) • Metric (.func/.shape/label etc) contain data arrays • Get each data array i with features = metric.darrays[i].data Kings ICL Smart imaging CDT Lecture series Advanced Machine Learning • Deep Learning tutorials for medical image data • In Pytorch • Available soon github/coursera @ecr05

91. MeTrICS Summary • Cognitive disorders are extremely challenging to model due to cortical and phenotypic heterogeneity • Surface based modelling of the brain has been shown to significantly improve the accuracy with which we can model cortical organization • Admittedly slightly more overhead in • getting data onto the surface • Using/understand existing surface data sets • Hopefully, this tutorial will relieve a lot of confusion and create some opportunities • Lots of potential for creative uses of Machine Learning of surface imaging data • Need help/want to collaborate? 92 emma.robinson@kcl.ac.uk @emrobSci ecr05

92. MeTrICS Acknowledgements My team: 94 • Cher Bass • Abdulah Fawaz • Kyriaki Kaza KCL • David Edwards • Jo Hajnal • Jonathan O’Muicheartaigh • Maria Deprez Other Contributors and collaborators: Imperial College: • Daniel Rueckert • Bernhard Kainz • Antonis Makropoulos • Andreas Schuh • Samuel Budd • Ira Ktena • Martin Rajchl Oxford • Stephen Smith • Mark Jenkinson • Saad Jbabdi WashU • Matthew Glasser • David Van Essen • Janine Bijsterbosch • Tim Coalson https://metrics- lab.github.io/

Analysing cortical organisation and its relation to cognition: A machine learning & discrete optimization approach

Emma Robinson

Analysing cortical organisation and its relation to cognition: A machine learning & discrete optimization approach