1.
MeTrICS
Analysing cortical
organisation and its
relation to cognition
A machine learning & discrete optimization
approach
Emma C. Robinson
King’s College London
emma.robinson@kcl.ac.uk.
@emrobSci ecr05
1
https://metrics-
lab.github.io/

2.
Overview
2
What makes brain imaging
analysis so challenging?
Improving precision through
cortical surface modelling
Comparing images across
populations using
Multimodal Surface
matching (MSM)
• (A discrete optimization-based tool
for correspondence matching)
Modelling the Human Brain
using Big Data and Machine
Learning
• The Human Connectome project’s
Multimodal Brain Parcellation
(Nature 2016)
Looking to the future…
• Can (Geodesic) Deep Learning
tools offer further precision?
Tutorial: Getting up and
running with surface-based
brain analysis

3.
MeTrICS
The
Cerebral
Cortex
• Highly convoluted,
Outer layer
• Location of neuronal cell
bodies
• Responsible for higher-
order cognition
• E.g. sensory
perception/spatial
resoning/language
understanding
Origin of brain
functional signal

4.
MeTrICS
The Cerebral Cortex
• Highly convoluted, Outer layer
• Responsible for higher-order
cognition
• Contains functionally specialised
units
• Connected in a network
4

5.
MeTrICS
The Cerebral Cortex
• Most evolved relative to non
human primates
• Implicated in neurological and
psychiatric disease
5DeFelipe, Javier. "The evolution of the brain, the human nature of cortical circuits, and intellectual
creativity." Frontiers in neuroanatomy 5 (2011): 29.
Pre-frontal

6.
MeTrICS
The
Cerebral
Cortex
• Highly variable in shape
• And functional
organisation
• Particularly within the
frontal lobe
• On top of this most
cognitive disorders are
highly heterogenous in
their presentations and
symptoms

7.
MeTrICS
A spatial
normalization
approach
• Traditionally, medical
imaging and radiology
has performed population
studies by spatially
normalizing data to a
global average
template
7

8.
MeTrICS
A spatial normalization approach
• One approach to doing this is to use Free-form Deformations (Rueckert 1999)
8
Here, the displacement at
different points in the image
is controlled by
deformation of a coarse
control-point grid
i.e. the deformation of intermediate point x’ will be determined
from a weighted sum of contributions of neighbouring control-
points, up to 3 hops away (third order) with weights determined
by distance)

9.
MeTrICS
A spatial
normalization
approach
• All brains are deformed to
match a global average
space
• Then imaging data is
compared across the
population at each spatial
location
• Which implicitly assumes
that brains can be exactly
matched…
9

10.
Surface-based
processing
10

11.
MeTrICS
Cortical constrained analysis
Best analysed as a surface
• Better captures geodesic distances along cortical sheet
• Improves registration
• Improves smoothing
11

12.
MeTrICS
Cortical constrained analysis
12
Volumetric smoothing mixes signals Surface-constrained smoothing averages
only GM signals
Best analysed as a surface
• Better captures geodesic distances along cortical sheet
• Improves registration
• Improves smoothing

13.
MeTrICS
FreeSurfer
• The first team to do this were FreeSurfer (Fischl et al)
• Full suite of tools for fitting triangulated meshes to inner and outer cortical
boundaries of MRI images
13
Step 1 – cortical segmentation in 3D
Tissue segmentation is traditionally performed
with Expectation Maximisation of a mixture of
Gaussians model
0.95
0.05
0.95
0.95
0.95 0.05
WMGMCSF

14.
MeTrICS
FreeSurfer
• The first team to do this were FreeSurfer (Fischl et al)
• Full suite of tools for fitting triangulated meshes to inner and outer cortical
boundaries of MRI images
14

15.
MeTrICS
FreeSurfer
• The first team to do this were FreeSurfer (Fischl et al)
• Full suite of tools for fitting triangulated meshes to inner and outer cortical boundaries
15
• Inflating to sphere
• And performing surface
registration

16.
MeTrICS
Big Data brain imaging
Available pipelines/data sets:
16
*soon
• Adults
• Children > 2 yrs
• Monkeys
• Neonates (29-45 weeks GA)
• Fetuses (to come)
• Young healthy adults
(20-30 yrs)
*FreeSurfer slide:
shorturl.at/acl45

17.
MeTrICS
The (Developing) Human
Connectome Projects
• Pushing the boundaries of image processing
• HCP 1200 data sets young adults
• High spatial and temporal resolution
• sMRI (T1 and T2)
• HARDI dMRI
• fMRI (1 hour resting state; 7 tasks)
• dHCP - 861 neonates (144 Pre-term), 266
foetuses (so far)
• sMRI (T1 and T2)
• HARDI dMRI (300 volumes, 3 shells)
• fMRI (20 mins resting state)
17Makropoulos and Robinson et al. The Developing Human Connectome Project: a Minimal
Processing Pipeline for Neonatal Cortical Surface Reconstruction. NeuroImage (2018)

18.
MeTrICSThe (Developing) Human
Connectome Projects
• Highly rich feature sets including:
• imaging
• Genetics-> 4.3 million SNP array
• Cognitive test scores/eye tracking
• Demographics
• Open access – links at end
18

19.
Multimodal Surface
Matching (MSM)
Correspondence matching with discrete optimisation
19

20.
MeTrICS
Surface
registration
• In FreeSurfer spheres
are deformed to
improve alignment of
coarse scale (sulcal
depth) patterns
• Performed on high-
resolution dense
meshes
• Optimised with
Gradient descent
20
Fischl, Bruce, et al. "High-resolution intersubject averaging and a coordinate
system for the cortical surface." Human brain mapping 8.4 (1999): 272-284.

21.
MeTrICS
Surface
registration
However
• cortical folding is highly
variable
• And may not reflect the
underlying functional
organization of the cortex
As such folding based alignment
doesn’t always do a good job of
aligning brain function
21
Fischl, Bruce, et al. "Cortical folding patterns and predicting
cytoarchitecture." Cerebral cortex 18.8 (2008): 1973-1980.

22.
MeTrICS
Surface
registration
However
• cortical folding is highly
variable
• And may not reflect the
underlying functional
organization of the cortex
As such folding based alignment
doesn’t always do a good job of
aligning brain function
22Nenning, Karl-Heinz, et al. "Diffeomorphic functional brain surface alignment: Functional
demons." NeuroImage 156 (2017): 456-465.

23.
Curvature
Task/rest fMRI
Myelin*
Structural
Connectivity
Sotiropoulos et al
NeuroImage 2016
Glasser, 2011. J.
Neurosci, 31
11597-11616
*reflects patterns of
cellular organisation
Rich multimodal feature sets

24.
MeTrICS
Multimodal Surface Matching
• Spherical framework for Multimodal cortical surface registration
• Use low resolution control point grids to constrain the deformation
• Optimised using discrete methods
24
x LABEL%POINT%
OPTIMAL%LABEL%
CONTROL%POINT%
CONTROL%GRID%
FIXED%REFERENCE%GRID%
A# B# C#
ROTATIONAL%DEFORMATION%R
lp
Robinson, Emma C., et
al Neuroimage 100
(2014): 414-426.
Robinson, Emma C., et al.
NeuroImage 167 (2018):
453-465.

25.
MeTrICS
𝐸 𝒍 = $
%∈'
𝑐% 𝑙% + 𝜆 $
%,- ∈.
𝑉%-(𝑙%, 𝑙-)
Discrete Optimisation
• More traditionally used for image segmentation
• Equivalent to discrete Markov Random Field models
25
cp(lp) are called unary potentials
(Data likelihood term)
𝑉%-(𝑙%, 𝑙-) are called pairwise potentials
Smoothness penalty
𝑙% represents some choice of labelling

26.
MeTrICS
Discrete Optimisation
• More traditionally used for image segmentation
• Based on Markov Random field models
Likelihood term
27
𝑐% 𝑙 = lnP I6 l)
e.g. The data likelihood term could estimate the
likelihood of seeing a data points with the intensity
of x6 coming from the Gaussian distribution for
label 𝑙

27.
MeTrICS
Discrete Optimisation
• More traditionally used for image segmentation
• Based on Markov Random field models
Smoothness penalty
• Discourage neighbouring data points from taking different labels
• But not so much that it discourages discontinuities where they are suggested by the data term
28
• Penalizes discontinuities between voxels of similar intensities if |Ip − Iq| < σ .
• If voxels are very different, |Ip − Iq| > σ, then the penalty is small.
𝑉%- 𝑙%, 𝑙- = 𝑒
9
:!9:"
#
;<#

28.
MeTrICS
Discrete
Optimisation
• Solved as a max-flow
min-cut problem
• Image voxels
represented as nodes
in a graph
• Source and Sink
nodes represent
foreground and
background
29
Seek a ‘cut’ (segmentation) which:
•Maximises the difference in intensities between classes
•Minimises the total weight of edges severed in the cut

29.
MeTrICS
Discrete
Optimisation
• Solved as a max-flow
min-cut problem
• Image voxels
represented as nodes
in a graph
• Source and Sink
nodes represent
foreground and
background
30
𝐸 𝒍 = '
!∈#
𝑐! 𝑙! 𝜆 '
!,% ∈&
𝑉!%(𝑙!, 𝑙%)
Summed cost of each edge ’cut’
equals MRF energy
𝒘𝒊𝒋 = 𝑽𝒊𝒋 𝒍𝒊, 𝒍𝒋
𝒘 𝒔𝒊 = 𝒄𝒊 𝒍𝒊 = 𝒔
𝒘𝒕𝒋 = 𝒄𝒋(𝒍𝒋 = 𝒕)

30.
MeTrICS
Discrete
Optimisation
• Solved as a max-flow
min-cut problem
• Image voxels
represented as nodes
in a graph
• Source and Sink
nodes represent
foreground and
background
31

31.
MeTrICS
Max-flow
Min Cut
• Shown for Toy problem
• ‘distances’ between
cities
32

32.
MeTrICS
Max-flow
Min Cut
• Shown for Toy problem
• ‘distances’ between cities
• Must assign flow so as to
equalize inflow and outflow
from each vertex
• Flow through network is
limited by lowest capacity
edge
33

33.
MeTrICS
Max-flow
Min Cut
• The Ford Fulkerson Method
• Start by assigning zero flow
to all edges
• Find example path
• Identify bottleneck capacity
along the path
• Subtract this flow capacity
from all edges in p
• Repeat until no more
augmenting paths can be found
34
[1] L. Ford and D. Fulkerson, Flows in
Networks. Princeton Univ. Press, 1962.

34.
MeTrICS
Max-flow
Min Cut
• The Ford Fulkerson Method
• Start by assigning zero flow
to all edges
• Find example path
• Identify bottleneck capacity
along the path
• Subtract this flow capacity
from all edges in p
• Repeat until no more
augmenting paths can be found
35
Max flow=8
[1] L. Ford and D. Fulkerson, Flows in
Networks. Princeton Univ. Press, 1962.

35.
MeTrICS
Max-flow
Min Cut
• The Ford Fulkerson Method
• Start by assigning zero flow
to all edges
• Find example path
• Identify bottleneck capacity
along the path
• Subtract this flow capacity
from all edges in p
• Repeat until no more
augmenting paths can be found
36
[1] L. Ford and D. Fulkerson, Flows in
Networks. Princeton Univ. Press, 1962.

36.
MeTrICS
Max-flow
Min Cut
• The Ford Fulkerson Method
• Start by assigning zero flow
to all edges
• Find example path
• Identify bottleneck capacity
along the path
• Subtract this flow capacity
from all edges in p
• Repeat until no more
augmenting paths can be found
37
[1] L. Ford and D. Fulkerson, Flows in
Networks. Princeton Univ. Press, 1962.

37.
MeTrICS
𝜶 −expansion
• Traditional max-flow min-cut can
be slow
• The most famous extension is
probably the 𝜶-expansion
algorithm by Yuri Boykov
• But only works for metric
smoothness penalties i.e.
𝑉 𝛼, 𝛽 = 0 𝛼 = β
𝑉 𝛼, 𝛽 = 𝑉 𝛽, 𝛼 ≥ 0
𝑉 𝛼, 𝛽 ≤ 𝑉 𝛼, 𝛾 + 𝑉 𝛾, 𝛽
• The 3rd (triangle penalty) implies
problem is convex and is rarely
true in practice
38
Boykov, Yuri, Olga Veksler, and Ramin Zabih. "Fast approximate energy
minimization via graph cuts." IEEE Transactions on pattern analysis and
machine intelligence 23.11 (2001): 1222-1239.

38.
MeTrICS
Fast-PD
• Solves a metric labeling
representation of the MRF for
{0,1} variables
• Optimised as a primal-dual
linear problem
• Primal approx. of integer (labelling)
problem
• Dual – relaxed continuous form
39
min cp (a)
a∈L
∑ xp (a)+ Vpq (a,b)xpa (a,b)
a,,b∈L
∑p,q∈E
∑p∈P
∑
xp (a) =1a∑
xpq (a,b) −a∑ xq (b) = 0
xpq (a,b) −b∑ xp (a) = 0
mincT
x
Ax = b, x ≥ 0
maxbT
y
AT
y ≤ c
PRIMAL DUAL
cT
x ≤ fbT
y
Primal-Dual form
Metric Labelling problem
Komodakis et al. Approximate Labeling via Graph Cuts
Based on Linear Programming, PAMI 2007

39.
MeTrICS
Higher-Order Clique Reduction
• Proposed by Ishikawa CVPR 2009, 2014
• Allows for high order order penalty terms through polynomial approximation
• e.g.
40
Glocker, Ben, et al. "Triangleflow: Optical flow with triangulation-based higher-order likelihoods." European Conference on
Computer Vision. Springer Berlin Heidelberg, 2010.

40.
MeTrICS
MSM
• Labels 𝑙" represent a finite
choice of possible
displacements for each
control point 𝑝
• 𝑐" 𝑙" estimates the
similarity of data patches
around each control point
e.g. using
• cross correlation
• Sum of square differences
• Normalised Mutual
Information 41
x LABEL%POINT%
OPTIMAL%LABEL%
CONTROL%POINT%
CONTROL%GRID%
FIXED%REFERENCE%GRID%
A# B# C#
ROTATIONAL%DEFORMATION%R
lp
(lp, lq, lr) = CC(l0
pqr, Jpqr) = 1
cov((l0
pqr, Jpqr)
l0
pqr Jpqr

41.
MeTrICS
MSM regularisation
MSM offers pairwise smoothness penalties:
𝑉%- 𝑙%, 𝑙- = log 𝑅D" 𝑅%
E
𝑅D# 𝑅-
F
= 𝜃%-
• Penalises differences between the proposed, cumulative rotational
displacements of points 𝑝 and 𝑞
• Here 𝑅'./ 𝑅'/ is rotation of control point from its original position to new location
given by label 𝑙!/ 𝑙
42
x𝑅D.
𝑝
𝑙%
x
(𝑅D/)E
𝑞
𝑙-
𝑅D. (𝑅D/)E~Ι
Accumulated
gradients

42.
MeTrICS
MSMstrain
Control point grid faces projected
onto inflated 32K surface
Surface normal
F
Wper =
µ
2
(Rk
+ R k 2) +

2
(Jk
+ J k 2)
• Penalise 𝑊%-M (biomechanical strain)
• Estimate 𝐹 (local affine deformation
of triangles) where from eigenvalues
𝜆O, 𝜆; obtain
• J= 𝜆1 𝜆2 (isotropic –area changes)
• R = 𝜆1,/𝜆2 (anisotropic – shape
changes)
• Implemented with HOCR (Ishikawa
2014) and FastPD
• but still very slow relative to
pairwise form
***Improves alignment of complex
multimodal data, whilst keeping
distortions within a plausible biological
range***
Robinson, Emma C., et al. NeuroImage 167 (2018): 453-465.

43.
MeTrICS
aMSM
• Rather than penalizing the
displacements of coordinates
on the sphere instead
penalize the
displacements of the
corresponding anatomy
• Allows for biomechanical
modelling of longitudinal
cortical growth
• Discrete displacement labels
still fit to spherical mesh!!
44
SAS SSS
a)
e)
b)
x LABEL POINT
OPTIMAL LABEL
CONTROL POINT
c)
f)
d)
MSS
SSS + G
TSS
DAS TAS
g)
F
Robinson, Emma C., et al. NeuroImage 167 (2018): 453-465.

44.
Applications of (MSM)
The Human Connectome Project Multimodal Parcellation and beyond …
45

45.
MeTrICS
aMSM for modelling cortical development
• 24 very preterm infants (born <30 weeks PMA, 15 male, 15 female)
• scanned 2-4 times before or at term-equivalent (36-40 weeks PMA)
46
Garcia, Kara E., Robinson E.C. et al.
"Dynamic patterns of cortical expansion
during folding of the preterm human
brain." PNAS (2018)
• Regions of statistically
significant differences
relative to global growth
were observed
• Growth rates in primary
motor, sensory, visual & insula,
decrease over time (green).
relative expansion increases in
the temporal lobe over time

46.
MSM cortical
atlassing
• Atlas = global average
model of cortical shape
and/or functional
organisation
• Bozek et al 2018 –
spatio-temporal atlas
of cortical growth over
the early neonatal period
47
Jelena Bozek et al. NeuroImage 179 (2018): 11-29.

47.
Generative
Modelling of
Cortical
development
• Surface-based features
can be used to improve
volumetric alignment
• From which voxel-wise
generative models can be
built
• Using Gaussian
Process regression of
• 3D warps
• T1 and T1w intensity
changes
48
Gaussian Process model of cortical
Growth
O'Muircheartaigh, J, Robinson EC et al. "Brain (2020).

48.
Generative
Modelling of
Cortical
development
49
O'Muircheartaigh, J, Robinson EC et al. "Brain (2020).
…and how it is impacted by
prematurity
• Surface-based features
can be used to improve
volumetric alignment
• From which voxel-wise
generative models can be
built
• Using Gaussian
Process regression of
• 3D warps
• T1 and T1w intensity
changes

49.
MeTrICS
Modelling cortical functional organisation
Independent Component Analysis (ICA)
Brain activity at each point in time 𝑋 𝑡, 𝑣 reflects a mixture 𝐴 of independent spatial maps 𝑆
50
X(t, v) =
LX
l
Al(t)Sl(v)
<latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit>
http://users.ics.aalto.fi/whyj/publications/thesis/thesis_node8.html

50.
MeTrICS
Modelling cortical functional organisation
Independent Component Analysis (ICA)
Brain activity at each point in time 𝑋 𝑡, 𝑣 reflects a mixture 𝐴 of independent spatial maps 𝑆
• Typically performed by pooling spatially aligned data across a group to boost SNR
51
X(t, v) =
LX
l
Al(t)Sl(v)
<latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit><latexit sha1_base64="xxgf4TsL5DIM1cU20XqUhL3G4J0=">AAACBHicbVC7SgNBFJ2NrxhfUUstBoOQgIRdEdRCiNpYWER0TSBZl9nJbDJk9sHM3UAIaWz8FRsLFVs/ws6/cZJsoYkH7uVwzr3M3OPFgiswzW8jMze/sLiUXc6trK6tb+Q3t+5VlEjKbBqJSNY9opjgIbOBg2D1WDISeILVvO7lyK/1mFQ8Cu+gHzMnIO2Q+5wS0JKb360X4aBXOmuqJHDFwzU+d0URSre690puvmCWzTHwLLFSUkApqm7+q9mKaBKwEKggSjUsMwZnQCRwKtgw10wUiwntkjZraBqSgClnML5iiPe10sJ+JHWFgMfq740BCZTqB56eDAh01LQ3Ev/zGgn4J86Ah3ECLKSTh/xEYIjwKBLc4pJREH1NCJVc/xXTDpGEgg4up0Owpk+eJfZh+bRs3RwVKhdpGlm0g/ZQEVnoGFXQFaoiG1H0iJ7RK3oznowX4934mIxmjHRnG/2B8fkDCbqWjw==</latexit>

51.
MeTrICS
ICA modes
• At high resolution these return
spatially-contiguous, localized
regions
• At coarse-scales these reflect
functionally co-ordinated
modules in the brain
52
Surface ICA modes derived from dHCP neonatal data,
aligned with MSM

52.
MeTrICS
Brain
Connectivity
Networks
• What wires together
‘fires’ together
• distant regions
coordinate through
synchronized signalling
• Estimate networks from
the partial (or full)
correlation of time-
series 𝑨𝒊 for each region
53Smith, Stephen M., et al. "Functional connectomics from resting-state
fMRI." Trends in cognitive sciences 17.12 (2013): 666-682.

53.
MeTrICS
MSM improves
alignment of
areal features
• Driving MSM alignment
with resting state fMRI
improves the sharpness of
task fMRI group maps
• Rest-fMRI is fMRI
acquired with participant
not doing anything in
particular
• Task-fMRI is acquired
with participant
performing a cognitive
task
54Smith, Stephen M., et al. "Functional connectomics from resting-state
fMRI." Trends in cognitive sciences 17.12 (2013): 666-682.

54.
MeTrICS
Improving
alignment of
areal features
• MSM can be driven with
multimodal feature sets
including
• Cortical folding
• Cortical myelination
• rfMRI ICA maps
• Here, 𝑐" 𝑙" is estimated
by summing up similarities
of multivariate feature
vectors for overlapping
patches
55
x LABEL%POINT%
OPTIMAL%LABEL%
CONTROL%POINT%
CONTROL%GRID
FIXED%REFERENCE%GRID%
A# B# C#
ROTATIONAL%DEFORMAT
e.g. similarities summed up for
feature vectors estimated at
each point on the fixed
reference grid within
overlapping patch
(moving points interpolated
using barycentric
interpolation)

55.
MeTrICSThe Human Connectome Project’s
Multimodal Parcellation of the Human Cerebral
Cortex
• Expert manual annotations of
180 functionally specialised
regions (per hemisphere) on
(MSM-aligned) group average
data
• 97 entirely new areas
• 83 areas previously reported by
histological studies
56Glasser, Matthew F., et al. "A multi-modal parcellation of human cerebral
cortex." Nature 536.7615 (2016): 171-178.

56.
MeTrICS
Topological
variability in the
human brain
Evidence from folding
• Van Essen, David C. "Neuroimage 28.3
(2005): 635-662.
And function
• Amunts, K., A. Schleicher, and K. Zilles.
Neuroimage 37.4 (2007): 1061-1065.
• Wang D et al Nature
neuroscience.2015;18(12):1853.
• Gordon EM Cerebral Cortex
2017;27(1):386-99.
• Glasser, Matthew F., et al. "A multi-
modal parcellation of human cerebral
cortex." Nature (2016).
57

57.
MeTrICS
Machine learning for cortical parcellation
• Given the known risk of topological variation
• The HCP MMPv1 propagates the group map to individuals via training of a Multi-Layer
Perception (Fully connected neural network)
• Binary (per region) classifications
• used to train classifier ONLY where subject data closely agrees with group
58
Hacker, Carl D., et al. "Resting state network estimation in
individual subjects." Neuroimage 82 (2013): 616-633.

58.
MeTrICSMachine learning for cortical parcellation
The HCP MMPv1: Output from Classifier for 4 example datasets
59
Group Average
Glasser, Matthew F., Tim Coalson, Emma C. Robinson et al. "A multi-
modal parcellation of human cerebral cortex." Nature (2016).

59.
MeTrICSMachine learning for cortical parcellation
The HCP MMPv1: semantic segmentation implemented as a U-net

60.
MeTrICS
Predicting
Phenotypes
• We can use the HCP
parcellation to start to
predict cognitive
phenotypes i.e. fluid
intelligence
• Using a random forest
trained on
• 110 Cortical imaging
features
• averaged for each of the
360 regions
61
• Cross-validated R2 =
0.347
• Feature Importance
mapped back to the
image space

61.
MeTrICS
Predicting Phenotypes
• Canonical Correlation:
• Explores relationship between two multivariate sets of variables
• For two matrices 𝑿 and 𝒀, where columns 𝑿𝒊 and 𝒀𝒊 represent different
variable sets for some example i (imaging data/demographic variables)
• Wish to find 2 new sets sets of basis vectors 𝑨 and 𝑩 (one for 𝑿, one for 𝒀) that
will rotate data into new basis 𝑨’, 𝑩:
that maximises correlations:
• 𝑨’𝑿 and 𝑩’𝒀 maximise
62
ρ = corr(A′
X, B′
Y)

62.
MeTrICS
Predicting Phenotypes
• Smith et al. Nature NeuroScience 2015
• Canonical correlation of functional connectivity matrices vs HCP demographics
63

63.
Looking to the future
(Geodesic) Deep Learning..
64

64.
MeTrICS
The limitations
of spatial
normalisation
• Topology-preserving
transforms are unsuited
to alignment of the
human brain
• However, in the absence
of such regularisation the
problem becomes
extremely ill-posed
65

65.
MeTrICS
The cost of
spatial
misalignment
• Residual spatial
misalignment may
reduce the sensitivity
of population studies
• Recent studies have also
suggested that
comparative studies
of brain networks are
confounded by these
errors
66
Bijsterbosch, et al. elife (2018 and 2019).

66.
MeTrICS
Spectral
alignment
• Spectral (Hyper)
alignment approaches
have been proposed
• But have low spatial
resolution and a tendency
to assign correspondences
between spatially distant
regions
(due to long range
functional correlations)
67
Langs, Georg, et al. Advances in neural information processing systems. 2010.
Nenning, Karl-Heinz, et al. "Diffeomorphic functional brain surface alignment: Functional
demons." NeuroImage 156 (2017): 456-465.

67.
MeTrICS
Could Deep Learning provide and answer?
• Learns to represent image through convolution with a hierarchy of increasing
complex filters
• Offers a degree of transformation invariance
68
Low level (early) filters
– small receptive field
– edge detectors
Mid level (early) filters
– larger receptive field
– textures
At later layers filters detect whole objects

68.
MeTrICS
Geodesic Deep
Learning
• Techniques for applying
convolutional operations
to graphs or manifolds
• Either:
• Spatial (e.g. Spherical
U-Net, which learns
hexagonal filters)
• Spectral e.g. ChebNet,
Caleynet
69
Seong, Si-Baek, et
al Frontiers in
Neuroinformatics 12
(2018): 42.
Zhao, Fenqiang, et al.
"Spherical U-Net on Cortical
Surfaces: Methods and
Applications." IPMI 2019.
CayleyNets: Graph Convolutional Neural Networks with
Complex Rational Spectral Filters

69.
MeTrICS
Geodesic Deep
Learning
70
• Prototyping on
prediction of
gestational age at scan
from neonatal cortical
imaging data
• DL outperforms
spatial-normalisation
+ ML
x Train mae=0.198; Test mae= 0.493
x Train mae=0.41; Test mae= 0.95
Derived through deep learning
Derived through traditional machine learning

70.
MeTrICS
Geodesic Deep
Learning
• Prototyping on
prediction of
gestational age at scan
from neonatal cortical
imaging data shows
promise
Increasing GA
32 37 41 44
Means
Features visualised using Grad CAM
Selvaraju,
Ramprasaath R.,
et al. ICCV. 2017.

71.
MeTrICS
Geodesic Deep
Learning…
On graphs
• Siamese networks can be used
to learn a metric by which
to compare functional
brain networks
• To generate classifiers for
gender and Autism
• Ktena et al Neuroimage
(2018)

72.
Tutorial
73
How do I start working with cortical surface imaging data

73.
MeTrICS
Getting data
The following ‘Big’ data cortical imaging data sets are open and subject to download
(subject to user agreements)
74
(no cortical processing yet)
1.
https://db.humanconnectome.org/
~ 1100 young
adult datasets
2.
558 session (505 subjs)
of Neonatal data:
http://www.developingconnec
tome.org/second-data-
release/
3.
But 40,000 images and
growing
https://bbams.ndph.ox.ac.uk/
ams/researcher_home.jsp

74.
MeTrICS
dHCP neonatal data release
2nd Data Release
• 558 session (505 subjects) of Neonatal data:
• T1, T2, fMRI and dMRI volumes, original, cleaned, pre-processed and mapped to
volumetric template space
• Cortical surface meshes and features
• http://www.developingconnectome.org/data-release/information-registration-and-download/
• Released as Torrent (troubleshooting documents available)
• Support via https://neurostars.org/tags/developing-hcp
75

75.
MeTrICS
HCP connectomedb
76
• Easiest to start with:
• Highest resolution,
lowest noise
• Easiest/most flexible
download platform
• Requires IBMs aspera
connect
• Then you can just choose
how many samples and
what data per sample
https://db.humanconnectome.org/
https://wiki.humanconnectome.org/ https://www.humanconnectome.org/contact-usHELP?

76.
MeTrICS
Downloading a test subject and target data
Select: A) Single subject data
77
Pray aspera works
first time
I used safari and needed
to restart
Once you know what files you need you can download on mass using curl
https://wiki.humanconnectome.org/display/PublicData/How+To+Access+Subject+Data+via+REST

77.
MeTrICS
Downloading test data
Select: Tutorial Data set
78

78.
MeTrICS
Connectome
Workbench
• wb_command
– image processing e.g. surface
based smoothing, resampling etc
• wb_view
– excellent volume/surface viewer
• Tutorials
• PDF available with tutorial dataset
download
• http://mvpa.blogspot.com/2017/08/g
etting-started-with-connectome.html
https://www.humanconnectome.org/software/connectome-workbench

79.
MeTrICS
BALSA
• https://balsa.wustl.edu/
• Database of HCP outputs
• Include HCP parcellation
dataset
Refers to a wb_view file fomat

80.
MeTrICS
HCP ‘Atlas’
spaces
• All data in the HCP has been
aligned to an average
(template) brain using either:
• MSMSulc (folding based
alignment)
• MSMall (folding, myelin and
function)
• There are 2 template spaces
• MNInonlinear (164k)
• fsaverage32k (32k)
MSMSulc
MSMall

81.
HCP ‘Atlas’
spaces
• All data in the HCP has been
aligned to an average
(template) brain using
either:
• MSMSulc (folding based
alignment)
• MSMall (folding, myelin
and function)
• There are 2 template spaces
• MNInonlinear (164k)
• fsaverage32k (32k)
• One Native space
82
AVERAGENATIVE

82.
MeTrICS
HCP data
• Each subject folder presents
data in Native space and
resampled onto templates
• Unless specifically stated (in
the filename) the files will be
transformed via MSMSulc
File types:
• Niftii (.nii.gz) volumes
• Gifti surfaces surf.gii)
• Gifti metrics
(.func/.shape/.label.gii)
• CIFITI (.nii)
ls 100307/MNINonLinear/fsaverage_LR32k/
100307.L.MyelinMap.32k_fs_LR.func.gii
100307.L.aparc.32k_fs_LR.label.gii
100307.L.aparc.a2009s.32k_fs_LR.label.gii
100307.L.atlasroi.32k_fs_LR.shape.gii
100307.L.corrThickness.32k_fs_LR.shape.gii
100307.L.curvature.32k_fs_LR.shape.gii
100307.L.inflated.32k_fs_LR.surf.gii
100307.L.midthickness.32k_fs_LR.surf.gii
100307.L.pial.32k_fs_LR.surf.gii
100307.L.rfMRI_REST.ica_d30_MSMAllDeDriftFinal_d26.SubjectsD.func.gii
100307.L.sphere.32k_fs_LR.surf.gii
100307.L.sulc.32k_fs_LR.shape.gii
100307.L.thickness.32k_fs_LR.shape.gii
100307.L.very_inflated.32k_fs_LR.surf.gii
100307.L.white.32k_fs_LR.surf.gii
100307.MyelinMap.32k_fs_LR.dscalar.nii
100307.aparc.32k_fs_LR.dlabel.nii
100307.aparc.a2009s.32k_fs_LR.dlabel.nii
100307.corrThickness.32k_fs_LR.dscalar.nii
100307.curvature.32k_fs_LR.dscalar.nii
100307.rfMRI_REST.ica_d30_MSMAllDeDriftFinal_d26.SubjectsD.dscalar.nii
100307.sulc.32k_fs_LR.dscalar.nii
100307.thickness.32k_fs_LR.dscalar.nii
CIFTI

83.
MeTrICS
CIFTI
• Left and right surface metrics
• Plus deep grey matter
volume
• fMRI inspired

84.
MeTrICS
How can I process new data onto surfaces
dHCP structural pipeline (minimal requirement T2w MRI)
• https://github.com/BioMedIA/dhcp-structural-pipeline
• Includes docker installation - contact j.cupitt@imperial.ac.uk
• Pial surface extraction can be a bit buggy for different acquisition parameters
• Support via https://neurostars.org/tags/developing-hcp
HCP structural pipeline (minimal requirement T1 and T2w MRI)
• https://www.humanconnectome.org/software/hcp-mr-pipelines
• If only T1 look to the original freesurfer
https://surfer.nmr.mgh.harvard.edu/fswiki/DownloadAndInstall
• https://www.humanconnectome.org/contact-us
dHCP Atlases
• https://brain-development.org/brain-atlases/atlases-from-the-dhcp-project/
• https://github.com/ecr05/dHCP_template_alignment (Script for running MSM surface-to-
template alignment for neonates)
HCP Atlases
• https://db.humanconnectome.org/data/projects/HCP_1200
85

85.
MeTrICS
Running MSM
Download from
• Github:
https://github.com/ecr05/MSM_HOC
R/releases
• FSL :
https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/
MSM
SAS SSS
a)
e)
b)
x LABEL POINT
OPTIMAL LABEL
CONTROL POINT
c)
f)
d)
MSS
SSS + G
TSS
DAS TAS
g)
F
Option 1 sMSM
Option 2 aMSM

86.
MeTrICS
Running
MSMSulc
sMSM Inputs:
-o output filepath
--inmesh path to input sphere
--refmesh path to target sphere
--indata path to input metric
--refdata path to target metric
--conf config
--trans for chaining trans-
formations
• Pairwise config
https://github.com/ecr05/MSM_HOCR/blob/master/
config/basic_configs/config_standard_MSMpair
• HOCR config
https://github.com/ecr05/MSM_HOCR/blob/master/
config/basic_configs/config_standard_MSM_strain
msm --
inmesh=111716/MNINonLinear/Native/111716.L.sphere.rot.native.surf.gii --
refmesh=HCP_WB_Tutorial_1.0/Q1-Q6_R440.L.sphere.164k_fs_LR.surf.gii --
indata=111716/MNINonLinear/Native/111716.L.sulc.native.shape.gii --
refdata=HCP_WB_Tutorial_1.0/Q1-Q6_R440.L.sulc.164k_fs_LR.shape.gii --
conf=config_standard_MSMpair -o 111716_Q1-Q6_R440.L.MSMSulc -v
MSMSulc:
wb_command -cifti-separate HCP_WB_Tutorial_1.0/Q1-
Q6_R440.sulc.164k_fs_LR.dscalar.nii COLUMN -metric CORTEX_LEFT
HCP_WB_Tutorial_1.0/Q1-Q6_R440.L.sulc.164k_fs_LR.shape.gii
Outputs:
• 111716_Q1-Q6_R440.L.MSMSulc.sphere.reg.surf.gii
• 111716_Q1-Q6_R440.L.MSMSulctransformed_and_reprojected.func.gii

87.
MeTrICS
Running
MSMall
sMSM Inputs:
-o output filepath
--inmesh path to input sphere
--refmesh path to target sphere
--indata path to input metric
--refdata path to target metric
--conf config
--trans for chaining trans-
formations
• Pairwise regularisation (not
recommended)
• HOCR config
https://github.com/ecr05/MSM_HOCR/blob/master/
config/HCP_multimodal_alignment/MSMAllStrainFi
nalconf1to1_1to3_2
MSMall*:
*Toy example not actual call – for exact details see HCP pipelines
msm --
inmesh=111716/MNINonLinear/Native/111716.L.sphere.rot.native.surf.gii --
trans=111716/MNINonLinear/Native/
111716.L.sphere.MSMSulc.native.surf.gii --in_register=111716/
MNINonLinear/fsaverage_LR32k/111716.L.sphere.32k_fs_LR.surf.gii
--refmesh=HCP_WB_Tutorial_1.0/Q1-Q6_R440.L.sphere.32k_fs_LR.surf.gii --
indata=../HCP_PARCELLATION/TRAININGDATA/featuresets/
111716.L.MultiModal_Features_MSMAll_2_d41_WRN_DeDrift.32k_fs_LR.func.gii
--refdata=../HCP_PARCELLATION/TRAININGDATA/featuresets/
MEAN.L.MultiModal_Features_MSMAll_2_d41_WRN_DeDrift.32k_fs_LR.func.gii
--conf=MSMAllStrainFinalconf1to1_1to3_2 -o 111716_Q1-Q6_R440.L.MSMall
• --trans allows you to initialise with MSMSulc
whilst regularizing over full deformation
• fMRI data only exists on 32k template space thus
–in_register allows for resampling of the initial
warp to the 32K mesh space

88.
MeTrICS
Running
aMSM
aMSM Inputs:
-o output filepath
--inmesh path to input sphere
--refmesh path to target sphere
--indata path to input metric
--refdata path to target metric
--conf config
--trans for chaining trans-
formations
--inanat. Input anatomy (i.e.
white surface)
--refanat Target anatomy
• Needs many iterations - Runs for
24 hours!!!
Hidden options
msm
--inmesh=111716/MNINonLinear/Native/
111716.L.
sphere.rot.native.surf.gii
--refmesh=HCP_WB_Tutorial_1.0/Q1-
Q6_R440.L.sphere.164k_fs_LR.surf.gii
--indata=111716/MNINonLinear/Native/
111716.L.sulc.native.shape.gii
--refdata=HCP_WB_Tutorial_1.0/Q1-
Q6_R440.L.sulc.164k_fs_LR.shape.gii --
conf=config_standard_aMSM
-o 111716_Q1-Q6_R440.L.MSMSulc -v
--inanat=111716/MNINonLinear/Native/
111716.L.white.native.surf.gii
--refanat=HCP_WB_Tutorial_1.0/Q1-
Q6_R440.L.white.164k_fs_LR.surf.gii
aMSM:
Significantly smooths deformations of the
anatomical mesh

89.
MeTrICS
A note on
configs
• --regoption determines mode
• 1 for pairwise
• 3 for sMSM strain
• 5 for aMSM strain
• --bulkmod ---shearmod are 𝜅 and 𝜇
in strain equation
• 𝜆 controls regularisation strengths
• --it = iterations
• 5 ok for pairwise (with –rescaleL)
• Otherwise use 10-20
• aMSM will need more

90.
MeTrICS
ML Tips and
Tricks
• You can load data into Python
using nibabel
https://nipy.org/nibabel/
metric=nibabel.load(‘some.func.gii’)
• Metric (.func/.shape/label etc)
contain data arrays
• Get each data array i with
features = metric.darrays[i].data
Kings ICL Smart imaging CDT
Lecture series Advanced Machine Learning
• Deep Learning tutorials for medical
image data
• In Pytorch
• Available soon github/coursera
@ecr05

91.
MeTrICS
Summary
• Cognitive disorders are extremely challenging to model due to cortical and
phenotypic heterogeneity
• Surface based modelling of the brain has been shown to significantly improve the
accuracy with which we can model cortical organization
• Admittedly slightly more overhead in
• getting data onto the surface
• Using/understand existing surface data sets
• Hopefully, this tutorial will relieve a lot of confusion and create some
opportunities
• Lots of potential for creative uses of Machine Learning of surface imaging data
• Need help/want to collaborate?
92
emma.robinson@kcl.ac.uk @emrobSci ecr05

92.
MeTrICS
Acknowledgements
My team:
94
• Cher Bass
• Abdulah Fawaz
• Kyriaki Kaza
KCL
• David Edwards
• Jo Hajnal
• Jonathan
O’Muicheartaigh
• Maria Deprez
Other Contributors and
collaborators:
Imperial College:
• Daniel Rueckert
• Bernhard Kainz
• Antonis Makropoulos
• Andreas Schuh
• Samuel Budd
• Ira Ktena
• Martin Rajchl
Oxford
• Stephen Smith
• Mark Jenkinson
• Saad Jbabdi
WashU
• Matthew Glasser
• David Van Essen
• Janine Bijsterbosch
• Tim Coalson
https://metrics-
lab.github.io/