This talk attemps to motivate the use of Graph Signal Processing to analyse neuroimaging data. After introducing recent paradigm shifts in neuroimaging research (network neuroscience and principal gradients of connectivity), we present our recent work in combining GSP and machine learning, which show substantial improvements in inference based approach using simple machine learning techniques. We finally open new perspectives regarding the potential of using GSP for interpretable neuroscientific research.

1. Graph Signal Processing: an interpretable
framework to link neurocognitive architectures
with brain activity
Nicolas Farrugia
June 4th, 2019
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 1 / 26

2. Prelude
Signal processing in Neuroscience
Interpreting (oscillatory) brain activity using Fourier or Wavelet
transforms.
Network Neuroscience
Applying graph theory metrics to study functional and structural
connectivity.
Graph Signal Processing for Cognitive Neurosciences
The best of both worlds ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 2 / 26

3. Prelude
Signal processing in Neuroscience
Interpreting (oscillatory) brain activity using Fourier or Wavelet
transforms.
Network Neuroscience
Applying graph theory metrics to study functional and structural
connectivity.
Graph Signal Processing for Cognitive Neurosciences
The best of both worlds ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 2 / 26

4. Prelude
Signal processing in Neuroscience
Interpreting (oscillatory) brain activity using Fourier or Wavelet
transforms.
Network Neuroscience
Applying graph theory metrics to study functional and structural
connectivity.
Graph Signal Processing for Cognitive Neurosciences
The best of both worlds ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 2 / 26

5. 1 Motivations
Functional role of brain oscillations
From brain regions to network neuroscience
Networks and connectome harmonics / gradients
2 Graph Signal Processing for Neuroimaging
Introduction to Graph Signal Processing
Inference using GSP derived measures
Predictive approaches using GSP derived feature vectors
GSP and interpretability
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 3 / 26

6. Spectral (time-frequency) analysis in cognitive
neuroscience
Hans Berger and early measurements of electro-encephalography
-> Observations on alpha (7.5 to 12.5 Hz) and beta (12.5 to 25 Hz)
frequency bands (and later, gamma frequency band from 25 to 40 Hz)
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 4 / 26

7. Spectral (time-frequency) analysis in cognitive
neuroscience
Fast-forward to 21st century...
Functional role of delta-theta phase in sensory prediction.
Arnal and Giraud, 2012, Trends in Cognitive Sciences
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 4 / 26

8. Spectral (time-frequency) analysis in cognitive
neuroscience
Fast-forward to 21st century...
Phase and cross-frequency coupling:
Sauseng et al. 2008, Neuroscience and Biobehavioral reviews
See also : non-sinusoidal oscillations (Cole and Woytek, 2017, TICS)
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 4 / 26

9. 1 Motivations
Functional role of brain oscillations
From brain regions to network neuroscience
Networks and connectome harmonics / gradients
2 Graph Signal Processing for Neuroimaging
Introduction to Graph Signal Processing
Inference using GSP derived measures
Predictive approaches using GSP derived feature vectors
GSP and interpretability
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 5 / 26

10. From regions to networks...
Primary visual cortex
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 6 / 26

11. From regions to networks...
Coactivation of brain areas - networks
Resting-state networks - spontaneous and task activity.
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 6 / 26

12. The network neuroscience revolution
Fornito et al. 2015, Nat. Neuro.
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 7 / 26

13. From networks to spectral graph analysis
Connectome harmonics - spectral analysis of structural connectivity.
Atasoy et al. 2016, Nat. Comms
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 8 / 26

14. From networks to spectral graph analysis
Connectivity gradients - Spectral analysis of functional connectivity.
Margulies et al. 2016, PNAS
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 8 / 26

15. From networks to spectral graph analysis
Functional interpretation of connectivity gradients.
Margulies et al. 2016, PNAS
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 8 / 26

16. 1 Motivations
Functional role of brain oscillations
From brain regions to network neuroscience
Networks and connectome harmonics / gradients
2 Graph Signal Processing for Neuroimaging
Introduction to Graph Signal Processing
Inference using GSP derived measures
Predictive approaches using GSP derived feature vectors
GSP and interpretability
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 9 / 26

17. What is Graph Signal Processing ?
GSP in four steps
Define a graph with N nodes,
Compute Graph Laplacian L (e.g. L = D − A) and find its
eigenvectors U
Define a signal x with N values, one per node.
Graph Fourier transform is defined by ˆx = U x
Analogy with (Discrete) Fourier Transform
Shuman et al. 2013, IEEE Signal Processing Magazine
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 10 / 26

18. Why using GSP for Neuroimaging ?
A straightforward use case
Brain signals on graphs built on structural (white matter) connectivity.
Huang et al. 2018
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 11 / 26

19. Why using GSP for Neuroimaging ?
A straightforward use case
Brain signals on graphs built on structural (white matter) connectivity.
Elegance
Network-centric tools adapted to the problem
Applicable to both inference-based and predictive approaches
Interpretations in term of graph frequencies
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 11 / 26

20. Why using GSP for Neuroimaging ?
A straightforward use case
Brain signals on graphs built on structural (white matter) connectivity.
Elegance
Network-centric tools adapted to the problem
Applicable to both inference-based and predictive approaches
Interpretations in term of graph frequencies
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 11 / 26

21. 1 Motivations
Functional role of brain oscillations
From brain regions to network neuroscience
Networks and connectome harmonics / gradients
2 Graph Signal Processing for Neuroimaging
Introduction to Graph Signal Processing
Inference using GSP derived measures
Predictive approaches using GSP derived feature vectors
GSP and interpretability
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 12 / 26

22. Inference using GSP metrics
What measures are authors currently using ?
Useful measures on graph signals
Smoothness (x Lx), also called Dirichlet energy,
Mutual information / F measure of GFT decompositions,
Concentration in certain frequency bands (Alignement / Liberality)
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 13 / 26

23. Inference using GSP metrics
Huang et al. 2018
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 13 / 26

24. Inference using GSP metrics
Statistical methods : group-wise inference
Dependent variables derived from GSP statistically related to
behavior.
Smoothness / Dirichlet energy (Smith et al. 2017)
Alignement and Liberality related to attention switching (Huang et
al., 2018)
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 13 / 26

25. Inference using GSP metrics
Resting state networks predicted by connectome harmonics.
Atasoy et al. 2016, Nat. Comms
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 13 / 26

26. 1 Motivations
Functional role of brain oscillations
From brain regions to network neuroscience
Networks and connectome harmonics / gradients
2 Graph Signal Processing for Neuroimaging
Introduction to Graph Signal Processing
Inference using GSP derived measures
Predictive approaches using GSP derived feature vectors
GSP and interpretability
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 14 / 26

27. Predictive approach using GSP
Proposed approach
Graph nodes defined by a brain parcellation
Graph edges defined by functional or structural connectivity
Feature Extraction on brain activity using GSP (e.g. GFT or Graph
Wavelets)
Feature selection
Supervised learning (using an interpretable method)
Ménoret, Farrugia, Pasdeloup and Gripon, 2017, GlobalSIP
Brahim and Farrugia, 2019, GRETSI
Pilavci and Farrugia, 2019, ICASSP
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 15 / 26

28. Predictive approach using GSP
Proposed approach
Graph nodes defined by a brain parcellation
Graph edges defined by functional or structural connectivity
Feature Extraction on brain activity using GSP (e.g. GFT or Graph
Wavelets)
Feature selection
Supervised learning (using an interpretable method)
Ménoret, Farrugia, Pasdeloup and Gripon, 2017, GlobalSIP
Brahim and Farrugia, 2019, GRETSI
Pilavci and Farrugia, 2019, ICASSP
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 15 / 26

29. Predictive approach using GSP
Proposed approach
Graph nodes defined by a brain parcellation
Graph edges defined by functional or structural connectivity
Feature Extraction on brain activity using GSP (e.g. GFT or Graph
Wavelets)
Feature selection
Supervised learning (using an interpretable method)
Ménoret, Farrugia, Pasdeloup and Gripon, 2017, GlobalSIP
Brahim and Farrugia, 2019, GRETSI
Pilavci and Farrugia, 2019, ICASSP
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 15 / 26

30. Predictive approach using GSP
Proposed approach
Graph nodes defined by a brain parcellation
Graph edges defined by functional or structural connectivity
Feature Extraction on brain activity using GSP (e.g. GFT or Graph
Wavelets)
Feature selection
Supervised learning (using an interpretable method)
Ménoret, Farrugia, Pasdeloup and Gripon, 2017, GlobalSIP
Brahim and Farrugia, 2019, GRETSI
Pilavci and Farrugia, 2019, ICASSP
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 15 / 26

31. Graph Fourier Transform on resting state data
Brahim and Farrugia, submitted
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 16 / 26

32. Graph Fourier Transform on resting state data
Resuls on ADHD200 dataset.
5 10 15 20 25 30 35 40 45
Features
0.0
0.2
0.4
0.6
0.8
1.0
Accuracy
STD+SG
STD
FC
EC
CC
NS
Mean
Mean+SG
5 10 15 20 25 30 35 40 45
Features
0.0
0.5
1.0
1.5
2.0
2.5
-log(P)
1.0 2.5
Brahim and Farrugia, submitted
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 16 / 26

33. Graph Fourier Transform on resting state data
ADHD-200
Approaches Acc (%) Sen (%) Spe (%)
STD 52.5±0.037 55±0.043 55±0.066
STD+GSP 85±0.019 85±0.036 85±0.036
FC 65±0.032 60±0.059 70±0.063
ABIDE NYU Langone
Approaches Acc (%) Sen (%) Spe (%)
STD 66.65±0.007 63.75±0.015 74.44±0.010
STD+GSP 70.36±0.007 67.68±0.011 75.67±0.012
FC 62.83±0.006 56.61±0.014 70.22±0.015
Table: Maximum classification rates of the different approaches for
ADHD-200 and NYU databases using SVM classifier with linear kernel (max
± standard error of the mean).
Brahim and Farrugia, submitted
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 16 / 26

34. Graph Fourier Transform on resting state data
Feature Selection (results from ABIDE NYU Langone)
Top: Ratios of selected ROIs across 10 folds, using STD (Accuracy 66.65±0.007 ).
Bottom : Average of selected GFT modes across 10 folds, using STD + GFT (Accuracy 70.36±0.007)
Brahim and Farrugia, submitted
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 16 / 26

35. Spectral Graph Wavelet Transform (SGWT)
Wavelet atoms are obtained by:
ψs,a = ΣN
n=1g(sλn )ˆδ(n)un (1)
where g(.) is the kernel function, s is the scale, ˆδ is the graph Fourier
transform of dirac delta and un are the columns of U.
SGWT of f is obtained using the inner product Wf (s, a) =< ψs,a , f >.
Optimizing spectral coverage using Warped Translate kernel
Pilavci and Farrugia, 2019, ICASSP
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 17 / 26

36. Spectral Graph Wavelet Transform (SGWT)
We tested this approach using an averaged functional connectivity
graph parcellated on 523 regions of BASC Atlas
Usings signals from :
A synthetic regression problem on a brain graph
fMRI data from Chang et al., predicting affective rating of
emotional pictures
Pilavci and Farrugia, 2019, ICASSPIMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 17 / 26

37. Spectral Graph Wavelet Transform (SGWT)
Pilavci and Farrugia, 2019, ICASSP
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 17 / 26

38. 1 Motivations
Functional role of brain oscillations
From brain regions to network neuroscience
Networks and connectome harmonics / gradients
2 Graph Signal Processing for Neuroimaging
Introduction to Graph Signal Processing
Inference using GSP derived measures
Predictive approaches using GSP derived feature vectors
GSP and interpretability
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 18 / 26

39. GSP and Interpretability
Inference
Generate and test hypothesis on graph frequency bands (similar
to M/EEG oscillations)
Spatial contributions of Graph Fourier modes
Prediction / data-driven
Visualize GSP features or contributions of Grapˆh Fourier modes
Use of interpretable ML methods with GSP features as inputs
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 19 / 26

40. GSP and Interpretability
Inference
Generate and test hypothesis on graph frequency bands (similar
to M/EEG oscillations)
Spatial contributions of Graph Fourier modes
Prediction / data-driven
Visualize GSP features or contributions of Grapˆh Fourier modes
Use of interpretable ML methods with GSP features as inputs
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 19 / 26

41. Generating new hypothesis using GSP (1/2)
Hypothesis using Frequencies
What would such an hypothesis look like ?
In the field of brain oscillations, empirical knowledge has been
built using frequency bands.
Should we start to make hypothesis on graph frequencies ?
Measures of (temporal) excursions from liberality or alignement
(Huang et al. 2018) - linking temporal frequencies and graph
frequencies
Can we directly make hypothesis on Graph fourier modes ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 20 / 26

42. Generating new hypothesis using GSP (1/2)
Hypothesis using Frequencies
What would such an hypothesis look like ?
In the field of brain oscillations, empirical knowledge has been
built using frequency bands.
Should we start to make hypothesis on graph frequencies ?
Measures of (temporal) excursions from liberality or alignement
(Huang et al. 2018) - linking temporal frequencies and graph
frequencies
Can we directly make hypothesis on Graph fourier modes ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 20 / 26

43. Generating new hypothesis using GSP (1/2)
Hypothesis using Frequencies
What would such an hypothesis look like ?
In the field of brain oscillations, empirical knowledge has been
built using frequency bands.
Should we start to make hypothesis on graph frequencies ?
Measures of (temporal) excursions from liberality or alignement
(Huang et al. 2018) - linking temporal frequencies and graph
frequencies
Can we directly make hypothesis on Graph fourier modes ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 20 / 26

44. Generating new hypothesis using GSP (2/2)
Hypothesis using Harmonics
In previous slides we have shown that propagation (diffusion)
patterns:
Were functionally meaningful,
Could predict spontaneous activity.
Analysis of experimental data could exploit decompositions based on
Harmonics
Could we exploit (connectome) harmonics to understand
spatiotemporal dynamics ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 21 / 26

45. Generating new hypothesis using GSP (2/2)
Hypothesis using Harmonics
In previous slides we have shown that propagation (diffusion)
patterns:
Were functionally meaningful,
Could predict spontaneous activity.
Analysis of experimental data could exploit decompositions based on
Harmonics
Could we exploit (connectome) harmonics to understand
spatiotemporal dynamics ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 21 / 26

46. Generating new hypothesis using GSP (2/2)
Hypothesis using Harmonics
In previous slides we have shown that propagation (diffusion)
patterns:
Were functionally meaningful,
Could predict spontaneous activity.
Analysis of experimental data could exploit decompositions based on
Harmonics
Could we exploit (connectome) harmonics to understand
spatiotemporal dynamics ?
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 21 / 26

47. What is missing: Dynamic (hyper?) graphs
Estimating a graph measure at each time point may not be valid
(Basset and Sizemore 2017.
Ongoing debate on time(?)-varying functional connectivity using sliding
windows(e.g. Hindriks et al. 2016)
Building dynamic graph and associated measures using slepians ?
(Huang et al. 2018), Preti and Van De Ville 2017
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 22 / 26

48. What is missing: Dynamic (hyper?) graphs
Estimating a graph measure at each time point may not be valid
(Basset and Sizemore 2017.
Ongoing debate on time(?)-varying functional connectivity using sliding
windows(e.g. Hindriks et al. 2016)
Building dynamic graph and associated measures using slepians ?
(Huang et al. 2018), Preti and Van De Ville 2017
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 22 / 26

49. What is missing: Dynamic (hyper?) graphs
Estimating a graph measure at each time point may not be valid
(Basset and Sizemore 2017.
Ongoing debate on time(?)-varying functional connectivity using sliding
windows(e.g. Hindriks et al. 2016)
Building dynamic graph and associated measures using slepians ?
(Huang et al. 2018), Preti and Van De Ville 2017
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 22 / 26

50. What is missing: Dynamic (hyper?) graphs
Estimating a graph measure at each time point may not be valid
(Basset and Sizemore 2017.
Ongoing debate on time(?)-varying functional connectivity using sliding
windows(e.g. Hindriks et al. 2016)
Building dynamic graph and associated measures using slepians ?
(Huang et al. 2018), Preti and Van De Ville 2017
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 22 / 26

51. Conclusions and perspectives
The promising part:
Connectome harmonics are relevant,
Certain graph frequency may be useful to generate new
hypothesis,
GSP may help levarage explanatory power in small datasets.
The difficult part...
Brain signals (oscillations) are highly non-linear and
non-stationnary,
Dynamic graphs are needed to integrate both the complex spatial
dynamics and non-stationary aspects,
Unique opportunity of collaborative research between Neuroscience
and AI!
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 23 / 26

52. Conclusions and perspectives
The promising part:
Connectome harmonics are relevant,
Certain graph frequency may be useful to generate new
hypothesis,
GSP may help levarage explanatory power in small datasets.
The difficult part...
Brain signals (oscillations) are highly non-linear and
non-stationnary,
Dynamic graphs are needed to integrate both the complex spatial
dynamics and non-stationary aspects,
Unique opportunity of collaborative research between Neuroscience
and AI!
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 23 / 26

53. Conclusions and perspectives
The promising part:
Connectome harmonics are relevant,
Certain graph frequency may be useful to generate new
hypothesis,
GSP may help levarage explanatory power in small datasets.
The difficult part...
Brain signals (oscillations) are highly non-linear and
non-stationnary,
Dynamic graphs are needed to integrate both the complex spatial
dynamics and non-stationary aspects,
Unique opportunity of collaborative research between Neuroscience
and AI!
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 23 / 26

54. Thanks!
Questions?
Thanks to Vincent Gripon, Abdelbasset Brahim, Yusuf Yigit Pilavci,
Mathilde Ménoret and Bastien Pasdeloup.
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 24 / 26

55. Prediction and Inference
"Inference typically focuses on isolating variables deemed
individually important above some chance level, often based on p
values."
"Prediction commonly aims at identifying variable sets that
together enable accurate guessing of outcomes based on other or
future data."
Bzdok and Ioannidis. 2019, Trends in Neurosciences
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 25 / 26

56. Graph Fourier Transform on task fMRI data
Feature Extraction using GSP
Supervised learning with dimensionality reduction using Graph
Fourier Transform and Graph Sampling (Menoret et al. 2017)
Table: Comparison of Graph Sampling (Semilocal graph), PCA, ICA and
ANOVA. Classification accuracy with 50 components for the Simulated and
Haxby datasets.
Method
Simulation Haxby
Easy Difficult Face-House Cat-Face
PCA 88.8% 65.5% 82.7% 63.6%
ICA 90.2% 65.3% 84.4% 67.0%
ANOVA 92.1% 67.3% 85.5% 65.5%
Graph sampling 90.9% 72.5% 88.2% 69.0%
Ménoret, Farrugia, Pasdeloup and Gripon, 2017, GlobalSIP
IMT-Atlantique GSP and interpretability in Neuroimaging June 4th, 2019 26 / 26