Introduction to checking reliability and validity of fMRI.

1. Reliability and validity of fMRI
ElhamFaghihzadeh(Ph.D. of Biostatistics)
faghihzadeh.elham@gmail.com
Department of Biostatistics and Epidemiology, Zanjan Universality of Medical science
&
NIAG, Imam Khomeini Hospital

2. Reliability and Validity
2

3. The accuracy, reliability and limitations of neuroimaging
depend on the purpose of its use
 Neuroscience research
 Medical (Such as Presurgical)
 Marketing
 Law
 Military
3

4. Structural vs Functional brain imaging
Structural brain imaging
Anatomy
Volume
Area
Spatial resolution 1x1x1 mm3
Functional brain imaging
 Brain function
 fMRI
 Perfusion imaging
 Diffusion imaging
 Molecular imaging
– MR spectroscopy
– SPECT
– PET
Spatial and Temporal resolution
4

5. Structural imaging
• Knowledge of
– Techniques
– Neuroanatomy
– Pathology
• Meaning of disease?
5

6. Technical Knowledge
6

7. Meaning of disease
What is the effect of the disease on the brain development
and function?
We don’t know!
7

8. Who has dementia?
Same Born Age
8

9. Functional imaging
• Technique
Indirect measure of brain
activity (BOLD)
• much larger than structural
datasets
9

10. 10
Arthurs, O.J. and Boniface, S., 2002. How well do we understand the neural origins of the fMRI BOLD
signal?. TRENDS in Neurosciences, 25(1), pp.27-31.
Pain
Motor
sensory
Cognitive
……
Neuronal Activity Hemodynamic Response
Neurovascular
Coupling
Detection of
fMRI scan
fMRI BOLD
Response
stimulus

11. Limitations and validity of fMRI
Theory behind the study & nature of the research question
Design and logistics of experiment
Reliability of task
Subject’s collaboration
– Head movement
– Task performance
 Field strength of the MRI scanner (and other hardware)
11

12. Number of subjects
Software
Registration
Statistics
Smoothing
Type of test
Thresholding
Region-of-interest analysis
Inferences to the population
12

13. Number of subjects
Clinical setting
Forensic use
Research
Marketing
Single
subject
Group
13

14. The effect of the number of
participants
14

15. Thresholding (Type of Errors)
Actual condition
Testresult
Active
Inactive
Inactive Active
True negative

True positive

False positive
Type I Error 
False negative
Type II Error ß
15

16. Thresholding
Several techniques can be used
Bonferroni correction
Permutation thresholding
Random field theory
False discovery rate
Designed to control for type 1
error
‘To make sure that we don’t say
there is an activation when there
isn’t one’
Type 2 error is important when
n=1!
‘To make sure that we don’t say
there
isn’t an activation when in fact there
is 16

17. Reliability Measurement
Reliability is a quantitative measurement that indexes the
stability of data values.
This is typically calculated as stability over time for test–
retest data, or stability of agreement across ratters.
17

18. An array of measures such as:
Intraclass Correlation (ICC), the ratio of the between-subject
variance and the total variance
Coefficient of variation (CV), the ratio between the SD and the
signal change between two conditions
been used to index the reliability of fMRI values over time.
18

19. 19

20. Three main possible ICC implementations on neuroimaging
data,
i. ICC for each activated ROI
ii. The within-subject ICC (Smoothing Problem)
iii.Voxel-wise ICCs to explore the reliability of an activated ROI
20

21. References:
• Seixas, D. and Lima, D., 2011. Accuracy, reliability, validity and limitations of functional and structural magnetic
resonance imaging data. Cortex; a journal devoted to the study of the nervous system and behavior, 47(10), p.1266.
• Raemaekers, M., Vink, M., Zandbelt, B., van Wezel, R.J.A., Kahn, R.S., Ramsey, N.F., 2007. Test–retest reliability of fMRI
activation during prosaccades and antisaccades. Neuroimage 36 (3), 532–542.
• Caceres, A., Hall, D.L., Zelaya, F.O., Williams, S.C. and Mehta, M.A., 2009. Measuring fMRI reliability with the intra-class
correlationn coefficient. Neuroimage, 45(3), pp.758-768.
• Matheson GJ. We need to talk about reliability: making better use of test-retest studies for study design and
interpretation. PeerJ. 2019 May 24;7:e6918. doi: 10.7717/peerj.6918. PMID: 31179173; PMCID: PMC6536112.
21

22. 22