Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
GENERALIZABILITY IN FMRI,
FAST AND SLOW
TalYarkoni
GENERALIZATION MATTERS
• Induction (reasoning from the specific to the
general) is central to the scientific method
• Most fi...
happy pandas
sad pandas
neutral pandas
LOGIC OF SUBTRACTION
-
Experimental manipulation Emotion-related neural changes
=
BRAINS ON PANDAS
WHAT SHOULD WE
CONCLUDE?
• We studied a small number of subjects responding
to a small number of panda images at one resea...
FAST GENERALIZATION
• Dream big!
• Paper title:“Increased amygdala activation in
response to affective mammalian expressio...
SLOW GENERALIZATION
• Careful now…
• Paper title:“Greater amygdala activation in 22 UT-
Austin undergraduates when viewing...
NOT A MATTER OF PREFERENCE
• We typically prefer our conclusions to hold for stimuli like
the ones we sampled, not just th...
Tom NicholsJake Westfall
THANKSTO…
RANDOMVS. FIXED EFFECTS
• Many definitions in the literature (Gelman, 2005)
• For our purposes: effects are fixed if the obs...
WHY BOTHER WITH
RANDOM EFFECTS?
• Suppose we have 100 trials for each of 20 subjects
—50 per experimental condition
• The ...
WHY STOP AT SUBJECTS?
• In fMRI, subjects are almost always models as
random effects
• Stimuli are virtually never modeled...
Westfall, Nichols, &Yarkoni (2017)
Westfall, Nichols, &Yarkoni (2017)
SIMULATION
• What happens if we don’t include random stimulus
effects?
• Assume:
• No true difference between sad & neutra...
Westfall, Nichols, &Yarkoni (2017)
Judd,Westfall, & Kenny (2012)
Westfall, Nichols, &Yarkoni (2017)
Westfall, Nichols, &Yarkoni (2017)
HEY, WHA’ HAPPENED?
Westfall, Nichols, &Yarkoni (2017)
Westfall, Nichols, &Yarkoni (2017)
THE BAD NEWS
• This problem is pervasive in the fMRI literature
• We estimate that it affects at least 50% of published st...
THE GOOD NEWS
• The behavior is (relatively) predictable
• When stimulus sample is large, RSM and standard
models converge...
Westfall, Nichols, & Yarkoni (2017)
GENERALIZINGTHE
GENERALIZATION PROBLEM
• Subjects and stimuli are not the only thing we care to
generalize over
• Same log...
INFERENCEVS. PREDICTION
• What goes for inference also holds for prediction
• Instead of random effects, we can talk about...
Woo, Chang, Lindquist, & Wager (2017)
Huth et al. (2016)
WHICH POPULATION?
• Modeling a factor as random allows us to generalized
to a population
• This doesn’t mean it’s the inte...
GENERALIZATION PATTERNS
• Given a design factor (e.g., stimuli, subjects, sites,
etc.), what does it take to support gener...
Generalization
target
Example Inference
Prediction/
classification
No population
Specific photos
of Amy and
Ramesh
Ignore F ...
CONCLUSIONS
• Generalization is hard
• We almost universally overgeneralize our conclusions in fMRI
• This has serious con...
A mathematician, a physicist, and an astronomer were
traveling north by train.They had just crossed the border
into Scotla...
Generalizability in fMRI, fast and slow
Generalizability in fMRI, fast and slow
Generalizability in fMRI, fast and slow
Upcoming SlideShare
Loading in …5
×

Generalizability in fMRI, fast and slow

1,998 views

Published on

Slides for keynote lecture at OHBM 2017.

Published in: Science
  • Be the first to comment

Generalizability in fMRI, fast and slow

  1. 1. GENERALIZABILITY IN FMRI, FAST AND SLOW TalYarkoni
  2. 2. GENERALIZATION MATTERS • Induction (reasoning from the specific to the general) is central to the scientific method • Most findings in neuroimaging are only interesting because we expect them to generalize broadly
  3. 3. happy pandas
  4. 4. sad pandas
  5. 5. neutral pandas
  6. 6. LOGIC OF SUBTRACTION - Experimental manipulation Emotion-related neural changes =
  7. 7. BRAINS ON PANDAS
  8. 8. WHAT SHOULD WE CONCLUDE? • We studied a small number of subjects responding to a small number of panda images at one research site with one experimenter • What universe of observations are we are entitled to generalize these conclusions to? • Can we just choose our generalization strategy? • Consider two different approaches: fast vs. slow
  9. 9. FAST GENERALIZATION • Dream big! • Paper title:“Increased amygdala activation in response to affective mammalian expressions relative to neutral mammalian expressions” • Seems pretty interesting! • But how many inductive leaps did we just take?
  10. 10. SLOW GENERALIZATION • Careful now… • Paper title:“Greater amygdala activation in 22 UT- Austin undergraduates when viewing images angsty_tree_panda.jpg and panda_delight.png than image panda_goes_meh.jpg” • Boooooooooring • Which of these titles do you prefer? Fast or slow?
  11. 11. NOT A MATTER OF PREFERENCE • We typically prefer our conclusions to hold for stimuli like the ones we sampled, not just those exact ones • Unfortunately, our intentions and preferences have very little to do with it • The statistical model must support the desired inference • If we want to generalize to new stimuli, our model must account for the variance introduced by the stimulus sampling process
  12. 12. Tom NicholsJake Westfall THANKSTO…
  13. 13. RANDOMVS. FIXED EFFECTS • Many definitions in the literature (Gelman, 2005) • For our purposes: effects are fixed if the observed levels are of intrinsic interest, and random if they are only interesting as exemplars of an underlying population • E.g., subjects are almost always treated as a random factor • We rare don’t care about these particular subjects • In an N-back task, load is considered fixed • We don’t sample 2-back and 3-back from a population of N’s
  14. 14. WHY BOTHER WITH RANDOM EFFECTS? • Suppose we have 100 trials for each of 20 subjects —50 per experimental condition • The appropriate one-sample t-test is conducted at the level of subjects, not trials.Why? • Answer: because we intend to generalize to new subjects, and trials are correlated within subjects
  15. 15. WHY STOP AT SUBJECTS? • In fMRI, subjects are almost always models as random effects • Stimuli are virtually never modeled as random (but see Donnet, Lavielle, & Poline, 2006; Bedny,Aguirre, &Thompson-Schill, 2007) • Yet the logic is (or should be) exactly the same for stimuli (Coleman, 1964; Clark, 1975)
  16. 16. Westfall, Nichols, &Yarkoni (2017)
  17. 17. Westfall, Nichols, &Yarkoni (2017)
  18. 18. SIMULATION • What happens if we don’t include random stimulus effects? • Assume: • No true difference between sad & neutral pandas (d = 0) • Roughly equal subject and stimulus variances ( ) • We sample 100 subjects and 16 stimuli (8 per condition)
  19. 19. Westfall, Nichols, &Yarkoni (2017)
  20. 20. Judd,Westfall, & Kenny (2012)
  21. 21. Westfall, Nichols, &Yarkoni (2017)
  22. 22. Westfall, Nichols, &Yarkoni (2017) HEY, WHA’ HAPPENED?
  23. 23. Westfall, Nichols, &Yarkoni (2017)
  24. 24. Westfall, Nichols, &Yarkoni (2017)
  25. 25. THE BAD NEWS • This problem is pervasive in the fMRI literature • We estimate that it affects at least 50% of published studies • The magnitude of the problem is often severe • In most of our test datasets, we see effect sizes drop by 50 - 90% • Under realistic assumptions, the FPR in the literature is likely to be high • No support for RSMs in any major analysis packages
  26. 26. THE GOOD NEWS • The behavior is (relatively) predictable • When stimulus sample is large, RSM and standard models converge • Moral of the story: use large stimulus samples! • It’s helpful to present different stimuli to different subjects
  27. 27. Westfall, Nichols, & Yarkoni (2017)
  28. 28. GENERALIZINGTHE GENERALIZATION PROBLEM • Subjects and stimuli are not the only thing we care to generalize over • Same logic applies to many other random factors • Scanner, research site, experimenter, font face, etc. • Most of these factors do not vary in a given experiment • The actual false positive rate has to be even higher • Unless you explicitly specify that you’re not interested in generalizing to new levels of the non-varying factors
  29. 29. INFERENCEVS. PREDICTION • What goes for inference also holds for prediction • Instead of random effects, we can talk about cross- validation schemes • In context of stimuli, we can train and test on different subsets of stimuli • And similarly for other random factors
  30. 30. Woo, Chang, Lindquist, & Wager (2017)
  31. 31. Huth et al. (2016)
  32. 32. WHICH POPULATION? • Modeling a factor as random allows us to generalized to a population • This doesn’t mean it’s the intended population! • In general, we don’t actually know what the population is • E.g., random subject effects allow us to generalize to new subjects from the same population • But those subjects can still be WEIRD! (Henrich et al., 2010)
  33. 33. GENERALIZATION PATTERNS • Given a design factor (e.g., stimuli, subjects, sites, etc.), what does it take to support generalization? • Conceptually similar approaches required for inference and prediction/classification
  34. 34. Generalization target Example Inference Prediction/ classification No population Specific photos of Amy and Ramesh Ignore F Ignore F A population The population of Ekman faces Model F as a random factor Train and test on different subsets of F Intended population The natural distribution of photos of facial expressions Sample widely; include more complex random effects; use post- stratification methods Sample widely; train and test on different sets of F; use sophisticated cross-validation schemes
  35. 35. CONCLUSIONS • Generalization is hard • We almost universally overgeneralize our conclusions in fMRI • This has serious consequences for false positive rates, predictive performance estimates, etc. • There are means of addressing the problem via both modeling (add random effects or cross-validate) and design (sample more levels) • We should probably all make an effort to generalize more slowly
  36. 36. A mathematician, a physicist, and an astronomer were traveling north by train.They had just crossed the border into Scotland, when the astronomer looked out of the window and saw a single black sheep in the middle of a field. “Scottish sheep are black,” he remarked. “No, my friend,” said the physicist.“Some Scottish sheep are black.”At which point the mathematician looked up from his paper and said: “In Scotland, there exists at least one field, in which there exists at least one sheep, at least one side of which is black.”

×