Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.

Slideshare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.

Successfully reported this slideshow.

Like this presentation? Why not share!

1,877 views

Published on

Slides for keynote lecture at OHBM 2017.

Published in:
Science

No Downloads

Total views

1,877

On SlideShare

0

From Embeds

0

Number of Embeds

380

Shares

0

Downloads

29

Comments

0

Likes

2

No embeds

No notes for slide

- 1. GENERALIZABILITY IN FMRI, FAST AND SLOW TalYarkoni
- 2. GENERALIZATION MATTERS • Induction (reasoning from the speciﬁc to the general) is central to the scientiﬁc method • Most ﬁndings in neuroimaging are only interesting because we expect them to generalize broadly
- 3. happy pandas
- 4. sad pandas
- 5. neutral pandas
- 6. LOGIC OF SUBTRACTION - Experimental manipulation Emotion-related neural changes =
- 7. BRAINS ON PANDAS
- 8. WHAT SHOULD WE CONCLUDE? • We studied a small number of subjects responding to a small number of panda images at one research site with one experimenter • What universe of observations are we are entitled to generalize these conclusions to? • Can we just choose our generalization strategy? • Consider two different approaches: fast vs. slow
- 9. FAST GENERALIZATION • Dream big! • Paper title:“Increased amygdala activation in response to affective mammalian expressions relative to neutral mammalian expressions” • Seems pretty interesting! • But how many inductive leaps did we just take?
- 10. SLOW GENERALIZATION • Careful now… • Paper title:“Greater amygdala activation in 22 UT- Austin undergraduates when viewing images angsty_tree_panda.jpg and panda_delight.png than image panda_goes_meh.jpg” • Boooooooooring • Which of these titles do you prefer? Fast or slow?
- 11. NOT A MATTER OF PREFERENCE • We typically prefer our conclusions to hold for stimuli like the ones we sampled, not just those exact ones • Unfortunately, our intentions and preferences have very little to do with it • The statistical model must support the desired inference • If we want to generalize to new stimuli, our model must account for the variance introduced by the stimulus sampling process
- 12. Tom NicholsJake Westfall THANKSTO…
- 13. RANDOMVS. FIXED EFFECTS • Many deﬁnitions in the literature (Gelman, 2005) • For our purposes: effects are ﬁxed if the observed levels are of intrinsic interest, and random if they are only interesting as exemplars of an underlying population • E.g., subjects are almost always treated as a random factor • We rare don’t care about these particular subjects • In an N-back task, load is considered ﬁxed • We don’t sample 2-back and 3-back from a population of N’s
- 14. WHY BOTHER WITH RANDOM EFFECTS? • Suppose we have 100 trials for each of 20 subjects —50 per experimental condition • The appropriate one-sample t-test is conducted at the level of subjects, not trials.Why? • Answer: because we intend to generalize to new subjects, and trials are correlated within subjects
- 15. WHY STOP AT SUBJECTS? • In fMRI, subjects are almost always models as random effects • Stimuli are virtually never modeled as random (but see Donnet, Lavielle, & Poline, 2006; Bedny,Aguirre, &Thompson-Schill, 2007) • Yet the logic is (or should be) exactly the same for stimuli (Coleman, 1964; Clark, 1975)
- 16. Westfall, Nichols, &Yarkoni (2017)
- 17. Westfall, Nichols, &Yarkoni (2017)
- 18. SIMULATION • What happens if we don’t include random stimulus effects? • Assume: • No true difference between sad & neutral pandas (d = 0) • Roughly equal subject and stimulus variances ( ) • We sample 100 subjects and 16 stimuli (8 per condition)
- 19. Westfall, Nichols, &Yarkoni (2017)
- 20. Judd,Westfall, & Kenny (2012)
- 21. Westfall, Nichols, &Yarkoni (2017)
- 22. Westfall, Nichols, &Yarkoni (2017) HEY, WHA’ HAPPENED?
- 23. Westfall, Nichols, &Yarkoni (2017)
- 24. Westfall, Nichols, &Yarkoni (2017)
- 25. THE BAD NEWS • This problem is pervasive in the fMRI literature • We estimate that it affects at least 50% of published studies • The magnitude of the problem is often severe • In most of our test datasets, we see effect sizes drop by 50 - 90% • Under realistic assumptions, the FPR in the literature is likely to be high • No support for RSMs in any major analysis packages
- 26. THE GOOD NEWS • The behavior is (relatively) predictable • When stimulus sample is large, RSM and standard models converge • Moral of the story: use large stimulus samples! • It’s helpful to present different stimuli to different subjects
- 27. Westfall, Nichols, & Yarkoni (2017)
- 28. GENERALIZINGTHE GENERALIZATION PROBLEM • Subjects and stimuli are not the only thing we care to generalize over • Same logic applies to many other random factors • Scanner, research site, experimenter, font face, etc. • Most of these factors do not vary in a given experiment • The actual false positive rate has to be even higher • Unless you explicitly specify that you’re not interested in generalizing to new levels of the non-varying factors
- 29. INFERENCEVS. PREDICTION • What goes for inference also holds for prediction • Instead of random effects, we can talk about cross- validation schemes • In context of stimuli, we can train and test on different subsets of stimuli • And similarly for other random factors
- 30. Woo, Chang, Lindquist, & Wager (2017)
- 31. Huth et al. (2016)
- 32. WHICH POPULATION? • Modeling a factor as random allows us to generalized to a population • This doesn’t mean it’s the intended population! • In general, we don’t actually know what the population is • E.g., random subject effects allow us to generalize to new subjects from the same population • But those subjects can still be WEIRD! (Henrich et al., 2010)
- 33. GENERALIZATION PATTERNS • Given a design factor (e.g., stimuli, subjects, sites, etc.), what does it take to support generalization? • Conceptually similar approaches required for inference and prediction/classiﬁcation
- 34. Generalization target Example Inference Prediction/ classiﬁcation No population Speciﬁc photos of Amy and Ramesh Ignore F Ignore F A population The population of Ekman faces Model F as a random factor Train and test on different subsets of F Intended population The natural distribution of photos of facial expressions Sample widely; include more complex random effects; use post- stratiﬁcation methods Sample widely; train and test on different sets of F; use sophisticated cross-validation schemes
- 35. CONCLUSIONS • Generalization is hard • We almost universally overgeneralize our conclusions in fMRI • This has serious consequences for false positive rates, predictive performance estimates, etc. • There are means of addressing the problem via both modeling (add random effects or cross-validate) and design (sample more levels) • We should probably all make an effort to generalize more slowly
- 36. A mathematician, a physicist, and an astronomer were traveling north by train.They had just crossed the border into Scotland, when the astronomer looked out of the window and saw a single black sheep in the middle of a ﬁeld. “Scottish sheep are black,” he remarked. “No, my friend,” said the physicist.“Some Scottish sheep are black.”At which point the mathematician looked up from his paper and said: “In Scotland, there exists at least one ﬁeld, in which there exists at least one sheep, at least one side of which is black.”

No public clipboards found for this slide

Be the first to comment