Reusable Science: How not to slip from the shoulders of giants


Published on

No research is done in a void: science is constantly expanding previous hypotheses, building upon past knowledge. We live in a digital age where information is ubiquitous, yet we struggle to preserve accurate machine readable and quantitative descriptions of our research compromising our capacity to use them in our inferences. In the following talk I will show how and why we incorporate assumptions in our studies based on three experiments we have conducted: (i) dissociating metacognitive subdomains in medial and lateral anterior prefrontal cortex, (ii) relating reading comprehension to individual differences in the default mode network, and (iii) exploring neural correlates of the content and form of self-generated thoughts. This will be followed by introducing a new inference method - probabilistic Regions of Interest (pROI) - which allows the use of prior knowledge in the form of a probabilistic map. This approach provides the middle ground between ROI and full brain analysis, by giving researchers more flexibility in formalizing priors. The quality of prior probability maps based on the literature can be improved by using unthresholded statistical maps instead of peak coordinates. To facilitate this we have created - a community - wide effort to collect unthresholded statistical maps. Taking the initiative a step further I will describe the concept of data papers - publications purely dedicated to datasets. Together those three mechanisms (pROI, and data papers) are a small but significant steps towards better, more reusable and reproducible science.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Stu
  • Distinct systemssystems
  • Anterior precuneus, Right inferior parietal cortex
  • DMN related to SGTs
  • Addandrewshanna
  • Positive (right insular cortex, right frontal operculum) vs. negative (striatum) correlations;
  • PCC hub
  • Add labels spell out
  • Mind wandering is related to brain regions that are part of brain networks other than default mode network.
  • Answer the question directly
  • It was a picture of a boa constrictor digesting an elephant
  • Think how much money and effort goes into one study100,000 USD to produce one paper:6-12 pages of authors interpretation of acquired data…without the data itselfBy not reporting subthreshold effects we are wasting (taxpayers) money!
  • Data sharing is like flossing – everyone knows is good, but no one does it.
  • Reusable Science: How not to slip from the shoulders of giants

    1. 1. Reusable Science: How not to slip from the shoulders of giants Chris Gorgolewski Max Planck Research Group: Neuroanatomy & Connectivity
    2. 2. Anatomy of a giant I. Example studies II. Probabilistic ROIs III.Sharing statistical maps IV.Data papers
    3. 3. Anatomy of a giant I. Example studies II. Probabilistic ROIs III.Sharing statistical maps IV.Data papers
    4. 4. Study I Medial and Lateral Networks in Anterior Prefrontal Cortex Support Metacognitive Ability for Memory and Perception Benjamin Baird, Jonathan Smallwood, Krzysztof J. Gorgolewski, and Daniel S. Margulies Journal of Neuroscience (in press)
    5. 5. Meta-cognition • Are we equally good in judging our performance of memory or perception tasks? • Is metacognition related to medial or lateral prefrontal cortex? Does it depend on modality?
    6. 6. Measuring metacognition
    7. 7. Metacognition of memory and perception are distinct systems
    8. 8. Sources of seed points Gilbert et al. 2006, Functional specialization within rostral prefrontal cortex (area 10): a meta-analysis. Journal of cognitive neuroscience
    9. 9. Sources of seed points Fleming, S. M., Weil, R. S., Nagy, Z., Dolan, R. J., & Rees, G. (2010). Relating introspective accuracy to individual differences in brain structure. Science (New York, N.Y.), 329(5998), 1541–3.
    10. 10. Meta-cognition of Perception vs. Meta-cognition of Memory Baird, Smallwood et al. (in press) JON
    11. 11. Double dissociation of metacognitive abilities
    12. 12. Study II The Default Modes of Reading: Modulation of posterior cingulate and medial prefrontal cortex connectivity associated with subjective and objective differences in reading experience Jonathan Smallwood, Krzysztof J. Gorgolewski, Johannes Golchert, Florence J.M. Ruby, Haakon G. Engen, Benjamin Baird, Melaina Vinski, Jonathan Schooler, Daniel S. Margulies Frontiers in Neuroscience (in press)
    13. 13. Reading comprehension • What is the relation between task focus and reading comprehension? • What role does Default Mode Network play in reading comprehension and task focus?
    14. 14. Task focus is inversely correlated with reading comprehension
    15. 15. Reading by Default Seed locations Andrews-hanna, J. R., Reidler, J. S., Sepulcre, J., Poulin, R., Buckner, R. L., & Temp, P. (2010). Functional-anatomic fractionation of the brain’s default network. Neuron, 65(4), 550–62. Smallwood, et al., Frontiers in Human Neuroscience
    16. 16. Reading by Default Smallwood, et al., Frontiers in Human Neuroscience
    17. 17. Reading by Default Smallwood, et al., Frontiers in Human Neuroscience
    18. 18. Why mind-wandering may disrupt reading
    19. 19. Study III A correspondence between the brain's intrinsic functional architecture and the content and form of self-generated thoughts Krzysztof J. Gorgolewski, Dan Lurie, Sebastian Urchs, Judy A. Kipping, R. Cameron Craddock, Michael P. Milham, Daniel S. Margulies, and Jonathan Smallwood PLoS One (submitted)
    20. 20. Mind wandering • What is the content and form of thoughts in mind wandering? • How does it relate to various aspects of intrinsic BOLD activity?
    21. 21. Questions about the content
    22. 22. Questions about the form
    23. 23. Future vs. Past = Words vs. Images
    24. 24. Resting state measures Group average Fractional Amplitude of Low Frequency Fluctuations Regional Homogeneity Degree Centrality
    25. 25. Not only Default Mode Network
    26. 26. Not only Default Mode Network
    27. 27. What these studies have in common?
    28. 28. Anatomy of a giant I. Example studies II. Probabilistic ROIs III.Sharing statistical maps IV.Data papers
    29. 29. Signal to Noise ratio
    30. 30. Looking in the wrong places
    31. 31. Lower SNR = we miss more stuff
    32. 32. Lower SNR = higher FDR threshold
    33. 33. How to improve power? • stronger effects? • fewer null/noise samples -> ROI
    34. 34. What is wrong with ROI analysis?
    35. 35. What is wrong with ROI analysis?
    36. 36. Binary nature of masks
    37. 37. Fifty Shades of Grey, Matter Fifty shades of grey A probabilistic view on the ROI analysis A probabilistic approach to ROI analysis Gorgolewski et al. PRNI 2013
    38. 38. Extensions and disclaimers • Kernel density estimation • Markov Random Field reguralization • Posterior maps cannot be used in meta analysis – circularity! • Prior maps are integral part of the analysis and need to be included in publications
    39. 39. Anatomy of a giant I. Example studies II. Probabilistic ROIs III.Sharing statistical maps IV.Data papers
    40. 40. Just coordinates? • Databases such as Neurosynth or BrainMap rely on peak coordinates reported in papers (only strong effects)
    41. 41. Are we throwing money away?
    42. 42. Data sharing?
    43. 43. Data sharing? • Ok, ok so we should share data. • We all know it’s good. • But almost no one does it. – You have to prepare data – You risk that your mistakes will be found!
    44. 44. “I swear I’ve heard it before” • In the past there were many attempts to propagate data sharing – For example fMRI DC: • Failed because of technical issues • …and the amount of time it took to prepare data for submission (a week, a very frustrating week) • fMRI DC was however too ambitious for its time: – They wanted to collect raw data and all metadata required to reproduce the analysis Van Horn & Gazzaniga (2013). Why share data? Lessons learned from the fMRIDC. NeuroImage
    45. 45. Baby steps • Everything is a question of cost and benefit – If we keep the cost low even small benefit (or just conviction that data sharing is GOOD) will suffice
    46. 46. simple data sharing • Minimize the cost! • We just want your statistical maps with minimum description (DOI) – If you want you can put more metadata, but you don’t have to • We streamline login process (external services such as Google, Facebook)
    47. 47. Benefits? • In return authors get interactive web based visualization of their statistical maps – Something they can embed on their lab website • We are keeping both cost and benefit low… – …but we also plan to work with journal editors to popularize the idea
    48. 48. Share your stat maps! ? Make science more reproducibl
    49. 49.
    50. 50. Anatomy of a giant I. Example studies II. Probabilistic ROIs III.Sharing statistical maps IV.Data papers
    51. 51. Motivation • Share your stat maps! vs . institutions scientists
    52. 52. Quality control • Share your stat maps! Complex datasets require elaborate descriptions
    53. 53. Solution – data papers • Authors get recognizable credit for their work. – Even smaller contributors such as RAs can be included. • Acquisition methods are described in detail. • Quality of metadata is being controlled by peer review.
    54. 54. Where to publish data papers? • Neuroinformatics (Springer) • Frontiers in Human Brain Methods (Nature Publishing (Frontiers Media) Group) • GigaScience (BGI, BioMed Central) • Scientific Data (Nature Publising Group, coming soon)
    55. 55. Read more • Probabilistic ROIS Gorgolewski et al. PRNI, 2013 • Gorgolewski et al. OHBM, 2013 • Data papers Gorgolewski et al. Frontiers in Brain Imaging Methods, 2012
    56. 56. Acknowledgements (my personal giants) Pierre-Louie Bazin Haakon Engen Satrajit Ghosh Russell A. Poldrack Jean-Baptiste Poline Yannick Schwarz Tal Yarkoni Michael Milham Daniel Margulies Benjamin Baird Jonathan Smallwood Johannes Golchert Florence J.M. Ruby Melaina Vinski Jonathan Schooler Dan Lurie Sebastian Urchs Judy A. Kipping R. Cameron Craddock MPI CBS Resting state group
    57. 57. THANK YOU!
    58. 58. Details
    59. 59. Details
    60. 60. Details