Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

OSFair2017 Training | Increasing Research Transparency using the Open Science Framework

187 views

Published on

Jennifer Freeman Smith talks about increasing research transparency using the Open Science Framework | OSFair2017 Workshop

Workshop title: Increasing Research Transparency using the Open Science Framework

Workshop overview:
Part of the challenge with making research more open and transparent is purely logistical. Where and how can the research be stored, organized, and shared most effectively when there are so many different tools, processes and policies in place? The OSF provides an open source, structured environment where researchers from all over the world, using their own tools and processes, can collaborate openly, transparently, and effectively.

DAY 3 PARALLEL SESSION 8

Published in: Science
  • Hello! I do no use writing service very often, only when I really have problems. But this one, I like best of all. The team of writers operates very quickly. It's called ⇒ www.WritePaper.info ⇐ Hope this helps!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! I can recommend a site that has helped me. It's called HelpWriting.net They helped me for writing my quality research paper on diabetes, and of course by keeping my all other needs fulfilled.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

OSFair2017 Training | Increasing Research Transparency using the Open Science Framework

  1. 1. cos.io | osf.io Jennifer Freeman Smith, PhD Transparency and Openness Training Coordinator Increasing Research Transparency Using the Open Science Framework
  2. 2. Improving Openness, Integrity, and Reproducibility of Scientific Research MISSION
  3. 3. Infrastructure Metascience Community
  4. 4. Everything we do is free and open. COS Strategic Plan: https://osf.io/x2w9h
  5. 5. More at: http://cos.io/about/our-sponsors
  6. 6. How does transparency improve science?
  7. 7. SOURCES OF ISSUES IN REPRODUCIBILITY • Methodological, statistical, and reporting practices • Structural and organizational practices • Rarely, intentional scientific misconduct
  8. 8. WHAT IS REPRODUCIBILITY? Computation Reproducibility: If we took your data and code/analysis scripts and reran it, we can reproduce the numbers/graphs in your paper Methods Reproducibility: We have enough information to rerun the experiment or survey the way it was originally conducted Results Reproducibility/Replicability: We use your exact methods and analyses, but collect new data, and we get the same statistical conclusion
  9. 9. If we seek to facilitate reproducibility, replicability, extension, and reuse…
  10. 10. We need to move beyond description of outcomes to description of process or, better, sharing actual process.
  11. 11. Open Access Open Data Open Materials Open Data Cleaning Scripts Open … Open Workflow
  12. 12. OPEN WORKFLOW • Increases process transparency • Increases accountability • Facilitates reproducibility • Facilitates metascience • Fosters collaboration • Fosters inclusivity • Fosters innovation • Protects against lock-in: Open + Accessible
  13. 13. It takes some effort to organize your research to be reproducible…the principal beneficiary is generally the author herself. Jon Claerbout Making Scientific Contributions Reproducible http://sepwww.stanford.edu/oldsep/matt/join/redoc/web/iris.html “ ”
  14. 14. Write Report Publish Report Search/ Discovery Develop Idea Design Study Collect Data Store Data Analyze Data OS F
  15. 15. OPEN SCIENCE FRAMEWORK http://osf.io Free, open source scientific commons
  16. 16. Collaboration, Documentation, Archiving, Sharing
  17. 17. Put data, materials, and code on the OSF
  18. 18. Automatic file versioning
  19. 19. Now Publish Report Search / Discovery Develop Idea Design Study Collect Data Store Data Analyze Data Write Report
  20. 20. Future Publish Report Search / Discovery Develop Idea Design Study Collect Data Store Data Analyze Data Write Report
  21. 21. Why open your workflow?
  22. 22. WHY OPEN YOUR WORKFLOW? • Improve reproducibility and replicability • Increased efficiency • Increases reuse and extension of knowledge • Public data can be combined with private data • Can influence scientists, entrepreneurs, policymakers, citizens
  23. 23. OPEN DATA CHALLENGES • How do we make our data accessible, understandable, reusable? • Which repository should I choose? • Who owns the data? Do I have a copyright on the raw data I collected? • If I reuse data from someone else, do I have to offer them co-authorship? • How should privacy issues be addressed?
  24. 24. SHARING IS A CONTINUUM • Data underlying just results reported in a paper • Data underlying publication + information about other variables collected • Data underlying publication + embargo on full dataset • All data collected for that study
  25. 25. Persistent citable identifiers
  26. 26. GUIDs make sharing simple Arnold BF, van der Laan MJ, Hubbard AE, Steel C, Kubofcik J, Hamlin KL, et al. (2017) https://doi.org/10.1371/journal.pntd.0005616
  27. 27. File downloads Forks See the Impact
  28. 28. NEXT STEPS 1.Build a test project on the OSF 2.Document from the beginning - or even right now 3.Talk to your collaborators – What is our data management plan? – What/when will we share?
  29. 29. Jennifer Freeman Smith, PhD Center for Open Science Charlottesville, VA, USA jennifer@cos.io @jfsmith434 Find this presentation at https://osf.io/ncdpa/ QUESTIONS AND COMMENTS
  30. 30. SUPPLEMENTAL SLIDES
  31. 31. The Research Lifecycle
  32. 32. POSITIVE RESULTS BY DISCIPLINE Fanelli D (2010) “Positive” Results Increase Down the Hierarchy of the Sciences. PLOS ONE 5(4): e10068. doi:10.1371/journal.pone.001 0068 http://journals.plos.org/ploso ne/article?id=10.1371/journal .pone.0010068
  33. 33. RESEARCHER DEGREES OF FREEDOM All data processing and analytical choices made after seeing and interacting with your data • Should I collect more data? • Which observations should I exclude? • Which conditions should I compare? • What should be my main DV? • Should I look for an interaction effect?
  34. 34. FALSE POSITIVE INFLATION Simmons, Nelson, & Simonsohn (2011)
  35. 35. EXPLORATORY VS. CONFIRMATORY ANALYSES Exploratory • Interested in exploring possible patterns/relationships in data to develop hypotheses Confirmatory • Have a specific hypothesis you want to test Preregistered analysis plans clarify which results are exploratory and which are confirmatory
  36. 36. Other ways the OSF supports open workflows
  37. 37. Preprints decouple publication and evaluation, allowing for the rapid dissemination of content.
  38. 38. OSF Preprints
  39. 39. Free, open (meta)dataset of research activity across the research lifecycle ~40M records from ~163 sources http://share.osf.io
  40. 40. PRE-REGISTRATION Documenting your research plan in a read-only public repository before you conduct the study. Pre-registration helps reduce the “file drawer effect” by increasing discoverability of unpublished studies.
  41. 41. PRE-REGISTRATION Benefits of pre-registering your study depend on how much information you include. At a minimum a preregistration should include the “what” of the study: • Research question • Population and sample size • General design • Variables you’ll be collecting, or dataset you’ll be using
  42. 42. PRE-ANALYSIS PLAN Details the analyses planned for hypothesis testing: Sample size Data processing and cleaning procedures Exclusion criteria Statistical analyses Including a pre-analysis plan in your pre-registration helps improve study accuracy and replicability by guarding against unintended false positive inflation.
  43. 43. $1,000,000 PREREGISTRATION CHALLENGE
  44. 44. $1,000,000 PREREGISTRATION CHALLENGE https://cos.io/prereg

×