This document provides an overview of open science practices to increase rigor and reproducibility in research. It begins with a discussion of current challenges to rigor, including an overemphasis on metrics, not publishing null findings, and a lack of replication. It then outlines several open science practices like pre-registration, open materials and methods, transparent statistics and data visualization, and open data. Benefits of these practices include reducing biases like p-hacking and increasing transparency, replication, and collaboration. Concerns include increased workload and losing proprietary advantages. Overall the presentation aims to promote discussion of adopting open science practices to strengthen the quality of research.

1. Kyle S. Burger RD, MPH, PhD
Open Science Practices to increase rigor
and Reproducibility

2. Acknowledgements
• Brian Brown (@NutriXiv)
• Grace Shearrer, PhD
• Jennifer Sadler
NutriXiv.org
NIBLunc.org

3. Roadmap
1. Scientific reinforcers

4. Roadmap
1. Scientific reinforcers
2. What is being what done now
i. Position statements/accessibility
ii. Open Science Practices
i. Pre-reg/RR
ii. Open methods
iii. Data visualization
iv. Open data
iii. Preprint (NutriXiv)

5. Goals
• Exposure to ‘What can I do’
– beyond a revolution creating a fully-automated scientific utopia of vast
advancement & stress-free, mentally healthful lifestyles (cr: Dr. Grace Shearrer)

6. Goals
• Exposure of ‘What can I do’
– beyond a revolution creating a fully-automated scientific utopia of vast
advancement & stress-free, mentally healthful lifestyles (cr: Dr. Grace Shearrer)
• Provide a brief tour of some open practices
• Elicit discussion (panel):
– Current scientific challenges
– Benefits/ ease your mind about challenges
– Promote aspects of these practices? SSIB/SBOC statement of
support? Position paper? Adopt measures?

7. 1. Metric-based evaluation of scientists
i. Emphasis on journals with high impact factors
ii. Emphasis on number of cites
iii. Emphasis on quantity of publications
Dysfunctional incentive structure for scientific rigor
& discovery

8. 1. Metric-based evaluation of scientists
2. Null findings = file drawer
i. Doomed to repeat the failed
(unknown) experiments
Dysfunctional incentive structure for scientific rigor
& discovery

9. 1. Metric-based evaluation of scientists
2. Null findings = file drawer
3. Innovation vs. Replication
Smaldino & McElreath RSOS 2016
Vinkers et al., Br. Med J 2015
Dysfunctional incentive structure for scientific rigor
& discovery

10. Dysfunctional incentive structure for scientific rigor
& discovery
1. Metric-based evaluation of scientists
2. Null findings = file drawer
3. Innovation vs. Replication

11. 1. Small samples in highly, controlled studies
2. Small effect sizes from large, less controlled studies
3. Increased # of tests on individual topics/reductionism
4. A wide variety of methods in use
5. High financial interests
6. High public interests and press coverage
Loannidis et al., PLOS Med 2005

12. It’s not that bad, right?
Of 49 highly cited phase III RCTs (compared to
subsequent controlled studies):
• 44% (20) replicated
• 16% (7) found better effects
• 16% (7) contradicted the effects
• 24% (11) no attempted replication
Ioannidis, JAMA 2005

13. It’s not that bad, right?

14. It’s not that bad, right?
Vogt et al., PLOS Bio 2016
Begley & Loannidis, Cir Res 2016

15. Impact of registered reports

16. Overarching (prelim) goal of open science
This research provides a meaningful step
toward understanding ingestive behavior.
“Reproducibility”

17. Repeatability:
Same lab,
Same approach
Replicability-ish:
Same lab,
Different approach
Replicability:
Different lab,
Same approach
Reproducibility:
Different lab,
Different approach
Theory validation

18. Repeatability:
Same lab,
Same approach
Confirmation:
Same lab,
Different approach
Replicability:
Different lab,
Same approach
Reproducibility:
Different lab,
Different approach
Theory validation

19. Repeatability:
Same lab,
Same approach
Confirmation:
Same lab,
Different approach
Replicability:
Different lab,
Same approach
Reproducibility:
Different lab,
Different approach
Theory validation

20. Repeatability:
Same lab,
Same approach
Confirmation:
Same lab,
Different approach
Replicability:
Different lab,
Same approach
Reproducibility:
Different lab,
Different approach

21. Repeatability:
Same lab,
Same approach
Confirmation:
Same lab,
Different approach
Replicability:
Different lab,
Same approach
Reproducibility:
Different lab,
Different approach
Theory validation

22. How much repeating is enough? Too much?
Waste of resources,
Cost of advancement
Wasted resources on false findings
Cost of advancement,
Lose public trust
Too little Too much

23. Things are changing, to the exhilaration of some &
apprehension of others.

24. The times are changing: Increases in Open Science
policies

25. The times are changing: efforts
• Effort towards standardization of
– Content of papers/registrations

26. The times are changing: Standardization of content
• Effort towards standardization of
– Content of papers/registrations

27. The times are changing: Standardization of content

28. The times are changing
• Effort towards standardization of:
– Content of papers/registrations
–Constructs
• Example: standardizing impulsivity/self-regulation

29. The times are changing
• Effort towards standardization of:
– Content of papers/registrations
–Constructs
• Example: standardizing impulsivity/self-regulation
–Protocols/Measures/Paradigms/Stimuli/data structure

30. The times are changing
• Effort towards standardization of:
– Content of papers/registrations
–Constructs
• Example: standardizing impulsivity/self-regulation
–Protocols/Measures/Paradigms/Stimuli/data structure

31. Preface to Open Science practices
• You’re going to disagree with some of this
– That’s good
• Evolving process: We can shape our stance our direction in a way
to highlight the quality of ingestive behavior related research

32. Preface to Open Science practices
• You’re going to disagree with some of this
– That’s good
• Evolving process: We can shape our stance our direction in a way
to highlight the quality of ingestive behavior related research
These approaches are not an all or nothing

33. Black box Glass box
1-slide summary of Open Science

34. Making our rigorous science behaviors
visible with open science practices

35. Badges will be our guide

36. Badges will be our guide

37. Making our rigorous science behaviors
visible with open science practices

40. HARKing & PHATing; other bad stuff
• P hacking: running iterations of analyses to find the best p-
value
• (undisclosed, not statistically corrected)

41. HARKing & PHATing; other bad stuff
• P hacking: running iterations of analyses to find the best p-
value
• (undisclosed, not statistically corrected)
• HARKing: Hypothesizing after results are known
• (passing it off as a priori; includes control variables)

42. HARKing & PHATing; other bad stuff
• P hacking: running iterations of analyses to find the best p-
value
• (undisclosed, not statistically corrected)
• HARKing: Hypothesizing after results are known
• (passing it off as a priori; includes control variables)
• PHATing: Post-Hoc Application of Theory
• (passing it off as a priori)

43. What is Pre-Registration
What is it?
• Time-stamped documentation of how you’ll treat &
analyze data prior:
• Collection or Analyses or after?
• On the manuscript level, not grant level

44. What is Pre-Registration
What is it?
• Time-stamped documentation of how you’ll treat &
analyze data prior:
• Collection or Analyses or after?
• On the manuscript level, not grant level
Why?
1. Distinguishes confirming hypotheses from exploring data
2. Reduces unreported “flexibility” in analysis
3. File drawer phenomena
4. Increase evaluation of hypotheses and methods, not
results.

45. Pre-Registration not a prison
• Something went weird? Awesome lead to a new
idea?
• Document it in
the pre-reg.

46. Pre-Registration
• “I already have to do clinicaltrials.gov”
• That can be a chore
• Grant/aims page level
• Purpose is slightly different (file drawer facing)

47. Pre-Registration
• “I already have to do clinicaltrials.gov”
• That can be a chore
• Grant/aims page level
• Purpose is slightly different (file drawer facing)
• Pre-registration can be at various level of detail (anything is
better than nothing)
• Consider the number of papers you write of of 1 grant, this
is this the manuscript level
• Beyond just the aims

48. Pre-registration tips
• See it as doing work in advance, NOT a time suck
• Be precise
– Analysis scripts > statistical models > hypotheses
– Can you write the code? Include it!

49. Pre-registration tips
• See it as doing work in advance, NOT a time suck
• Be precise
– Analysis scripts > statistical models > hypotheses
– Can you write the code? Include it!
• Admit uncertainty
– Analysis/results ‘flow’ projection.
• If ‘this’ then ‘that’ statements esp. with data driven
approaches. Exercise in critical thinking
– Label post hoc as post hoc

50. Pre-registration
Benefits
• Nullifies p-hacking / HARKing/
PHATing
• Identification of barriers before
things start
• Critical thinking exercise,
educational opportunity/part of
instruction
• (ideally) shift focus to
hypothesis/methods from results
• Decreases file draw phenomena

51. Pre-registration
Benefits
• Nullifies p-hacking / HARKing/
PHATing
• Identification of barriers before
things start
• Critical thinking exercise,
educational opportunity/part of
instruction
• (ideally) shift focus to
hypothesis/methods from results
• Decreases file draw phenomena
Concerns
• Feels like more work
• Stuff might change
• Can I get scoped?

52. Making our rigorous science behaviors
visible with open science practices

53. Open Materials/Methods
• Methods presented in publications are frequently
inadequate for replication
• Sharing code, toolboxes, step-by-step protocols, cell lines,
increases replicability & reproducibility

54. Open Materials/Methods
• Methods presented in publications are frequently
inadequate for replication
• Sharing code, toolboxes, step-by-step protocols, cell lines,
increases replicability & reproducibility
• OSF: great for project organization
• esp. for mentor/mentee’s
• OSF is 1 stop shopping!

55. Benefits
• Increases efficiency of science
• Makes replications studies
viable
• Comparable results across
labs (new avenue of study)
• Testing against established
methods
Open Materials/Methods

56. Benefits
• Increases efficiency of science
• Makes replications studies
viable
• Comparable results across
labs (new avenue of study)
• Testing against established
methods
Concerns
• What if someone picks it
apart?
• Can be a little more work
(requires planning)
• It’s ‘my’ code/protocol
– (give away my secrets!?)
Open Materials/Methods

57. • We all do stats
• Statistics are becoming more complex
Unintentional opaque results: Statistics & Data
Visualization

58. • We all do stats
• Statistics are becoming more complex
• You too mechanistic/preclinical scientists:
• Congrats on your new confounding (biological)
variable sex!!!
• Welcome to interactions, and mediation and
moderation models J
Unintentional opaque results: Statistics & Data
Visualization

59. • Stat Packages (good practices)
• Don’t use excel for your stats, like ever...
• Scripting the stat = understanding the model
• Open source stat packages (accessibility)
• Still share code with paywalled programs
(SAS, SPSS, PRISM, STATA)
• Effect sizes are just as/more important as p values
Unintentional opaque results: Statistics & Data
Visualization

60. • Stat Packages (good practices)
• Don’t use excel for your stats, like ever...
• Scripting the stat = understanding the model
• Open source stat packages (accessibility)
• Still share code with paywalled programs
(SAS, SPSS, PRISM, STATA)
• Effect sizes are just as/more important as p values
Unintentional opaque results: Statistics & Data
Visualization

61. Statistics knowledge: 54.26 (16.76)
Attractiveness rating: 47.83 (26.93)
Unintentional opaque results: Statistics & Data
Visualization

62. Statistics knowledge: 54.26 (16.76)
Attractiveness rating: 47.83 (26.93)
r = -0.06
Unintentional opaque results: Statistics & Data
Visualization

63. Statistics knowledge: 54.26 (16.76)
Attractiveness rating: 47.83 (26.93)
r = -0.06
Unintentional opaque results: Statistics & Data
Visualization

64. Boxplots & traditional bar graphs
Unintentional opaque results: Statistics & Data
Visualization

65. Boxplots & traditional bar graphs
Unintentional opaque results: Statistics & Data
Visualization

66. female
male
Unintentional opaque results: Statistics & Data
Visualization
Boxplots & traditional bar graphs

67. female male
Unintentional opaque results: Statistics & Data
Visualization
Boxplots & traditional bar graphs

68. Violin plots
female male
Unintentional opaque results: Statistics & Data
Visualization

69. female
male
Raincloud plots
Unintentional opaque results: Statistics & Data
Visualization

70. Raincloud plots
Unintentional opaque results: Statistics & Data
Visualization
Raincloud plots

71. Making our rigorous science behaviors
visible with open science practices

72. • It’s simply sharing data
• but a little more complicated
Open Data

73. • It’s simply sharing data
• but a little more complicated
• Can be more than an Excel sheet:
• Individual raw data
• Searchable, large databases
• Completely deidentified, aggregate data (format)
• Interactive meta-analytic tools (upload
statistical/outcome outputs)
Open Data

74. FAIR Guiding Principles for Scientific Data
Management & Stewardship
Findable Accessible Interoperable Reusable
1 (meta)data are assigned
a globally unique and
persistent identifier
1 (meta)data are
retrievable by their
identifier using a
standardized
communications protocol
1 (meta)data use a formal,
accessible, shared, and
broadly applicable
language for knowledge
representation
1 meta(data) are richly
described with a plurality
of accurate and relevant
attributes
2 data are described with
rich metadata (defined by
R1 below)
1.1 the protocol is open, free,
and universally
implementable
2 (meta)data use
vocabularies that follow
FAIR principles
1.1 (meta)data are released
with a clear and
accessible data usage
license
3 metadata clearly and
explicitly include the
identifier of the data it
describes
1.2 the protocol allows for an
authentication and
authorization procedure,
where necessary
3 (meta)data include
qualified references to
other
(meta)data
1.2 (meta)data are
associated with detailed
provenance
4 (meta)data are registered
or indexed in a
searchable
Resource
2 metadata are accessible,
even when the data are
no longer available
4 (meta)data can be
exchanged through a
standard format
1.3 (meta)data meet domain-
relevant community
standards

75. Benefits
• Science more time efficient
• Increases rigor via replication, meta-
analyses
• Basis of power analyses
• Ethical considerations
• ~68% increase in cites
• Reduces redundancies in projects
• Science more cost effective
Open Data

76. Benefits
• Science more time efficient
• Increases rigor via replication, meta-
analyses
• Basis of power analyses
• Ethical considerations
• ~68% increase in cites
• Reduces redundancies in projects
• Science more cost effective
Concerns
• Human subjects
• It’s my data, what if I get
scooped?
• Do I get credit?
• Parasites!!!
• Proprietary data sets
• Infrastructure (& noise) for
big data
Open Data

77. Some nifty open science stuff

78. Brief open science tour!
Cr: Dr. Grace Shearrer & Jenny Sadler (PhD est. May 2020)

79. Preprints & Publishing
An ingestive behavior, nutrition
and health preprint service
(Cr: Brian Brown)

80. • Preprints & postprints
• Posting copies of manuscripts
• Currently under review
• Submitted version of a published paper
(permitted; post-embargo)
• Service
• Open access searchable data base (moderated)
• doi (appears in google scholar)
• Links to published article
What is A Preprint Service

81. Why should I share care about preprints:
Grants
• NIH encourages it

82. Why should I share care about preprints:
Grants
• NIH encourages it
“NIH supports the use of preprints & encourages
reviewers to read and evaluate them as typical
research. As study section reviewers, you are capable
of evaluating science independent of peer-review. You
could be the manuscript reviewer/editor anyway.”

83. Why should I share care about preprints:
Access
• Accessibility & Paywalls

84. Why should I share care about preprints:
Access
• Accessibility & Paywalls

85. Why should I share care about preprints:
Access
• Accessibility & Paywalls

86. Why should I share care about preprints:
Citations
• Preprint have persistent DOIs that transfer to the
published article = more rapid & larger impact!

87. Why NutriXiv?
• Why not Pubmed for postprints?
– Yup, why not Pubmed?
– Recently allows for supplemental material
• If you get into pre-registration, project management,
data/code sharing, you can centralize the these with the
preprint at Open Science Framework.
– ‘Redundant’ manuscripts are fine

88. Why NutriXiv?
• What about BioRxiv?
– Highly generalized and submission get lost in the throughput
– We have specialized journals that are our ‘go-to’ aimed for the
audience that is mostly likely to benefit (AND indexed on
google scholar etc... i.e., still a wide distribution)
– Stuff is going on at BioRxiv

89. Why NutriXiv?
• Together we can shape the what and how about SSIB-
related preprints. What do you want out of a preprint?
– Drive the innovation of our discipline
• Our team will promote your work!

90. Why NutriXiv?
• Together we can shape the what and how about SSIB-
related preprints. What do you want out of a preprint?
– Drive the innovation of our discipline
• Our team will promote your work!
• Our future
– On your 3rd submission we’ll send you some
cool stickers!!
– On your 10th submission we’ll send you a
free t-shirt!!
– email distribution list?

91. Preprints
Benefits
• Increased rate of
dissemination
• Increased accessibility
• Increased exposure to the
science
• Increased citations

92. Benefits
• Increased rate of
dissemination
• Increased accessibility
• Increased exposure to the
science
• Increased citations
Concerns
• Copyright concerns
• What if my paper changes
during peer review?
• ‘Misinformation’ to the
public?
• Is this were I drop papers I
can’t get published?
Preprints

93. NutriXiv Demo!
• Increase access and the rate that your science is read
to you public with preprint services
Nutrixiv.org
@NutriXiv

97. The next level: Registered
reports

98. The next level: Registered reports

99. The next level: Registered reports

100. The next level: Registered reports

101. Impact of registered reports

102. It’s not that bad, right?