Beyond the EU: DORA and NIS 2 Directive's Global Impact
Sharing, Reproducibility, Replication – AN NIH View
1. Sharing, Reproducibility, Replication – AN NIH View
ACS National Meeting
March 24, 2015
Philip E. Bourne, PhD, FAMCI
Associate Director for Data Science, NIH
Department of Health and Human Services
With Thanks to Larry Tabak 1
2. The Growing Challenge
Noted by research community; in multiple publications
Across research areas
Especially in preclinical research
2
3. Challenges to Ensuring Rigor and Transparency
in Reporting Science: Additional Contributors
Insufficient Reporting
“P-Hacking”
Lack of Consideration of Sex as a Biological Variable
3
4. Challenges to Ensuring Rigor and Transparency
in Reporting Science: Underlying Issues
4
Publish or perish! Grant support
Impact factor Innovation
Significance Novelty:
No negative data
Poor training
Incentives
6. Principles for Addressing the
Underlying Issues
Raise community awareness
Enhance formal training
Protect the quality of funded and published research
by adoption of more systematic review processes
Increase stability for investigators
6
7. Addressing Underlying Issues:
Raise Community Awareness
Workshop in June 2014 with Journal Editors to
identify common opportunity areas
Workshop in July 2014 with PhRMA to identify areas
of common interest with industry
Obtained input on barriers to reproducibility re:
research reagents
Meetings with professional societies and institutions
7
8. Addressing Underlying Issues:
Raise Community Awareness
Over 130 journals endorsed the principles, which were
broadly shared in November 2014 through editorials and
other notifications
8http://nih.gov/about/endorsing-jounals.htm
9. Addressing Underlying Issues:
Trans-NIH Pilots
9
Pilot Focus Types of Efforts Being Developed
Evaluation of scientific premise in
grant applications
New Funding Opportunities with additional
review criteria regarding scientific premise
Checklist and reporting guidelines Reviewer checklists regarding reporting
standards and scientific rigor
Changes to biosketch Biosketch pilot with focus on accomplishments
and not just publications
Approaches to reduce "perverse
incentives” to publish
Exploring award options with a longer period of
support for investigators
Supporting replication studies New Funding Opportunities for replication
studies, and options to assess whether pre-
clinical findings should be replicated
Training Developing materials on experimental design
Other efforts Use of Prize Challenges to encourage
reproducibility of results, PubMed Commons
12. Elements of The Digital Enterprise
Communities Policies
Infrastructure
• Intersection:
• Sustainability
• Efficiency
• Collaboration
• Training
13. Policies: Now & Forthcoming
Data Sharing
Genomic data sharing announced
Data sharing plans on all research awards
Data sharing plan enforcement
Machine readable plan
Repository requirements to include grant
numbers
http://www.nih.gov/news/health/aug2014/od-27.htm
14. Policies - Forthcoming
Data Citation
Goal: legitimize data as a form of scholarship
Process:
Machine readable standard for data citation (done)
Endorsement of data citation for inclusion in NIH bib
sketch, grants, reports, etc.
Example formats for human readable data citations
Slowly work into NLM/NCBI workflow
dbGaP in the cloud (soon!)
17. The Commons: Compute
Platforms
The Commons
Conceptual
Framework
Public
Cloud
Platforms
Super Computing
(HPC) Platforms
Other
Platforms
?
Google, AWS (Amazon)
Microsoft (Azure), IBM,
other?
In house compute
solutions
Private clouds, HPC
– Pharma
– The Broad
– Bionimbus
Traditionally low access
by NIH