This document discusses challenges to rigor and transparency in scientific research reporting. It outlines principles to address underlying issues like publish or perish incentives and lack of training. The NIH is raising community awareness through workshops and endorsements of reporting principles. NIH is also piloting efforts like evaluating scientific premise in grants, developing reporting checklists, and funding replication studies. Specific to data science, the NIH discusses policies around data sharing, citation and genomic data. It presents the concept of a digital commons for computing platforms, digital objects and search/indexing to support data sharing and analysis.

1. Sharing, Reproducibility, Replication – AN NIH View
ACS National Meeting
March 24, 2015
Philip E. Bourne, PhD, FAMCI
Associate Director for Data Science, NIH
Department of Health and Human Services
With Thanks to Larry Tabak 1

2. The Growing Challenge
 Noted by research community; in multiple publications
 Across research areas
 Especially in preclinical research
2

3. Challenges to Ensuring Rigor and Transparency
in Reporting Science: Additional Contributors
 Insufficient Reporting
 “P-Hacking”
 Lack of Consideration of Sex as a Biological Variable
3

4. Challenges to Ensuring Rigor and Transparency
in Reporting Science: Underlying Issues
4
Publish or perish! Grant support
Impact factor Innovation
Significance Novelty:
No negative data
Poor training
Incentives

5. 5

6. Principles for Addressing the
Underlying Issues
 Raise community awareness
 Enhance formal training
 Protect the quality of funded and published research
by adoption of more systematic review processes
 Increase stability for investigators
6

7. Addressing Underlying Issues:
Raise Community Awareness
 Workshop in June 2014 with Journal Editors to
identify common opportunity areas
 Workshop in July 2014 with PhRMA to identify areas
of common interest with industry
 Obtained input on barriers to reproducibility re:
research reagents
 Meetings with professional societies and institutions
7

8. Addressing Underlying Issues:
Raise Community Awareness
 Over 130 journals endorsed the principles, which were
broadly shared in November 2014 through editorials and
other notifications
8http://nih.gov/about/endorsing-jounals.htm

9. Addressing Underlying Issues:
Trans-NIH Pilots
9
Pilot Focus Types of Efforts Being Developed
Evaluation of scientific premise in
grant applications
New Funding Opportunities with additional
review criteria regarding scientific premise
Checklist and reporting guidelines Reviewer checklists regarding reporting
standards and scientific rigor
Changes to biosketch Biosketch pilot with focus on accomplishments
and not just publications
Approaches to reduce "perverse
incentives” to publish
Exploring award options with a longer period of
support for investigators
Supporting replication studies New Funding Opportunities for replication
studies, and options to assess whether pre-
clinical findings should be replicated
Training Developing materials on experimental design
Other efforts Use of Prize Challenges to encourage
reproducibility of results, PubMed Commons

10. http://www.ncbi.nlm.nih.gov/pubmedcommons/
Addressing Underlying Issues:
PubMed Commons
10
System allowing researchers to share opinions on
publications indexed by PubMed

11. Issues Specific to Data Science
11

12. Elements of The Digital Enterprise
Communities Policies
Infrastructure
• Intersection:
• Sustainability
• Efficiency
• Collaboration
• Training

13. Policies: Now & Forthcoming
 Data Sharing
 Genomic data sharing announced
 Data sharing plans on all research awards
 Data sharing plan enforcement
 Machine readable plan
 Repository requirements to include grant
numbers
http://www.nih.gov/news/health/aug2014/od-27.htm

14. Policies - Forthcoming
 Data Citation
 Goal: legitimize data as a form of scholarship
 Process:
 Machine readable standard for data citation (done)
 Endorsement of data citation for inclusion in NIH bib
sketch, grants, reports, etc.
 Example formats for human readable data citations
 Slowly work into NLM/NCBI workflow
 dbGaP in the cloud (soon!)

15. BD2K
Center
BD2K
Center
BD2K
Center
BD2K
Center
BD2K
Center
BD2K
Center
DDICC
Software
Standards
Infrastructure - The
Commons
Labs
Labs
Labs
Labs

16. The Commons
Digital Objects
(with UIDs)
Search
(indexed metadata)
Computing
Platform
TheCommons
Vivien Bonazzi
George Komatsoulis

17. The Commons: Compute
Platforms
The Commons
Conceptual
Framework
Public
Cloud
Platforms
Super Computing
(HPC) Platforms
Other
Platforms
?
 Google, AWS (Amazon)
 Microsoft (Azure), IBM,
other?
 In house compute
solutions
 Private clouds, HPC
– Pharma
– The Broad
– Bionimbus
 Traditionally low access
by NIH

18. The Commons:
Business Model
[George Komatsoulis]

19. NIH…
Turning Discovery Into Health
philip.bourne@nih.gov