Talk given at the National Science Foundation on issues related to replication and data sharing in the psychological sciences (February 22, 2014)

1. Replication & Data Sharing:
The Publication Perspective
Bradford W. Hesse, PhD
Chair-Elect,APA Publications & Communications Board
American Psychological
Association
Friday, February 14, 14

2. APA Monitor: Feb 2013*
Complex Problem
• Bias toward positive findings
➡ “P-Hacking,” with false
positives in literature
➡ The “file drawer” problem,
with false negatives
obscured
• Insufficient detail for method
replication
• Lack of incentives to replicate
• Disincentives for data sharing
* Winerman, Lea. (2013). Interesting results: Can they be replicated? Monitor on Psychology, 44, 38-41.
Friday, February 14, 14

3. Task Force on Data Sharing*
Necessary for
verification
* Data sharing memo,April 4, 2012
Restriction in purpose: No reason for
sharing other than “verification”
Burdensome, costs to be borne
by replicating “professional”
Friday, February 14, 14

4. Trends Toward Open Science*
* e.g., Hesse, B.W., Croyle, R.T., & Buetow, K. H. (2011). Cyberinfrastructure and the biomedical
sciences.Am J Prev Med, 40(5 Suppl 2), S97-102.
•Computer infrastructure
•NSF, NIH requirements
•Open Science mandates
•Emergence of “Big” problems
•Emergence of “Big Data”
•Team Science to solve problems
Friday, February 14, 14

5. 2 Sides of Same Coin*
Data Sharing
• Enables replication
• Promotes aggregation for
knowledge synthesis,
hypothesis generation,
programmatic decisions,
and generalizability testing
• Opens data for analysis
with more powerful
analytic techniques than
possible originally
• Encourages a culture of
openness in science
Replication
• Technical replication
➡ Method validation
➡ Verification
• Conceptual replication
➡ Theory building
➡ Generalizability
➡ Model testing
• Culture change
➡ Self-correcting
➡ Science as public trust
* Replication and Data Sharing (RADS) Task Force, October 2013
Friday, February 14, 14

6. * Replication and Data Sharing (RADS) Task Force, October 2013
Proposed Policy Revision*
Friday, February 14, 14

7. Consider:Article of the Future*
* http://www.articleofthefuture.com/about
Direct Access to
Data
Interactive
Content
Links to Author
community (e.g., ORCID)
Ability to publish
ePub only material
Friday, February 14, 14

8. APA’s Cyber-infrastructure*
• Credible and ubiquitous portal to psychological sciences
• Expanded scope promotes team science, with bidirectional links
between data services (e.g., PubMed & PsycINFO)
• Direct access to bibliographic tools (e.g., *.pdf’s, cited
references, reference software exportation) elevates scientific
productivity, reduces friction to good scholarship
* Advised by Electronic Resources Advisory Committee (ERAC)
Friday, February 14, 14

9. Replication ePubs in all Journals*
• Subject to same, rigorous review as print pubs
• Not considered against impact factor
• May be highlighted by editor as important contribution to field
• May be open to online comments at some point in future
* Proposal by Council of Editors
online only
Friday, February 14, 14

10. APA’s Open Science
• Free and open to the public
• Standard, finely nuanced
descriptions of study’s
rationale, method, results, and
interpretation
• Data collaboration
➡ data underlying analysis
available
➡ data generators retain
authorship rights
• Technical + Public abstracts
• Open comment discussions
Archives of Scientific Psychology*
* Courtesy Harris Cooper (Duke University) and GaryVandenBos (APA)
Friday, February 14, 14

11. Ten Prescriptions*
Improve the transparency of research
1. Require research reports to include complete
accounts of method, analyses and results
2. Require data sharing
3. Clarify rights and responsibilities of parties to data-
sharing agreements
4. Construct a prospective database of IRB-
approved research
*Cooper, Harris & VandenBos, Gary. (2013). Ten Prescriptions To Foster Better Social Science in
the Internet Environment. Final Report on NIH Grant. Duke University. Durham, NC.
Friday, February 14, 14

12. Change the reward structure of research
5. Value replication
6. Train researchers in record keeping and data
management
7. Use approaches to evaluations of research (and
researchers) that promote transparent science
Ten Prescriptions*
*Cooper, Harris & VandenBos, Gary. (2013). Ten Prescriptions To Foster Better Social Science in
the Internet Environment. Final Report on NIH Grant. Duke University. Durham, NC.
Friday, February 14, 14

13. Improve interaction with the global
audience
8. Take responsibility for open access publication
costs
9. Provide the means for interaction between
researchers and an audience of non-researchers
10.Make special efforts to include members of the
international community in training
Ten Prescriptions*
*Cooper, Harris & VandenBos, Gary. (2013). Ten Prescriptions To Foster Better Social Science in
the Internet Environment. Final Report on NIH Grant. Duke University. Durham, NC.
Friday, February 14, 14

14. Task Force Recommendations
Coordinate with other stakeholders in psychological science
Work with APA’s Board of Scientific Affairs to engage Office of
Human Subjects Protection & State IRB’s
Draft publishable document for Replication and Data Sharing
Guidelines (similar to Journal Article Reporting Standards)
Friday, February 14, 14

15. Reinventing
Discovery
The real question is: How do we
use the methods and discoveries
of the psychological sciences to
solve problems of replicability,
generativity, & integration across
all sciences?
Friday, February 14, 14