Meta-research and rethinking peer review at eLife. Peter Rodgers, Features Editor, eLife.

1. Meta-research
and rethinking
peer review at eLife
New Frontiers for
Research on Research
September 20, 2019
Peter Rodgers
Features Editor, eLife

2. Peer review at eLife
• ~30% of submissions sent for full peer review
• All reviewers see all reviews (and identities) and the
Reviewing Editor leads a discussion
– i) Ask for revision (~60%): list of essential revisions sent to
author (ie, not full reviews). Only ask for extra work if it is
needed to support main conclusions and can be done in
about two months
– ii) Reject (~40%)
• Reviewing Editor usually makes decision on revision
• Publish decision letter and author response
• Aim is to be fast, fair and transparent

3. Study 1: Author-reviewer homophily in
peer review
• Murray, Siler, Larivière, Chan, Collings, Raymond,
Sugimoto. 2019. bioRxiv: 400515
• 28876 initial submissions and 7192 full submissions
• Outcomes were more favourable for male authors
– Acceptance rate for male corresponding authors = 15.6%
– Acceptance rate for female corresponding authors = 13.8%
– Similar bias seen for last authors but no difference between male
and female first authors

4. Study 1: Author-reviewer homophily in
peer review
• Outcomes were more favourable for authors affiliated with
institutions in North America and Europe
• Both groups were over-represented among gatekeepers.
• Outcomes were influenced by homophily
– When all reviewers were male, accept rate was 55.9% for male
last authors and 51.1% for female last authors
– For mixed-gender review teams the disparity was smaller and not
significant
– Last author-reviewer homogeny had accept rate of 57.4%; non-
homogeny had accept rate of 47.4%

5. Study 2: Results from a peer-review trial
• For papers sent for peer review as part of trial, authors
could decide how they responded to the reviews
• Published paper would include assessment from Editor
• 313 papers in trial; 612 regular papers; not randomized
• Data still being analyzed; initial findings reported in two
blog posts on: elifesciences.org/inside-elife
• Encourage rate = 22.4% (cf 26.4% for regular papers)
• Results suggest different decision outcomes based on
career stage.

6. What are we doing based on the results
of these studies?
• Working to improve the diversity of our editorial boards
– BRE is 68% male/32% female
– Aiming for 60%/40% by end of 2019 and 50/50 in the long term
– Aiming for better geographical/career-stage diversity
• Collecting data on career stage
• Explore blinding in the early stages of the process
• Explore if eLife review process is more or less biased
because of the consultation between reviewers
• Publish-Review-Curate
• Work with bioRxiv, ASAPbio, Center for Open Science
(RP:CB)

7. 7

8. Meta-research papers in eLife
• Centralized scientific communities are less likely to generate
replicable results (Danchev et al. 2019. eLife 8:e43094)
• Gender variations in citation distributions in medicine are very small
and due to self-citation and journal prestige (Andersen et al. 2019.
eLife 8:e45374)
• Gender bias in scholarly peer review (Helmer et al. 2017. eLife
6:e21718)
• Gender inequalities among authors who contributed equally
(Broderick & Casadevall. 2019. eLife 8:e36399)
• The readability of scientific texts is decreasing over time (Plaven-
Sigray et al. 2017. eLife 6:e27725)
• Use of the Journal Impact Factor in academic review, promotion, and
tenure evaluations (McKiernan et al. 2019. eLife 8:e47338)

9. Thank you
Email: p.rodgers@elifesciences.org

10. DISTRIBUTION OF GATEKEEPERS AND AUTHORS BY GENDER
(TOP) AND GEOGRAPHY. FIG. 3 FROM MURRAY ET AL.

11. ACCEPTANCE RATES FOR FULL SUBMISSIONS BY GENDER (LEFT) AND
FATE OF SUBMISSIONS BY GEOGRAPHY. FIG. 4 FROM MURRAY ET AL

12. 12
AN INTERESTING RESULT FROM ACTON ET AL.

13. SPARE
SLIDES!!

14. Outline
• Peer review at eLife
• Study 1: Author-reviewer homophily in peer review
• Study 2: Results from a peer-review trial
• What next?
• Meta-research papers in eLife

15. About eLife
● 700+ submissions per month
● 120+ publications per month
● Mike Eisen = EIC
● 60 Senior Editors
● 430+ Reviewing Editors

16. Editorial process
Full
submission
Peer review
Decision
after peer
review
Revision
assessed by
BRE
Assign to Reviewing Editor, who serves as a
reviewer and contacts other reviewers
Consultation amongst reviewers
Essential revisions listed
Limit rounds of revision