Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
October 2015
Peer review in 2015
A global view
A white paper from Taylor & Francis
The UKSG webinar
Elaine Devine Senior C...
Peer review in 2015 | A global view (October 2015)
1. Most important motivation to publish in peer reviewed journals:
maki...
October 2015
1 Methodology
2
Ideal world & real
world objectives
authorservices.taylorandfrancis.com/
peer-review-in-2015
...
43,000
Science
Technology
Medicine
43,000
Humanities
& Social
Sciences
1 Methodology
2
Ideal world & real
world objectives...
43,000
Science
Technology
Medicine
43,000
Humanities
& Social
Sciences
1 Methodology
2
Ideal world & real
world objectives...
2,398
STM
responses
4,750
HSS
responses
1 Methodology
2
Ideal world & real
world objectives
Peer review in 2015 | A global...
Confidence Interval:
1.95%
@
Confidence Level:
95%
Confidence Interval:
1.34%
@
Confidence Level:
95%
1 Methodology
2
Idea...
2% confidence interval at 95%
confidence level for the population
of all 2013 published authors
95% Result
-2% +2%
1 Metho...
2,398
STM
responses
4,750
HSS
responses
18%
63%
18%16%
63%
21%
All survey respondents:
STM
 Authors
 Reviewers
 Editors...
1 Methodology
2
Ideal world & real
world objectives
Numbers: 6 focus groups, 46 participants
Locations: UK, China and Sout...
In an ideal world…
…to what extent do you agree or
disagree the following objectives
should be the purpose of peer review?...
In an ideal world…
…to what extent do you agree or
disagree the following objectives
should be the purpose of peer review?...
In an ideal world…
…to what extent do you agree or
disagree the following objectives
should be the purpose of peer review?...
In the real world…
…to what extent do you agree or disagree
that peer review is currently achieving
the following objectiv...
In the real world…
…to what extent do you agree or disagree
that peer review is currently achieving
the following objectiv...
In the real world…
…to what extent do you agree or disagree
that peer review is currently achieving
the following objectiv...
HSSSTM
4
5
6
7
8
9
10
4 5 6 7 8 9 10
RealWorld
Ideal World
4
5
6
7
8
9
10
4 5 6 7 8 9 10
RealWorld
Ideal World
1 Methodolo...
HSSSTM
1
2
38
5
4
9
7
10
6
Detect plagiarism
1
2
38
5
4
9
7
10
6
Ideal world mean score Real world mean score
1 Methodolog...
HSSSTM
1
2
38
5
4
9
7
10
6
Improve quality of published article
1
2
38
5
4
9
7
10
6
Ideal world mean score Real world mean...
HSSSTM
1
2
38
5
4
9
7
10
6
Provide polite feedback
1
2
38
5
4
9
7
10
6
Ideal world mean score Real world mean score
1 Meth...
HSSSTM
1
2
38
5
4
9
7
10
6
Correct spelling, punctuation & grammar
1
2
38
5
4
9
7
10
6
Ideal world mean score Real world m...
HSS
Most important objective…
Expectation exceeds reality…
Expectation matches reality…
Reality exceeds expectation…
Polit...
1 Methodology
2
Ideal world & real
world objectives
“The worst reviews are short,
snitty, patronising and not remotely
use...
How common are the following
situations in peer review?
Regional bias
Seniority bias
Gender bias
1 Methodology
2
Ideal wor...
How common are the following
situations in peer review?
HSSSTM
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
Gender bias
Regio...
How common are the following
situations in peer review?
HSSSTM
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
Gender bias
Regio...
How common are the following
situations in peer review?
HSSSTM
1 2 3 4 5 6 7 8 9 10
1 2 3 4 5 6 7 8 9 10
Reviewers delay a...
1 2 3 4 5 6 7 8 9 10Higher frequency
of occurrences
reported
1 Methodology
2
Ideal world & real
world objectives
3
Ethical...
1 Methodology
2
Ideal world & real
world objectives
Peer review in 2015 | A global view
A white paper from Taylor & Franci...
As an editor, what do you
consider a realistic amount
of time to expect reviewers
to deliver their initial report?
0.4%
6%...
As an editor, what do you
consider a realistic amount
of time to expect reviewers
to deliver their initial report?
0.4%
6%...
HSS
7%
16%
40%
30%
5%
1%
7 days 14 days 30 days 2 months 6 months 7+ months
As a reviewer, how long was
the duration betwe...
HSS
7%
16%
40%
30%
5%
1%
7 days 14 days 30 days 2 months 6 months 7+ months
As a reviewer, how long was
the duration betwe...
HSS
As an author, how long did
you wait after submission
before you received the peer
reviewer’s initial comments?
1%
4%
9...
HSS
As an author, how long did
you wait after submission
before you received the peer
reviewer’s initial comments?
1%
4%
9...
HSS
96% 93%
Editors
2 months
is realistic
Reviewers
2 months to
deliver report
Authors
2 months to
receive report
1 Method...
HSS
96% 93%
44%
Editors
2 months
is realistic
Reviewers
2 months to
deliver report
Authors
2 months to
receive report
1 Me...
STMHSS
94% 95%
60%
Editors
2 months
is realistic
Reviewers
2 months to
deliver report
Authors
2 months to
receive report
9...
1 Methodology
2
Ideal world & real
world objectives
Peer review in 2015 | A global view
A white paper from Taylor & Franci...
1 Methodology
2
Ideal world & real
world objectives
3
Ethical
concerns
4
Timing
discrepancies
5
Different models of
peer r...
1 Methodology
2
Ideal world & real
world objectives
STMHSS
Double blind:
Neither the author’s nor the
reviewers’ names are...
1 Methodology
2
Ideal world & real
world objectives
3
Ethical
concerns
4
Timing
discrepancies
5
Different models of
peer r...
1 Methodology
2
Ideal world & real
world objectives
3
Ethical
concerns
4
Timing
discrepancies
5
Different models of
peer r...
1 Methodology
2
Ideal world & real
world objectives
3
Ethical
concerns
4
Timing
discrepancies
5
Different models of
peer r...
1 Methodology
2
Ideal world & real
world objectives
STMHSS
Post-publication:
Online readers comment on,
or rate the paper ...
1 Methodology
2
Ideal world & real
world objectives
On double blind
“You have to be quite secure about
your career to un-b...
authorservices.taylorandfrancis.com/
peer-review-in-2015
To come:
1. Motivations to review
2. Training and support
3. Geog...
Upcoming SlideShare
Loading in …5
×

UKSG webinar: Peer review in 2015: a global view - Key findings from the Taylor & Francis white paper with Elaine Devine and Will Frass, Taylor & Francis

1,200 views

Published on

Peer review: much discussed, much covered, much maligned, but what do researchers really think of the system at the heart of scholarly research? Is it as broken as we are sometimes led to believe? How much of a discrepancy is there between expectation and reality? How do views differ between authors, editors and reviewers? Or across the disciplines – humanities, social sciences, science and medicine?

Building on previous research by the Publishing Research Consortium and Sense about Science, Taylor & Francis conducted a global survey and focus groups in 2015 with authors, reviewers and editors around the world. This webinar will present a snapshot of the findings, particularly focusing on views on the purpose of peer review, perceptions on ethical issues, opinions on different models of review, and on the ‘mechanics’ of the system.

Published in: Education
  • Be the first to comment

UKSG webinar: Peer review in 2015: a global view - Key findings from the Taylor & Francis white paper with Elaine Devine and Will Frass, Taylor & Francis

  1. 1. October 2015 Peer review in 2015 A global view A white paper from Taylor & Francis The UKSG webinar Elaine Devine Senior Communications Manager (Author Relations) Will Frass Senior Research Executive authorservices.taylorandfrancis.com/ peer-review-in-2015
  2. 2. Peer review in 2015 | A global view (October 2015) 1. Most important motivation to publish in peer reviewed journals: making contribution to the field and sharing research with others. 2. Most important motivation to review: playing a part in the academic process and improving papers. 3. The benefit of peer review: towards improving an article rated as 8 or above out of 10 (most important aspect in ideal and real world).
  3. 3. October 2015 1 Methodology 2 Ideal world & real world objectives authorservices.taylorandfrancis.com/ peer-review-in-2015 5 Different models of peer review 3 Ethical concerns 4 Timing discrepancies
  4. 4. 43,000 Science Technology Medicine 43,000 Humanities & Social Sciences 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  5. 5. 43,000 Science Technology Medicine 43,000 Humanities & Social Sciences 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  6. 6. 2,398 STM responses 4,750 HSS responses 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  7. 7. Confidence Interval: 1.95% @ Confidence Level: 95% Confidence Interval: 1.34% @ Confidence Level: 95% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review 2,398 STM responses 4,750 HSS responses Peer review in 2015 | A global view A white paper from Taylor & Francis
  8. 8. 2% confidence interval at 95% confidence level for the population of all 2013 published authors 95% Result -2% +2% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis
  9. 9. 2,398 STM responses 4,750 HSS responses 18% 63% 18%16% 63% 21% All survey respondents: STM  Authors  Reviewers  Editors HSS  Authors  Reviewers  Editors 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis
  10. 10. 1 Methodology 2 Ideal world & real world objectives Numbers: 6 focus groups, 46 participants Locations: UK, China and South Africa (early 2015) Participants: Editors, authors and reviewers Minimum of two articles peer reviewed (with Taylor & Francis or any other publisher) Disciplines: sciences, technology, medicine, social sciences, and humanities. Qualitative research 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis
  11. 11. In an ideal world… …to what extent do you agree or disagree the following objectives should be the purpose of peer review? 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis
  12. 12. In an ideal world… …to what extent do you agree or disagree the following objectives should be the purpose of peer review? 5 6 7 8 9 10 Check methodology Provide polite feedback Judge novelty Relevent to scope Check factual accuracy Detect fraud Detect plagiarism Ideal World – rating out of 10 HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  13. 13. In an ideal world… …to what extent do you agree or disagree the following objectives should be the purpose of peer review? 5 6 7 8 9 10 Check methodology Provide polite feedback Judge novelty Relevent to scope Check factual accuracy Detect fraud Detect plagiarism Ideal World – rating out of 10 HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  14. 14. In the real world… …to what extent do you agree or disagree that peer review is currently achieving the following objectives effectively? 5 6 7 8 9 10 Check methodology Provide polite feedback Judge novelty Relevent to scope Check factual accuracy Detect fraud Detect plagiarism Ideal World – rating out of 10 5 6 7 8 9 10 RealWorld–ratingoutof10 HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  15. 15. In the real world… …to what extent do you agree or disagree that peer review is currently achieving the following objectives effectively? 5 6 7 8 9 10 Check methodology Provide polite feedback Judge novelty Relevent to scope Check factual accuracy Detect fraud Detect plagiarism Ideal World – rating out of 10 5 6 7 8 9 10 RealWorld–ratingoutof10 HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  16. 16. In the real world… …to what extent do you agree or disagree that peer review is currently achieving the following objectives effectively? 5 6 7 8 9 10 Check methodology Provide polite feedback Judge novelty Relevent to scope Check factual accuracy Detect fraud Detect plagiarism Ideal World – rating out of 10 5 6 7 8 9 10 RealWorld–ratingoutof10 HSS Relevant to Aims & Scope 1 Methodology 2 Ideal world & real world objectives Ethical concerns3 4 Timing discrepancies 5 Different models of peer review
  17. 17. HSSSTM 4 5 6 7 8 9 10 4 5 6 7 8 9 10 RealWorld Ideal World 4 5 6 7 8 9 10 4 5 6 7 8 9 10 RealWorld Ideal World 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns Timing discrepancies4 5 Different models of peer review
  18. 18. HSSSTM 1 2 38 5 4 9 7 10 6 Detect plagiarism 1 2 38 5 4 9 7 10 6 Ideal world mean score Real world mean score 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  19. 19. HSSSTM 1 2 38 5 4 9 7 10 6 Improve quality of published article 1 2 38 5 4 9 7 10 6 Ideal world mean score Real world mean score 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  20. 20. HSSSTM 1 2 38 5 4 9 7 10 6 Provide polite feedback 1 2 38 5 4 9 7 10 6 Ideal world mean score Real world mean score 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  21. 21. HSSSTM 1 2 38 5 4 9 7 10 6 Correct spelling, punctuation & grammar 1 2 38 5 4 9 7 10 6 Ideal world mean score Real world mean score 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  22. 22. HSS Most important objective… Expectation exceeds reality… Expectation matches reality… Reality exceeds expectation… Politeness Detect Fraud Correcting spelling, punctuation & grammar Relevant to the Aims & Scope Improve quality of published paper STM 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  23. 23. 1 Methodology 2 Ideal world & real world objectives “The worst reviews are short, snitty, patronising and not remotely useful. The best are critically engaged, add something and improve the quality.” Editor, Linguistics, United Kingdom “Editors should be more pre-emptive in detecting plagiarism & other types of fraud.” Researcher, Medical Research, United Kingdom HSSSTM Peer review in 2015 | A global view A white paper from Taylor & Francis 5 Different models of peer review 4 Timing discrepancies 3 Ethical concerns
  24. 24. How common are the following situations in peer review? Regional bias Seniority bias Gender bias 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis
  25. 25. How common are the following situations in peer review? HSSSTM 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Gender bias Regional bias Seniority bias 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  26. 26. How common are the following situations in peer review? HSSSTM 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Gender bias Regional bias Seniority bias 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  27. 27. How common are the following situations in peer review? HSSSTM 1 2 3 4 5 6 7 8 9 10 1 2 3 4 5 6 7 8 9 10 Reviewers delay assessment Reviewers take ideas Reviewers use false identities 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies Different models of peer review5
  28. 28. 1 2 3 4 5 6 7 8 9 10Higher frequency of occurrences reported 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review Peer review in 2015 | A global view A white paper from Taylor & Francis Lower frequency of occurrences reported
  29. 29. 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis “Peer review can be used as a gatekeeping mechanism to keep certain views out of circulation. In which article are there not personal views?” Researcher, Anthropology, South Africa “Some of the reviewers don’t exist. The author forges a name, creates an identity, applies for a new mailbox and reviews their paper themselves.” Researcher, Healthcare, China “I used to be at a university which is low ranking in my current field. When I was there I couldn’t get a paper accepted but now I am at a well-respected institution, I feel some papers are accepted too easily!” Researcher, Environmental Science, UK 4 Timing discrepancies 5 Different models of peer review 3 Ethical concerns
  30. 30. As an editor, what do you consider a realistic amount of time to expect reviewers to deliver their initial report? 0.4% 6% 48% 41% 4% 0.0% 7 days 14 days 30 days 2 months 6 months 7+ months HSS 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  31. 31. As an editor, what do you consider a realistic amount of time to expect reviewers to deliver their initial report? 0.4% 6% 48% 41% 4% 0.0% 7 days 14 days 30 days 2 months 6 months 7+ months 6% HSS 54% 96% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  32. 32. HSS 7% 16% 40% 30% 5% 1% 7 days 14 days 30 days 2 months 6 months 7+ months As a reviewer, how long was the duration between your acceptance to review and the delivery of your initial report? 96% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  33. 33. HSS 7% 16% 40% 30% 5% 1% 7 days 14 days 30 days 2 months 6 months 7+ months As a reviewer, how long was the duration between your acceptance to review and the delivery of your initial report? 96%7% 23% 63% 93% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  34. 34. HSS As an author, how long did you wait after submission before you received the peer reviewer’s initial comments? 1% 4% 9% 30% 42% 13% 7 days 14 days 30 days 2 months 6 months 7+ months 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  35. 35. HSS As an author, how long did you wait after submission before you received the peer reviewer’s initial comments? 1% 4% 9% 30% 42% 13% 7 days 14 days 30 days 2 months 6 months 7+ months 96% 93%1% 5% 14% 44% 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  36. 36. HSS 96% 93% Editors 2 months is realistic Reviewers 2 months to deliver report Authors 2 months to receive report 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  37. 37. HSS 96% 93% 44% Editors 2 months is realistic Reviewers 2 months to deliver report Authors 2 months to receive report 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  38. 38. STMHSS 94% 95% 60% Editors 2 months is realistic Reviewers 2 months to deliver report Authors 2 months to receive report 96% 93% 44% Editors 2 months is realistic Reviewers 2 months to deliver report Authors 2 months to receive report 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  39. 39. 1 Methodology 2 Ideal world & real world objectives Peer review in 2015 | A global view A white paper from Taylor & Francis 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  40. 40. 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review STMHSS Authors Reviewers Editors Uncomfortable Very comfortable
  41. 41. 1 Methodology 2 Ideal world & real world objectives STMHSS Double blind: Neither the author’s nor the reviewers’ names are known to each other 8.6 8.9 Authors Reviewers Editors 8.1 8.48.0 Uncomfortable Very comfortable 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  42. 42. 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review STMHSS Single blind: Only the author’s name is known to the reviewer, but the reviewers’ names are not known to the author 4.1 4.0 Authors Reviewers Editors 5.4 5.9 6.5 Uncomfortable Very comfortable 4.4
  43. 43. 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review STMHSS Open: Both the authors’ and reviewers’ names are known to each other 5.3 5.1 Authors Reviewers Editors 6.0 5.8 & 5.9 Uncomfortable Very comfortable 5.7
  44. 44. 1 Methodology 2 Ideal world & real world objectives 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review STMHSS Open and published: Both the authors and reviewers’ names are known to each other and the reviewers’ signed reports are published 4.7 Authors Reviewers Editors 5.7 5.4 Uncomfortable Very comfortable 5.3
  45. 45. 1 Methodology 2 Ideal world & real world objectives STMHSS Post-publication: Online readers comment on, or rate the paper following publication 4.4 Authors Reviewers Editors 5.7 5.1 Uncomfortable Very comfortable 4.7 4.9 3 Ethical concerns 4 Timing discrepancies 5 Different models of peer review
  46. 46. 1 Methodology 2 Ideal world & real world objectives On double blind “You have to be quite secure about your career to un-blind yourself. I don’t want to offend a future employer or someone sitting on an interview panel…” Researcher, Environmental Sciences, UK On open and published “I think this is the most transparent way… It may put some pressure on the reviewer, but it also gives him/her credit…” Reviewer, Humanities, Lebanon On post-publication “This method is limited to those who can actually read the articles (are subscribed) online unless the articles are open access..” Reviewer, Agriculture & Food Science, Zimbabwe 4 Timing discrepancies 5 Different models of peer review 3 Ethical concerns
  47. 47. authorservices.taylorandfrancis.com/ peer-review-in-2015 To come: 1. Motivations to review 2. Training and support 3. Geographical analysis @tandfauthorserv #tfPeerReview elaine.devine@tandf.co.uk Authors Reviewers Editors

×