The peer review process

The peer review process (II)

Ana Marušić
Croatian Medical Journal
University of Split School of Medicine

What is the peer review process?

Einstein’s answer to Physical
Review in 1936

Reviewer comments

• Comments to editor
• Comments to author
• Reviewer recommendations and comments may
not agree, may even be contradictory
• Reviewers are consultants, not decision makers
• Editors are decision makers, ask them if you
have questions about reviewer comments,
editorial decision, or the process

Ethical responsibilites of reviewers

• Declare competing interests
• Ensure that reviewer is qualified
(= a peer)
• Inform the editor who actually did the
review (e.g. if passed onto a colleague)
• Treat material in confidence
• Take steps to avoid biased review
• Deliver courteous and timely reviews

Reviewer misconduct

• Cistron submitted DNA sequence for
interleukin-1 (IL-1) to Nature
• Paper reviewed by Gillis (Immunex): reject
• Sequence published in PNAS (corrected)
• Cistron and Immunex file patents for IL-1
• Immunex patent contains 7 errors from
original (rejected) Nature paper
• Cistron sues Immunex ($21mn settlement)
Rennie 1999

Advice to novice reviewers
http://ori.dhhs.gov/education/products/niu_peerr
eview/mistakes/index.htm#

Advice to novice reviewers:
Confidentiality

• Researchers who agree to participate as
reviewers take on the responsibility to
provide authors and/or applicants with the
best reviews possible. There is a trust
recognized between the reviewer and
author(s) of any work reviewed.

Confidentiality
• The reviewer has completed a review within the
prescribed time frame and sent the review response form
back to the editor. It contained comments intended for the
editor, including a recommendation to accept with minor
revisions, and helpful suggestions that the authors of the
manuscript might incorporate to improve the paper. This
reviewer, who shares an office with another researcher,
has left the manuscript with the accompanying comments,
in a common area that is used by both researchers. The
document remains in the area for a two week period
during which time his officemate happens to peruse the
manuscript.

Confidentiality

Has the reviewer acted responsibly?

No, this represents a violation of confidentiality.
Yes, since the reviewer has already submitted the review
form with comments.

Impartiality

• The selection criteria for peer reviewers
include relevant research expertise, sound
judgment, good communication skills, and
objectivity. Funding agencies and journal
editors rely on peer reviewers to make
recommendations to accept, reject, or
revise proposals/manuscripts based
primarily on their scientific merit.

Impartiality
• A chemistry professor specializing in efforts to develop a
safe and efficient chemical warfare agent decontamination
system is solicited by a journal editor to review a manuscript
that is very close to work she is currently engaged in.
• The abstract describes experiments that are quite similar to
the experiments being conducted by the chemistry
professor's collaborators. The potential reviewer realizes
that there are only a handful of researchers specializing in
this area who could competently provide an assessment of
the manuscript, and feels that accepting the request to
review would be the responsible thing to do.

Impartiality
• At this point, the chemistry professor has just seen the
title and abstract. However, the chemistry professor also
realizes that by reviewing the complete paper, she will
be privy to protocols that could aid in completing her own
work.
How should the chemistry professor proceed?
• Agree to review the manuscript, then decide whether she
can remain impartial after reading the complete
document.
• Notify the editor about her concerns about impartiality
before reading the complete document.

Responsiveness

• Participation in peer review is considered a
valuable service to science in general and to
one's field of study in particular. The main
mission of peer review is to assess the quality of
research to be conducted, in the form of
proposal submissions, or research completed in
the form of manuscript submissions. Reviewers
base their recommendations to funding agencies
or journal editors on their research expertise and
experience.

Responsiveness
• An editor for a high profile cellular biology journal contacts a
biologist with an expertise in cellular regeneration, and
request that he review a manuscript on this topic. The
biologist is very intrigued, because the authors of the paper
describe an innovative approach in the abstract.
• The editor asked that the review be completed and
submitted within two weeks. Although the time frame would
be reasonable during a different time of the year, the
biologist has several research investigations currently
demanding his attention as well as a presentation at an
international conference to complete. Although the biologist
estimates he will be unable to review the manuscript for at
least three to four weeks, he accepts the editor's request.

Responsiveness

Has the reviewer responded in a responsible manner?

• Yes, since he has every intention of providing a
comprehensive review.
• No, despite his intention to provide a comprehensive
review, he knows he will not do it within the agreed time
frame.

Disclosure of competing interests

• While conflict is a common human
experience, it can become an issue in
research integrity when the conflicts
(financial, personal, political), are between
interest and duties. According to Shamoo
(1992, 1993) and Resnik (2001), most of
the concerns with COIs arise because
personal interest can undermine duties
relating to scientific objectivity.

• A researcher from an academic institution receives a request by
a program officer to participate in a round of reviews for
proposals in an area in which she has considerable
background. The review process is a single blind review where
she will know the name of the authors of the proposals, but the
reviewers will remain anonymous to the authors.
• The researcher agrees to participate because she feels the
opportunity to review proposals will enhance her professional
service record. She is given a number of proposals to read, but
is assigned to take the lead on reviewing one proposal in
particular. As the researcher proceeds with reviewing this
assigned proposal, she realizes that she may have a conflict of
interests with one of the authors.


• Earlier in her career, she and the author in question had a
mutual affiliation with a research endeavor at a private
research lab. There collaborative efforts resulted in several
unpublished reports. Since the affiliation was more than five
years ago, and the duration of the affiliation was less than
nine months, the researcher is uncertain whether she should
identify and certify this on the pre-meeting and post-meeting
Conflict of Interest Certification forms (COI_Information.pdf,
2005).


How should the reviewer proceed with this proposal?

• Say nothing to the program officer and submit the
review without any mention of a possible COI.
• Say nothing to the program officer until submitting the
review that includes a disclosure of possible COI.
• Decides to say nothing unless it is brought up by the
program officer.
• Reveal this conflict of interest immediately and let the
program officer determine how to proceed.

Review quality
• Proposal applicants and authors rely on peer
reviewers providing objective, comprehensive
and fair assessments of their proposals and
submitted manuscripts. Reviewer comments and
recommendations can significantly enhance the
quality of research to be conducted, or make
necessary modifications of conclusion drawn.
The quality of the review depends on the
required level of demonstrated expertise,
research objectivity, and sufficient time spent in
the activity.

Review quality
• A researcher at a private research institute maintains a
busy schedule with responsibilities conducting and
coordinating multiple research projects. She receives a
request to participate in a round of proposal reviews for a
national research funding agency. Because she has
previously received funding from this agency, she feels
obliged to participate, even though there is a strong
likelihood she will not be able to provide a
comprehensive review of the assigned proposals.
Despite her reservations, the researcher's strong sense
of obligation may lead her to accept the request against
her better judgment.

Review quality
How should the researcher proceed?
• Decide to review proposals despite her reservations.
• Immediately decline to review proposals, but provide an
explanation to the program officer.
• Wait several weeks before deciding not to review
proposals, providing an explanation to the review
committee for her inability to participate and the delayed
response.
• Wait several weeks before deciding not to review the
proposals, and provides no explanation to the review
committee for either herunavailability to participate or her
delayed response.

Constructive criticism

• Journal editors attempt to select peer
reviewers based on their competencies in
areas relevant to the submitted
manuscript. Useful and practical
comments can be incorporated in later
revisions, enhance the quality of the
research, and hopefully, improve the
chances for acceptance of a submitted
paper.

• A peer reviewer is perusing a submission that details a
study with the objective of examining the superiority of
the newly developed instrument to measure a general
mental health of a person. The author, a graduate
student submitting her first manuscript, designed the
study with 20 healthy volunteers, 1) each one used the
standard Medical Outcome Study (MOS) mental health
scale soon after they entered the study and 2) after 3
months they used the newly developed instrument to
measure the response.

• The data consisted of pre (MOS scale) and post (new
instrument) study results on 20 subjects. The author
decided to test whether the mean post-study result is better
than the mean pre-study result using a standard 'pooled' t-
test.
• The peer reviewer, who has a substantial background in
statistics, disagrees with the selection of the standard
'pooled' t-test. The reviewer comments that “had the author
invested in a basic statistic text and bothered to read it, he
would have selected this test instead of wasting the
reviewer's time”.

Has the reviewer provided useful information to the
author?

• Yes, the reviewer's comment provide useful information on
how to address the problem.
• No, the reviewer's comment did not provide useful
information on how to address the problem.

Objectivity

• According to Rockwell (2006), researchers
serving as reviewers are obliged to judge
in a fair manner and objectively the quality
and significance of the work under review.
"He/she is obligated to support and
encourage publication of work of high
quality while appropriately challenging
flawed work."

Objectivity
• A senior researcher with an established reputation in a
specialized field is routinely sought after by funding
agencies and editorial boards to review proposals and
submitted articles. Her current level of demand contrasts
sharply with her early professional experience following
completion of her doctorate, when the researcher's quest
for a research position took almost 1 ½ years. She holds
a personal bias against several institutions that chose
not to hire her following completion of her doctorate. The
researcher has agreed to review a number of proposals
for a research funding agency.

Objectivity
• As she begins reviewing her first proposal, she
recognizes that the principal investigator is affiliated with
one of the institutions she holds a personal bias against.
There is a question as to whether her bias against the
institution may also extend to researchers affiliated with
this institution.

Objectivity
How should the researcher proceed?
• Engage in a self-reflection regarding her ability to
impartially review this proposal.
• Decide to review the proposal without any self-reflection.
• Decide to recuse herself from reviewing this proposal
without any self-reflection.
• After engaging in self-reflection, decide that her bias
impacts objectivity and ask to recuse herself from
reviewing the proposal.

Questions?
ana.marusic@mefst.hr

BMJ Question 1. is the paper important?

• Has the research addressed a question that
had to be answered, or is it just “another
brick in the wall”?
• The question matters more than the answer.
If the question was important and the
answer is valid, then it doesn’t matter if the
answer is negative or boring.
• Is this something that clinicians or scientists,
policy makers, or the public need to know,
remembering that there’s more for them to
know than they can possibly know?

BMJ Question 2. Is the paper original?

• Ideally, you know the literature in the field covered in
the paper you have been asked to review; wise to
conduct a literature search before you write your
review
• Almost all of a series of RCTs described as “the
first” in major journals were not the first.
• Has this never been done before?
• If the question has been addressed before does this
add importantly (for example, a much bigger or
better designed study; first time in this population)?

BMJ Question 2. Is the paper original?
(continued)

• Remember that some things that are
“well known” are not based on any
evidence.
• If you think the research unoriginal
please give us references to previous
work. Don’t just say “it’s unoriginal.”
• If there are other important studies that
the authors don’t reference, please
provide references

BMJ Questions 3 and 4. Is the
Introduction section appropriate?
• Good explanation of study
context/background?
• Is there a brief, relevant literature review?
• Are the aims of the study, and study
hypothesis/question stated clearly?

BMJ Question 5: Are the research
methods valid?
• Identify the strengths as well as the
weaknesses of the study methods
• Is the design right for answering the
research question?
• Were the data collected adequately? Was
the sampling right?
• Are the methods described adequately
and completely?
• Are the analyses right? Should they be
redone?

BMJ Question 5: are the research
methods valid? (continued)

• If you are not strong in statistics, just
say so and focus on you’re the areas
you know best
• At a minimum, check the data in the
abstract, text, and tables for
consistency; add up some of the data in
the tables and point out obvious errors

BMJ Question 5: Are the methods valid
– do the paper and the information
reported follow ethical requirements?

• For research involving human participants
– Was the study reviewed and approved by an
ethics review committee?
– Was informed consent appropriately obtained?
• For research involving animals
– Were guidelines for the protection of animals in
experiments followed
• For case reports
– Are patients privacy protected? If patients are
identifiable, was appropriate permission obtained

BMJ Question 6. Are the results
presented adequately?
• Outcomes or observations in logical
order?
• For quantitative research - are data
presented numerically, with measures of
statistical significance and variability?
• For qualitative research – are observations
reported in categories or themes?

BMJ Question 7. Is the Discussion
section appropriate?
• Does it provide an answer to the research
question?
• Are the findings discussed in light of other
relevant literature?
• Are the study limitations clearly and
completely described?

BMJ Question 8: are the conclusions
reasonable?
• Are the conclusions supported by the data
or evidence in the paper?
• Do the conclusions go beyond the merits
of the paper

BMJ Question 9 – are the tables and
figures appropriate?
• Are they cited in the text?
• Are they overly simple or too complicated?
• Are the data in the tables or figures
redundant with each other or the text?
• Are the data clearly presented?
• Check data for consistency
• Are there too many or too few tables and
figures?

BMJ Question 10: are the references
appropriate?

• Are the references up to date?
• Are they relevant?
• Are they any important, relevant
reference that are missing?

BMJ Question 11: is the paper
appropriate for the journal’s readers?
• Know the journal and the journal’s
readership
• If uncertain, look for the journal’s mission
statement or description (often available
online); may also be in the journal’s
instructions for authors
• If you’re still uncertain, don’t comment on
this point

BMJ Question 12. Are the title and
abstract adequate?
• Assess title and abstract after you have
reviewed the manuscript, tables, and
figures
• Does the title give a clear message about
the study? Is it too long? Too short? Is a
subtitle needed?
• Is the abstract structured? Consistent with
the text, tables, and figures? Complete?

BMJ Question 13: is the writing clear?

• Do not point out every typo, spelling error,
and grammatical mistake
• Provide general comment on the clarity of
the paper
• Is the manuscript too long or too short?
• Remember to be constructive

Manuscript and cover letter

• Manuscript presents facts and
figures.

• Cover letter is your advertisment.

Cover letter
1. Manuscript title and authors’ names
2. Statement that the manuscript has not
been published or is under consideration
for publishing elsewhere (abstract up to
400-500 words is not considered previous
publication)
3. Reasons why you think the journal should
publish you article
4. (description of individual author’s
contributions)

Editor’s Decisions

• Acceptance
• Revise and Resubmit
• Rejection

How to Reply to Request for Revision

• Address editor’s concerns
• Address reviewers’ concerns
• Itemize replies to editor’s and and reviewer’s
comments
• If you disagree with a comment, explain
• Do not ignore comments
• Ask for deadline or timing of when revision
should be submitted
• Request for revision is not a guarantee of
acceptance

Sample author response letter – JAMA
March 6, 2010

Dear Dr. Editor,

We are pleased to submit our revised manuscript “Smoking Status is a
Clinical Indicator for Alcohol Misuse in Adults” (JAMA06-4240) for
your consideration for publication in JAMA.

We thank you and the reviewers for the careful consideration that was
given to the original version of the manuscript. We have addressed the
issues raised by each reviewer in the revision and describe how we have
addressed each issue below.

Each of the co-authors has reviewed and approved of the revision. Please
let us know if you have any additional questions. My contact information
is listed below. Thank you for the opportunity to revise this manuscript.

Sincerely,

Dr Author

Comments of Reviewer A
1. Introduction: This is much too long. It appears to have been lifted straight
out of your grant proposal.
The introduction has been shortened to 400 words.

2. Methods, page 9: Please explain why you included these control variables.
In light of this comment, the analyses are no longer adjusted for these control
variables. As suggested, we conducted additional analysis to examine whether
the original associations, and we found that our adjusted and unadjusted
outcomes were not substantially different. See manuscript page 8 for this
additional information.

3. Results, page 10: How did you calculate the response rate?
The response rate was derived by multiplying the household response rate
(89%) by the person response rate (93%) and the sample frame response rate
(99%). This method consistent with the JAMA recommended source document
for calculating response rates for multistage sample designs. See manuscript
page 9 for this explanation.

4. Confidence intervals for all measures of effect would improve the
presentation of the results.
We have provided either standard errors or confidence intervals for all
reported effects in the text and the tables.

Conclusions: What you should expect
from the editorial process

• Prompt acknowledgment and timely
assessment
• If reviewed, a rigorous review
• Confidentiality
• Fairness
• Clear, straightforward, and timely
communication

Timeline for publishing research

The peer review process

More Related Content

What's hot

Similar to The peer review process

More from Universitat de Barcelona - Facultat d’Informació i Mitjans Audiovisuals

Recently uploaded

The peer review process