What every new reviewer should know about peer review

@SAJS_Official @ASSAf_Official
#PeerReview
PEER
What every new reviewer should know
about peer review
E
V
I
E
W
09 September 2024
How to cite:
Presenter’s last name, initials. Title of presentation [PowerPoint slides]. 2024. Available from: URL
© 2024. The Author(s). Published under a Creative Commons Attribution Licence.

09 September 2024
Peer review in scholarly journals
P E E R
E
V
I
E
W
PROGRAMME
10:00 Welcome and introduction
10:05 Where to start with peer review
Leslie Swartz, SUN
Editor-in-Chief: SAJS
10:50 Peer review models and their technologies
Susan Veldsman, ASSAf
Director: Scholarly Publication Unit
11:20 Tools for recognising and rewarding peer review
Nadia Grobler
Online Publishing Systems Administrator: SAJS
11:35 Q&A
11:55 Thanks and closure
#PeerReview

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Presenters
Leslie Swartz is a Professor of Psychology at Stellenbosch University. His research focuses on disability
rights and mental health issues in southern Africa. He has a particular interest in developing
academic writing skills and increasing access to academic publishing. He is Editor-in-Chief of the
South African Journal of Science and of the Scandinavian Journal of Disability.
Susan Veldsman is Director of the Scholarly Publication Unit at the Academy of Sciences in South
Africa, responsible for driving the Open Science agenda, to raise the visibility, discoverability and
accessibility of South African scholarly journals, improve the quality of SA research output, and
support the development of policy frameworks in order to facilitate optimal use and access to
publicly funded research. She received the Electronic Publishing Trust (EPT) award for her
outstanding contribution to the promotion of Open Access in Developing Countries. Recently, she
was the Co-chair of the IAP Report on “Combatting Predatory Journals and Conferences”, which
was launched on the 16th March 2022. She also received special recognition for her role in
establishing the SciELO SA platform in South Africa at the most recent SciELO 25 years celebrations.
Nadia Grobler is the Online Publishing Systems Administrator for the South African Journal of Science.
She holds a degree in Information Science from the University of Pretoria and has expertise in
scholarly publishing systems, tools, and services.

#PeerRevie
w
What every new reviewer should
know about peer review
PEER
E
V
I
E
W
09 September 2024
Where to start with peer
review
Leslie Swartz

Some
principles
Peer review is central to the academic enterprise, and
is as important as writing your own articles
There is currently a crisis in peer reviewing – it is hard
for journals to find peer reviewers, and without peer
review journals cannot continue to function
We are all peers
We all have a responsibility to keep the peer review
system going
A rule of thumb: for every time I send an article for
review, I should be prepared to review another article

What do I want from a peer reviewer?
• I want someone who is
• Competent
• Constructive
• Fair
• Kind
• Clear
• Focused on making my work better
• Not trying to make me a version of themselves

“I am not good enough”
Academics and impostor
syndrome – who do NOT
think that they are
impostors?
The academic hierarchy
(“only professors….”)
What do you really need to
know, and what don’t you
need to know?
It is fine to point out the
limitations of your
knowledge in your review
(the editor may well have
chosen a range of
competencies)
You don’t have
to pretend

“I’m not
sure I am
the right
person to
review
this”
• Make a list of what you think you can and
cannot do
• How important is what you can do to
helping the author and the journal?
• How much of a barrier is what you can’t do
for your ability to be helpful?
Read the paper quickly, and then:
• Don’t be shy to contact the editor and raise
any concerns – we editors are grateful to
you and want to work with you!
If still in doubt

Your role
as a
reviewer is
that of a
‘peer
mentor’
(Way, et
al, 2021)
You are not expected to know everything,
but you can make it clear what you do and
don’t know
Four main
areas:
Do I understand the
methods?
Do I have a good idea of
who the audience is for
this journal? Can I stand
as a ‘representative’ of
that audience?
Do I know enough about
the content to
comment?
Have the authors told
me enough about the
context of the work that
I can make an
assessment?
Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM,
Mavis B. Fundamentals of Scholarly Peer Review: A
Workshop for Health Professions Educators on Practicing
Scholarly Citizenship. MedEdPORTAL. 2021 Aug 2;17:11174.

Using the abstract
Read the abstract
carefully, and a few
times if needs be.
You should be able to
find an indication of
the key parts of the
article, in the abstract.

Parts of an article/abstract
• WHY the authors did the
study
• HOW they did the study
• WHAT they did
(steps)
• WHAT they found
• WHAT it means for
readership of this journal
• RATIONALE,
includes literature review
• METHOD
• PROCEDURE
• RESULTS
• DISCUSSION AND
CONCLUSION

Once you have the whole manuscript:
Further key questions
How do the
different parts of
the article link
together?
For example, does
the research
question follow
clearly from the
rationale?
Do the conclusions
follow from the
data, or are they
more general
claims?
Are the authors
clear about what
they can know
from their work
and what they can
NOT know?

How do we assess whether research is good/publishable?
We do NOT assess the
quality of research based on
whether we agree with the
conclusions.
We may suggest rejection of
a submission where we DO
agree with the conclusions
We may suggest acceptance
of a submission where we
STRONGLY DISAGREE with
the conclusions
The key consideration is
about the QUALITY OF THE
METHODS and the
RELATIONSHIP BETWEEN
THE ARGUMENT AND THE
METHODS

Ethical
considerations
• Is there a clear statement about ethical
permissions from an accredited
Institutional Review Board/Ethics
Committee?
• Is there are statement, where
applicable, about how Large Language
Models/AI have been used?
• Beyond these concerns, do you as a
reviewer have worries?
• Ethical questions are not just tick-box
questions – you must engage with them
• Always raise ethical concerns even if
you are not sure if they are relevant

Format/
length/
style etc
If you suggest revisions, can these all
be done within the word limit for the
journal?
But do consider them, especially
Is the paper
way over
length?
Is the referencing
style the same as
used in the journal?
Writing style ok for
journal (technical
language, acronyms etc)?
These issues are usually looked at by
the editorial team

Recommendation examples (varies
from journal to journal)
Accept with no changes (rare, but don’t be afraid to make this recommendation!)
Accept
Minor revisions
Minor
revisions
Revise and resubmit/major revisions
Revise and
resubmit
Reject (very common)
Reject
In ALL cases, you must give reasons for your recommendation
Give reasons

How not to give feedback
In summary, this article
adds nothing to knowledge
I can’t see why the author
bothered
This is not academic
This is a bad article

Give the kind of feedback you
would like to receive
As constructive as possible
Always give reasons for your views
Make clear suggestions for improvements
As much detail as possible
Generally speaking, a longer reviewer report is a better
reviewer report

The
language
and style of
peer review
As a general principle:
say what you want the
authors to do and not
what you don’t want
them not to do
Write as simply and
clearly as you can
Remember: most
people writing in
English language
journals are not first
language English
There will be times
where you are not sure,
or just have a
preference/suggestion
– say so
Make clear what for
you is an absolute
requirement and what
is something you would
like/prefer to see

• Write in a supportive tone, but be definitive
• Summarize your understanding of the work
• Explain your overall impressions (recommendation /
ratings)
• Base your overall impressions on actual content
• Indicate strengths and weaknesses, providing specific
examples
• Provide suggestions for improvement
• Be detailed and clear
• Explain any descriptors, such as “insufficient method”
Tips for Writing Comments*
* Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisor’s
completed clinical evaluation reports. Med Educ 2008; 42:816-22.
NOTE: THIS SLIDE IN ITS ENTIRETY IS COPIED FROM Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM, Mavis B. Fundamentals of Scholarly Peer Review: A
Workshop for Health Professions Educators on Practicing Scholarly Citizenship. MedEdPORTAL. 2021;17:11174. https://doi.org/10.15766/mep_2374-
8265.11174

You are not
alone…
• Get help from others and give help
to others, but ONLY ONCE YOU HAVE
ASKED THE EDITOR IF YOU MAY DO
THIS
• Concerns:
• Confidentiality and respect for
authors
• “Ghost reviewing” and
exploitation of more junior
reviewers, hidden from sight
• In this – remember your rights, those
of the authors, and those of the
journal

You are
not
alone…
• Consider some options
• Ask your supervisor/a more
experienced colleague to
include you in peer reviewing
when they get approached
• Approach a more senior
colleague/supervisor to help you
when you get asked to peer
review
• Develop peer review buddy
systems (peer to peer)

Peer review buddy systems (1)
• Link in with “shut up and write” – writing support systems
• Journal clubs also useful
• A good place to start, even before you do peer reviews
yourselves:
• Make a commitment that you and your buddies will “peer
review” reviews you yourselves receive from journals.
• Assess the reviews you receive in terms of competence,
respectfulness, constructiveness, etc; and
• Suggest in your group ways in which the reviews could
have been improved

Peer
review
buddy
systems
(2)
First, establish rules
of confidentiality for
your group
Always get
permission from the
editor to use a peer
system
Step 1: Each
(independently)
sketch out areas to
be covered in review
Step 2: Distribute the
tasks and each do
your task
Step 3: Review what
one another have
done
Step 4: Assemble full
review

A PLEA
FROM A
JOURNAL
EDITOR
Please do not undermine the
peer review process while it is
underway, BUT
Do not
undermine
the process
Please do give journals feedback
on peer review experiences,
good and bad
Do give
feedback
Remember: we need to peer
review peer review!
Remember

#PeerReview
PEER
E
V
I
E
W
09 September 2024
Peer Review Models and
their Technologies
Susan Veldsman
Director: Scholarly Publishing

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Peer Review?
• Peer review is designed to assess the validity, quality and often the originality of articles for
publication. Its ultimate purpose is to maintain the integrity of science by filtering out invalid or
poor quality articles……. and helping to improve that work through the peer review process.
 Publisher’s perspective: a filter for content, directing better quality articles to better quality
journals and so creating (better) journal brands.
 Editor’s feedback: "Pointing out the specifics about flaws in the paper’s structure is paramount.
Are methods valid, is data clearly presented, and are conclusions supported by data?”
 Critics feedback: The peer review system is not without criticism. Studies show that even
after peer review, some articles still contain inaccuracies and demonstrate that most rejected
papers will go on to be published somewhere else.
Criticism should be understood within the context of peer review as a human activity. The occasional errors
of peer review are not reasons for abandoning the process altogether – the mistakes would be worse
without it.

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Different designs in the use of peer review
• Journal peer review
 despite its complexity peer review can “serve as a filter to separate or potentially harmful
content out papers with “irrelevant, trivial, weak, misleading, or potentially harmful
content” while also “improving the clarity, accuracy and utility” of potential papers”
 The process is time intensive and
 takes the effort of mostly volunteer staff to evaluate papers for publication and construct
objective and useful feedback

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
• Researcher evaluation for academic achievement and career, as measured
by publications and grant funding, are also dependant on the judgement of
one’s peers in a presumed fair appraisal process
• Mixed-method designs (peer review and bibliometrics) e.g. ASSAf Peer Review
Panels: Journals, institutional audits and system reviews
o Predominant use of peer review only in foresight studies, retrospective case
studies, ex ante proposal and manuscript reviews.
o Predominant use of bibliometrics only in cross-country comparative studies,
institutional rankings and disciplinary field reviews.
• Combination of self-evaluation with external evaluation components
(institutional audits, OECD reviews)
Different designs in the use of peer review (2)

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Different forms of peer review
• Single blind system
The referee knows the identity of the author and the institution that submitted the work.
The author does not know the identity of the reviewer.
Advantages:
• Preventing bias or favoritism based on the authors' reputation, allowing for honest and critical
feedback,
• Reducing the risk of plagiarism or misuse of confidential information by the reviewers.
Disadvantages:
• Bias or discrimination based on gender, ethnicity, nationality, or seniority.
• Limit diversity and creativity in research by favoring conventional perspectives.
• Junior or less-known authors may be discouraged from submitting to prestigious journals or
challenging established theories

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Different forms of peer review (2)
• Double blind system
In this system the identity of both the author and the reviewer are hidden.
Advantages
 Improves peer review
 Tprocess allowed reviewers to be more objective and that
 Eliminating bias or discrimination based on identity and encouraging innovation and originality by allowing authors to present novel or controversial
ideas.
 Increases the confidence and credibility of the peer review process by reducing the potential for conflicts of interest or personal agendas.
Disadvantages
 Double-blind review may be difficult or impossible to conceal the identity of the authors or reviewers, especially in small or specialized fields.
 Hinder communication and collaboration between authors and reviewers, as well as
 Delay the publication process by requiring additional time and effort to anonymize manuscripts and check for inadvertent clues.
Reviewers in anonymous reviews carry no responsibility for those views but in practice, a decision can be appealed by the author.
The process takes place behind closed doors (or, rather, password privileges) and reviews are not published

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Different forms of peer review(3)
• Triple blind review
All identifiers are removed from submitted manuscripts so that handling editors, authors and
reviewers remain blind from each other
• Open peer review
In this system the identity of both the author and reviewer are exposed
Reviewer’s names and their reports are published alongside the paper.
Open Access Infrastructure for Research in Europe (OpenAIRE) (2016: 4) has defined this process –
“although often narrowly defined as peer review where author/reviewer identities are disclosed to
one another – is best understood as an umbrella term describing a variety of innovations which can
“open up” the traditional peer review process by modifying one or more of aspects to make it more
inclusive, transparent and/or accountable”

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
• Open peer review (OPR) is moving into the mainstream and increasing trend
• Not new, 20 years a feature of BMJ and BMC
• Gaining ground through Open Science
• Often poorly understood
• Surveys of researcher attitudes show important barriers to implementation.
• As more journals move to implement and experiment with the myriad of innovations
covered by this term,
• There is a clear need for best practice guidelines to guide implementation
Open Peer Review

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
• Open peer review (henceforth OPR) can take place in different
stages of the review process—pre- or post-publication
• Offering extended communication and knowledge exchange
between researchers.
• Yet, the openness of these systems often differs in term of what is
revealed to whom and when
• the main core elements were revealing reviewer identities (open
identities) and
• Publishing reviews (open reports).
Open Peer Review

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
OpenAIRE (2016:5) defines the primary aspects are:
Open Identities: Authors and reviewers are aware of each other's identity.
Open Reports: Review reports are published alongside the relevant article.
Open Participation: The wider community to able to contribute to the review
process.
Open Peer review: different elements of open

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Open Peer Review: Secondary
characteristics
Open pre-review manuscripts: Manuscripts are made immediately available
(e.g., via pre-print servers like ArXiv) in advance of any formal peer review
procedures
Open final-version commenting: Review or commenting on final “version of
record” publications
Open interaction: Direct reciprocal discussion between author(s) and reviewers,
and/or between reviewers, is allowed and encouraged
Open platforms: Review is de-coupled from publishing in that it is facilitated by
a different organizational entity than the venue of publication”.

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
• These elements can often complement each other
• Can be combined in various ways, to present a broad continuum of
‘openness’.
• For the manuscript under review,
• “Some journals might only include optional open identities, others
• Might publish the entire multi-staged review process
The review reports and the authors’ responses, and the revised
manuscript(s), while inferring links between the earlier released version(s)
and the final version of record” (OpenAIRE, 2016:5).
Different versions of “open”

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Advantages Disadvantages
Reviewers will be more tactful and constructive Open review is the likely accumulation of “enemies” between
authors and reviewers
Reviewers with vested interest in suppressing the publication of a manuscript
could be easier unmasked by authors
Reviewers refuse to participate in OPR proceed or have concerns
over them
Reviewers are mentioned in footnotes of papers and therefore do a better review
(recognition)
No protection to both authors and reviewers
Reviewers and authors are held accountable for their communications – enables
positive feedback
Younger reviewers feels exposed, especially if they have to review
the manuscripts of more senior researchers
Over time the quality of papers improve There are no criteria that identifies the reviewer as “peer”
Opportunity for reviewers to gain credit for and cite their contributions towards
science communication
The changing landscape of academic and professional publication
present formidable challenges to the industry and some portended
correctiveness that affects the quality assurance of peer review
processes
Scholarly community get an insight into the communication between
author/reviewer – which leads to expanded contextual conversations and enriches
science communication
Open peer review in part reflects contemporary disregard or
contempt for expertise
Identify fraudulent data, pointing out errors The dumbing down of intellectual life in general
Quicker publication and dissemination of scientific findings Increase of unfiltered noise and low quality work
More pressure on editor to mediate exchange and assume responsibility for the
final decision
The perception that “junk” is also growing at a pace. Hence and
determining what to read there is a greater need for formal peer
review, perceived selectivity, and other less formal mechanisms such
as “reputation” to filter such noise for time pressed scholars keeping
up to date
Easier identification of misconduct
It will make processes more transparent – it will also address the problem of
“predatory” publishers
Open peer review allows the reader to see how the science unfold.

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Implementing Open Peer Review
• Set your open peer review goals
• Decide which elements you want to implement
• Listen to research communities—be conscious of, and sensitive to community differences;
• Plan technologies and costs
- Assess technological feasibility and possibilities-- electronic editorial office and production/publication
systems and workflows cannot currently be easily configured for OPR elements, they may be difficult
and/or expensive to implement.
- Assess the different costs-- lack of infrastructure to facilitate automated workflows for many of the
elements of OPR, staff, publication of peer review information could be as simple as manually compiling
review components and publishing a single document as Additional file 1 or as complex as an automated
(XML) workflow where each element is published separately
• Set priorities and consider a phased approach
• Engage the community, especially via ‘open champions’
• Communication! editors have a duty of care to ensure reviewers and authors fully understand the systems of
peer review in which they participate and its potential advantages and disadvantages
• Accept changes take time
• After evaluation, share results
• Consider piloting or make open identities optional

09 September 2024
P E E R
E
V
I
E
W
#PeerReview

Author: The author can also see the reviewer’s name, but
only after the reviewer has accepted the request. The
author cannot see the assigned potential reviewer(s).
Reviewer: The reviewer will be able to see
the author’s name if they click on “View
All Submission Details.” They do not need
to accept the review request, in order to
see this information. The reviewer can
make their decision on whether they want
to review a manuscript, with knowledge
about the author’s identity.
Communications: Since the review is open,
authors and reviewers can communicate
with each other using the discussion
feature. Even if they don’t include the editor
as a “participant” to the discussion, editors
can always view discussions.
The author, reviewer, and editor can all see
messages between reviewer and author.
PKP Open Journal Systems

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
• Publons was founded by Andrew Preston and Daniel Johnston in 2012 to address
the static state of peer-reviewing practices in academic research publishing, in
view of encouraging collaboration and speeding scientific development.
• Publons was then acquired by Clarivate Analytics in 2017.
• August 18, 2022 Publons was fused into WoS.
• Publons serving as a platform to recognize and reward mostly voluntary peer review
services
• “Building upon our commitment to help you get credit for peer review, we are also
committed to helping you gain recognition for more of your contributions to scholarship
throughout your career. That's why we display your publications and citations from the
Web of Science Core Collection alongside your peer reviews and editor records”.
• Now called Web of Science Researcher Profile
Publons

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
• Publons was founded by Andrew Preston and Daniel Johnston in 2012 to address the
static state of peer-reviewing practices in academic research publishing, in view of
encouraging collaboration and speeding scientific development.
• Publons was then acquired by Clarivate Analytics in 2017.
• The platform will facilitate more transparency, recognition, and training to peer
review, helping to solve some of the critical problems facing research today.
• Speeding up research by helping editors find skilled and motivated peer reviewers,
and tackling issues such as peer review fraud.
• Publons Peer Review Awards are recognitions for top peer reviewers and editors.
Publons' Awards started in 2016. In 2017 an award program called the Sentinel
Award was added, for outstanding advocacy, innovation or contribution to
scholarly peer review (da Silva, 2017:4).
Publons

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Integration amongst systems (workflow and users)
Subscription prices—costs
No standardisation between platforms
Expertise of peer reviewers not guaranteed
Embedding peer review acknowledgement in national
awarding and recognition structures
Critique

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Conclusion
The integrity and excellence of scholarly
communication are foundational to the
advancement of knowledge, science and the
development of academia. Recognising the critical
role that editors, reviewers, and publishers play in
this process

#PeerReview
PEER
E
V
I
E
W
09 September 2024
Tools for recognising and
rewarding peer review
Nadia Grobler

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Why do we need to recognise and reward peer review?
Voluntary
nature
Time and
expertise
Sustainable
peer review
system
Encouraging
participation

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Methods for recognising and rewarding peer review
Publication of
annual
reviewer lists
Thank you
letters and
certificates
Discounts and
waivers
Annual awards
and recognition
events
Editorial
team/board
invitations
Peer reviewer profiles

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Peer reviewer profiles
Why do you need one?
• To keep track of peer review activities
• To be discovered by editors, and invited for peer review
• To ensure you are matched to review papers that align with your expertise
• To complement your academic record
What is it?
A record of an individual’s peer review activities, typically showcasing their review
interests, number of reviews completed, the journals reviewed for, the quality and
timeliness of reviews, and any recognition or rewards received.

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Elements of a reviewer profile
• Personal information: Name and Surname, affiliation, and contact details.
• Areas of expertise: List research interests and keywords.
• Review history: Details of number of reviews completed, the journals reviewed
for, and quality and timeliness of reviews.
• Certificates/recognition: Display badges, certificates, and awards from journals.

What does it offer to reviewers? Registration and automation
-Track and verify peer review activity
(profile)
-Earn credits and rewards*
-Certification
-Inclusion in the RC reviewer database
*Dependent on journal subscription
-Register and setup your profile here:
https://www.reviewercredits.com/reviewer-signup/
-Verified reviews automatically added to profile if the
journal has RC integration.
Reviewer recognition
via WoS researcher
profile
(profile)
-Inclusion in the WoS reviewer database
-Register and setup your profile here:
https://access.clarivate.com/register?app=wos
journal has WoS integration.
(profile)
journal has ORCiD integration, so ensure you have
one setup beforehand: https://orcid.org/register
09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Platforms that track and verify reviews

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Examples Source: ReviewerCredits website

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Examples Source: Clarivate website

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Examples Source: ORCiD website

09 September 2024
P E E R
E
V
I
E
W
#PeerReview
Last thoughts….
• Keep your profile current with recent reviews and achievements.
• Ensure that all information is accurate: Personal details, research interests etc.
• Make sure to include this in your CV, personal website, applications etc.
• Make sure you adhere to journal confidentially policies, when manually adding reviews.

What every new reviewer should know about peer review

More Related Content

Similar to What every new reviewer should know about peer review

More from Academy of Science of South Africa

Recently uploaded

What every new reviewer should know about peer review