Peer review in scholarly journals

1. @SAJS_Official @ASSAf_Official #PeerReview Peer review in scholarly journals PEER E V I E W 13 September 2023

2. @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W PROGRAMME Welcome and introduction What is peer review and what is it not? Jemma Finch, UKZN Associate Editor: SAJS What reviewers should know about peer review Nkosinathi Madondo, MUT Associate Editor Mentee: SAJS What authors should know about peer review Shane Redelinghuys, Wits Associate Editor Mentee: SAJS Q&A What everyone should know about AI and peer review Martin Bekker Wits University Where to start with peer review Leslie Swartz, SUN Editor-in-Chief: SAJS Q&A Thanks and closure

3. @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W Meet the presenters

4. @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W

5. What is peer review and what is it not? Jemma Finch @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W

6. @SAJS_Official @ASSAf_Official #PeerReview

7. ‘Peer review is secretive. Only those present in the deliberative chambers know exactly what happens there’ (Lamont, 2009, p. 2) @SAJS_Official @ASSAf_Official #PeerReview ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

8. @SAJS_Official @ASSAf_Official #PeerReview Let’s make things more transparent…. • What is peer review? • What is it not? • What are we striving for?

9. What is peer review? • Check quality, novelty, significance and presentation • Constructive • Collegial and respectful • Detailed and comprehensive @SAJS_Official @ASSAf_Official #PeerReview

10. What is peer review? • Independent, unbiased • Anonymous (depending on the model) • Confidential • Professional service @SAJS_Official @ASSAf_Official #PeerReview

11. What is it not? • Gatekeeping • Spelling and grammar check • Opinions unaccompanied by evidence • Personal • Paid (usually) @SAJS_Official @ASSAf_Official #PeerReview

12. What is it not? • A chance to: • Pull apart other people’s work • Show how clever you are • Advance competing interests • Help your buddies • Boost your citations @SAJS_Official @ASSAf_Official #PeerReview

13. What are we striving for? • Diverse reviewers: origin, gender, career stage • Transparency • Fairness @SAJS_Official @ASSAf_Official #PeerReview

14. Further reading • Small Pond Science Blog ‘Why I don’t have trouble finding peer reviewers’ https://smallpondscience.com/2022/07/20/why-i-dont-have-trouble-finding-peer- reviewers/#more-27097 • Stephen Heard’s Scientist Sees Squirrel Blog ‘How (as an editor) I choose lists of reviewers’ https://scientistseessquirrel.wordpress.com/2018/12/20/how-as-an- editor-i-choose-lists-of-reviewers/ • Lamont 2009: How Professors Think: Inside the Curious World of Academic Judgment. Harvard University Press. • Day & Gastel 2012: How to write and publish a scientific paper. Cambridge University Press. • Cargill & O’Connor 2013: Writing Scientific Research Articles. Wiley Blackwell. @SAJS_Official @ASSAf_Official #PeerReview

15. What reviewers need to know about peer review? nathi madondo 13 September 2023 @SAJS_Official @ASSAf_Official #PeerReview Peer review in scholarly journals P E E R E V I E W

16. Mytalkwill beframedaroundtheidea oftakingthescholarship forward @SAJS_Official @ASSAf_Official #PeerReview

17. Taking the scholarship forward @SAJS_Official @ASSAf_Official #PeerReview A cursory report cannot A detailed report can In a way, not only reviewing the quality of the manuscript but also your quality as a reviewer (Boughey, 2022).

18. What isthe quality of a manuscript? Well presented and clear argument Topic has the potential to inform current debate Methodological coherence Conceptual depth @SAJS_Official @ASSAf_Official #PeerReview

19. What quality is not? Unsubstantiated claims – sweeping statements Some authors try to make an easy publication to earn subsidy @SAJS_Official @ASSAf_Official #PeerReview

20. The paper to review should be in the area of expertise of the reviewer to be able to judge its quality To avoid cursory report @SAJS_Official @ASSAf_Official #PeerReview To judge quality …

21. So, to review is to take the scholarship forward @SAJS_Official @ASSAf_Official #PeerReview

22. Concluding thoughts A big challenge to take the scholarship forward - neo-liberal agenda @SAJS_Official @ASSAf_Official #PeerReview

23. References Boughey, C. (2023, March 31). NRF rating process with Professor Chrissie Boughey [video file]. You tube. https://youtu.be/uIqdjb3z1l8 @SAJS_Official @ASSAf_Official #PeerReview

24. What authors should know about peer review Shane Redelinghuys @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W

25. Initial assessment ‘Under review’ 2-16 weeks ~100 days

26. Know your publisher’s policies • Manuscript submission instructions (formatting, referencing style, word limit) • Attention to detail will improve the aesthetics of your manuscript – simplify reading to appreciate the content • Streamlines the review process, minimise turnaround time • Publishing agreements (e.g., open access, publication fees) & ethical principles

27. • Is it acceptable?  Yes!  Won’t affect the handling of your paper or the review process  Do so respectfully  First, use the online tracking system of the journal to track status  Will hear back very soon if manuscript not suited for the journal  ‘Under review’ – reviewers have been invited / waiting for reviewers to accept review / waiting for reviewers to submit their review reports / reviews being assessed by AE Follow-up on submission

28. • When is a good time to do so?  Subject matter – dependent, technicality of your paper  After 3-6 months Follow-up on submission An analysis of ~3 million papers indexed in PubMed from 1981 to 2015 indicated that the median time-until-acceptance has remained consistent at around 100 days (i.e., approx. 3 months)

29. • A platform on which researchers tell others about their experiences with a particular journal • A tool to help you determine which journal to submit to, based on researcher- provided statistics:  Time until first review received  Total handling time  Time until desk rejection (rejection without review)  Number of review reports received  Quality of reviews (ranked out of 5) (subjective)  Number of rounds of review SciRev – An extra layer of transparency?

32. • Don’t take it personal • Manuscript rejection – does not mean it is the end of the road for your paper • Consider why – take reviews into account – May be too specialised/general • Decide if the work is worth resubmitting (most of the time it is) • Improve work by taking a different approach with your analysis – brainstorm with peers Rejection – what to do?

33. @SAJS_Official @ASSAf_Official #PeerReview Sources • Peer review ethics: Six things every author should know https://www.apa.org/about/division/digest/publishing/peer-review-ethics • The history of publishing delays https://blog.dhimmel.com/history-of-delays/ • The peer review process https://authorservices.wiley.com/Reviewers/journal-reviewers/what-is-peerreview/the-peer-review-process.html • Stichting SciRev https://scirev.org/

34. Martin Bekker School of Electronic and Information Engineering, University of the Witwatersrand Martin Bekker // martin.bekker@wits.ac.za LLMs tools and you, the peer reviewer

35. Our time together How do LLMs work? What are they good at? What are they bad at? How to get the best out of LLMs What do I do if something smells fishy? What level of AI-support is acceptable? How should AI be referenced? (How) should prompts be shared?

36. Warning: Strong Language and Sensitive Content The following presentation contains examples of language and content that may be offensive, including swearing, violent language, and racist undertones. These examples are presented solely for the purpose of analysis and discussion, highlighting the impact of such language in certain contexts. The intention is not to endorse or promote any form of discrimination, violence, or offensive behaviour. It is important to approach these examples with sensitivity and understanding, recognizing that they may cause discomfort or distress. We encourage you to exercise discretion when considering the appropriateness of this content for your audience. Please note that the presentation aims to foster awareness, education, and critical thinking regarding the use of offensive language and its potential effects. We strongly condemn any form of discrimination, violence, or hate speech, and advocate for respectful and inclusive communication. By proceeding with this presentation, you acknowledge that you have been provided with this warning and disclaimer, and that you understand the purpose and context in which these examples are being presented. Martin Bekker // martin.bekker@wits.ac.za

37. Martin Bekker // martin.bekker@wits.ac.za

38. Large Language Models Many available today A corpus of text A stochastic model A base model Safety layers Reinforcement Learning through Human Feedback Martin Bekker // martin.bekker@wits.ac.za

40. Strengths // What it is good at Repackaging of ideas Summating text Style transfer Fixing grammar Brainstorming / Dinner Ideas / Essay Ideas Martin Bekker // martin.bekker@wits.ac.za

44. Weaknesses Mathematical model ≠ not a linguistic model Stochastic parrot Hallucinations Causal reasoning Jailbreaking Martin Bekker // martin.bekker@wits.ac.za

52. Hidden Harms Human Labour Exploitation Environmental Monopolising power (Fundamental?) Un-understandability Commodity Fetishism Martin Bekker // martin.bekker@wits.ac.za

53. Threats Claims of IP non-consent (and theft) Racism, sexism, hate speech (based on training biases) Poor advice (negligent, dangerous, harmful) Mass-scale cheating Martin Bekker // martin.bekker@wits.ac.za

58. How to get the best out of LLMs Only use for what it’s good (narrow!) Only use once the thinking’s done (generally) Never let it have the final say

59. • GTPZero • Quillbot.com • Quetext.com • AIdetector.com • Scale.ai • Originality.ai Martin Bekker // martin.bekker@wits.ac.za Prompt: Write an essay on {insert topic} in {insert wordcount} in the style of an 18-year old second langue speaker. Use at least 10 unusual words, and two sentences must be translated directly from Afrikaans, using Afrikaans grammar. Academics (ad students!) are human It’s dangerous to think one can outsmart tech (most of the time) Critical thinking under threat

60. What do I do if something smells fishy? 1. Understand where your publication stands vis-à-vis LLMs (default should be ‘none’ 2. Seek acknowledgements 3. Search (a few) citations 4. Talk to author

61. How should AI-help be referenced? If usage permitted - Norms (e.g. APA) Acknowledgements Level Github / Repository {but consider how would journal check for paraphrased plagiarism?}

62. Prompts Online repository Google Drive Github Supplemetary documentation Platform, date, version (if possible)

63. Use Ban Grammar and spelling assistance / Proofing Co-creation / Editor Brainstorming and planning ‘Over to you’ / No holds barred Show your prompts Responsible for your work Martin Bekker // martin.bekker@wits.ac.za

64. Happy reviewing Martin Bekker // martin.bekker@wits.ac.za

65. Where to start with peer review Leslie Swartz @SAJS_Official @ASSAf_Official #PeerReview 13 September 2023 Peer review in scholarly journals P E E R E V I E W

66. A recap, and some principles Peer review is central to the academic enterprise, and is as important as writing your own articles There is currently a crisis in peer reviewing – it is hard for journals to find peer reviewers, and without peer review journals cannot continue to function We are all peers We all have a responsibility to keep the peer review system going A rule of thumb: for every time I send an article for review, I should be prepared to review another article

67. What do I want from a peer reviewer? • I want someone who is • Competent • Constructive • Fair • Kind • Clear • Focused on making my work better • Not trying to make me a version of themselves

68. “I am not good enough” Academics and impostor syndrome – who do NOT think that they are impostors? The academic hierarchy (“only professors….”) What do you really need to know, and what don’t you need to know? It is fine to point out the limitations of your knowledge in your review (the editor may well have chosen a range of competencies You don’t have to pretend

69. “I’m not sure I am the right person to review this” • Make a list of what you think you can and cannot do • How important is what you can do to helping the author and the journal? • How much of a barrier is what you can’t do for your ability to be helpful? Read the paper quickly, and then: • Don’t be shy to contact the editor and raise any concerns – we editors are grateful to you and want to work with you! If still in doubt

70. Your role as a reviewer is that of a ‘peer mentor’ (Way, et al, 2021) You are not expected to know everything, but you can make it clear what you do and don’t know Four main areas: Do I understand the methods? Do I have a good idea of who the audience is for this journal? Can I stand as a ‘representative’ of that audience? Do I know enough about the content to comment? Have the authors told me enough about the context of the work that I can make an assessment? Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM, Mavis B. Fundamentals of Scholarly Peer Review: A Workshop for Health Professions Educators on Practicing Scholarly Citizenship. MedEdPORTAL. 2021 Aug 2;17:11174.

71. You are not alone… • Get help from others and give help to others, but ONLY ONCE YOU HAVE ASKED THE EDITOR IF YOU MAY DO THIS • Concerns: • Confidentiality and respect for authors • “Ghost reviewing” and exploitation of more junior reviewers, hidden from sight • In this – remember your rights, those of the authors, and those of the journal

72. You are not alone… • Consider some options • Ask your supervisor/a more experienced colleague to include you in peer reviewing when they get approached • Approach a more senior colleague/supervisor to help you when you get asked to peer review • Develop peer review buddy systems (peer to peer)

73. Peer review buddy systems (1) • Link in with “shut up and write” – writing support systems • Journal clubs also useful • A good place to start, even before you do peer reviews yourselves: • Make a commitment that you and your buddies will “peer review” reviews you yourselves receive from journals. • Assess the reviews you receive in terms of competence, respectfulness, constructiveness, etc; and • Suggest in your group ways in which the reviews could have been improved

74. Peer review buddy systems (2) First, establish rules of confidentiality for your group Always get permission from the editor to use a peer system Step 1: Each (independently) sketch out areas to be covered in review Step 2: Distribute the tasks and each do your task Step 3: Review what one another have done Step 4: Assemble full review

75. • Write in a supportive tone, but be definitive • Summarize your understanding of the work • Explain your overall impressions (recommendation / ratings) • Base your overall impressions on actual content • Indicate strengths and weaknesses, providing specific examples • Provide suggestions for improvement • Be detailed and clear • Explain any descriptors, such as “insufficient method” Tips for Writing Comments* * Dudek NL, Marks MB, Wood TJ, Lee AC. Assessing the quality of supervisor’s completed clinical evaluation reports. Med Educ 2008; 42:816-22. NOTE: THIS SLIDE IN ITS ENTIRETY IS COPIED FROM Way DP, Bierer SB, Cianciolo AT, Gruppen L, Riddle JM, Mavis B. Fundamentals of Scholarly Peer Review: A Workshop for Health Professions Educators on Practicing Scholarly Citizenship. MedEdPORTAL. 2021;17:11174. https://doi.org/10.15766/mep_2374- 8265.11174

76. A PLEA FROM A JOURNAL EDITOR Please do not undermine the peer review process while it is underway, BUT Do not undermine the process Please do give journals feedback on peer review experiences, good and bad Do give feedback Remember: we need to peer review peer review! Remember

Peer review in scholarly journals

Peer review in scholarly journals

Recommended

More Related Content

Similar to Peer review in scholarly journals

Similar to Peer review in scholarly journals(20)

More from Academy of Science of South Africa

More from Academy of Science of South Africa(20)

Recently uploaded

Recently uploaded(20)

Peer review in scholarly journals