Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real-life examples of manuscript reviews Comparison and contrast of useful vs. not so useful reviewer comments


Published on

Aileen Davis, PhD
Senior Scientist and Division Head,
Health Care and Outcomes Research,
Krembil Research Institute,
University Health Network and
Professor, University of Toronto

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

Real-life examples of manuscript reviews Comparison and contrast of useful vs. not so useful reviewer comments

  1. 1. Real-life examples of manuscript reviews Comparison and contrast of useful vs. not so useful reviewer comments Aileen Davis, PhD Senior Scientist and Division Head, Health Care and Outcomes Research, Krembil Research Institute, University Health Network and Professor, University of Toronto
  2. 2. Aileen Davis, PhD Senior Scientist and Division Head, Health Care and Outcomes Research Krembil Research Institute, University Health Network, Toronto, ON, Canada Disclosure Information I have financial relationship(s) with: Canadian Institutes of Health Research, Grant Funding Other relevant disclosures: Member at Large, OARSI Board Associate Editor, Osteoarthritis and Cartilage Editorial Board, Arthritis Care & Research AND My presentation does not include discussion of off-label or investigational use.
  3. 3. Outline 1. What language is appropriate in a review? 2. Context: editor needs author needs 3. Common issues with examples 4. Summary and Conclusion
  4. 4. Caution and Confidentiality • Acknowledge the reviewers; their work results in dissemination of excellent science • Examples and quotes have been taken from various sources, including reviews of my own work • Some are verbatim, other reworded or constructed purely for this talk; often just a very small piece of a quality review • All anonymized but can’t ensure that some of these won’t be recognized • Focus on the issue • Please do not photograph the quotes
  5. 5. Language of critique Respectful language: - makes communication more effective - it prevents misunderstandings and conflicts - shows attention, uses descriptive words and is problem-oriented - allows for thoughtful reflection
  6. 6. To the editor….. A point well-taken but…. 50 lines of text in 6 paragraphs for one issue: ‘…concerns with the use of internet-based sampling panels for normative data.’ - ‘in my experience’ used 11 times - be aware that ‘even with a representative sample that constructing weights was (is) complex.’ - ‘worked with a professor from X with decades of experience with surveys.’ - ‘discussed for decades by survey professionals.’
  7. 7. (Unintended) Bias in language ‘A few edits would help to make the presentation more American.’ OARSI is an international organization! Most journals serve an international audience English may not be a first language for many BUT even native English speakers struggle for writing clarity at times OAC is an English language journal and need sufficient quality of writing for comprehension and clarity Editor prerogative to ask authors to have their work edited Reviewer can suggest to the editor that it might be helpful
  8. 8. Content of the Review
  9. 9. Context: editor perspective • Innovation/new knowledge vs incremental • Objectives, methods including analysis and interpretation align • Sufficient methodological rigour
  10. 10. Comments to the editor • Chosen to review because of expertise and knowledge of literature in the area • Editor relies on you to address the novelty/importance of the work • Okay to confirm existing studies if limited work in the area
  11. 11. R1: The paper by [xxx] is describing the results on healthcare costs out of a [xxx] trial. The results are well analyzed and the conclusions are very clear, inclusive taking the limitations of the study into account. The paper reads well and gives a clear view on the results obtained in the study. Points to consider: - Are there any data on pain medication available? This would add value to the results and would also give further insight in costs post surgery - On the results (line 188): Could the numbers of missing diaries and the numbers of excluded datasets be provided? Overall a well written paper with good conclusions. R2: This is a very well written account. I have few issues (unusually) with the statistical presentation, and any questions that arose were generally covered by the authors. This is a very well written account of a well-constructed study. Minor point only. Participant reported data was included if 7/9 diaries were completed, and an imputation model used where individuals are missing only 1-2 diaries. To what extent can missing data also be assumed in the diaries which were completed, and how was this handled by the authors? R3: This study examining the ninety-day and one-year postoperative healthcare utilization and costs following total knee arthroplasty is a well conducted study. Manuscript is well written and is relevant to the journal. I have no major issues and nothing much to add to this paper. Happy Editors, Authors and Reviewers: Short and sweet does it!
  12. 12. Context: authors • Guidance to optimize the presentation and impact of the work - substantive vs clarifications in writing - additional analyses to enhance interpretation - ‘so what’ message - key limitations missing that influence interpretation
  13. 13. Substantive issues to improve/optimize the work Using administrative data…. I was surprised that no sensitivity analyses were carried out concerning the OA diagnoses, particularly given the long time span (1998 to 2003) over which exposure status could be ascertained. Given issues around the making an OA diagnosis, other researchers have recommended that algorithms requiring more than one visit be required for studies using administrative data. (See for example Rahman MM et al., Int J Rheumatol 2O16; 6475318 which shows higher specificity and PPV with at least 2 diagnoses within 2 years (or a hospital admission) -a lit search will give a few other studies). If OA is truly related to increased mortality, there will likely be stronger relationships with increased specificity. Given the long time frame over which OA was ascertained there should be ample sample carry out such a sensitivity analysis.
  14. 14. Substantive issues cont… Given the richness of the data sources used, this paper also represents a missed opportunity. There are at least two studies in the literature which show that women with OA have higher risks of cardiovascular mortality than men (see Rahman MM et al., Arthritis Care & Research 2013;65:1951-1958. Schieir O et al., Arthritis Care & Research 2016;68:811-818). These findings need confirmation in other populations. I strongly suggest that the authors of this paper explore and report on analyses stratified by sex. If there is a substantial sex difference in mortality risk (either overall or cardiovascular) this clearly would have important implications.
  15. 15. Clarifications in writing and content ‘Please clarify if studies permitting participants to walk with walking aids were included’ ‘ Please clarify if studies with participants with confirmed OA in lower joint other than the knee were included’ ‘Please clarify is findings/studies from principle component or principal pattern analysis were included’ What is the obvious overarching clarification needed? Does it need separate line by line critique?
  16. 16. Structure of the review • Avoid providing a line by line review
  17. 17. Missing content • Thoughtfully review methods, results and interpretation • Do methods and results reporting match, e.g. data reported as collected but not in results? • Results not discussed? Corollary….. • Due diligence: Are you sure you aren’t asking for something already in the paper?
  18. 18. Reviewer: The participants in these two cohorts are highly educated. This limits the generalizability of the study. Response: We are unsure what the reviewer would like us to do based on the above. The following is currently in the limitations section of the discussion: Our study has several limitations. A high proportion of people in our cohorts were highly educated and this may not reflect other cohorts. However, the Organisation for Economic Co-operation and Development reports that Canada ranks first in the proportion of people with higher education (i.e. more than high school) with Canadian women representing the highest proportion (33) such that generalizability to the Canadian population is likely less problematic. We have not made any revisions to the manuscript given we have addressed this issue in the initial version submitted.
  19. 19. Setting up a debate Providing thoughts/opinion that leads to debate versus manuscript revisions are not helpful to the author ‘I personally don’t like the concept of MCID. I’d appreciate the authors thoughts/comments on this.’
  20. 20. Staying within the context of the work • Request for discussion of issue(s) that take work beyond purpose and results of work Context - Systematic review of MCID and PASS values for the WOMAC in total hip and knee replacement to determine if values are similar across countries - Method and sample heterogeneity such that comparisons not possible - Discussion focused on this and need for consensus on methods
  21. 21. Staying within the context of the work cont. • Reviewer request: We are in the era of precision medicine. Please discuss the methods for this and how the MCID fits with precision medicine, if at all, given that we will be looking at interventions only based on the individual’s outcome. How do you deal with such a request?
  22. 22. Comments taking you outside expertise • Be sure of your knowledge, in the context of the request to authors • Look it up if you don’t know/aren’t sure
  23. 23. Figure 1: Density plots comparing distribution of pre-surgical WOMAC pain for cohort 1 (03/2006-03/2008) and cohort 2 (11/2012-03/2015) Cohort 2 Tertile 1 Tertile 2 Tertile 3 Cohort 1 Pre-surgery WOMAC pain Density
  24. 24. From the journal editor…. ‘Figure 1 is confusing. Could you please display this information differently. A table with the descriptive statistics would be easier for the reader.’ ‘Use of bandwidth is unclear. Please avoid using jargon.’
  25. 25. Criticism/concern but what to do? Classic cryptic comments ‘The authors could shorten the Introduction/this section of the manuscript.’ Page x, lines y-z: ‘The authors need to expand this section.’ Page x, lines y-z: ‘The paper would benefit if more information were provided.’ ‘The title isn’t a good reflection of the work.’ ‘The authors need to additionally discussion x, y, z topics in the Discussion.’ (and the word count becomes?) ‘This work would be better split into two papers.’ ‘I disagree with the conclusion.’ (with no justification)
  26. 26. Summary and Conclusion Writing a thoughtful, respectful review: - write as if you were the recipient editor or author - if important work to publish, provide guidance to assist authors in optimizing the approach, interpretation and discussion - don’t ‘eat’ the authors
  27. 27. Thank you!