Aileen Davis, PhD Senior Scientist and Division Head, Health Care and Outcomes Research, Krembil Research Institute, University Health Network and Professor, University of Toronto

1. Real-life examples of manuscript reviews
Comparison and contrast of
useful vs. not so useful reviewer comments
Aileen Davis, PhD
Senior Scientist and Division Head,
Health Care and Outcomes Research,
Krembil Research Institute,
University Health Network and
Professor, University of Toronto

2. Aileen Davis, PhD
Senior Scientist and Division Head, Health Care and Outcomes Research
Krembil Research Institute, University Health Network, Toronto, ON, Canada
Disclosure Information
I have financial relationship(s) with:
Canadian Institutes of Health Research, Grant Funding
Other relevant disclosures:
Member at Large, OARSI Board
Associate Editor, Osteoarthritis and Cartilage
Editorial Board, Arthritis Care & Research
AND
My presentation does not include discussion of off-label or investigational use.

3. Outline
1. What language is appropriate in a review?
2. Context: editor needs
author needs
3. Common issues with examples
4. Summary and Conclusion

4. Caution and Confidentiality
• Acknowledge the reviewers; their work results in
dissemination of excellent science
• Examples and quotes have been taken from various
sources, including reviews of my own work
• Some are verbatim, other reworded or constructed
purely for this talk; often just a very small piece of a
quality review
• All anonymized but can’t ensure that some of these
won’t be recognized
• Focus on the issue
• Please do not photograph the quotes

5. Language of critique
Respectful language:
- makes communication more effective
- it prevents misunderstandings and conflicts
- shows attention, uses descriptive words and is
problem-oriented
- allows for thoughtful reflection

6. To the editor…..
A point well-taken but….
50 lines of text in 6 paragraphs for one issue:
‘…concerns with the use of internet-based sampling
panels for normative data.’
- ‘in my experience’ used 11 times
- be aware that ‘even with a representative
sample that constructing weights was (is) complex.’
- ‘worked with a professor from X with decades
of experience with surveys.’
- ‘discussed for decades by survey professionals.’

7. (Unintended) Bias in language
‘A few edits would help to make the presentation more American.’
OARSI is an international organization!
Most journals serve an international audience
English may not be a first language for many BUT even native
English speakers struggle for writing clarity at times
OAC is an English language journal and need sufficient quality of
writing for comprehension and clarity
Editor prerogative to ask authors to have their work edited
Reviewer can suggest to the editor that it might be helpful

8. Content of the Review

9. Context: editor perspective
• Innovation/new knowledge vs incremental
• Objectives, methods including analysis and
interpretation align
• Sufficient methodological rigour

10. Comments to the editor
• Chosen to review because of expertise and
knowledge of literature in the area
• Editor relies on you to address the
novelty/importance of the work
• Okay to confirm existing studies if limited work in
the area

11. R1: The paper by [xxx] is describing the results on healthcare costs out of a [xxx] trial. The results are well
analyzed and the conclusions are very clear, inclusive taking the limitations of the study into account. The
paper reads well and gives a clear view on the results obtained in the study.
Points to consider:
- Are there any data on pain medication available? This would add value to the results and would also give
further insight in costs post surgery
- On the results (line 188): Could the numbers of missing diaries and the numbers of excluded datasets be
provided?
Overall a well written paper with good conclusions.
R2: This is a very well written account. I have few issues (unusually) with the statistical presentation, and
any questions that arose were generally covered by the authors.
This is a very well written account of a well-constructed study.
Minor point only.
Participant reported data was included if 7/9 diaries were completed, and an imputation model used where
individuals are missing only 1-2 diaries. To what extent can missing data also be assumed in the diaries
which were completed, and how was this handled by the authors?
R3: This study examining the ninety-day and one-year postoperative healthcare utilization and costs
following total knee arthroplasty is a well conducted study. Manuscript is well written and is relevant to the
journal. I have no major issues and nothing much to add to this paper.
Happy Editors, Authors and Reviewers:
Short and sweet does it!

12. Context: authors
• Guidance to optimize the presentation and impact
of the work
- substantive vs clarifications in writing
- additional analyses to enhance interpretation
- ‘so what’ message
- key limitations missing that influence
interpretation

13. Substantive issues to
improve/optimize the work
Using administrative data….
I was surprised that no sensitivity analyses were carried out
concerning the OA diagnoses, particularly given the long time
span (1998 to 2003) over which exposure status could be
ascertained. Given issues around the making an OA diagnosis,
other researchers have recommended that algorithms
requiring more than one visit be required for studies using
administrative data. (See for example Rahman MM et al., Int J
Rheumatol 2O16; 6475318 which shows higher specificity and
PPV with at least 2 diagnoses within 2 years (or a hospital
admission) -a lit search will give a few other studies). If OA is
truly related to increased mortality, there will likely be
stronger relationships with increased specificity. Given the
long time frame over which OA was ascertained there should
be ample sample carry out such a sensitivity analysis.

14. Substantive issues cont…
Given the richness of the data sources used, this paper
also represents a missed opportunity. There are at least
two studies in the literature which show that women
with OA have higher risks of cardiovascular mortality
than men (see Rahman MM et al., Arthritis Care &
Research 2013;65:1951-1958. Schieir O et al., Arthritis
Care & Research 2016;68:811-818). These findings need
confirmation in other populations. I strongly suggest that
the authors of this paper explore and report on analyses
stratified by sex. If there is a substantial sex difference in
mortality risk (either overall or cardiovascular) this
clearly would have important implications.

15. Clarifications in writing and content
‘Please clarify if studies permitting participants to walk
with walking aids were included’
‘ Please clarify if studies with participants with confirmed
OA in lower joint other than the knee were included’
‘Please clarify is findings/studies from principle component
or principal pattern analysis were included’
What is the obvious overarching
clarification needed?
Does it need separate line by line
critique?

16. Structure of the review
• Avoid providing a line by line review

17. Missing content
• Thoughtfully review methods, results
and interpretation
• Do methods and results reporting match, e.g. data
reported as collected but not in results?
• Results not discussed?
Corollary…..
• Due diligence: Are you sure you aren’t asking for
something already in the paper?

18. Reviewer: The participants in these two cohorts are
highly educated. This limits the generalizability of the
study.
Response: We are unsure what the reviewer would like us
to do based on the above. The following is currently in the
limitations section of the discussion:
Our study has several limitations. A high proportion of
people in our cohorts were highly educated and this may not
reflect other cohorts. However, the Organisation for Economic
Co-operation and Development reports that Canada ranks first
in the proportion of people with higher education (i.e. more
than high school) with Canadian women representing the
highest proportion (33) such that generalizability to the
Canadian population is likely less problematic.
We have not made any revisions to the manuscript given we
have addressed this issue in the initial version submitted.

19. Setting up a debate
Providing thoughts/opinion that leads to debate
versus manuscript revisions are not helpful to the
author
‘I personally don’t like the concept of MCID. I’d
appreciate the authors thoughts/comments on this.’

20. Staying within the context of the
work
• Request for discussion of issue(s) that take work
beyond purpose and results of work
Context
- Systematic review of MCID and PASS values for the
WOMAC in total hip and knee replacement to
determine if values are similar across countries
- Method and sample heterogeneity such that
comparisons not possible
- Discussion focused on this and need for consensus on
methods

21. Staying within the context of the
work cont.
• Reviewer request:
We are in the era of precision medicine. Please
discuss the methods for this and how the
MCID fits with precision medicine, if at all,
given that we will be looking at interventions
only based on the individual’s outcome.
How do you deal with such a request?

22. Comments taking you outside
expertise
• Be sure of your knowledge, in the context of
the request to authors
• Look it up if you don’t know/aren’t sure

23. Figure 1: Density plots comparing distribution of pre-surgical WOMAC pain
for cohort 1 (03/2006-03/2008) and cohort 2 (11/2012-03/2015)
Cohort 2
Tertile 1 Tertile 2 Tertile 3
Cohort 1
Pre-surgery WOMAC pain
Density

24. From the journal editor….
‘Figure 1 is confusing. Could you please display this
information differently. A table with the descriptive
statistics would be easier for the reader.’
‘Use of bandwidth is unclear. Please avoid using
jargon.’

25. Criticism/concern but what to do?
Classic cryptic comments
‘The authors could shorten the Introduction/this section
of the manuscript.’
Page x, lines y-z: ‘The authors need to expand this
section.’
Page x, lines y-z: ‘The paper would benefit if more
information were provided.’
‘The title isn’t a good reflection of the work.’
‘The authors need to additionally discussion x, y, z topics
in the Discussion.’ (and the word count becomes?)
‘This work would be better split into two papers.’
‘I disagree with the conclusion.’ (with no justification)

26. Summary and Conclusion
Writing a thoughtful, respectful review:
- write as if you were the recipient editor or
author
- if important work to publish, provide
guidance to assist authors in optimizing the
approach, interpretation and discussion
- don’t ‘eat’ the authors

27. Thank you!