Slides based on preprint by Besançon et al, considering rush of COVID-19 research, much of poor quality

1. Open science saves lives:
Lessons from the COVID-19 pandemic
Lonni Besançon 1,2*, Nathan Peiffer-Smadja3,4, Corentin Segalas 5, Haiting Jiang6, Paola
Masuzzo 7, Cooper Smout 7, Maxime Deforet 8 and Clémence Leyrat 5,9
1Faculty of Information Technology, Monash University, Melbourne, Australia
2Media and Information Technology, Linköping University, Norrköping, Sweden
3Université de Paris, IAME, INSERM, F-75018 Paris, France
4National Institute for Health Research Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial
Resistance, Imperial College London, London, United Kingdom
5Department of Medical Statistics, London School of Hygiene and Tropical Medicine, London, United Kingdom
6School of Health Policy and Management, Nanjing Medical University, No.101 Longmian Avenue, Nanjing 211166, P.R.China.
7IGDORE, Institute for Globally Distributed Open Research and Education
8Sorbonne Université, CNRS, Institut de Biologie Paris-Seine (IBPS), Laboratoire Jean Perrin (LJP), F-75005, Paris, France
9Inequalities in Cancer Outcomes Network, Department of Non-Communicable Disease Epidemiology, London School of Hygiene
and Tropical Medicine, London, United Kingdom

4. Illustration of the problem
• First appeared on MedRxiv as a preprint
• Accepted a day after submission by the
Int. J. Antimicrobial Agents
• Concerns raised
• ISAC commissioned postpublication
reviews, published 4 months later
• One review concluded: "this study
suffers from major methodological
shortcomings which make it nearly if not
completely uninformative. Hence, the
tone of the report, in presenting this as
evidence of an effect of
hydroxychloroquine and even
recommending its use, is not only
unfounded, but, given the desperate
demand for a treatment for COVID-19,
coupled with the potentially serious side-
effects of hydroxychloroquine, fully
irresponsible."
• Paper not retracted, on the grounds that
it “gives room for scientific debate”.
Editor-in-Chief
Editor
Journal of International Society of Antimicrobial Chemotherapy
(ISAC), published by Elsevier

5. • Here analysed data on preprints and published articles, and on retracted
articles
• Since 31st December 2019, 80,000 preprints and peer-reviewed articles on
COVID-19
• Period of rapid change – important to ensure change is for the better
• If Open Science not embraced properly, danger of:
• Misleading information that could affect medical treatments
• Waste of resources
• Confusion from conflicting information

7. Problem 1: Methods/stats issues
29 COVID-19-related papers have been retracted*.
In 8 cases based on data analysis or study design
Problem 2: Duplication of research
No study registration means that people unaware of
ongoing studies
Bad for patients, if lots of little studies
Problem 3: Ethical concerns
• Over-recruitment in trials
• Human trials of vaccine before animal trials
• Neglect of adverse effects (e.g. cardiotoxicity of
hydroxychloroquine and azithromycin).
• Lack of consent (Gautret et al)
Problems
*We don’t know how this retraction rate compares to
other subject areas, though we do know
editors/publishers are very reluctant to retract
articles

8. Problem 1: Methods/stats issues
29 COVID-19-related papers have been retracted.
In 8 cases based on data analysis or study design
Problem 2: Duplication of research
No study registration means that people unaware of
ongoing studies
Bad for patients, if lots of little studies
Problem 3: Ethical concerns
• Over-recruitment in trials
• Human trials of vaccine before animal trials
• Neglect of adverse effects (e.g. cardiotoxicity of
hydroxychloroquine and azithromycin).
• Lack of consent (Gautret et al)
Problems
NB. Most issues they raise apply to papers on other topics.
Question of whether this study needed a ‘control’ topic – if so what
would it be?
Or is it sufficient to use rush of COVID papers to illustrate the case?

9. Study preregistration
• Helps avoid study duplication
• Makes it possible to check if study
conducted as planned – limits p-hacking
• But not peer-reviewed
Registered reports
Peer reviewer check of study design and
data analysis plans
Both of these allow relevant work to be
found for meta-analysis
Solutions

10. Study preregistration
• Helps avoid study duplication
• Makes it possible to check if study
conducted as planned – limits p-hacking
• But not peer-reviewed
Registered reports
Peer reviewer check of study design and
data analysis plans
Both of these allow relevant work to be
found for meta-analysis
Solutions
In psychology, we don’t tend to
worry about study duplication.
Should we?

11. Fast-track peer review
at some journals (inc. Wiley,
Elsevier, Sage, PeerJ)
Palayew et al.:COVID-19 mss
beween Jan-April 2020
median 6 days review
Problems
Who is going doing the peer
review?! Editors have big
problems finding reviewers during
pandemic
Re Surgisphere: ”The most striking consequence of this affair, however, is that it
may have made scientists, editors, readers, organizations
and reviewers waste precious research time, when a rapid response is needed.”
I.e. time saved in review process just costs much more time down the line.

12. Conflict of interest
(only focussed on editorial COIs*)
NB Categories not mutually exclusive. Also, unclear if this is COVID-specific
*Many people would NOT regard it as COI if editorial board member published in journal, OR if editor’s paper was
handled independently

13. Not specifically COVID-related
Who should police editors?
Are publishers doing enough?

14. Difficult to evaluate published
results, especially if code+data
not shared.
Example of Surgisphere –
paper claming
hydroxychloroquine increased
mortality in COVID-19 patients
Problems

15. Solutions
Open review – share reviewers report and author
response. Particularly important for rapid review cases.
Need for expert statistical reviewers
Specific recommendations:
• Authors should highlight editorial affiliations with
journal
• Explicit statement about how peer review
conducted
• Make referee reports available
• Systematic review of statistical methods
Post-publication peer review, e.g. Pubpeer
Problem that this is time-consuming but not rewarded

16. Solutions
Open data/open source
‘Data available on request’ is not enough!
Also need source code
4 retracted papers from Surgisphere would not have
been published if these policies were adhered to
Suggests that if data can’t be made public, need a
trusted 3rd party to establish the existence of the raw
data and to validate the results
Ensure links to pre-registration, repositories, are
available
Who should pay for setting up the ‘trusted 3rd party’?

17. Preprints treated as a reliable source
Problems

18. Gautret et al: 1,458 shares in the media
including 54 in the news (as of 13th July)
Created high demand for
a treatment that has not been proven
beneficial and has many potential harmful
side-effects.
Misuse of preprints may discourage
researchers from preprinting in future
Among 6 retracted preprints, the median
total of shares was 723 [49−2488]
N.B. Science Media Centre has long
been critical of preprints, and this kind of
problem has added to their concerns.

19. Solutions
• Advantages of preprints are too great to give up on them.
• Recommend that press releases should be made only with respect to
peer-reviewed studies
• Otherwise, journalists should search for potential COIs, and check
registered information, look for peer review of the work
• Journalists and scientists working together
• Need high quality journalism training
Assumes good faith by journalists – often under pressure from their editors
to create sensational stories, or pursue a political agenda

20. Questions for discussion
• What are the barriers to implementing recommendations by Besancon et al?
• Will there be unintended negative consequences of these recommendations?
• Which recommendations are most urgent/most achievable?