PICO annotation of intervention reviews:
a case study of Cochrane Airways
Elizabeth Stovold1, Deirdre Beecher2
1Cochrane Airways, Population Health Research Institute, St George’s, University of London, UK
2CET/Informatics & Technology Services, Cochrane, Milan, Italy
As part of the Cochrane Linked Data projecti, Cochrane Review Groups (CRGs) are
annotating their reviews using the PICO (population, intervention, comparison, outcome)
annotation tool. The aim of this is to make our research easier to find, share and reuse.
Cochrane Airwaysii was involved in the initial annotation pilot project in 2015, and
continues to work with the core PICO annotation team. Cochrane Airways has a portfolio
of approximately 400 reviews and protocols covering a range of obstructive airways
conditions. We are working towards annotating all our reviews at methods level,
together with the included studies in reviews published between 2015 and 2018.
To describe a process for annotating a portfolio of intervention reviews with PICO, using
Cochrane Airways as a case study
CRG Funding Acknowledgement: The National Institute for Health Research (NIHR) is the largest single funder of Cochrane Airways.
Disclaimer: The views and opinions expressed therein are those of the authors and do not necessarily reflect those of the NIHR, the NHS or the Department of Health.
We have described a process for managing the annotation of a portfolio of reviews, which involves prioritisation, identification of vocabulary issues, a small team of annotators. We hope sharing our
experience of undertaking this work will be of benefit to other Groups starting to annotate their reviews.
Completing this work will make it will easier for patients and consumers to search for and find research of interest, in particular information about review and study outcomes. In the future, this work
will also help to link reviews with guidelines
• Pilot phase: we chose a single topic (asthma) to annotate at methods
level. We then started the next topic (COPD).
• Post pilot: annotations are prioritised as follows:
1. All new protocols published 2018 onwards at methods level
2. All new reviews/review updates published 2018 onwards at
3. Included studies in reviews/review updates published 2018 onwards
4. Reviews/review updates published 2015-2017 at methods level
5. All other reviews (publishedbefore 2015) at methods level
6. Included studies in reviews/review updates published 2015-2017
• Results: total completed annotations
Methods level Study level
367 complete 394 complete
• We have inserted a task into the Airways editorial workflows in Archie
1. Annotation at methods level following publication of new protocol
2. Annotation of methods & study level following publicationof new
3. Annotation of methods & study level following publicationof new
• Results: completed annotations for 2018 publications
Methods level Study level
26 annotations complete 75 annotations complete
(from 10 reviews/review
for new protocols,
reviews & updates
• We have set up a shared google spreadsheet to record the annotation
status of each review/protocol
• We use the following categories to indicate the status of a review
All complete; complete at methods level; Blocked; In discussion;
Withdrawn; superseded; waiting for publicationof full review; Not for
annotation (DTA; overview; prognosis; qualitative)
• Results: tracking status of annotations
55 All complete (methods & studies)
298 complete at methods level
8 in discussion
7 waiting for publication of full review
10 not for annotation
• We have set up a shared google spreadsheet to record preferred terms
for each PICO element:
• Vocabulary issues are reported to the Cochrane PICO annotation team.
Examples of these are:
• Missing terms
• Duplicate terms
• Orphan terms (not linked to a broader or narrower term)
• Results: all vocabulary issues
5 no action required
• Annotation by information specialist (IS):
• No QA required.
• Vocabulary queries are discussed with a colleague, or posted on the
Slack annotation channel to seek suggestions from other annotators.
• Annotations with queries are marked as ‘in discussion’until resolved.
• Annotation by other annotator:
• QA required by the IS.
• If no changes or minor changes needed corrections are made by the
IS and the annotation is completed.
• If there are major changes, they are marked for reannotation, with
notes, and the annotator is notified.
• Results: the team
1 information specialist
3 additional annotators: from the Complementary Medicine
Field; Cochrane Airways editorial base; and a review
author/member of the Cochrane central executive team
The team – working
Process & Results
i. Cochrane Linked Data. https://linkeddata.cochrane.org/home
ii. Cochrane Airways. https://airways.cochrane.org/