Sustainability and Cochrane Reviews: How technology can help


Published on

My plenary talk at the UK Contributors' Meeting held in Loughborough, UK, 20-21 March 2012

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Improvements in internal process, of course, affect consumers of our information downstream. I see sustainability as working both ways – internally via improved production process, priority-setting, etc. and externally in the health info marketplace reaching our audiences and changing health care. But, of course, “internal” involves working with partners, other orgs, etc.
  • From the intro to the Collaboration’s information technology strategy paper. Indeed, we were the first all-electronic medical journal. We were never in print and technology has advanced our cause. BUT, are we using technology to the betterment of our cause NOW? So, BUT....
  • Estimate is now nearly 10 years old!
  • Some technologies we’re already using, and they’re helping...
  • Refer to Phil Alderson talk, if possible...
  • The red text indicates where in the Review XML the data is extracted from, not that you can read this!
  • A mockup of the final product
  • So, in the data model and ontology we‘ve been working on in the Star Trek program of work (details to follow!), we‘re looking at drawing together the links between the PICOs. So, pulling together data via queries to easily answer questions like: “What‘s been compared to what and in what population for what outcome, etc.??“
  • So, could we use linked data technology to assist in the first step in doing overviews and network MA, drawing the map and visualizing the geometry of the interventions and comparisons...
  • Nice segue into...
  • Simple example with search...
  • First, we‘ll need to start thinking outside of the “container“ of the Review...
  • When data from Reviews is removed its context within the Review, we need to be careful that the context „travels“ with it...
  • Some of these developments require us, an organisation, to think differently about our content. The one-size-fits-all “container“ of the Cochrane Review will need to be flexible in order for use to meet new user demands and to allow for content that travels freely (any device, any platform, any context), retains its context and meaning so that people know they can trust it and allows us to create new “products“ to meet these new user demands.
  • The technologies that underpin this can be used at the level of the web itself AND within and across individual datasets. We are investigating both.Current web = web of documentsData in documents (mostly HTML/XML, etc.) not structuredLinked data allows for structuring data so that both humans and computers can understand itFor example, Cochrane Review XML is highly-structured but relationships not explicitIf they were, we could query across dataset and link other datasets for complex queriesThe web turns into a giant database (the vision, anyway)Search results display can be improvedContent enriched with external data and re-packaging of our dataMany other possiblitiesCurrent web = web of documentsMostly in HTML/XMLGood for telling computers how to display informationNot good for embedding meaning and relationships between bits of informationLinked Data technologies allow for structuring data so that both humans and computers can understand itIt’s all about structured data
  • Every presentation about the web and technology nowadays has to include a cloud image!!
  • The CRS is an exciting new project that could facilitate better linking of data. (Refer to something Gordon said in his talk, if possible/applicable).
  • Explain Star Trek joke and general framework here...
  • Explain Star Trek joke and general framework here...
  • We use diagrams like this to conceptualize the relationships. This is the findings ontology developed by Lorne Becker.
  • This is Rachel Marshall from the CEU’s work on modeling the PICOs and understanding the concepts and relationships between Reviews and studies...
  • Here is the Marshall-o-gram with GRADE and RoB per outcome modelled...
  • Reviews include comparisons. Comparisons include outcomes that are compared. Etc. This is by no means complete but was meant to assist in a “proof of concept“ exercise we called “interrogating the XML“ of Cochrane Reviews.
  • These questions build on each other in increasing complexity...
  • Just so you can visualize the potential application of this: In this Pubmed search, each Cochrane box has pointers to the Cochrane review(s) that included that trial.
  • Just so you can visualize the potential application of this: In this Pubmed search, each Cochrane box has pointers to the Cochrane review(s) that included that trial.
  • What sorts of bias are most prevalent in this particular body of research/clinical question of interest?
  • We could look at answering these kinds of questions which involve external datasets and mashups...
  • Cochrane could take the lead and model the entire knowledge space of Evidence-based Health Care in these semantic standards and create a giant triple store with our data. Then, others in the EbHC would use our ontologies and refer to our data and thus we would drive “the conversation” around the data.
  • Then, once we start throwing in other datasets, the triple store becomes even more powerful (note: not sure I drew all lines between all datasets, but you get the idea)…
  • For example, Volkswagen have done this for the car industry. They modeled the domain of car options with a car options ontology and are now positioned with “first mover” advantage in the car industry in leveraging semantic technologies.
  • This crazy image is the Linked Data cloud which shows all the various datasets currently in the web of data. The pink area is the life sciences area and includes PubMed, DrugBank and others.
  • Sustainability and Cochrane Reviews: How technology can help

    1. 1. Sustainability & Cochrane Reviews How technology can helpChris MavergamesDirector of Web DevelopmentThe Cochrane Collaboration
    2. 2. Structure of this talk• Sustainability of Review production (internal sustainability) ▫ Technologies to assist Review production ▫ PICOtron and Overviews of Reviews• Sustainability in the health information marketplace (external sustainability) ▫ “Nimble” content and thinking outside Review “container” ▫ Linked data and Star Trek• Summary
    3. 3. Information Technology Strategy“Information and Communication Technology (ICT) is at the heart of The Cochrane Collaboration ... To a very large extent, the success of The Cochrane Collaboration has been based on its investment in ICT.” - From the Collaboration‟s information technology strategy paper (ISSF)Our data and systems are great, but...
    4. 4. Can we make themachines work a bit harder?
    5. 5. Some critical points• For internal processes ▫ How to automate w/o creating more work ▫ Important that methodology remains sound• For external consumers ▫ Retain context of findings and results ▫ Create better understanding, not confusion
    6. 6. Sustainability ofReview production (and updating)
    7. 7. Producing Reviews - challenges• We have this incredibly complex, methodologically rigorous process and aim to be global and comprehensive with a mostly-volunteer workforce• We‟ve just cleared 5,000 reviews and its taken nearly 20 years, and these need to be maintained and updated• Estimates say we need min. 10,000 Reviews to be comprehensive (estimate 10 years old!)
    8. 8. Cochrane Reviews are fantasticbut...• …creating them is a long and laborious process
    9. 9. Cochrane Reviews are fantasticbut...• …creating them is a long and laborious process Help!
    10. 10. Review production assistance• Reference managers ▫ EndNote, Zotero, Connotea, Mendeley and others• Screening/appraising references ▫ ScreenToGo App (email: for public beta)• Automated abstract screening/appraisal ▫ Semi-automated appraisal Support Vector Machines (SVMs)• Software covering multiple steps in systematic review production ▫ EPPI-Reviewer ( ▫ Distiller SR ( ▫ EROS (Early Review Organizing Software ▫ Automated data extraction from Cochrane Reviews (reading the XML)• GRADEpro• CRS Slide courtesy: Rachel Marshall
    11. 11. What if authors could...?• Pull in Risk of Bias assessment information already done on trials or studies• See studies already included or excluded in other Reviews• Other links between studies and Reviews via the CRS – more later in Star Trek section• Use datasets like Drugbank and Diseasome for auto-completion or help filling out fields in RevMan for standardisation
    12. 12. Could we use technology to assistwith…?• Using MeSH mappings to show coverage of Reviews• Using study links to show where gaps exist• Priority-setting• Creating derivative “views” or products from Cochrane data
    13. 13. PICOtron
    14. 14. PICOtron and Cochrane ClinicalAnswers• Project to semi-automate populating Cochrane Clinical Answers, a derivative product of The Cochrane Library• Python script written by Iain Marshall that extracts data from Cochrane Reviews• Script automatically and randomly reconfigures Review titles into one of 30 question formats
    15. 15. PICOtron• Data is pulled from various areas of the Review XML• Depending on whether the result is significant and favored intervention, a sentence (narrative result per PICO) explaining these results is automatically generated• Then, an author checks these and combines this information across PICOs into a “clinical answer“
    16. 16. PICOtron output
    17. 17. Overviews of Reviews &Network Meta-analysis
    18. 18. Help in preparing of Overviews of Reviewsand Network Meta-analysis driven by linked data Image from Lorne Becker
    19. 19. Example of a trial level paroxetine synthesis sertraline citalopram 12 new generation escitalopram fluoxetine antidepressants: fluvoxamine Which ones are the milnacipran most efficacious? venlafaxine reboxetine bupropion mirtazapine duloxetineThanks to Georgia Salanti. Source:
    20. 20. Network of Randomized Trials sertraline milnacipran reboxetine paroxetine mirtazapine duloxetine fluvoxamine escitalopram citalopram bupropion venlafaxine fluoxetineThanks to Georgia Salanti. Source:
    21. 21. Network of Randomized Trials sertraline milnacipran reboxetine No trials paroxetine comparing mirtazapine reboxetine to bupropion duloxetine ? available in Cochrane Reviews fluvoxamine escitalopram citalopram bupropion venlafaxine fluoxetineThanks to Georgia Salanti. Source:
    22. 22. Other ideas?Perhaps we can gain insight into how toimprove Review production processes fromhow our users actually use our output? segue ...
    23. 23. Sustainability in the health information marketplace
    24. 24. “The problem is not information overload but filter failure.” – Clay Shirky• Cochrane provides a vital curation, filtering and evaluation/quality-assessment role and we need to make this clear
    25. 25. Again, Cochrane Reviews arefantastic BUT...• There are problems that limit their use by some people ▫ Difficult to wade through all of the text ▫ Difficult to understand the figures, terminology, and other bits of the Review ▫ Hard to compare interventions without reading multiple Reviews ▫ Moving from studies in CENTRAL to Reviews that included that study difficult ▫ Can be difficult to find the Review you seek
    26. 26. • Search for “Prozac” – no reviews• Search for “fluoxetine” – 27 reviewsSearching The Cochrane Library
    27. 27. Ideally, we‟d restructure ourcontent for different users• Beginning to do this now: ▫ for consumers ▫ Cochrane Clinical Answers for clinicians• BUT ▫ Takes a lot of work to reformulate reviews & authors, CRGs, etc are busyWouldn‟t it be nice if we could automate or semi-automate some of these processes?
    28. 28. Thinking outside Review container
    29. 29. Thinking outside Review container
    30. 30. Making our content “nimble”Structured and linked data can help makeour content “nimble”Nimble content can: • Travel Freely • Retain Context Meaning • Create New Products - R. Lovinger, Razorfish
    31. 31. Linked data
    32. 32. What is linked data? Semantic Web is made up of: Linked Data & Web of Data Which all together comprise Web 3.0
    33. 33. Current web = Web of documents
    34. 34. Docs are linked not data in docs
    35. 35. Machines aren„t good at readingweb pages• Data on the web is meant for human consumption• Machines need the data to be structured• Once structured, information can be more easily shared within datasets and across web pages
    36. 36. It„s about...• Taking the complex relationships, interactions and dependencies in our data and modeling them in semantic web language and concepts for machine processing...• So that we can do things like: ▫ Gain insights into our data ▫ Help with priority setting ▫ Repackage it for different users ▫ Later, perhaps machine can infer new knowledge from our data and/or when our data is combined with other datasets
    37. 37. Cochrane Register of Studies
    38. 38. CRS and CENTRAL• Lack of unique study IDs a real problem• CRS solves this by providing a unique ID for all studies that can be referenced• Better linking of data about trials and to Reviews ▫ Example: Using forest plots to generate related studies lists for CENTRAL• Possibilities with linking to external sources such as PubMed
    39. 39. Insert witty Star Trek reference here!
    40. 40. Findings ontology Image: Lorne Becker
    41. 41. Marshall-o-gram
    42. 42. Marshall-o-gram
    43. 43. Cochrane Review ontology
    44. 44. We can…• Ask questions that use data from several different reviews• Improve search• Make it easier for people to find Cochrane Reviews• Link data from studies and Reviews better• Enhance the experience of our users by including data from other datasets
    45. 45. A question using multiple reviews I’ve done a search for trials on a particular intervention for dementia. I want to know which of the trials have been included in a Cochrane Review and a summary of the risks of bias for the entire set of trials.
    46. 46. Links to the relevant Review forthose trials that were included
    47. 47. This study was one of 38 studies included in the Cochrane Review, <Title Here>. Click here to see the full reviewLinks to the relevant Review forthose trials that were included
    48. 48. RoB Summary for Cochrane Reviews ondementia These figures summarize Risks of Bias from the trials included in the reviews in your search
    49. 49. Enhancing the User Experience Make search work better
    50. 50. You Say “Paracetamol”I Say “Acetaminophen”• Or, one could say any of these:Abenol (CA), Acephen, AnadinParacetamol (UK), Apo-Acetaminophen (CA), Aspirin Free Anacin, Atasol (CA), Calpol (UK), Cetaphen, Childrens Tylenol Soft Chews, Disprol (UK), Exdol (CA), Feverall, Galpamol (UK), Genapap, Genebs, Infants Pain Reliever, Mandanol (UK), Nortemp, Pain Eze, Panadol (UK), Robigesic (CA), Silapap, Tycolene, Tylenol 8 Hour, Tylenol, Tylenol Arthritis, Uni- Ace, Valorin
    51. 51. Cochrane leading in Web 3.0?
    52. 52. CRS/ CDSR CENTRAL HTAs DARE CMREbHC Semantic Platform
    53. 53. CRS/ CDSR CENTRAL UMLS Drug Bank Diseasome HTAs DARE Symptom CMR * BBC Health Ontology OntologyEbHC Semantic Platform * Not yet created
    54. 54. Cochrane and EbHC ontology?
    55. 55. Will Cochrane have a bubblehere someday?
    56. 56. Summary• Technology IS at the heart of what we do• For both internal and external applications, we can leverage these tools to further our mission• Requires that we think differently about the “container“ of the Review• Our data needs to become “nimble“ to meet future user needs• We should proceed slowly, incrementally - What are the “quick wins“?• Cochrane has the chance to lead in Web 3.0
    57. 57. Thank you