Cochrane Present Tech - Cochrane Future Tech


Published on

Ida Sim, from #CochraneTech Symposium, Québec 2013

Published in: Health & Medicine
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • It's an honor to be here, at Cochrane's 20thanniversary. The first 1000 Cochrane reviews took 8 years, until 2001, to be completed. 20 years on, there are now 5665 total reviews. On the one hand, this is outstanding progress, but on the other, it's not nearly enough. In 2003, Mallet and Clarke estimated that at least 10,000 systematic reviewers were needed just to cover the existing evidence at that time, in 2003. They projected that Cochrane would produce these 10,000 reviews sometime between 2010-2015. But being only about half way there, that projection is unlikely to be met. And of course, now many thousands more systematic reviews are needed to cover the tens of thousands of additional RCTs that have been published since 2003. It isn't news to you this mismatch between demand and production of Cochrane reviews, nor is it news to you that the problem is getting worse. The hope is that technology will come to the rescue. ~~~~~~~~~~
  • The question for today's conference is …I'd like to consider this question in 2 parts.One is what we can do with technology in the next 1-2 years, Cochrane Present Tech; and the 2nd part is Cochrane Future Tech for 3-5 years out.
  • Let's set the stage. A good practitioner of EBM always starts with a clear question. Considering Cochrane as an intervention, what is Cochrane's PICOT question?
  • First, the intervention is synthesized evidence generated by the Cochrane Way, which we'll say more about later.
  • Cochrane evidence is primarily targeted to health care systems, and clinicians.
  • to improve population-level health outcomes and cost.
  • The main comparison to Evidence-Based Medicine the Cochrane Way is Eminence-basedmedicine, which is self-descriptive.
  • And thetimeframe? You've heard that it takes an average of 17 years for clinical trial results to be put into practice. Whatever the delay is for primary studies, the delay is even more for systematic reviews. I couldn't find any studies on the time it takes for Cochrane reviews to impact care, but it's probably "too many years."
  • So this is the PICOT question for Cochrane Present. Cochrane Future is coming fast, but let's leave that aside right now, and look first at how technology can help the Cochrane in the next 1-2 years.
  • Here's a high level workflow of systematic reviewing, with a Cochrane Way to doing each of these steps. I know you're all very familiar with this so I'll just highlight a few points. First, there is a lot of iteration in the early steps, where a question is initially framed then revised in response to the details of the studies that are available. Second, there's a loop for updating and revising the reviews. Finally, that Dissemination and Implementation is often after and divorced from the process of conducting the reviews.
  • In the lightning talks later this morning, we will be hearing about technology being used for specific pain points in this workflow, like citation screening, supporting collaboration and supporting Dissemination and Implementation. There are many other technology projects we could discuss for other pain points here, but the elephant in the room is that things are changing, and you might be wondering how we should solve today's pain points in a way that will remain relevant for tomorrow.
  • Because the PICOT question for Cochrane Future will be quite different from today's. The interventionwill still be the Cochrane Way, which we will have to re-imagine.
  • For outcomes, cost is becoming an ever bigger driver.
  • This has implications for the type of evidence we want. The biggest driver of health care costs is chronic diseases. To reduce costs, health care systems are going to where the money is, and are focusing on chronic care. And because by definition chronic disease is 24/7 in patient's daily lives, and 1/3 of deaths are due to poor health behaviors that only patients can change, health care systems need to engage patients in their self-care.Nowadays, patients are savvy consumers and expect personalized medicine, and they will want evidence at the individual-patient level to help them self-manage.
  • There will thus be more demand for "personalized evidence," from patients and their families, for individual-level estimates of treatment effect.
  • As health care systems increasingly focus on self-care and improving personal behaviors, personal digital technologies like apps and sensors will play a large role in transforming health care. These technologies evolve rapidly, with a "metabolic rate" that is like nothing we've seen before. Here's a figure from a very nice paper by Riley and colleagues showing that a typical RCT published in 2012 was probably designed and submitted for funding in 2006, when the Wii came out, and it spans the introduction of the iPhone, Android, iPad and Siri by the time it's analyzed and published.The pace of traditional research is simply too slow, and is also not compatible with the US Institute of Medicine's goal of a continuous learning health system, where care and research drive each other in tight iterative feedback loops.
  • So the time frame for Cochrane Future is going from Too many years to continuous.
  • And finally, the comparison intervention for Cochrane won't be Eminence-based medicine anymore, but Big Data. The buzz around Big Data seems to take correlation for causation, that simply by having massive amounts of EHR data together in the cloud, we can somehow get all our answers. Clinical trials, in contrast seem so much less modern, yet we know that careful methodology is more not less important for interpreting Big Data and especially for synthesizing inferences drawn from Big Data.
  • I'd like to illustrate these dimensions of change through three projects I'm working on.
  • The first is the PREEMPT Project which addresses the highly prevalent problem of chronic pain. This is a frustrating problem to treat because there are few studies on which pain meds are better, and the studies only yield population-level estimates, which is what we get from standard RCTs.
  • In our project, a chronicpatient sits down with their doctor to select the pain regimens they want to compare, for example…
  • define the length of time on each treatment – 2 weeks -- and the number of times they'll take each treatment – 3 times. This would be a 6-week n-of-1 trial.
  • and they can customize the outcomes to what is most relevant to them. For example, opiates can dull our thinking and concentration in different ways for different patients. In this project, we allow patients to select the cognitive symptom that is most relevant to them, like . Immediately after setup, the patient can download the Trialist app onto his/her cell phone which guides them through their custom n-of-1 study, 6 weeks in this case.Allowing patients to choose their outcomes is patient-centered, keeps them engaged in continuing the study, and gives them evidence that THEY want and will find useful. Of course, only some clinical questions can be answered with n-of-1 studies but this is a valuable study design that is much more feasible now with new technology.
  • n-of-1 studies can be even more powerful if we go from just one study
  • to many n-of-1 studies, so that more people can get individual estimates of treatment effect,
  • to summing the evidence from these studies for example using Zucker'sBayesian meta-analysis methods
  • So n-of-1studies illustrate some differences from the traditional RCT. First, it is conducted as an integral part of clinical care. Patients are full partners, participating in defining the question, choosing the treatments to compare, and the outcomes measured. The results are estimates of individual-level effects, which can be aggregated to yield estimates of population-level effects.
  • My second example is a project that Open mHealth is doing with Kaiser San Diego to improve the self-management of diabetes in patients with depression. It's a very complex intervention:Patients track their glucose, BP, weight and activity using various wearable sensors. They self-report depression and other symptoms on the cell phone, behavioral and social metrics are collected in an app called, a company called FrameHealth delivers incentives and motivations based on a psychological profiling of the patient. All this data is put together with EHR data to support self-care, family and friends care, and the Kaiser care team.
  • These kinds of multi-modal interventions are going to become more common for managing chronic diseases and indeed for shaping the entire health care system. We desperately need evidence of what works and what doesn't, but these complex interventions are made up of many moving parts that should be evaluated as they are being designed, developed, and deployed. For example, did you know that Google runs hundreds of little RCTs every day on little tweaks to their search algorithm and interfaces? They randomize you to getting this or that type of result or a font or a placement of an ad, and after oh several hours, they've accrued several hundred thousand subjects and a result that they use immediately to keep improving their search site. eHealth interventions need to adopt this agile development and evaluation approach. This type of evidence is valuable to synthesize and share to improve the eventual effectiveness of eHealth. The analogy is to drug development. If we only shared Phase 3 studies of drugs, we would be missing out on very useful evidence earlier in the drug development pipeline, about toxicity and pharmacokinetics for instance. It just that the Phase 1 and 2 studies for eHealth will be small, fast, and very iterative.
  • The differences in this example are
  • My final example is the Health eHeart project, which you can think of as a "virtual Framingham" cohort study. We are aiming to enroll 1,000,000 patients worldwide who have or are interested in cardiovascular disease.
  • We are capturing from these patients: What's different about this study is that we're trading carefully obtained, precisely phenotyped data on fewerpatients for messier data on orders of magnitude more patients. Clearly, this approach opens up exciting possibilities for research but also introduces many methodological concerns.
  • Given all these changes, what will the future Cochrane look like? Because we're in Canada, and because I'm Canadian and from Edmonton,
  • I have to quote Wayne Gretsky:
  • Grossly simplified, this is where the puck is today. We have PDFsreporting on studies completed years ago, and Cochrane converts that to Forest plots in PDF, by drawing the most sound inference from the totality of the available evidence.
  • This is where the puck willbe in 3-5 years, The challenge still is to draw the most sound inference from the totality of available evidence, but both the input and the outputs will be different, as shown by our examples. like in n-of-1, the results will be more personalized, and the users will want synthesized evidence FOR THEM INDIVIDUALLYlike the Kaiser project, data will be large, disparate, very personal data, generating clinical and non-clinical evidence with varying degrees rigor. Evidence synthesis will need to accommodate this range of rigor, we can't just stick with standard RCTs, and to help users understand and adjust for HeH, large less precisely controlled studies that trade numbers for precision, drawing in many cases on the promise of Big Data. And patients and the health care system will want evidence on a continuous release schedule. The Cochrane Way will have to change to meet these new expectations. It's not clear how, but I think there are some basic principles that Cochrane should build to on the way there.
  • These basic principles are data science related, operational and methodological.
  • On the data front, we have to move beyond PDFs, and beyond a singular focus on results reports. We need to embrace data publishing as I'm sure John Wilbanks will be talking about at lunch. We need to publish study protocols as computable models, and publish results as computable data, so we can automate as much of the workflow as possible. We need to use text mining to extract meaning from prose, because there will always be, and should always be, prose for humans to read.An absolutely critical principle is that data cannot be in silos if we want to get the most out of evidence to improve health. We should pursue linked open data, again John will discuss, and the context for this data needs to be fully described. That is, we need metadata, or data about data. A simple example is a data point, let's say an odds ratio. The metadata, or context, for that odds ratio may include whether that odds ratio came from an RCT or an observational study. You can see that without good metadata, naked data can be downright dangerous. We also need ontologies. You can think of ontologies as standard computable ways of describing more complex abstractions, like what is a study question (so we can index and search on it), or my work on an Ontology of Clinical Research to describe study designs, or the Cochrane Ontology's standard description of the parts of a systematic review. I know this is all really abstract so I'll boil it down to a goal:
  • The overall goal is to capture data in clearly described computable form that we can repurpose for multiple needs today, and for unknown needs tomorrow.
  • Operationally for Cochrane Future Tech, a big challenge is to know what to have computers do, and what to leave humans do. Systematic review work is a mix of brain-numbingly tedious work mixed with highly skilled methodology work and clinical insight. The best technology solutions are going to be well-designed hybrid human-computer systems, which means seeking out and working with our colleagues in computer science, cognitive psychology and design t build the best hybrid systems. User interaction design is important. e.g., ExACT system.Part of this challenge is to support distributed collaborative knowledge work, tapping into crowds.
  • Finally, there are many methodological frontiers for Cochrane Future. One is methods for generating more precise individual-level treatment estimates. Another is systematic review methods for complex interventions and continuous evaluation methods. Because the Big Data world will be full of messy, biased, confounded data, we need critical appraisal methods that can scale to the volume of evidence we will see. This will likely be semi-automated methods. I'm sure you can think of more.
  • As I see it, these the points to keep in mind for the ambitious notion of a Human Evidence Project.First and foremost is to use technology not as an end to itself, but to use it to combine together human work and expertise, data and metadata, and computational methods to generate the most sound invidiual and population-level summary evidence for continuous learning.Mapping these basic principles to the workflow, we need ontologies of study questions and protocols to help with question framing, linked open data and text mining for assembling studies, we can say good riddance to abstraction if we publish in data and nanopublications, we need new statistical methods, and ontologies for diseases, care processes, decision making, etc. for effective D & I. Cochrane might not take on this whole challenge, but I can't see any other group besides Cochrane that can or should take the lead on this grand vision.
  • In conclusion,
  • Cochrane Present Tech - Cochrane Future Tech

    1. 1. Cochrane Present Tech Cochrane Future Tech Ida Sim, MD, PhD University of California San Francisco Open mHealth September 17, 2013
    2. 2. • In 2003, estimate of 10-45,000 reviews needed to cover existing evidence as of 2003 – projected Cochrane to hit 10,000 reviews between 2010 and 2015 Mallet and Clarke, EBM 2003(8):100-1 Total Protocols Total Reviews Total Updated Reviews1000th Cochrane review 5665 total reviews today
    3. 3. How can we better leverage technology and knowledge to both help us prepare systematic reviews more efficiently but also deliver the outputs better to our end-users?
    4. 4. Setting the Stage: PICOT P opulation I ntervention C omparison O utcome T ime frame
    5. 5. Intervention P I Synthesized evidence, the "Cochrane Way" C O T
    6. 6. (Target) Population P Health care systems, clinicians I Synthesized evidence, the Cochrane Way C O T
    7. 7. Outcome P Health care systems, clinicians I Synthesized evidence, the Cochrane Way C O Population-level health outcomes & costs T
    8. 8. Comparison P Health care systems, clinicians I Synthesized evidence, the Cochrane Way C Eminence-based medicine O Population-level health outcomes & costs T
    9. 9. Time frame P Health care systems, clinicians I Synthesized evidence from the Cochrane Way C Eminence-based medicine O Population-level improvement in health, costs T Too many yearsT
    10. 10. Setting the Present Stage Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many Cochrane Future P I C O T
    11. 11. Cochrane Way: Workflow
    12. 12. Technology for Pain Points
    13. 13. Setting the Future Stage Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P I Cochrane Way C O T
    14. 14. Additional Outcomes Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P I Cochrane Way C O cost T
    15. 15. Chronic Diseases Drive Cost • 46% of morbidity and 59% mortality worldwide due to chronic diseases1 • Health systems targeting chronic care – 1/3 of deaths due to poor health behaviors – need to engage patients in self-care • Patients expect personalized medicine – want evidence at the individual-patient level WHO | Facts related to Chronic Disease
    16. 16. Additional Outcomes and Population Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P add patients & families I Cochrane Way C O add ind-level health T
    17. 17. Time Frame • Personal digital technologies will play large role in chronic disease and transforming health – 20,000 health apps on iTunes, 8,000 on Google Play – technologies evolve rapidly • US Institute of Medicine goal of a continuous Learning Health System Riley et al. Clinical and Translational Medicine 2013, 2:10
    18. 18. New Time Frame Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P add patients & families I Cochrane Way C O add ind-level health T continuous
    19. 19. New Comparison Intervention Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P add patients & families I Cochrane Way C Big Data O add ind-level health T continuous
    20. 20. Three Illustrative Projects • N-of-1 studies for chronic pain • Kaiser: Diabetes and depression management • Health eHeart virtual cohort
    21. 21. PREEMPT Project • Chronic Pain is highly prevalent (>100 million Americans) and difficult to treat • Few studies on comparative effectiveness of analgesics, yielding only average population-level estimates R01-NR013938, PI R. Kravitz 50 people 100 people oxycodone Pain frequency, intensity 50 people hydrocodone population Pain frequency, intensity
    22. 22. none of us are average hydrocodone betteroxycodone better
    23. 23. which works better for you? N-of-1 study design: within-subject crossover design Kravitz, et al. Contemp Clin Trials 2009; 30:436-445 BPI individual pain intensity You pain intensity oxycodone hydrocodone hydrocodone oxycodone oxycodone hydrocodone
    24. 24. there you are! hydrocodone betteroxycodone better
    25. 25. n = 1
    26. 26. (n 1).N
    27. 27. Zucker DR et al. J Clin Epidemiol. 2010;63(12):1312-23. (n 1).N Σ
    28. 28. flip direction of research inference population
    29. 29. Differences from Traditional Research • Occurs in the course of clinical care • Patient participation, addresses patient questions • Patient choice in outcomes measured • Individual treatment effect • Aggregate to population-level effect
    30. 30. S. California Kaiser Complete Care Improving self-management of diabetes in patients with depression
    31. 31. Continuous Evaluation • eHealth interventions will play a substantial role in chronic care management and in shaping health care systems • Evaluations need to occur while they are being designed, developed, and deployed. Catwell and Sheikhl, PLoS Med 2009; 6(8):e1000126 Usability Studies Preliminary effectiveness User Requirements Pilot Field Testing Rigorous effectiveness
    32. 32. Differences from Traditional Research • Evidence is needed on intermediate non- clinical outcomes (e.g., effective design features) as well as end clinical effectiveness • Many of these evaluations will not be published in academic journals
    33. 33. • Enrolling 1,000,000 patients, capturing – self report data: food intake, mood – sensor data: weight, BP, activity – social data: Facebook – EHR data: including text mining for CV events – biospecimens and CV tests for San Francisco area patients • Trades precise data on fewer patients for messier data on 1000x more patients – will be supplemented with targeted data collection for specific studies
    34. 34. Future Cochrane Way? Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P add patients & families I Cochrane Way? C Big Data O add ind-level health T continuous
    35. 35. "A good hockey player plays where the puck is. A great hockey player plays where the puck is going to be." Wayne Gretsky, Edmonton Oilers
    36. 36. Where the Puck is Today Studies in PDF Forest plots in PDF Cochrane Way Draws the most sound inference from the totality of the available evidence
    37. 37. Where the Puck will be Evidence = study protocol + results Synthesized evidence continuous, multi- modal, personal data more personalized questions large, less controlled studies individual-level evidence range of acceptable rigor continuous release schedule Cochrane Way Drawing the most sound inference from the totality of the available evidence
    38. 38. Build to Basic Principles • Data sciences • Operational • Methodological
    39. 39. Data Sciences • Beyond PDF reports of study results – embrace data publishing • publish protocols as computable models • publish all results as open computable data – text mining to extract meaning from prose • No data silos: linked open data • Describe the data: metadata and ontologies for – study questions – study design (Ontology of Clinical Research) – systematic reviews (Cochrane Ontology)
    40. 40. Data Sciences Goal • Capture open data in clearly described computable form that can be re-purposed for multiple needs today, and unknown needs tomorrow
    41. 41. Operational • Pursue hybrid semi-automated approaches – seek out and work with the best designers of human-computer interaction • Support distributed, collaborative knowledge work, tapping into crowds
    42. 42. Methods Needed • Heterogeneity of treatment effect • Continuous evaluation methods • Large-scale assessment of biases and confounders • Better decision support for assessing biases and confounders • ...many more
    43. 43. The "Human Evidence Project" + + Methods Data Metadata + Most sound individual and population-level summary evidence for continuous learning
    44. 44. Conclusion • How to get from Cochrane Present to Cochrane Future? – new methods built on open data and ontologies – design hybrid human-computer systems • In new Big Data world, methodologically sound but pragmatic, sustainable evidence synthesis will be critical • Need Cochrane now more than ever