Downloaded 12 times
Uploaded byCochrane.Collaboration
Cochrane Present Tech - Cochrane Future Tech
The document discusses leveraging technology to improve the Cochrane review process and better deliver evidence to end-users. It proposes moving from a traditional review model to a continuous learning model that incorporates individual patient data, produces more personalized evidence, and accounts for rapidly evolving digital technologies and health apps. Specific ideas include using n-of-1 trials for chronic pain, evaluating eHealth interventions during development, and conducting large virtual cohort studies using diverse data sources. The goal is to generate the most sound individual and population-level evidence through open, linked data and continuous review methods.
More Related Content
Similar to Cochrane Present Tech - Cochrane Future Tech
Cochrane Present Tech - Cochrane Future Tech
- 1. Cochrane Present Tech CochraneFuture Tech Ida Sim, MD, PhD University of California San Francisco Open mHealth September 17, 2013
- 2. • In 2003,estimate of 10-45,000 reviews needed to cover existing evidence as of 2003 – projected Cochrane to hit 10,000 reviews between 2010 and 2015 Mallet and Clarke, EBM 2003(8):100-1 Total Protocols Total Reviews Total Updated Reviews1000th Cochrane review 5665 total reviews today
- 3. How can webetter leverage technology and knowledge to both help us prepare systematic reviews more efficiently but also deliver the outputs better to our end-users?
- 4. Setting the Stage:PICOT P opulation I ntervention C omparison O utcome T ime frame
- 5. Intervention P I Synthesized evidence,the "Cochrane Way" C O T
- 6. (Target) Population P Healthcare systems, clinicians I Synthesized evidence, the Cochrane Way C O T
- 7. Outcome P Health caresystems, clinicians I Synthesized evidence, the Cochrane Way C O Population-level health outcomes & costs T
- 8. Comparison P Health caresystems, clinicians I Synthesized evidence, the Cochrane Way C Eminence-based medicine O Population-level health outcomes & costs T
- 9. Time frame P Healthcare systems, clinicians I Synthesized evidence from the Cochrane Way C Eminence-based medicine O Population-level improvement in health, costs T Too many yearsT
- 10. Setting the PresentStage Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many Cochrane Future P I C O T
- 11.
- 12. Technology for PainPoints
- 13. Setting the FutureStage Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P I Cochrane Way C O T
- 14. Additional Outcomes Cochrane Present Phealth systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P I Cochrane Way C O cost T
- 15. Chronic Diseases DriveCost • 46% of morbidity and 59% mortality worldwide due to chronic diseases1 • Health systems targeting chronic care – 1/3 of deaths due to poor health behaviors – need to engage patients in self-care • Patients expect personalized medicine – want evidence at the individual-patient level WHO | Facts related to Chronic Disease http://www.who.int/dietphysicalactivity/publications/facts/chronic/en/
- 16. Additional Outcomes andPopulation Cochrane Present P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P add patients & families I Cochrane Way C O add ind-level health T
- 17. Time Frame • Personaldigital technologies will play large role in chronic disease and transforming health – 20,000 health apps on iTunes, 8,000 on Google Play – technologies evolve rapidly • US Institute of Medicine goal of a continuous Learning Health System Riley et al. Clinical and Translational Medicine 2013, 2:10
- 18. New Time Frame CochranePresent P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P add patients & families I Cochrane Way C O add ind-level health T continuous
- 19. New Comparison Intervention CochranePresent P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P add patients & families I Cochrane Way C Big Data O add ind-level health T continuous
- 20. Three Illustrative Projects •N-of-1 studies for chronic pain • Kaiser: Diabetes and depression management • Health eHeart virtual cohort
- 21. PREEMPT Project • ChronicPain is highly prevalent (>100 million Americans) and difficult to treat • Few studies on comparative effectiveness of analgesics, yielding only average population-level estimates R01-NR013938, PI R. Kravitz 50 people 100 people oxycodone Pain frequency, intensity 50 people hydrocodone population Pain frequency, intensity
- 22. none of usare average hydrocodone betteroxycodone better
- 23. which works betterfor you? N-of-1 study design: within-subject crossover design Kravitz, et al. Contemp Clin Trials 2009; 30:436-445 BPI individual pain intensity You pain intensity oxycodone hydrocodone hydrocodone oxycodone oxycodone hydrocodone
- 27. there you are! hydrocodonebetteroxycodone better
- 28.
- 29.
- 30. Zucker DR etal. J Clin Epidemiol. 2010;63(12):1312-23. (n 1).N Σ
- 31. flip direction ofresearch inference population
- 32. Differences from TraditionalResearch • Occurs in the course of clinical care • Patient participation, addresses patient questions • Patient choice in outcomes measured • Individual treatment effect • Aggregate to population-level effect
- 33. S. California KaiserComplete Care Improving self-management of diabetes in patients with depression
- 34. Continuous Evaluation • eHealthinterventions will play a substantial role in chronic care management and in shaping health care systems • Evaluations need to occur while they are being designed, developed, and deployed. Catwell and Sheikhl, PLoS Med 2009; 6(8):e1000126 Usability Studies Preliminary effectiveness User Requirements Pilot Field Testing Rigorous effectiveness
- 35. Differences from TraditionalResearch • Evidence is needed on intermediate non- clinical outcomes (e.g., effective design features) as well as end clinical effectiveness • Many of these evaluations will not be published in academic journals
- 37. • Enrolling 1,000,000patients, capturing – self report data: food intake, mood – sensor data: weight, BP, activity – social data: Facebook – EHR data: including text mining for CV events – biospecimens and CV tests for San Francisco area patients • Trades precise data on fewer patients for messier data on 1000x more patients – will be supplemented with targeted data collection for specific studies
- 38. Future Cochrane Way? CochranePresent P health systems, clinicians I Cochrane Way C eminence-based med O pop-level health & cost T too many years Cochrane Future P add patients & families I Cochrane Way? C Big Data O add ind-level health T continuous
- 39. "A good hockeyplayer plays where the puck is. A great hockey player plays where the puck is going to be." Wayne Gretsky, Edmonton Oilers
- 40. Where the Puckis Today Studies in PDF Forest plots in PDF Cochrane Way Draws the most sound inference from the totality of the available evidence
- 41. Where the Puckwill be Evidence = study protocol + results Synthesized evidence continuous, multi- modal, personal data more personalized questions large, less controlled studies individual-level evidence range of acceptable rigor continuous release schedule Cochrane Way Drawing the most sound inference from the totality of the available evidence
- 42. Build to BasicPrinciples • Data sciences • Operational • Methodological
- 43. Data Sciences • BeyondPDF reports of study results – embrace data publishing • publish protocols as computable models • publish all results as open computable data – text mining to extract meaning from prose • No data silos: linked open data • Describe the data: metadata and ontologies for – study questions – study design (Ontology of Clinical Research) – systematic reviews (Cochrane Ontology)
- 44. Data Sciences Goal •Capture open data in clearly described computable form that can be re-purposed for multiple needs today, and unknown needs tomorrow
- 45. Operational • Pursue hybridsemi-automated approaches – seek out and work with the best designers of human-computer interaction • Support distributed, collaborative knowledge work, tapping into crowds
- 46. Methods Needed • Heterogeneityof treatment effect • Continuous evaluation methods • Large-scale assessment of biases and confounders • Better decision support for assessing biases and confounders • ...many more
- 47. The "Human EvidenceProject" + + Methods Data Metadata + Most sound individual and population-level summary evidence for continuous learning
- 48. Conclusion • How toget from Cochrane Present to Cochrane Future? – new methods built on open data and ontologies – design hybrid human-computer systems • In new Big Data world, methodologically sound but pragmatic, sustainable evidence synthesis will be critical • Need Cochrane now more than ever
Editor's Notes
- #3 It's an honor to be here, at Cochrane's 20thanniversary. The first 1000 Cochrane reviews took 8 years, until 2001, to be completed. 20 years on, there are now 5665 total reviews. On the one hand, this is outstanding progress, but on the other, it's not nearly enough. In 2003, Mallet and Clarke estimated that at least 10,000 systematic reviewers were needed just to cover the existing evidence at that time, in 2003. They projected that Cochrane would produce these 10,000 reviews sometime between 2010-2015. But being only about half way there, that projection is unlikely to be met. And of course, now many thousands more systematic reviews are needed to cover the tens of thousands of additional RCTs that have been published since 2003. It isn't news to you this mismatch between demand and production of Cochrane reviews, nor is it news to you that the problem is getting worse. The hope is that technology will come to the rescue. ~~~~~~~~~~http://www.cochrane.org/cochrane-reviews/cochrane-database-systematic-reviews-numbershttp://www.thecochranelibrary.com/view/0/AboutTheCochraneLibrary.html
- #4 The question for today's conference is …I'd like to consider this question in 2 parts.One is what we can do with technology in the next 1-2 years, Cochrane Present Tech; and the 2nd part is Cochrane Future Tech for 3-5 years out.
- #5 Let's set the stage. A good practitioner of EBM always starts with a clear question. Considering Cochrane as an intervention, what is Cochrane's PICOT question?
- #6 First, the intervention is synthesized evidence generated by the Cochrane Way, which we'll say more about later.
- #7 Cochrane evidence is primarily targeted to health care systems, and clinicians.
- #8 to improve population-level health outcomes and cost.
- #9 The main comparison to Evidence-Based Medicine the Cochrane Way is Eminence-basedmedicine, which is self-descriptive.
- #10 And thetimeframe? You've heard that it takes an average of 17 years for clinical trial results to be put into practice. Whatever the delay is for primary studies, the delay is even more for systematic reviews. I couldn't find any studies on the time it takes for Cochrane reviews to impact care, but it's probably "too many years."
- #11 So this is the PICOT question for Cochrane Present. Cochrane Future is coming fast, but let's leave that aside right now, and look first at how technology can help the Cochrane in the next 1-2 years.
- #12 Here's a high level workflow of systematic reviewing, with a Cochrane Way to doing each of these steps. I know you're all very familiar with this so I'll just highlight a few points. First, there is a lot of iteration in the early steps, where a question is initially framed then revised in response to the details of the studies that are available. Second, there's a loop for updating and revising the reviews. Finally, that Dissemination and Implementation is often after and divorced from the process of conducting the reviews.
- #13 In the lightning talks later this morning, we will be hearing about technology being used for specific pain points in this workflow, like citation screening, supporting collaboration and supporting Dissemination and Implementation. There are many other technology projects we could discuss for other pain points here, but the elephant in the room is that things are changing, and you might be wondering how we should solve today's pain points in a way that will remain relevant for tomorrow.
- #14 Because the PICOT question for Cochrane Future will be quite different from today's. The interventionwill still be the Cochrane Way, which we will have to re-imagine.
- #15 For outcomes, cost is becoming an ever bigger driver.
- #16 This has implications for the type of evidence we want. The biggest driver of health care costs is chronic diseases. To reduce costs, health care systems are going to where the money is, and are focusing on chronic care. And because by definition chronic disease is 24/7 in patient's daily lives, and 1/3 of deaths are due to poor health behaviors that only patients can change, health care systems need to engage patients in their self-care.Nowadays, patients are savvy consumers and expect personalized medicine, and they will want evidence at the individual-patient level to help them self-manage.http://buyersguide.macrumors.com/
- #17 There will thus be more demand for "personalized evidence," from patients and their families, for individual-level estimates of treatment effect.
- #18 As health care systems increasingly focus on self-care and improving personal behaviors, personal digital technologies like apps and sensors will play a large role in transforming health care. These technologies evolve rapidly, with a "metabolic rate" that is like nothing we've seen before. Here's a figure from a very nice paper by Riley and colleagues showing that a typical RCT published in 2012 was probably designed and submitted for funding in 2006, when the Wii came out, and it spans the introduction of the iPhone, Android, iPad and Siri by the time it's analyzed and published.The pace of traditional research is simply too slow, and is also not compatible with the US Institute of Medicine's goal of a continuous learning health system, where care and research drive each other in tight iterative feedback loops. http://www.imedicalapps.com/2013/07/apple-android-medical-app/
- #19 So the time frame for Cochrane Future is going from Too many years to continuous.
- #20 And finally, the comparison intervention for Cochrane won't be Eminence-based medicine anymore, but Big Data. The buzz around Big Data seems to take correlation for causation, that simply by having massive amounts of EHR data together in the cloud, we can somehow get all our answers. Clinical trials, in contrast seem so much less modern, yet we know that careful methodology is more not less important for interpreting Big Data and especially for synthesizing inferences drawn from Big Data.
- #21 I'd like to illustrate these dimensions of change through three projects I'm working on.
- #22 The first is the PREEMPT Project which addresses the highly prevalent problem of chronic pain. This is a frustrating problem to treat because there are few studies on which pain meds are better, and the studies only yield population-level estimates, which is what we get from standard RCTs.
- #25 In our project, a chronicpatient sits down with their doctor to select the pain regimens they want to compare, for example…
- #26 define the length of time on each treatment – 2 weeks -- and the number of times they'll take each treatment – 3 times. This would be a 6-week n-of-1 trial.
- #27 and they can customize the outcomes to what is most relevant to them. For example, opiates can dull our thinking and concentration in different ways for different patients. In this project, we allow patients to select the cognitive symptom that is most relevant to them, like . Immediately after setup, the patient can download the Trialist app onto his/her cell phone which guides them through their custom n-of-1 study, 6 weeks in this case.Allowing patients to choose their outcomes is patient-centered, keeps them engaged in continuing the study, and gives them evidence that THEY want and will find useful. Of course, only some clinical questions can be answered with n-of-1 studies but this is a valuable study design that is much more feasible now with new technology.
- #29 n-of-1 studies can be even more powerful if we go from just one study
- #30 to many n-of-1 studies, so that more people can get individual estimates of treatment effect,
- #31 to summing the evidence from these studies for example using Zucker'sBayesian meta-analysis methods
- #33 So n-of-1studies illustrate some differences from the traditional RCT. First, it is conducted as an integral part of clinical care. Patients are full partners, participating in defining the question, choosing the treatments to compare, and the outcomes measured. The results are estimates of individual-level effects, which can be aggregated to yield estimates of population-level effects.
- #34 My second example is a project that Open mHealth is doing with Kaiser San Diego to improve the self-management of diabetes in patients with depression. It's a very complex intervention:Patients track their glucose, BP, weight and activity using various wearable sensors. They self-report depression and other symptoms on the cell phone, behavioral and social metrics are collected in an app called Ginger.io, a company called FrameHealth delivers incentives and motivations based on a psychological profiling of the patient. All this data is put together with EHR data to support self-care, family and friends care, and the Kaiser care team.
- #35 These kinds of multi-modal interventions are going to become more common for managing chronic diseases and indeed for shaping the entire health care system. We desperately need evidence of what works and what doesn't, but these complex interventions are made up of many moving parts that should be evaluated as they are being designed, developed, and deployed. For example, did you know that Google runs hundreds of little RCTs every day on little tweaks to their search algorithm and interfaces? They randomize you to getting this or that type of result or a font or a placement of an ad, and after oh several hours, they've accrued several hundred thousand subjects and a result that they use immediately to keep improving their search site. eHealth interventions need to adopt this agile development and evaluation approach. This type of evidence is valuable to synthesize and share to improve the eventual effectiveness of eHealth. The analogy is to drug development. If we only shared Phase 3 studies of drugs, we would be missing out on very useful evidence earlier in the drug development pipeline, about toxicity and pharmacokinetics for instance. It just that the Phase 1 and 2 studies for eHealth will be small, fast, and very iterative.
- #36 The differences in this example are
- #37 My final example is the Health eHeart project, which you can think of as a "virtual Framingham" cohort study. We are aiming to enroll 1,000,000 patients worldwide who have or are interested in cardiovascular disease.
- #38 We are capturing from these patients: What's different about this study is that we're trading carefully obtained, precisely phenotyped data on fewerpatients for messier data on orders of magnitude more patients. Clearly, this approach opens up exciting possibilities for research but also introduces many methodological concerns.
- #39 Given all these changes, what will the future Cochrane look like? Because we're in Canada, and because I'm Canadian and from Edmonton,
- #40 I have to quote Wayne Gretsky:
- #41 Grossly simplified, this is where the puck is today. We have PDFsreporting on studies completed years ago, and Cochrane converts that to Forest plots in PDF, by drawing the most sound inference from the totality of the available evidence.
- #42 This is where the puck willbe in 3-5 years, The challenge still is to draw the most sound inference from the totality of available evidence, but both the input and the outputs will be different, as shown by our examples. like in n-of-1, the results will be more personalized, and the users will want synthesized evidence FOR THEM INDIVIDUALLYlike the Kaiser project, data will be large, disparate, very personal data, generating clinical and non-clinical evidence with varying degrees rigor. Evidence synthesis will need to accommodate this range of rigor, we can't just stick with standard RCTs, and to help users understand and adjust for weaknesses.like HeH, large less precisely controlled studies that trade numbers for precision, drawing in many cases on the promise of Big Data. And patients and the health care system will want evidence on a continuous release schedule. The Cochrane Way will have to change to meet these new expectations. It's not clear how, but I think there are some basic principles that Cochrane should build to on the way there.
- #43 These basic principles are data science related, operational and methodological.
- #44 On the data front, we have to move beyond PDFs, and beyond a singular focus on results reports. We need to embrace data publishing as I'm sure John Wilbanks will be talking about at lunch. We need to publish study protocols as computable models, and publish results as computable data, so we can automate as much of the workflow as possible. We need to use text mining to extract meaning from prose, because there will always be, and should always be, prose for humans to read.An absolutely critical principle is that data cannot be in silos if we want to get the most out of evidence to improve health. We should pursue linked open data, again John will discuss, and the context for this data needs to be fully described. That is, we need metadata, or data about data. A simple example is a data point, let's say an odds ratio. The metadata, or context, for that odds ratio may include whether that odds ratio came from an RCT or an observational study. You can see that without good metadata, naked data can be downright dangerous. We also need ontologies. You can think of ontologies as standard computable ways of describing more complex abstractions, like what is a study question (so we can index and search on it), or my work on an Ontology of Clinical Research to describe study designs, or the Cochrane Ontology's standard description of the parts of a systematic review. I know this is all really abstract so I'll boil it down to a goal:
- #45 The overall goal is to capture data in clearly described computable form that we can repurpose for multiple needs today, and for unknown needs tomorrow.
- #46 Operationally for Cochrane Future Tech, a big challenge is to know what to have computers do, and what to leave humans do. Systematic review work is a mix of brain-numbingly tedious work mixed with highly skilled methodology work and clinical insight. The best technology solutions are going to be well-designed hybrid human-computer systems, which means seeking out and working with our colleagues in computer science, cognitive psychology and design t build the best hybrid systems. User interaction design is important. e.g., ExACT system.Part of this challenge is to support distributed collaborative knowledge work, tapping into crowds.
- #47 Finally, there are many methodological frontiers for Cochrane Future. One is methods for generating more precise individual-level treatment estimates. Another is systematic review methods for complex interventions and continuous evaluation methods. Because the Big Data world will be full of messy, biased, confounded data, we need critical appraisal methods that can scale to the volume of evidence we will see. This will likely be semi-automated methods. I'm sure you can think of more.
- #48 As I see it, these the points to keep in mind for the ambitious notion of a Human Evidence Project.First and foremost is to use technology not as an end to itself, but to use it to combine together human work and expertise, data and metadata, and computational methods to generate the most sound invidiual and population-level summary evidence for continuous learning.Mapping these basic principles to the workflow, we need ontologies of study questions and protocols to help with question framing, linked open data and text mining for assembling studies, we can say good riddance to abstraction if we publish in data and nanopublications, we need new statistical methods, and ontologies for diseases, care processes, decision making, etc. for effective D & I. Cochrane might not take on this whole challenge, but I can't see any other group besides Cochrane that can or should take the lead on this grand vision.
- #49 In conclusion,