Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Understanding Comparative Effectiveness

727 views

Published on

How to understand and put into perspective a comparative effectiveness report

  • Be the first to comment

Understanding Comparative Effectiveness

  1. 1. Demystifying Comparative Effectiveness Research: A Case Study Learning Guide<br />Robert W. Dubois, MD, PhD<br />Chief Medical Officer<br />Cerner LifeSciences<br />1<br />
  2. 2. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report <br />Case studies and lessons learned<br />Randomized trial-incomplete answer<br />Meta-analysis-misleading conclusion<br />Observational study-incorrect change in clinical practice<br />Final thoughts<br />2<br />
  3. 3. Institute of Medicine Definition*<br />“…The purpose of CER is to assist consumers, clinicians, purchasers and policy makers to make informed decisions that will improve health care at both the individual and population levels.”<br />*IOM 2009 Initial National Priorities for Comparative Effectiveness Research<br />3<br />
  4. 4. Not All Studies Are Created (or interpreted) Equally<br />
  5. 5. 5<br />
  6. 6. Proceed With Caution…<br />6<br />
  7. 7. When the reports arrive…<br />
  8. 8. It may feel like this…<br />
  9. 9. Categorizing is good…<br />
  10. 10. Understanding is even better…<br />
  11. 11. Topics<br />Opening thoughts<br />How to approach CER<br />11<br />
  12. 12. How to Approach CER<br />
  13. 13. Step 1: Consider for Whom the Findings are Applicable<br />An H1N1 study in school age kids may (or may not) help an internist with his/her adult population.<br />A study of patients with severe rheumatoid arthritis may (or may not) apply to a population of milder patients.<br />A study of diabetic control after one year may (or may not) apply to the longer term implications of a policy change.<br />Will the study results generalize to your environment?<br />13<br />
  14. 14. Questions for Step 1<br />Does the study resemble your population of interest (e.g., age, gender, SES, disease profile)?<br />Does the clinical setting resemble your own (e.g., primary care, inner city)?<br />Is the study conducted in the “real world” or in a highly controlled environment?<br />Are the outcomes appropriate to your needs?<br />14<br />
  15. 15. How to Approach CER<br />
  16. 16. There are 3 Common Types of CER<br />Randomized trials<br />Meta-analysis<br />Observational studies<br />16<br />Was the right design chosen?<br />
  17. 17. 17<br />
  18. 18. Does the design match the study question?<br />RCT: may not meet your “real world” needs.<br />Observational study: may have too many uncontrollable factors to be valid.<br />Meta-analysis: just combining studies because they are “there” may not be appropriate.<br />18<br />
  19. 19. Step 2: Consider Whether Aspects of the StudyDesign Might Affect the Results<br />Does the study design match the question being asked?<br />Was it carried out with adequate rigor?<br />Each study design has its own issues to consider.<br />One size does not fit all.<br />19<br />
  20. 20. One size does not fit all…<br />
  21. 21. One size does not fit all…<br />
  22. 22. How to Approach CER<br />
  23. 23. “How sure are we… really”<br />
  24. 24. http://www.hasyudeen.com/2008/02/network-centric-competition-in-flat.html<br />24<br />
  25. 25. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report (a.k.a. “Readers’ Guide”<br />Case studies and lessons learned<br />Randomized trial-incomplete answer<br /><ul><li>Step 3-new information
  26. 26. Step 1-revisit the relevant population</li></ul>25<br />
  27. 27. Can it Work?<br />Randomized Controlled Trials (RCTs)<br />Can an intervention work under certain controlled conditions?<br />Characteristics:<br />Planned experimental framework<br />Defined treatment options<br />Specified outcomes<br />May be single or double blinded<br />26<br />
  28. 28. RCT Design<br />
  29. 29. RCTs Have Strengths and Weaknesses<br />Strengths:<br />Substantial internal validity<br />Reduced likelihood of bias or confounding<br />Gold standard for clinical research<br />Pathway to FDA approval<br />Weaknesses:<br />May not generalize <br />Modest size reduces ability to observe rare events.<br />Placebo may not be the best “real world” comparator.<br />Some patients leave the trial or “cross over” to the other therapy.<br />28<br />
  30. 30. RCT Case Study: Cetuximab<br />Clinical Situation<br />Colorectal cancer is the 3rd most common cancer and cause of cancer death.<br />Only an 11% survival if metastatic<br />Treatment is not curative.<br />Understanding of cancer genetics led to biomarker testing and targeted therapies.<br />Cetuximab is an anti-epidermal growth factor receptor agent (EGFR).<br />29<br />
  31. 31. Cetuximab RCT*: Was This the Full Story?<br />572 patients with mCRC<br />Non-blinded RCT<br />Cetuximab+ supportive care vs. supportive care alone<br />Survival in Months<br />*Jonker N Engl J Med 2007<br />30<br />
  32. 32. Not the Full Cetuximab Story<br />Basic science research suggested that the KRAS gene and its mutations could influence efficacy.<br />Subgroup analysis performed based upon testing tissue samples<br />Those with non-mutated gene had much greater benefit from cetuximab (5 months greater overall survival).<br />31<br />
  33. 33. Not the Full Cetuximab Story<br />FDA narrowed the drug’s indications to patients with a non-mutated gene.<br />Recommendations to test patients for the genetic marker<br />Therapy targeted to patients most likely to respond.<br />Patients unlikely to respond avoid ill effects of the drug. <br />32<br />
  34. 34. RCT Tips for the Consumer<br />RCTs assess whether an intervention can work in a controlled environment.<br />Although RCTs are viewed as the “gold standard,” the initial impression of the cetuximab study was incomplete.<br />Multiple studies designed similarly would have likely yielded similar incomplete conclusions.<br />A subgroup analysis based upon genetic markers yielded different results and conclusions.<br />No results are permanent.<br />33<br />
  35. 35. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report (a.k.a. “Readers’ Guide”<br />Case studies and lessons learned<br />Randomized trial<br />Meta-analysis-misleading conclusion<br />Step 2: methods used (and not used)<br />34<br />
  36. 36. Meta-Analysis<br />“If one RCT is good, then more RCTs must be better…”<br />Quantitative and statistical combination of study results<br />Highest level of evidence<br />Useful when different studies have different results<br />35<br />
  37. 37. Bagshaw SM, Ghali WA. Acetylcysteine for prevention of contrast-induced nephropathy after intravascular angiography: a systematic review and meta-analysis. BMC Med. 2004;2(1):38.<br />Metanalysis Clarifies the Benefit<br />
  38. 38. Wilcken, MJA. 2007;186 (7):368-370.<br />Metanalysis Clarifies the Benefits<br />
  39. 39. Meta-Analysis<br />Strengths<br />Increases the effective sample size<br />Provides statistically stronger conclusions<br />Detects lower frequency events and more subtle distinctions<br />Weaknesses<br />Creates an impression of “truth”<br />Easy to do wrong…<br />If care is not taken, results may be invalid.<br />38<br />
  40. 40. Meta-Analysis Case Study: Avandia<br />Clinical Situation<br />Blood sugar control reduces certain diabetic complications.<br />But, oral drugs have been associated with an increased risk of heart disease (tolbutamide).<br />TZDs seemed safer than sulfonylureas.<br />Avandia approved by the FDA in 1999.<br />39<br />
  41. 41. Original ArticleEffect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes<br />Steven E. Nissen, M.D., and Kathy Wolski, M.P.H.<br />N Engl J Med<br />Volume 356(24):2457-2471<br />June 14, 2007<br />40<br />
  42. 42. What was Done and Found?<br />A meta-analysis examined the impact of Avandia on cardiac events.<br />Out of 116 studies, 42 met the authors’ inclusion criteria.<br />15,565 Avandia patients<br />12,282 comparison patients<br />41<br />
  43. 43. Avandia Adversely Impacts Heart Disease<br />Compared with the placebo, estrogen plus progestin resulted in:<br />Increased risk of heart attack<br />Increased risk of stroke<br />Increased risk of blood clots<br />Increased risk of breast cancer<br />Reduced risk of colorectal cancer<br />Fewer fractures<br />No protection against mild cognitive impairment and increased risk of dementia (study included only women 65 and older)<br />Nissen SE, Wolski K. N Engl J Med. 2007;356:2457-2471.<br />
  44. 44.
  45. 45. Was This the Right Answer?<br />“Thou shalt not combine heterogeneous studies…”<br />Some studies had placebo and others active comparators.<br />Some studies observed one arm longer than the other.<br />Excluded “zero” event trials<br />44<br />
  46. 46. Was This the Right Answer?<br />The choice of statistical methods impacts the results.<br />Re-analysis with an alternative approach showed no increased risk.<br />Inclusion of the “zero event” studies also eliminated any statistical differences.<br />New trial results (RECORD) showed no difference in deaths but an increase in heart failure.<br />45<br />
  47. 47. Meta-analysis Tips for the Consumer<br />Results are highly dependent on the studies included and excluded.<br />Statistical methodology can impact study results.<br />Nothing is permanent - emerging data may change the conclusions.<br />46<br />
  48. 48. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report (a.k.a. “Readers’ Guide”<br />Case studies and lessons learned<br />Randomized trial<br />Meta-analysis<br />Observational study-incorrect change in clinical practice<br />Step 1: population differed in subtle ways from a more typical one<br />Step 2: intervention and comparison groups were different<br />47<br />
  49. 49. Observational Studies<br />Answers the question: will it (likely) work (not can it work)?<br />Examines the effects of treatment without formal randomization<br />Performed prospectively or retrospectively<br />48<br />
  50. 50. Observational Studies<br />Framingham Study identified heart disease risk factors.<br />Commonly uses existing “real world” databases<br />Administrative or billing data<br />Electronic health records<br />49<br />
  51. 51. Observational Studies<br />Strengths<br />Assess health care in the “real world”<br />Lower cost and faster to perform<br />Large sample sizes feasible<br />Hypothesis generating<br />Weaknesses<br />Administrative databases have minimal clinical detail and may contain errors.<br />Uncontrolled design leads to potential bias or confounding.<br />Subject to “data dredging”<br />Cannot prove cause and effect<br />50<br />
  52. 52. Observational Case Study: Hormone Treatment<br />Clinical Situation<br />Heart disease is the leading cause of death in women above age 50<br />Estrogen falls after menopause<br />May account for the postmenopausal heat disease risk<br />51<br />
  53. 53. What was Done and Found?<br />Nurses’ Health Study (NHS) collected data from 48,470 women aged 30-55.<br />Focused on the impact of hormones on outcomes<br />Ongoing surveys assessed risk factors and health outcomes.<br />Estrogen use associated with lower heart disease risk<br />52<br />
  54. 54. Implications of the Nurses’ Health Study<br />
  55. 55. Was This the Right Answer?<br />
  56. 56. Was This the Right Answer?<br />?<br />
  57. 57. The clinical trials were designed to test the effects of postmenopausal hormone therapy, diet modification, and calcium and vitamin D supplements on heart disease, fractures, and breast and colorectal cancer.<br />56<br />
  58. 58. Women’s Health Initiative<br />RCT with 27,347 post-menopausal women<br />Mean age was 63.6 years.<br />57<br />
  59. 59. Women’s Health Initiative Muddied the Waters…<br />RCT with 27,347 post-menopausal women<br />Mean age was 63.6 years.<br />Study halted early.<br />Increased risk of clots, breast cancer and stroke.<br />Estrogen-progestin users: increased risk of heart disease <br />Estrogen only users: no CV benefits.<br />58<br />
  60. 60. Was This the Right Answer?<br />?<br />
  61. 61. Why the Different Results?<br />Theory 1: WHI population was older than the NHS cohort (started hormones much later)<br />Theory 2: NHS women who took hormones were healthier<br />Higher education<br />Higher SES<br />Leaner<br />Lacked prior cardiac disease<br />Theory 3: The NHS women who took hormones differed from those who did not<br />60<br />
  62. 62. The Differences Were Explainable (Partly)<br />NHS data re-analyzed…<br />Less benefit for older women starting HRT later.<br />WHI re-analyzed…<br />Some benefit for younger women starting HRT early.<br />61<br />Steps 1 and 2<br />With observational studies, make sure…<br /><ul><li>The population resembles yours
  63. 63. The intervention and comparison groups are similar</li></li></ul><li>Observational Study Tips for the Consumer<br />Useful to understand “real world” benefits and harms<br />Look for confounding factors (Step 2)<br />Nothing is permanent - emerging data may change the conclusions<br />62<br />
  64. 64. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report (a.k.a. “Readers’ Guide”<br />Case studies and lessons learned<br />Final thoughts<br />63<br />
  65. 65. Final Thoughts<br />Different study types can offer different understandings.<br />64<br />
  66. 66. Final Thoughts<br />Different study types can offer different understandings.<br />Results matter.<br />Avandia sales fell after meta-analysis published.<br />Hormone treatment plummeted after results of the WHI.<br />Cetuximab use is growing.<br />65<br />
  67. 67. Final Thoughts<br />Different study types can offer different understandings.<br />Results matter.<br />Consider the potential impact upon policy making.<br />Influence guidelines?<br />Impact reimbursement?<br />Stable clinical area or advancing rapidly?<br />66<br />
  68. 68. Demystifying Comparative Effectiveness Research<br />67<br />
  69. 69. Demystifying Comparative Effectiveness Research<br />68<br />
  70. 70. Demystifying Comparative Effectiveness Research: A Case Study Learning Guide<br />Reader’s Guide Check Lists<br />Robert W. Dubois, MD, PhD<br />Chief Medical Officer<br />Cerner LifeSciences<br />69<br />
  71. 71. How to Approach CER<br />
  72. 72. Use Check Lists With Caution<br />The lists are not all inclusive.<br />A “yes” to all is not all that you need…<br />All items are not created equally.<br />71<br />
  73. 73. Step 1: Are the Findings Applicable?<br />
  74. 74. Step 2: Design Affect the Results?<br />
  75. 75. Step 2: Design Affect the Results?<br />
  76. 76. Step 2: Design Affect the Results?<br />
  77. 77. Step 3: Findings Change With New Research?<br />

×