Understanding Comparative Effectiveness

647 views
593 views

Published on

How to understand and put into perspective a comparative effectiveness report

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
647
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
18
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Understanding Comparative Effectiveness

  1. 1. Demystifying Comparative Effectiveness Research: A Case Study Learning Guide<br />Robert W. Dubois, MD, PhD<br />Chief Medical Officer<br />Cerner LifeSciences<br />1<br />
  2. 2. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report <br />Case studies and lessons learned<br />Randomized trial-incomplete answer<br />Meta-analysis-misleading conclusion<br />Observational study-incorrect change in clinical practice<br />Final thoughts<br />2<br />
  3. 3. Institute of Medicine Definition*<br />“…The purpose of CER is to assist consumers, clinicians, purchasers and policy makers to make informed decisions that will improve health care at both the individual and population levels.”<br />*IOM 2009 Initial National Priorities for Comparative Effectiveness Research<br />3<br />
  4. 4. Not All Studies Are Created (or interpreted) Equally<br />
  5. 5. 5<br />
  6. 6. Proceed With Caution…<br />6<br />
  7. 7. When the reports arrive…<br />
  8. 8. It may feel like this…<br />
  9. 9. Categorizing is good…<br />
  10. 10. Understanding is even better…<br />
  11. 11. Topics<br />Opening thoughts<br />How to approach CER<br />11<br />
  12. 12. How to Approach CER<br />
  13. 13. Step 1: Consider for Whom the Findings are Applicable<br />An H1N1 study in school age kids may (or may not) help an internist with his/her adult population.<br />A study of patients with severe rheumatoid arthritis may (or may not) apply to a population of milder patients.<br />A study of diabetic control after one year may (or may not) apply to the longer term implications of a policy change.<br />Will the study results generalize to your environment?<br />13<br />
  14. 14. Questions for Step 1<br />Does the study resemble your population of interest (e.g., age, gender, SES, disease profile)?<br />Does the clinical setting resemble your own (e.g., primary care, inner city)?<br />Is the study conducted in the “real world” or in a highly controlled environment?<br />Are the outcomes appropriate to your needs?<br />14<br />
  15. 15. How to Approach CER<br />
  16. 16. There are 3 Common Types of CER<br />Randomized trials<br />Meta-analysis<br />Observational studies<br />16<br />Was the right design chosen?<br />
  17. 17. 17<br />
  18. 18. Does the design match the study question?<br />RCT: may not meet your “real world” needs.<br />Observational study: may have too many uncontrollable factors to be valid.<br />Meta-analysis: just combining studies because they are “there” may not be appropriate.<br />18<br />
  19. 19. Step 2: Consider Whether Aspects of the StudyDesign Might Affect the Results<br />Does the study design match the question being asked?<br />Was it carried out with adequate rigor?<br />Each study design has its own issues to consider.<br />One size does not fit all.<br />19<br />
  20. 20. One size does not fit all…<br />
  21. 21. One size does not fit all…<br />
  22. 22. How to Approach CER<br />
  23. 23. “How sure are we… really”<br />
  24. 24. http://www.hasyudeen.com/2008/02/network-centric-competition-in-flat.html<br />24<br />
  25. 25. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report (a.k.a. “Readers’ Guide”<br />Case studies and lessons learned<br />Randomized trial-incomplete answer<br /><ul><li>Step 3-new information
  26. 26. Step 1-revisit the relevant population</li></ul>25<br />
  27. 27. Can it Work?<br />Randomized Controlled Trials (RCTs)<br />Can an intervention work under certain controlled conditions?<br />Characteristics:<br />Planned experimental framework<br />Defined treatment options<br />Specified outcomes<br />May be single or double blinded<br />26<br />
  28. 28. RCT Design<br />
  29. 29. RCTs Have Strengths and Weaknesses<br />Strengths:<br />Substantial internal validity<br />Reduced likelihood of bias or confounding<br />Gold standard for clinical research<br />Pathway to FDA approval<br />Weaknesses:<br />May not generalize <br />Modest size reduces ability to observe rare events.<br />Placebo may not be the best “real world” comparator.<br />Some patients leave the trial or “cross over” to the other therapy.<br />28<br />
  30. 30. RCT Case Study: Cetuximab<br />Clinical Situation<br />Colorectal cancer is the 3rd most common cancer and cause of cancer death.<br />Only an 11% survival if metastatic<br />Treatment is not curative.<br />Understanding of cancer genetics led to biomarker testing and targeted therapies.<br />Cetuximab is an anti-epidermal growth factor receptor agent (EGFR).<br />29<br />
  31. 31. Cetuximab RCT*: Was This the Full Story?<br />572 patients with mCRC<br />Non-blinded RCT<br />Cetuximab+ supportive care vs. supportive care alone<br />Survival in Months<br />*Jonker N Engl J Med 2007<br />30<br />
  32. 32. Not the Full Cetuximab Story<br />Basic science research suggested that the KRAS gene and its mutations could influence efficacy.<br />Subgroup analysis performed based upon testing tissue samples<br />Those with non-mutated gene had much greater benefit from cetuximab (5 months greater overall survival).<br />31<br />
  33. 33. Not the Full Cetuximab Story<br />FDA narrowed the drug’s indications to patients with a non-mutated gene.<br />Recommendations to test patients for the genetic marker<br />Therapy targeted to patients most likely to respond.<br />Patients unlikely to respond avoid ill effects of the drug. <br />32<br />
  34. 34. RCT Tips for the Consumer<br />RCTs assess whether an intervention can work in a controlled environment.<br />Although RCTs are viewed as the “gold standard,” the initial impression of the cetuximab study was incomplete.<br />Multiple studies designed similarly would have likely yielded similar incomplete conclusions.<br />A subgroup analysis based upon genetic markers yielded different results and conclusions.<br />No results are permanent.<br />33<br />
  35. 35. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report (a.k.a. “Readers’ Guide”<br />Case studies and lessons learned<br />Randomized trial<br />Meta-analysis-misleading conclusion<br />Step 2: methods used (and not used)<br />34<br />
  36. 36. Meta-Analysis<br />“If one RCT is good, then more RCTs must be better…”<br />Quantitative and statistical combination of study results<br />Highest level of evidence<br />Useful when different studies have different results<br />35<br />
  37. 37. Bagshaw SM, Ghali WA. Acetylcysteine for prevention of contrast-induced nephropathy after intravascular angiography: a systematic review and meta-analysis. BMC Med. 2004;2(1):38.<br />Metanalysis Clarifies the Benefit<br />
  38. 38. Wilcken, MJA. 2007;186 (7):368-370.<br />Metanalysis Clarifies the Benefits<br />
  39. 39. Meta-Analysis<br />Strengths<br />Increases the effective sample size<br />Provides statistically stronger conclusions<br />Detects lower frequency events and more subtle distinctions<br />Weaknesses<br />Creates an impression of “truth”<br />Easy to do wrong…<br />If care is not taken, results may be invalid.<br />38<br />
  40. 40. Meta-Analysis Case Study: Avandia<br />Clinical Situation<br />Blood sugar control reduces certain diabetic complications.<br />But, oral drugs have been associated with an increased risk of heart disease (tolbutamide).<br />TZDs seemed safer than sulfonylureas.<br />Avandia approved by the FDA in 1999.<br />39<br />
  41. 41. Original ArticleEffect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes<br />Steven E. Nissen, M.D., and Kathy Wolski, M.P.H.<br />N Engl J Med<br />Volume 356(24):2457-2471<br />June 14, 2007<br />40<br />
  42. 42. What was Done and Found?<br />A meta-analysis examined the impact of Avandia on cardiac events.<br />Out of 116 studies, 42 met the authors’ inclusion criteria.<br />15,565 Avandia patients<br />12,282 comparison patients<br />41<br />
  43. 43. Avandia Adversely Impacts Heart Disease<br />Compared with the placebo, estrogen plus progestin resulted in:<br />Increased risk of heart attack<br />Increased risk of stroke<br />Increased risk of blood clots<br />Increased risk of breast cancer<br />Reduced risk of colorectal cancer<br />Fewer fractures<br />No protection against mild cognitive impairment and increased risk of dementia (study included only women 65 and older)<br />Nissen SE, Wolski K. N Engl J Med. 2007;356:2457-2471.<br />
  44. 44.
  45. 45. Was This the Right Answer?<br />“Thou shalt not combine heterogeneous studies…”<br />Some studies had placebo and others active comparators.<br />Some studies observed one arm longer than the other.<br />Excluded “zero” event trials<br />44<br />
  46. 46. Was This the Right Answer?<br />The choice of statistical methods impacts the results.<br />Re-analysis with an alternative approach showed no increased risk.<br />Inclusion of the “zero event” studies also eliminated any statistical differences.<br />New trial results (RECORD) showed no difference in deaths but an increase in heart failure.<br />45<br />
  47. 47. Meta-analysis Tips for the Consumer<br />Results are highly dependent on the studies included and excluded.<br />Statistical methodology can impact study results.<br />Nothing is permanent - emerging data may change the conclusions.<br />46<br />
  48. 48. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report (a.k.a. “Readers’ Guide”<br />Case studies and lessons learned<br />Randomized trial<br />Meta-analysis<br />Observational study-incorrect change in clinical practice<br />Step 1: population differed in subtle ways from a more typical one<br />Step 2: intervention and comparison groups were different<br />47<br />
  49. 49. Observational Studies<br />Answers the question: will it (likely) work (not can it work)?<br />Examines the effects of treatment without formal randomization<br />Performed prospectively or retrospectively<br />48<br />
  50. 50. Observational Studies<br />Framingham Study identified heart disease risk factors.<br />Commonly uses existing “real world” databases<br />Administrative or billing data<br />Electronic health records<br />49<br />
  51. 51. Observational Studies<br />Strengths<br />Assess health care in the “real world”<br />Lower cost and faster to perform<br />Large sample sizes feasible<br />Hypothesis generating<br />Weaknesses<br />Administrative databases have minimal clinical detail and may contain errors.<br />Uncontrolled design leads to potential bias or confounding.<br />Subject to “data dredging”<br />Cannot prove cause and effect<br />50<br />
  52. 52. Observational Case Study: Hormone Treatment<br />Clinical Situation<br />Heart disease is the leading cause of death in women above age 50<br />Estrogen falls after menopause<br />May account for the postmenopausal heat disease risk<br />51<br />
  53. 53. What was Done and Found?<br />Nurses’ Health Study (NHS) collected data from 48,470 women aged 30-55.<br />Focused on the impact of hormones on outcomes<br />Ongoing surveys assessed risk factors and health outcomes.<br />Estrogen use associated with lower heart disease risk<br />52<br />
  54. 54. Implications of the Nurses’ Health Study<br />
  55. 55. Was This the Right Answer?<br />
  56. 56. Was This the Right Answer?<br />?<br />
  57. 57. The clinical trials were designed to test the effects of postmenopausal hormone therapy, diet modification, and calcium and vitamin D supplements on heart disease, fractures, and breast and colorectal cancer.<br />56<br />
  58. 58. Women’s Health Initiative<br />RCT with 27,347 post-menopausal women<br />Mean age was 63.6 years.<br />57<br />
  59. 59. Women’s Health Initiative Muddied the Waters…<br />RCT with 27,347 post-menopausal women<br />Mean age was 63.6 years.<br />Study halted early.<br />Increased risk of clots, breast cancer and stroke.<br />Estrogen-progestin users: increased risk of heart disease <br />Estrogen only users: no CV benefits.<br />58<br />
  60. 60. Was This the Right Answer?<br />?<br />
  61. 61. Why the Different Results?<br />Theory 1: WHI population was older than the NHS cohort (started hormones much later)<br />Theory 2: NHS women who took hormones were healthier<br />Higher education<br />Higher SES<br />Leaner<br />Lacked prior cardiac disease<br />Theory 3: The NHS women who took hormones differed from those who did not<br />60<br />
  62. 62. The Differences Were Explainable (Partly)<br />NHS data re-analyzed…<br />Less benefit for older women starting HRT later.<br />WHI re-analyzed…<br />Some benefit for younger women starting HRT early.<br />61<br />Steps 1 and 2<br />With observational studies, make sure…<br /><ul><li>The population resembles yours
  63. 63. The intervention and comparison groups are similar</li></li></ul><li>Observational Study Tips for the Consumer<br />Useful to understand “real world” benefits and harms<br />Look for confounding factors (Step 2)<br />Nothing is permanent - emerging data may change the conclusions<br />62<br />
  64. 64. Topics<br />Opening thoughts<br />Framework for reviewing a new CER report (a.k.a. “Readers’ Guide”<br />Case studies and lessons learned<br />Final thoughts<br />63<br />
  65. 65. Final Thoughts<br />Different study types can offer different understandings.<br />64<br />
  66. 66. Final Thoughts<br />Different study types can offer different understandings.<br />Results matter.<br />Avandia sales fell after meta-analysis published.<br />Hormone treatment plummeted after results of the WHI.<br />Cetuximab use is growing.<br />65<br />
  67. 67. Final Thoughts<br />Different study types can offer different understandings.<br />Results matter.<br />Consider the potential impact upon policy making.<br />Influence guidelines?<br />Impact reimbursement?<br />Stable clinical area or advancing rapidly?<br />66<br />
  68. 68. Demystifying Comparative Effectiveness Research<br />67<br />
  69. 69. Demystifying Comparative Effectiveness Research<br />68<br />
  70. 70. Demystifying Comparative Effectiveness Research: A Case Study Learning Guide<br />Reader’s Guide Check Lists<br />Robert W. Dubois, MD, PhD<br />Chief Medical Officer<br />Cerner LifeSciences<br />69<br />
  71. 71. How to Approach CER<br />
  72. 72. Use Check Lists With Caution<br />The lists are not all inclusive.<br />A “yes” to all is not all that you need…<br />All items are not created equally.<br />71<br />
  73. 73. Step 1: Are the Findings Applicable?<br />
  74. 74. Step 2: Design Affect the Results?<br />
  75. 75. Step 2: Design Affect the Results?<br />
  76. 76. Step 2: Design Affect the Results?<br />
  77. 77. Step 3: Findings Change With New Research?<br />

×