Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Upcoming SlideShare
×

# How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies

226 views

Published on

Published in: Health & Medicine, Technology
• Full Name
Comment goes here.

Are you sure you want to Yes No
Your message goes here
• Be the first to comment

• Be the first to like this

### How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies

1. 1. HOW TO BUILD A MODEL TO PREDICT REIMBURSEMENT DECISIONS BY MAJOR HTA/CER AGENCIES Kermit Daniel, PhD – Chief Analytics Officer, Context Matters, Inc. December 2013 Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED
2. 2. BUILDING A PREDICTIVE MODEL OF HTA BEHAVIOR • Why build a model? • Can we build a model? • How do we build a model? • How will we know if we built a good model? Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 2
3. 3. WHY BUILD A MODEL? A GOOD MODEL WILL • Improve predictions • Disentangle multiple influences • Provide actionable guidance for how to increase the likelihood of a positive reimbursement decision Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 3
4. 4. WHY BUILD A MODEL? A MODEL IS • Useful simplification • Mathematical: Y = Xβ + ε Y: Probability of a positive recommendation X: Observable influences of recommendations β: Parameters we will estimate ε: Effect of influences we don’t observe Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 4
5. 5. CAN WE BUILD A MODEL? WHAT HAS TO BE TRUE? • Agencies behave in ways that can be described by a simple model that we can estimate o Consistency - non-random, consistent behavior • Through time • Across therapeutic areas • Across agencies o Transparency – we can observe decision factors (or good proxies) Y = Xβ + ε THIS EXPLAINS A LOT THIS EXPLAINS A LITTLE Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 5
6. 6. CONSISTENCY ACROSS AGENCIES AGENCIES FREQUENTLY AGREE WHEN PRESENTED WITH THE SAME DRUG/INDICATION TO REVIEW FREQUENCY OF AGREEMENT BETWEEN AGENCY PAIRS CADTH HAS NICE PBAC SMC 68% 78% 85% 76% PBAC 63% 72% 85% NICE 61% 65% HAS 58% Note: Reviews of 94 drugs reviewed by at least two agencies between January 2005 – February 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 6
7. 7. CONSISTENCY BETWEEN DECISION AND ASSESSMENTS THE CLINICAL AND/OR ECONOMIC ASSESSMENT AND THE REIMBURSEMENT DECISION ARE USUALLY CONSISTENT AGENCY PAIR Same Clinical Outcome Different Clinical Assessment PBAC:SMC P value Same Economic Outcome Different Economic Assessment 0.05* P value 0.00* Agree on decision 27 4 20 7 Disagree on decision 5 4 1 8 CADTH:PBAC 0.00* 0.14 Agree on decision 16 4 11 8 Disagree on decision 2 11 3 9 SMC:HAS Agree on decision 0.01* 32 9 4 9 Disagree on decision Note: Reviews of 94 drugs reviewed by at least two agencies between January 2005 – February 2013. Asterisk and shading indicates significance at 5% or better. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 7
8. 8. TRANSPARENCY WE OBSERVE A LOT ABOUT REVIEWS, INCLUDING . . . DRUGS CLINICAL STUDIES SUBMISSIONS • Chemical type • Indications • Regulatory history • Orphan status • Number, dates • Design, e.g., goal, comparators, size, subpopulations, length • HTA assessment • Number • Professional, patient, other group support AGENCIES ECONOMIC MODELS CONCLUSIONS • Identity • Region • Agency-specific factors, e.g., additional benefit • Manufacturer comparators & assumptions • Agency model • HTA assessment • Recommendation • Restrictions • Factors discussed, e.g., study limitations, PRO use Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 8
9. 9. HOW DO WE BUILD A MODEL? HOW DO WE DECIDE WHAT DATA TO INCLUDE? • Predict outcomes that matter • Capture implications of agency objectives and constraints • Incorporate what we observe about agency behavior – but this is dangerous! Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 9
10. 10. PREDICT OUTCOMES THAT MATTER POSITIVE DECISIONS OFTEN ADD RESTRICTIONS Oncology NICE Non-Oncology 55% 54% 46% 45% Positive Decision Not More Restrictive Positive Decision More Restrictive Note: Based on 150 NICE reviews of 72 drugs for 34 diseases between January 2007 – August 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 10
11. 11. CAPTURE IMPLICATIONS OF WHAT WE OBSERVE ABOUT AGENCY BEHAVIOR PRIMARY OUTCOMES THAT ARE PROs ARE MORE LIKELY TO BE MENTIONED IN THE DECISION RATIONALE Neurology and Respiratory Indications Note: Based on a Chi-squared test, the difference between the observed frequencies and the expected frequencies were statistically significant at the .01 level. Data span 2005 – April 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 11
12. 12. THE DANGER OF A “DATA MINING” APPROACH RANDOM VARIABLES Correlation with Y X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 -0.35 -0.30 0.28 0.17 -0.23 0.20 -0.14 -0.01 0.07 0.22 -0.21 -0.07 -0.33 0.14 0.03 -0.02 0.28 -0.19 0.12 X20 X21 X22 X23 X24 X25 X26 X27 X28 X29 X30 X31 X32 X33 X34 X35 X36 X37 X38 0.09 0.21 -0.02 -0.22 0.19 0.38 0.15 -0.05 0.15 -0.39 -0.24 0.30 0.00 -0.20 -0.19 0.12 0.13 -0.27 -0.12 X39 X40 X41 X42 X43 X44 X45 X46 X47 X48 X49 X50 X51 X52 X53 X54 X55 X56 X57 -0.26 0.38 0.04 -0.41 -0.25 -0.32 -0.27 0.06 0.21 0.34 0.02 0.21 -0.36 0.22 -0.12 0.06 0.13 0.12 0.26 X58 X59 X60 X61 X62 X63 X64 X65 X66 X67 X68 X69 X70 X71 X72 X73 X74 X75 X76 Possible Drivers -0.31 -0.03 0.01 -0.23 -0.12 -0.27 0.21 0.03 0.10 -0.22 -0.36 -0.17 0.08 0.15 -0.52 -0.01 0.05 -0.04 0.10 X77 X78 X79 X80 X81 X82 X83 X84 X85 X86 X87 X88 X89 X90 X91 X92 X93 X94 -0.22 -0.03 -0.11 0.08 0.07 -0.20 0.01 -0.21 -0.23 -0.27 0.06 0.04 0.30 0.06 -0.20 0.09 0.11 0.42 X25 X29 X40 X42 X51 X68 X72 X94 Shading indicates correlation with Y is significant at 5%. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 12
13. 13. THE DANGER OF A “DATA MINING” APPROACH WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL • All estimates significant at better than 5% • Two are significant at .1% or better reg Y X72 X42 X25 X40 • Very unlikely to have occurred by chance (p=1/10,000) • Explains 2/3 of the variation in Y X51; Source | SS df MS -------------+-----------------------------Model | 1684.41859 5 336.883717 Residual | 909.007507 24 37.8753128 -------------+-----------------------------Total | 2593.42609 29 89.4284859 Number of obs F( 5, 24) Prob > F R-squared Adj R-squared Root MSE = = = = = = 30 8.89 0.0001 0.6495 0.5765 6.1543 -----------------------------------------------------------------------------Y | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------X72 | -1.654714 .5923167 -2.79 0.010 -2.877196 -.4322327 X42 | -1.369594 .5741604 -2.39 0.025 -2.554603 -.1845855 X25 | 1.435235 .6360141 2.26 0.033 .1225669 2.747904 X40 | 1.329745 .5173819 2.57 0.017 .2619215 2.397569 X51 | -1.138341 .5299355 -2.15 0.042 -2.232074 -.044608 _cons | 677.9564 239.1864 2.83 0.009 184.3 1171.613 -----------------------------------------------------------------------------Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 13
14. 14. THE DANGER OF A “DATA MINING” APPROACH WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL 45° (predicted = actual) Actual vs. Predicted 60 50 R2 = .65 40 Predicted 30 20 10 0 0 10 20 30 40 50 60 Actual Source: Y and all Xs are independent random variables. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 14
15. 15. THE DANGER OF A “DATA MINING” APPROACH THE MODEL IS USELESS: IT HAS NO PREDICTIVE VALUE Out-of-Sample Prediction 60 50 40 Best Prediction Predicted 30 20 10 0 0 10 20 30 40 50 60 Actual Source: Y and all Xs are independent random variables. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 15
16. 16. HOW WILL WE KNOW IF WE BUILT A GOOD MODEL? HOW DO WE JUDGE OUR MODEL? • Predicts well • Predicts better than alternatives Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 16
17. 17. POSITIVE DECISIONS BY AGENCY POSITIVE DECISION RATES (2007-2013) Positive Decision Rate HAS 95% NICE 72% CADTH 67% SMC 67% PBAC 62% Note: Based on 213 reviews of about 40 therapeutic areas between 2005 – May 2013 reviewed by NICE and at least one other major agency: SMC, PBAC, HAS, and CADTH. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 17
18. 18. HOW WILL WE KNOW IF WE BUILT A GOOD MODEL? HOW DO WE JUDGE OUR MODEL? • Predicts well • Predicts better than alternatives • Provides actionable guidance for how to increase the likelihood of a positive reimbursement decision Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 18