HOW TO BUILD A MODEL TO PREDICT REIMBURSEMENT
DECISIONS BY MAJOR HTA/CER AGENCIES
Kermit Daniel, PhD – Chief Analytics Off...
BUILDING A PREDICTIVE MODEL OF HTA
BEHAVIOR
• Why build a model?
• Can we build a model?
• How do we build a model?
• How ...
WHY BUILD A MODEL?
A GOOD MODEL WILL

• Improve predictions
• Disentangle multiple influences
• Provide actionable guidanc...
WHY BUILD A MODEL?
A MODEL IS

• Useful simplification
• Mathematical: Y = Xβ + ε
Y: Probability of a positive recommendat...
CAN WE BUILD A MODEL?
WHAT HAS TO BE TRUE?

• Agencies behave in ways that can be
described by a simple model that we can
...
CONSISTENCY ACROSS AGENCIES

AGENCIES FREQUENTLY AGREE WHEN PRESENTED WITH THE SAME
DRUG/INDICATION TO REVIEW

FREQUENCY O...
CONSISTENCY BETWEEN DECISION AND ASSESSMENTS
THE CLINICAL AND/OR ECONOMIC ASSESSMENT AND THE REIMBURSEMENT
DECISION ARE US...
TRANSPARENCY

WE OBSERVE A LOT ABOUT REVIEWS, INCLUDING . . .

DRUGS

CLINICAL STUDIES

SUBMISSIONS

• Chemical type
• Ind...
HOW DO WE BUILD A MODEL?
HOW DO WE DECIDE WHAT DATA TO
INCLUDE?
• Predict outcomes that matter
• Capture implications of a...
PREDICT OUTCOMES THAT MATTER
POSITIVE DECISIONS OFTEN ADD RESTRICTIONS

Oncology

NICE

Non-Oncology

55%

54%
46%

45%

P...
CAPTURE IMPLICATIONS OF WHAT WE
OBSERVE ABOUT AGENCY BEHAVIOR

PRIMARY OUTCOMES THAT ARE PROs ARE MORE LIKELY TO BE MENTIO...
THE DANGER OF A “DATA MINING” APPROACH
RANDOM VARIABLES

Correlation
with Y
X1
X2
X3
X4
X5
X6
X7
X8
X9
X10
X11
X12
X13
X14...
THE DANGER OF A “DATA MINING” APPROACH
WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL
• All estimates
significant at better...
THE DANGER OF A “DATA MINING” APPROACH
WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL

45°
(predicted = actual)

Actual vs....
THE DANGER OF A “DATA MINING” APPROACH
THE MODEL IS USELESS: IT HAS NO PREDICTIVE VALUE

Out-of-Sample Prediction
60
50
40...
HOW WILL WE KNOW IF WE BUILT A GOOD
MODEL?
HOW DO WE JUDGE OUR MODEL?

• Predicts well
• Predicts better than alternatives...
POSITIVE DECISIONS BY AGENCY
POSITIVE DECISION RATES (2007-2013)
Positive Decision Rate
HAS

95%

NICE

72%

CADTH

67%

S...
HOW WILL WE KNOW IF WE BUILT A GOOD
MODEL?
HOW DO WE JUDGE OUR MODEL?

• Predicts well
• Predicts better than alternatives...
Upcoming SlideShare
Loading in...5
×

How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies

111
-1

Published on

Published in: Health & Medicine, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
111
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILDING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
  • I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
  • Agency agreement – main agencies
  • Clinical assessment agreement, economic assessment agreement. NOTE for this presentation we did not have a joint table of agreement on the clinical AND economic in relationship to the decision.
  • I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
  • I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
  • LESS SALEY
  • LESS SALEY
  • I DON’T HAVE A MODEL TODAY, BUT WE ARE STARTING TO BUILD A MODEL, SO I WILL TALK ABOIUT HOW WE’RE GOING ABOUT BUILING ONE. I’M GONG TO ORGANIZE THIS AROUND HOW WE ARE ANSWERING 4 IMPORTANT QUESTIONS. (NOT THE ONLY QEUSTIONS TO ASK, MAYBE NOT THE MOST IMPORTANT, BUT THEY ARE 4 IMPORTANT QUESTIONS0[WHYBUILD IS WHERE I CAN TALK ABOUT CONTEXT MATTERS?]
  • Number of reports: 78, 15, 39,39,42 (top to bottom)
  • How to Build a Model to Predict Major Reimbursement Decisions by Major HTA / CER Agencies

    1. 1. HOW TO BUILD A MODEL TO PREDICT REIMBURSEMENT DECISIONS BY MAJOR HTA/CER AGENCIES Kermit Daniel, PhD – Chief Analytics Officer, Context Matters, Inc. December 2013 Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED
    2. 2. BUILDING A PREDICTIVE MODEL OF HTA BEHAVIOR • Why build a model? • Can we build a model? • How do we build a model? • How will we know if we built a good model? Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 2
    3. 3. WHY BUILD A MODEL? A GOOD MODEL WILL • Improve predictions • Disentangle multiple influences • Provide actionable guidance for how to increase the likelihood of a positive reimbursement decision Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 3
    4. 4. WHY BUILD A MODEL? A MODEL IS • Useful simplification • Mathematical: Y = Xβ + ε Y: Probability of a positive recommendation X: Observable influences of recommendations β: Parameters we will estimate ε: Effect of influences we don’t observe Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 4
    5. 5. CAN WE BUILD A MODEL? WHAT HAS TO BE TRUE? • Agencies behave in ways that can be described by a simple model that we can estimate o Consistency - non-random, consistent behavior • Through time • Across therapeutic areas • Across agencies o Transparency – we can observe decision factors (or good proxies) Y = Xβ + ε THIS EXPLAINS A LOT THIS EXPLAINS A LITTLE Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 5
    6. 6. CONSISTENCY ACROSS AGENCIES AGENCIES FREQUENTLY AGREE WHEN PRESENTED WITH THE SAME DRUG/INDICATION TO REVIEW FREQUENCY OF AGREEMENT BETWEEN AGENCY PAIRS CADTH HAS NICE PBAC SMC 68% 78% 85% 76% PBAC 63% 72% 85% NICE 61% 65% HAS 58% Note: Reviews of 94 drugs reviewed by at least two agencies between January 2005 – February 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 6
    7. 7. CONSISTENCY BETWEEN DECISION AND ASSESSMENTS THE CLINICAL AND/OR ECONOMIC ASSESSMENT AND THE REIMBURSEMENT DECISION ARE USUALLY CONSISTENT AGENCY PAIR Same Clinical Outcome Different Clinical Assessment PBAC:SMC P value Same Economic Outcome Different Economic Assessment 0.05* P value 0.00* Agree on decision 27 4 20 7 Disagree on decision 5 4 1 8 CADTH:PBAC 0.00* 0.14 Agree on decision 16 4 11 8 Disagree on decision 2 11 3 9 SMC:HAS Agree on decision 0.01* 32 9 4 9 Disagree on decision Note: Reviews of 94 drugs reviewed by at least two agencies between January 2005 – February 2013. Asterisk and shading indicates significance at 5% or better. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 7
    8. 8. TRANSPARENCY WE OBSERVE A LOT ABOUT REVIEWS, INCLUDING . . . DRUGS CLINICAL STUDIES SUBMISSIONS • Chemical type • Indications • Regulatory history • Orphan status • Number, dates • Design, e.g., goal, comparators, size, subpopulations, length • HTA assessment • Number • Professional, patient, other group support AGENCIES ECONOMIC MODELS CONCLUSIONS • Identity • Region • Agency-specific factors, e.g., additional benefit • Manufacturer comparators & assumptions • Agency model • HTA assessment • Recommendation • Restrictions • Factors discussed, e.g., study limitations, PRO use Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 8
    9. 9. HOW DO WE BUILD A MODEL? HOW DO WE DECIDE WHAT DATA TO INCLUDE? • Predict outcomes that matter • Capture implications of agency objectives and constraints • Incorporate what we observe about agency behavior – but this is dangerous! Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 9
    10. 10. PREDICT OUTCOMES THAT MATTER POSITIVE DECISIONS OFTEN ADD RESTRICTIONS Oncology NICE Non-Oncology 55% 54% 46% 45% Positive Decision Not More Restrictive Positive Decision More Restrictive Note: Based on 150 NICE reviews of 72 drugs for 34 diseases between January 2007 – August 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 10
    11. 11. CAPTURE IMPLICATIONS OF WHAT WE OBSERVE ABOUT AGENCY BEHAVIOR PRIMARY OUTCOMES THAT ARE PROs ARE MORE LIKELY TO BE MENTIONED IN THE DECISION RATIONALE Neurology and Respiratory Indications Note: Based on a Chi-squared test, the difference between the observed frequencies and the expected frequencies were statistically significant at the .01 level. Data span 2005 – April 2013. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 11
    12. 12. THE DANGER OF A “DATA MINING” APPROACH RANDOM VARIABLES Correlation with Y X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12 X13 X14 X15 X16 X17 X18 X19 -0.35 -0.30 0.28 0.17 -0.23 0.20 -0.14 -0.01 0.07 0.22 -0.21 -0.07 -0.33 0.14 0.03 -0.02 0.28 -0.19 0.12 X20 X21 X22 X23 X24 X25 X26 X27 X28 X29 X30 X31 X32 X33 X34 X35 X36 X37 X38 0.09 0.21 -0.02 -0.22 0.19 0.38 0.15 -0.05 0.15 -0.39 -0.24 0.30 0.00 -0.20 -0.19 0.12 0.13 -0.27 -0.12 X39 X40 X41 X42 X43 X44 X45 X46 X47 X48 X49 X50 X51 X52 X53 X54 X55 X56 X57 -0.26 0.38 0.04 -0.41 -0.25 -0.32 -0.27 0.06 0.21 0.34 0.02 0.21 -0.36 0.22 -0.12 0.06 0.13 0.12 0.26 X58 X59 X60 X61 X62 X63 X64 X65 X66 X67 X68 X69 X70 X71 X72 X73 X74 X75 X76 Possible Drivers -0.31 -0.03 0.01 -0.23 -0.12 -0.27 0.21 0.03 0.10 -0.22 -0.36 -0.17 0.08 0.15 -0.52 -0.01 0.05 -0.04 0.10 X77 X78 X79 X80 X81 X82 X83 X84 X85 X86 X87 X88 X89 X90 X91 X92 X93 X94 -0.22 -0.03 -0.11 0.08 0.07 -0.20 0.01 -0.21 -0.23 -0.27 0.06 0.04 0.30 0.06 -0.20 0.09 0.11 0.42 X25 X29 X40 X42 X51 X68 X72 X94 Shading indicates correlation with Y is significant at 5%. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 12
    13. 13. THE DANGER OF A “DATA MINING” APPROACH WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL • All estimates significant at better than 5% • Two are significant at .1% or better reg Y X72 X42 X25 X40 • Very unlikely to have occurred by chance (p=1/10,000) • Explains 2/3 of the variation in Y X51; Source | SS df MS -------------+-----------------------------Model | 1684.41859 5 336.883717 Residual | 909.007507 24 37.8753128 -------------+-----------------------------Total | 2593.42609 29 89.4284859 Number of obs F( 5, 24) Prob > F R-squared Adj R-squared Root MSE = = = = = = 30 8.89 0.0001 0.6495 0.5765 6.1543 -----------------------------------------------------------------------------Y | Coef. Std. Err. t P>|t| [95% Conf. Interval] -------------+---------------------------------------------------------------X72 | -1.654714 .5923167 -2.79 0.010 -2.877196 -.4322327 X42 | -1.369594 .5741604 -2.39 0.025 -2.554603 -.1845855 X25 | 1.435235 .6360141 2.26 0.033 .1225669 2.747904 X40 | 1.329745 .5173819 2.57 0.017 .2619215 2.397569 X51 | -1.138341 .5299355 -2.15 0.042 -2.232074 -.044608 _cons | 677.9564 239.1864 2.83 0.009 184.3 1171.613 -----------------------------------------------------------------------------Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 13
    14. 14. THE DANGER OF A “DATA MINING” APPROACH WE APPEAR TO HAVE A POWERFUL PREDICTIVE MODEL 45° (predicted = actual) Actual vs. Predicted 60 50 R2 = .65 40 Predicted 30 20 10 0 0 10 20 30 40 50 60 Actual Source: Y and all Xs are independent random variables. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 14
    15. 15. THE DANGER OF A “DATA MINING” APPROACH THE MODEL IS USELESS: IT HAS NO PREDICTIVE VALUE Out-of-Sample Prediction 60 50 40 Best Prediction Predicted 30 20 10 0 0 10 20 30 40 50 60 Actual Source: Y and all Xs are independent random variables. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 15
    16. 16. HOW WILL WE KNOW IF WE BUILT A GOOD MODEL? HOW DO WE JUDGE OUR MODEL? • Predicts well • Predicts better than alternatives Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 16
    17. 17. POSITIVE DECISIONS BY AGENCY POSITIVE DECISION RATES (2007-2013) Positive Decision Rate HAS 95% NICE 72% CADTH 67% SMC 67% PBAC 62% Note: Based on 213 reviews of about 40 therapeutic areas between 2005 – May 2013 reviewed by NICE and at least one other major agency: SMC, PBAC, HAS, and CADTH. Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 17
    18. 18. HOW WILL WE KNOW IF WE BUILT A GOOD MODEL? HOW DO WE JUDGE OUR MODEL? • Predicts well • Predicts better than alternatives • Provides actionable guidance for how to increase the likelihood of a positive reimbursement decision Copyright © 2013 Context Matters, Inc. ALL RIGHTS RESERVED 18
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×