The document discusses the limitations of traditional statistical methods like sensitivity, specificity, and AUC for evaluating prediction models and tests. It argues that these metrics do not consider clinical consequences and do not help clinicians determine which test to use. The document introduces decision curve analysis as an alternative that incorporates the relative importance of sensitivity vs specificity through a threshold probability parameter. Decision curve analysis calculates net benefit across a range of threshold probabilities to assess clinical value, addressing limitations of traditional metrics.

2. The Kattan challenge A clinician comes to you with two models (or tests) and wants to know which to use. What statistical method do you use to help answer the clinician?

3. Traditional biostatistical metrics Sensitivity Specificity PPV NPV LR+ LR- AUC (Youden) Brier (mean squared error) % Free PSA < 20% 91% 40% 35% 92% 1.52 0.23 0.65 0.47 hK2 > 75 pg / ml 51% 78% 45% 82% 2.32 0.63 0.64 0.29

6. Prostate cancer High risk (10%) Low Risk (90%)

7. Prostate cancer Use of the model to determine treatment would avoid 47 unnecessary treatments at the expense of failing to treat 45 patients who do require treatment.

15. Calibration plot

21. Application to models with a continuous endpoint 1. Select a p t 2. Positive test defined as 3. Calculate “Clinical Net Benefit” as:

22. Kallikrein panel: weight false positives by 20% ÷ (1 – 20%) Cancers found Unecessary Biopsies Net benefit Biopsy all men with elevated PSA 277 723 277 - 723 ÷ 4 96.25 Biopsy if risk ≥ 20% on panel 211 276 211 - 276 ÷ 4 142

25. Decision curve analysis 4. Vary p t over an appropriate range Vickers & Elkin Med Decis Making 2006;26:565–574 1. Select a p t 2. Positive test defined as 3. Calculate “Clinical Net Benefit” as:

26. Decision curve analysis

28. Decision curve analysis

31. Statistical analysis vs. decision analysis Traditional statistical analysis Traditional decision analysis Mathematics Simple Can be complex Additional data Not required Patient preferences, costs or effectiveness Endpoints Binary or continuous Continuous endpoints problematic Assess clinical value? No Yes

32. Statistical analysis vs. decision analysis Traditional statistical analysis Traditional decision analysis Decision curve analysis Mathematics Simple Can be complex Simple Additional data Not required Patient preferences, costs or effectiveness Informal, general estimates Endpoints Binary or continuous Continuous endpoints problematic Binary or continuous Assess clinical value? No Yes Yes