ROC curve analysis implementation on drought data

2. Contents
ROC Curve

3.  ‘The 170th relative Korean combination drought, need for better prediction and alarming service
- (’18, 3month prediction) meteorological prediction 10.0% agriculture, 0.0% living/eating, 16.8% industrial
 For field crop, [prediction]alarm should not be performed, [current]situation alarm only.
 Prediction duration 2018’ Jan. criteria 2018. 4. ~ 2019. 3. alarm/prediction [current]situation is only 1 area
 August rice paddy drought situation is only at 1 place, even wrong
 It is hard to make prediction/alarm on just on place which is definitely incomplete in accuracy for 1 case study
2018’ May <3month
prediction>
2018’ August <Current>
구 분
2018’ May
<3 month prediction>
구 분 2018’ August <current>
alarm
(Yellow)
-
alarm
(Yellow)
<rice paddy>
[전남] 나주시
<field crop>
[충북] 영동군
[전남] 무안군,장성군,완도군,
신안군
[경북] 안동시
[경남] 거창군
[제주] 제주시,서귀포시
severe
(Orange)
-
severe
(Orange)
-
Very
severe
(Red)
-
Very
severe
(Red)
-

4. The 170st governmental
discussion TF workforce
Drought
alarm/prediction
workshop
Statistician consultant
• Chung-buk university Dr. Hur
tae-young professor
The 174th TF workforce
discussion
6.7. 6.14. 7.8.
7.10.
7.26.
Drought prediction/alarm
seminar and professional
workforce
7.19.
Statistician consultant
• Chung-buk university Dr. Hur
tae-young professor

5. Contents
ROC Curve

6.  If performed prediction, then happened TP
 If prediction, then wrong which does not happened FP
 If not predicted, then happened at the area FN
 If not predicted, then drought did not happen TN
observation
YES NO
prediction
YES TP FP
NO FN TN
* TP: True Positive, FN: False Negative (meaning)

7. All data information, including drought happening
Rare event/ disaster, inadequate index
Current not happened drought, 3 months perspective which 3
months after it does not happen, possibility (prob.)
1-SPE ROC Curve x axis
3 months after, it predicted not to happen and reality
it does not happened, possibility (prob.)
3 months prediction that drought does happen,
reality it happens, possibility (prob.)
Current not happened, 3 month before 3 months after
it would happened, possibility (prob.)
ROC Curve y axis
𝐴𝐶𝐶 =
𝑇𝑃 + 𝑇𝑁
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁
𝑆𝐸𝑁 =
𝑇𝑃
𝑇𝑃 + 𝐹𝑁
𝑃𝑃𝑉 =
𝑇𝑃
𝑇𝑃 + 𝐹𝑃
𝑆𝑃𝐸 =
𝑇𝑁
𝑇𝑁 + 𝐹𝑃
𝑁𝑃𝑉 =
𝑇𝑁
𝑇𝑁 + 𝐹𝑁
Observation
YES NO
prediction
YES TP FP
NO FN TN

8. ACC SEN SPE PPV NPV CSI
1 month
predict
0.991 0.821 0.993 0.639 0.997 0.561
3 month
predict
0.996 0.500 0.996 0.111 0.999 0.100
ACC SEN SPE PPV NPV CSI
1 month
predict
0.998 0.857 0.998 0.600 0.999 0.545
3 month
predict
0.995 0.000 0.995 0.000 0.999 0.000
ACC SEN SPE PPV NPV CSI
1 month
predict
0.971 0.902 0.976 0.705 0.994 0.655
3 month
predict
0.923 0.646 0.930 0.185 0.991 0.168

9.  ROC Curve is the classification technique that is used to assess the
performance of the model
 (Sensitivity) and (Specificity) relation basis
- x axis : 1-spe, y axis: sen
- prediction, inaccuracy included with all information
 Among different kinds of classification models,
select the best model, based on the criteria.
 Most objective conclusion can be made
sen
1-spe

10. Model prediction and real answer comparison for data
Real infection
(Y/N)
Prediction
probability
P 0.6
N 0.7
P 0.4
N 0.2
 Real infection and prediction probability is aligned from top to bottom
(Here, prediction probability is the model assumed to get real infected, possibility)
Real infection
(Y/N)
Prediction
probability
N 0.7
P 0.6
P 0.4
N 0.2

11. Threshold=0.7
Real infection
(Y/N)
Prediction
probability
Model
prediction
N 0.7 P
P 0.6 N
P 0.4 N
N 0.2 N
(x,y)=(
𝟏
𝟐
, 𝟎)
X axis
𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑅𝑎𝑡𝑒 =
𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑝𝑜𝑠𝑠.
𝑟𝑒𝑎𝑙 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 𝑝𝑜𝑠𝑠.
=
1
2
Y axis
𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑅𝑎𝑡𝑒 =
𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑝𝑜𝑠𝑠.
𝑟𝑒𝑎𝑙 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑝𝑜𝑠𝑠.
=
0
2

12. Threshold=0.6
Real infection
(Y/N)
Prediction
probability
Model
prediction
N 0.7 P
P 0.6 P
P 0.4 N
N 0.2 N
(x,y)=(
𝟏
𝟐
,
𝟏
𝟐
)
X축
𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑅𝑎𝑡𝑒 =
𝐹𝑎𝑙𝑠𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 수
실제 𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒 수
=
1
2
Y축
𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 𝑅𝑎𝑡𝑒 =
𝑇𝑟𝑢𝑒 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 수
실제 𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 수
=
1
2

13. For all Threshold
Real infection
(Y/N)
Prediction
probability
Model
prediction
N 0.7 P
P 0.6 P
P 0.4 P
N 0.2 P
 All Threshold is shown in the coordinates, ROC curve of AUC (AUC;Area Under The
Curve) Calculate to find the accuracy level (index) to assess

14. Graph : ROC curve
2x2 contingency table : Confusion Matrix
X-axis : sen
Y-axis : 1 - spe
Graph of adverse relationship
1-특이도
민
감
도
AUC
(area under the
curve) value
1-특이도
민
감
도
Value Result
.90-1 excellent(A)
.80-.90 good(B)
.70-.80 fair(C)
.60-.70 poor(D)
0-.60 bad(F) 1-특이도
민
감
도
Excellent
Fair
bad
< AUC vale assessment> < ROC curve example >
Graph by the threshold
1. Area calculation
2. model selection
3.

15. 1 month prediction of Korea
prediction AUC=0.907
3 month prediction of Korea
prediction AUC=0.748
1·3 month prediction accuracy performance assessment conclude
over 0.7 which is almost good

16. 1 month prediction of Korea drought prediction
AUC= 0.928
3month prediction of Korea drought
prediction AUC=0.502
1 month prediction accuracy level is over 0.9, excellent
3 month prediction accuracy level is 0.5, poor

17. 1month prediction AUC=0.939 3month prediction AUC=0.788
1·3 month prediction accuracy performance assessment conclude
over 0.7 which is almost good

18. Contents
ROC Curve

19. Drought data sen, spe calculation
Confidence level calculation
(99%, 95%, 90% is usual)
SPE, SEN, Confidence level used to calculate the
optimized data number calculation

20. Observation number
Observation number
P: specific duration drought probability, happened

21. observation
YES NO total
prediction
YES
21
True Positive(TP)
56
False Positive(FP)
77
NO
53
False Negative(FN)
1,874
True Negative(TN)
1,927
total 74 1,930 2,004
 (Sensitivity)=
𝑻𝑷
𝑻𝑷+𝑭𝑵
= 0.284
 (Specificity)=
𝑻𝑵
𝑻𝑵+𝑭𝑷
= 0.971
 n(observation =yes)=74
 N=2004
 n(observation=no)=1930

22. Observation number

23. Observation number

24. 2017’
agriculture
95% CI 90% CI
SEN
Drought observation
numbers
74 ≤313 ≥55
Total
observation
2,004 ≤8,462 ≥1,479
SPE
Drought, not
being observed
1,930 ≥44 ≥8
Total
observation
2,004 ≥46 ≥8
2017’ agriculture drought 3 months prediction assessment ROC index –
SEN: 0.284, SPE: 0.971 falls in the range of 90% CI

25. observation
YES NO total
predi
ction
YES 0 10 77
NO 1 1,993 1,927
total 1 2,003 2,004
 SEN=0
 SPE = 0.995
 n(observation=yes)=1
 n(observation=no)=2003
 N=2004
SEN= 0 so that the least number of data needed is 0?? How?
Past data analysis yields predicted SEN could be used to come up with necessary
observation or apply the 0.5 (max value) into the calculation

26. 2018’ agriculture
drought
95%CI 90%CI
SEN
Drought observation
number
1 ≤385 ≤68
Total data numbers 2,004 ≤769,857 ≤134,750
SPE
Drought, not
being observed
2,003 ≥8 ≥2
Total data numbers 2,004 ≥8 ≥2
2018’ agriculture drought 3 month prediction ROC index of SEN = 0 due to insufficient
data observational numbers so that we could not trust the SEN, SPE
as it does not fall into CI