The document discusses various evaluation metrics that can be used for binary classification and click prediction, including AUC, RIG, LogLoss, precision, recall, and F1. It notes that AUC ignores predicted probabilities and considers type 1 and type 2 errors equally. RIG is bad for comparing models with different data distributions but can be used to compare multiple models trained on the same data. The document also provides a reference for more information on offline and online predictive model performance evaluations.

1. EVALUATION METRICS
FOR CLICK PREDICTION
Evgeniy Zhurin, RuTarget

2. Binary Classification Error Measurement
1) AUC
2) RIG
3) LogLoss
4) Precision/Recall
5) F1
6) PE, MSE, MAE

8. AUC
1) ignores the predicted probability values
2) usually we are interested in parts of
roc curve
3) considers Type 1 error and Type 2
error weights equivalently
4) dependent on the underlying distribution
of data

12. RIG
1) bad to compare two model
performances with different
distributions
2) can be used to compare the relative
performance of multiple models trained
and tested on the same data
3) is not informative, because score also
depends on the data distribution

14. OR WRITE A SIMULATOR

15. Thanks!
J. Yi, Y. Chen, J. Li, S. Sett, and T. W. Yan.
Predictive model performance: Offline and
online evaluations. In KDD, pages 1294–1302,
2013.
http://chbrown.github.io/kdd-2013-
usb/kdd/p1294.pdf