This document summarizes key concepts in machine learning evaluation including: 1. Common evaluation metrics like accuracy, precision, recall, and ROC curves. 2. Offline evaluation techniques like cross-validation to estimate model performance. 3. Hyperparameter tuning to optimize model configuration. 4. A/B testing to evaluate model impact by comparing control and experiment groups. 5. Causal effect measurement techniques like propensity score matching to estimate treatment effects.

1. Sonpvh – VNG 2017-july-11
LITERATURE REVIEW

2. 1. The machine learning workflow
2. Evaluation Metrics
3. Offline Evaluation mechanisms
4. Hyperparameter turning
5. A/B Testing
6. Casual effect measurament
1

3. 2
Why is it so complicated?
1. Offline evaluation: accuracy, precision,
recall …
Online evaluation: business metrics
2. Distribution drift: the distribution of data
changes overtime, so keep track the models
performance on the validation metrics of live
data.

4.  Ex: spam detection, prostitute detection…
Confusion matrix
Per-class accuracy
3
ROC: receiver operating characteristic
AUC: area under the curve

5.  Ex: search ranker, personalized recommendation ..
4
"The precision is the proportion of recommendations that
are good recommendations, and recall is the proportion of
good recommendations that appear in top
recommendations."

6.  Ex: prediction…

5

7.  Evaluation metrics # model log loss function: Train a personalized recommender
by minimizing the loss between its predictions and observed ratings, and then use
this recommender to produce a ranked list of recommendations. AVOID
 Skewed data, imbalanced, classes, outliers, rare data: analysis carefully before
doing anything else
6

8. 1. The machine learning workflow
2. Evaluation Metrics
3. Offline Evaluation mechanisms
4. Hyperparameter turning
5. A/B Testing
6. Casual effect measurament
7

9. 8

10. 9
Cross validation:
Independently and Identically distributed

11. 1. The machine learning workflow
2. Evaluation Metrics
3. Offline Evaluation mechanisms
4. Hyperparameter turning
5. A/B Testing
6. Casual effect measurament
10

12. 11
 Model parameter: y = WT x
 Hyper-parameter (nuisance parameters): optimization state.
 Ex:
 Linear regression: regularization parameter,
 Decision trees: desired depth and number of leaves.
 SVMs: misclassification penalty term
 ..

13. 1. The machine learning workflow
2. Evaluation Metrics
3. Offline Evaluation mechanisms
4. Hyperparameter turning
5. A/B Testing
6. Casual effect measurament
12

14. 1. Split into randomized control/experimentation groups.
2. Observe behavior of both groups on the proposed methods.
3. Compute test statistics.
4. Compute p-value.
5. Output decision.
13

15. 1. Baggage of the old: should do A/A testing first
2. Choose metrics, indexes (business design)
3. Did you count right?
4. How many observations do you need?
5. Is the distribution of the metric Gaussian?
6. Variances equal?
7. Multiple models, multiple hypotheses: A/A1/A2/…/B testing
8. How long to run the test?
9. Catching distribution drift: stationarity assumption
14

16. 1. The machine learning workflow
2. Evaluation Metrics
3. Offline Evaluation mechanisms
4. Hyperparameter turning
5. A/B Testing
6. Casual effect measurament
15

17.  D = E(Yi(1)|Ti=1)- E(Yi(0)|Ti=0) + [E(Yi(0)|T=1) – E(Yi(0) |Ti=1)]
D: casual effect
Y(1),Y(0): outcome
T =1/0: treatment/control group
E(): expectation function
 D = ATE + B
 ATE = E(Yi(1)|Ti=1)- E(Yi(0)|Ti=1) Average Treatment Effect
 B = [E(Yi(0)|T=1) – E(Yi(0) |Ti=0)]
 We cannot observe E(Yi(0)|T=1) => cannot evaluate B => cannot evaluate D.
16

18. 17
TOT = E[Yi(1) – Yi(0)|Ti=1] = E[Yi(1)|Ti=1] - E[Yi(0)|Ti=1]
Because of random sample, we have: E[Yi(0)|Ti=1] = E[Yi(0)|Ti=0]
So E[Yi(1)|Ti=1] - E[Yi(0)|Ti=0] = D

19. 18
1. (Conditional) independence
2. Common support
3. TOT = EP(X)|T=1 {E[Y(1)|T=1, P(X)] – E[Y(0)|T=0, P(X)]}

20. 19

21.  cov(T, Z) != 0
 cov(Z, ε) = 0 (exclusion restriction)
 Ti = γZi + φXi + ui , predict Ti
 Yi = αXi + βTi + εi
20