Slides of the PyData conference in Eindhoven about common pitfalls leading to wrongly estimated model performance

1. Common pitfalls leading to wrongly
estimated model performance
Jan van der Vegt
Cubonacci
PyData Eindhoven – November 30th 2019

2. Overview
▪ Problem statement
▪ Generalization
▪ Assumptions
▪ Validation schemes
▪ Other pitfalls

3. Problem statement

5. Generalization
How well does our model adapt to new, previously unseen data, drawn
from the same distribution as the one used to create the model.
How well does our model predict on unseen data
How well is our model going to predict in production
How much value is this model going to generate

6. Evaluation
How well does our model predict on unseen data
𝐼 𝑓𝑛 =
𝑋×𝑌
𝑉 𝑓𝑛 𝑥 , 𝑦 𝜌 𝑥, 𝑦 𝑑𝑥𝑑𝑦

7. Evaluation
Model (train) Unseen
(validation)
𝐼 𝑓𝑛 =
𝑋×𝑌
𝑉 𝑓𝑛 𝑥 , 𝑦 𝜌 𝑥, 𝑦 𝑑𝑥𝑑𝑦
𝐼𝑆 𝑓𝑛 =
1
𝑛
𝑖=1
𝑛
𝑉 𝑓𝑛 𝑥 , 𝑦 = 𝐼 𝑓𝑛 + 𝜺
𝔼𝜀 = 0

8. Thought experiment 𝑦 = 𝒂𝑥 + 𝒃
𝑦 − 𝑦 2
a b MSE
0.7 0.1 4.5
0.2 0.8 0.4
-0.6 1.6 2.6
0.1 0.4 1.9

9. Thought experiment
𝐼𝑆 𝑓𝑛 =
1
𝑛
𝑖=1
𝑛
𝑉 𝑓𝑛 𝑥 , 𝑦 = 𝐼 𝑓𝑛 + 𝜺
Best model
Fair evaluation
𝔼𝜀 < 0

10. Train | validate | test
Model (train) Unseen
(validation)
𝐼𝑆 𝑓𝑛 =
1
𝑛
𝑖=1
𝑛
𝑉 𝑓𝑛 𝑥 , 𝑦 = 𝐼 𝑓𝑛 + 𝜺
Unseen
(test)

11. Validation schemes

12. Nested crossvalidation
Model A
Model B
Model A
4.5
2.5
3.5
3.5

13. (Broken) assumptions
Identical, independently distributed
X1 X2 y
1 2 0
2 1 1
3 5 1
4 3 0
5 4 0
𝐹 𝑋𝑖 < 𝑥 = 𝐹(𝑋𝑗 < 𝑥) 𝐹 𝑋𝑖 < 𝑥 = 𝐹 𝑋𝑖 < 𝑥𝑖 𝑋𝑗 = 𝑥𝑗)

14. Validation schemes

15. Validation schemes

16. Validation schemes

17. Validation schemes

18. Training size bias
0% 80%

19. Duplication
X1 X2 y
2 3 0
3 4 1
1 1 1
4 2 0
4 2 0

20. Leakage
X1 X2 y
2 3 0
3 4 1
1 1 1
4 2 0
X1 X2 y
2 3 0
3 4 1
1 1 1
4 2 0
y
0
1
1
0

21. Leakage
X1 X2 y
2 Cat A 0
3 Cat A 1
1 Cat B 1
4 Cat B 1
X1 CatTargetAvg
2 0.5
3 0.5
1 1
4 1
y
0
1
1
1

22. Leakage
X1 X2 y
2 Cat A 0
3 Cat A 1
1 Cat B 1
4 Cat B 1
X1 CatTargetAvg
2 0.5
3 0.5
1 1
4 1
X1 CatTargetAvg
2 1
3 0
1 1
4 1

23. Takeaway
Estimating performance is about simulating real life
Think about:
▪ Assumptions
▪ Data generation
▪ Data availability
▪ The goal

24. Thank you for your time
https://www.linkedin.com/in/jan-van-der-vegt/
jan.vandervegt@cubonacci.com