The document discusses statistical learning techniques including bias-variance tradeoff, overfitting, regularization, and validation. It covers theory topics such as the Vapnik-Chervonenkis dimension and central limit theorem. Regularization heuristics like weight decay and different weights for layers are examined. The document reiterates key concepts of statistical learning, bias-variance tradeoff, overfitting, regularization, and validation throughout.

1. R&D – VNG 19 Sep 2017

2.  Statistical learning
 Bias – variance Tradeoff
 Overfitting
 Regularization
 Validation
1

3. 2

4.  Theory of generalization
Vapnik-Chervonenkid
3
ℙ 𝑬𝒊𝒏(𝒈 − 𝑬 𝒐𝒖𝒕(𝒈 > 𝓔 ≤ 𝟒𝒎 𝓗 𝟐𝑵 −
𝟏
𝟖
𝜺 𝟐 𝑵

5. 4
 Central limit theorem

6. 5
 The Gauss Markov Theorem:

7.  Statistical learning
 Bias – variance Tradeoff
 Overfitting
 Regularization
 Validation
6

8. 7

9. 8

10. 9

11. 10

12.  Statistical learning
 Bias – variance Tradeoff
 Overfitting
 Regularization
 Validation
11

13. 12

14. 13

15. 14

16.  Statistical learning
 Bias – variance Tradeoff
 Overfitting
 Regularization
 Validation
15

17. 16

18. 17
SOFT CONSTRAINT

19. 18
Heuristics
- Weight decay
- Different layer get different weight
- Tikhonov regulerizer

20. 19
Practical rule Idea

21. 20

22. 21

23.  Statistical learning
 Bias – variance Tradeoff
 Overfitting
 Regularization
 Validation
22

24. 23

25. 24

26. 25
OPTIMISTICS BIAS