Forecasting is a cornerstone of profitability for many industry. Overfitting is a subtle phenomenon that lead to very deceptive measurements about the expected accuracy of your forecasts. This talk is a gentle and not too technical intuitive introduction of the overfitting problem

1. Overfitting : your forecasts may not
be as good as the measure tells you
Joannes Vermorel, 2009-04-19, www.lokad.com

2. Overfitting
Subtle problem
Too little known
Key facts
• it costs money
• no work-around
Joannes Vermorel, 2009-04-19, www.lokad.com

3. Forecasting accuracy?
Wait and see?
Not tractable
• Can’t wait.
• Method is changing.
Joannes Vermorel, 2009-04-19, www.lokad.com

4. Forecasting model
Joannes Vermorel, 2009-04-19, www.lokad.com

5. Linear model (error > 50%)
(numbers are made-up, just an illustration)
Joannes Vermorel, 2009-04-19, www.lokad.com

6. More complexity (error ≈ 10%)
Joannes Vermorel, 2009-04-19, www.lokad.com

7. Too much complexity (error < 1%)
Joannes Vermorel, 2009-04-19, www.lokad.com

8. Comparing models
10% error 1% error
Model “fits” Model “overfits”
Joannes Vermorel, 2009-04-19, www.lokad.com

9. Comparing models
10% empirical error 1% empirical error
Model “fits” Model “overfits”
Joannes Vermorel, 2009-04-19, www.lokad.com

10. Comparing models
real error = ? real error = ?
Model “fits” Model “overfits”
Joannes Vermorel, 2009-04-19, www.lokad.com

11. Real error
Def: it’s the error on the
data you don’t have.
Paradox?
Joannes Vermorel, 2009-04-19, www.lokad.com

12. Old school: bias vs. variance
high bias medium bias low bias
low variance medium variance high variance
Joannes Vermorel, 2009-04-19, www.lokad.com

13. Bounding the real error
1995: The Nature of Statistical Learning Theory
by Vladimir Vapnik (AT&T Bell Labs)
Real Error < Empirical Error + Structural Risk
Joannes Vermorel, 2009-04-19, www.lokad.com

14. Empirical Error and Structural Risk
Low risk Medium risk High risk
Joannes Vermorel, 2009-04-19, www.lokad.com

15. Empirical Error and Structural Risk
(numbers are made-up, just an illustration)
Emp. Error = 50% Emp. Error = 10% Emp. Error = 1%
Stuct. Risk = 10% Stuct. Risk = 20% Stuct. Risk = 99%
Real error < 60% Real error < 30% Real error < 100%
Joannes Vermorel, 2009-04-19, www.lokad.com

16. Estimating the structural risk
1% error
This criterion is
• Theoretical
• Difficult
• Yet unavoidable
Model “overfits”
Joannes Vermorel, 2009-04-19, www.lokad.com

17. Estimating the structural risk
1% error
Overfitting is probable if
• Data is complex
• Models are complex
No magic: happens also
with linear models
Model “overfits”
Joannes Vermorel, 2009-04-19, www.lokad.com

18. Conclusions
Without taking into account the structural risk,
your measure of the forecast error can be deceptive.
Shameless plug : No time to compute your structural risks?
Let Lokad handle your forecasts, so that you don’t have to.
Ask your questions at http://forums.lokad.com/
Academic References:
1984: A Theory of the Learnable. Communications of the ACM,
27(11):1134--1142, L. Valiant
1995: The Nature of Statistical Learning Theory
Springer, by Vladimir Vapnik (AT&T Bell Labs)
Joannes Vermorel, 2009-04-19, www.lokad.com