Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

10 Ways Backtests Lie by Tucker Balch

3,818 views

Published on

"I’ve never seen a bad backtest” — Dimitris Melas, head of research at MSCI. Quantitative Analysts rely heavily on backtests as a means of validating their trading strategies. All too often, strategies look great in simulation but fail to live up to their promise in live trading. There are a number of reasons for these failures, some of which are beyond the control of a quant developer. But other failures are caused by common but insidious mistakes. In this talk I’ll review a list of 10 pitfalls in strategy development and testing that can result in optimistic backtests. I’ll also present methods for detecting and avoiding them. This talk will be of interest to quant developers and also non-quants who are interested to know what to look out for when presented with remarkably successful backtests.

This presentation was part of the QuantCon 2015 Conference hosted by Quantopian. Visit us at: www.quantopian.com.

Published in: Economy & Finance

10 Ways Backtests Lie by Tucker Balch

  1. 1. 10 WAYS BACKTESTS LIE TUCKER BALCH, PH.D. PROFESSOR, GEORGIA TECH CO-FOUNDER AND CTO, LUCENA RESEARCH
  2. 2. OR… Mistakes quant developers make that cause backtests to be inaccurate.
  3. 3. WHAT I’LL COVER Introductions What is a backtest? How backtests lie: 1.  In sample backtesting 2.  Survivor bias 3.  Assume you can observe the close and trade at the close 4.  Ignoring market impact 5.  Assume you can buy $10M of a $1M company 6.  Data mining fallacy 7.  Stateful strategy luck 8.  Buy at the open 9.  Don’t trust complex models 10.  Don’t forward test
  4. 4. ABOUT THE SPEAKER •  Professor of Interactive Computing at Georgia Institute of Technology. •  Teach courses in Artificial Intelligence and Finance. •  Teach MOOCs on Machine Learning for Trading. •  Published over 120 research publications related to Robotics and Machine Learning. •  Co-founder of Lucena Research.
  5. 5. ABOUT THE SPEAKER •  Professor of Interactive Computing at Georgia Institute of Technology. •  Teach courses in Artificial Intelligence and Finance. •  Teach MOOCs on Machine Learning for Trading. •  Published over 120 research publications related to Robotics and Machine Learning. •  Co-founder of Lucena Research. New book ->
  6. 6. ABOUT LUCENA RESEARCH •  We are a fin-tech company who employ experts in Computational Finance, Quantitative Analysis, and Software Development. •  We deliver investment decision support technology at a fraction of the cost of an in house quant shop. •  Python-based infrastructure. •  Erez Katz, CEO •  Eric Davidson, VP http://lucenaresearch.com LUCENA RESEARCH!
  7. 7. BUILDING A MODEL FROM DATA
  8. 8. BUILDING A MODEL FROM DATA
  9. 9. BUILDING A MODEL FROM DATA
  10. 10. BUILDING A MODEL FROM DATA
  11. 11. BUILDING A MODEL FROM DATA
  12. 12. BUILDING A MODEL FROM DATA
  13. 13. BUILDING A MODEL FROM DATA
  14. 14. BUILDING A MODEL FROM DATA
  15. 15. BACKTESTING TO VALIDATE THE MODEL
  16. 16. BACKTESTING TO VALIDATE THE MODEL
  17. 17. BACKTESTING TO VALIDATE THE MODEL
  18. 18. BACKTESTING TO VALIDATE THE MODEL
  19. 19. BACKTESTING TO VALIDATE THE MODEL
  20. 20. BACKTESTING TO VALIDATE THE MODEL
  21. 21. BACKTESTING TO VALIDATE THE MODEL Roll forward cross validation Out of sample validation
  22. 22. 10 WAYS BACKTESTS LIE
  23. 23. 1. IN SAMPLE BACKTESTING Description: Backtesting over the same data you used to train your model.
  24. 24. 1. IN SAMPLE BACKTESTING Description: Backtesting over the same data you used to train your model. This method is doomed to succeed spectacularly!
  25. 25. 1. IN SAMPLE BACKTESTING Description: Backtesting over the same data you used to train your model. Training Testing
  26. 26. 1. IN SAMPLE BACKTESTING How to avoid? Training Testing
  27. 27. 1. IN SAMPLE BACKTESTING How to avoid? More generally, build safeguards and procedures to prevent testing over the same data you train over. E.g., Train over 2007, test over 2008-forward.
  28. 28. 2. SURVIVOR BIAS Description: Selective use of data in a statistical study that emphasizes examples that are “alive” at the end of the study. The significance of the bias depends on how important survival is to the quantity being measured.
  29. 29. 2. SURVIVOR BIAS Example: Company claims: “Our drug reduces the blood pressure of those who take the drug over time.” 5 year study: •  Randomly select 500 cardiac patients •  Administer drug to them •  Measure their blood pressure monthly Results: •  160/110 average first month •  135/80 average at end of study Do you believe this is a good drug?
  30. 30. 2. SURVIVOR BIAS Problem: 58 of the patients they started with have died since the start of the study. Note: 58 of the members of the S&P 500 in 2008 are now delisted. Not just out of the S&P 500, but gone as companies. 11.6%
  31. 31. 2. SURVIVOR BIAS
  32. 32. 2. SURVIVOR BIAS
  33. 33. 2. SURVIVOR BIAS
  34. 34. 2. SURVIVOR BIAS Green: Current S&P 500, Purple: Point in time S&P 500 --Lucena Research
  35. 35. 2. SURVIVOR BIAS How to prevent? •  Use historic index membership. •  Pair with SBF-free data. •  Use these indices as your universe for testing.
  36. 36. 3. OBSERVING THE CLOSE Description: You assume you can observe information recorded at market close, and trade on it. Examples: •  Closing price/volume •  Technicals based on price/volume
  37. 37. 3. OBSERVING THE CLOSE Description: You assume you can observe information recorded at market close, and trade on it. Examples: •  Closing price/volume •  Technicals based on price/volume This is a specific case of “look ahead bias.” Other examples: •  Earnings reports •  News feeds
  38. 38. 3. OBSERVING THE CLOSE How to prevent? Ensure that information with timestamp X cannot be acted on until X+1. Example: Data marked January 15 cannot be traded until the open on January 16.
  39. 39. 4. IGNORING MARKET IMPACT Description: The act of trading affects price. Historical data does not include your trades and is therefore not an accurate representation of the price you would get.
  40. 40. 4. IGNORING MARKET IMPACT
  41. 41. 4. IGNORING MARKET IMPACT
  42. 42. 4. IGNORING MARKET IMPACT
  43. 43. 4. IGNORING MARKET IMPACT Swetha Shivakumar, Georgia Tech
  44. 44. 4. IGNORING MARKET IMPACT How to prevent? Include a “slippage” or “market impact” model in your backtests.
  45. 45. 5. BUY $10M OF A $1M COMPANY Description: Backtest allows a strategy to buy (or short) as much of a symbol as it wants.
  46. 46. 5. BUY $10M OF A $1M COMPANY Description: Backtest allows a strategy to buy (or short) as much of a symbol as it wants. There often is real alpha in thinly traded stocks.
  47. 47. 5. BUY $10M OF A $1M COMPANY Description: Backtest allows a strategy to buy (or short) as much of a symbol as it wants. There often is real alpha in thinly traded stocks. This is a specific example of the more general issue of capacity limitations.
  48. 48. 5. BUY $10M OF A $1M COMPANY How to avoid? Ensure the backtester prohibits trading more dollar volume than actually was available on that day. Add slippage/market impact models to penalize buying too much.
  49. 49. 6. DATA MINING FALLACY Description: If you generate and test enough strategies you’ll eventually find one that “works” in a backtest. The quality of the strategy cannot be distinguished from random luck.
  50. 50. 6. DATA MINING FALLACY Description: If you generate and test enough strategies you’ll eventually find one that “works” in a backtest. The quality of the strategy cannot be distinguished from random luck. Example: Look for “skilled” coin flipper among 10,000 candidates.
  51. 51. 6. DATA MINING FALLACY How to avoid? You can’t! However you can and should forward test before committing significant capital.
  52. 52. 7 THROUGH 10 7. Stateful strategy luck 8. Buy at the open 9. Trust complex models 10. Don’t forward test
  53. 53. THANK YOU! www.lucenaresearch.com

×