Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Using Bayesian Optimization to Simultaneously Tune Multiple Metrics - Quantcon 2018

827 views

Published on

Bayesian Optimization is an efficient way to optimize model parameters, especially when evaluating different parameters is time-consuming or expensive. Trading pipelines often have many tunable configuration parameters that can have a large impact on the efficacy of the model and are notoriously expensive to train and backtest.



In traditional optimization a single metric like a Sharpe Ratio is being optimized over a potentially large set of configurations with the goal of a single, best configuration being produced. In this talk we’ll explore real world extensions to this where multiple competing objectives need to be optimized, a portfolio of multiple solutions may be required, constraints on the underlying system make certain configurations unviable, and more. We’ll present work from recent ICML and NIPS workshop papers and detailed examples.



We’ll compare the results of Bayesian Optimization to these optimization problems to standard techniques like grid search, random search, and expert tuning across several datasets.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Using Bayesian Optimization to Simultaneously Tune Multiple Metrics - Quantcon 2018

  1. 1. Model Optimization with Competing Objectives QuantCon 2018 Scott Clark scott@sigopt.com
  2. 2. OUTLINE 1. Why is Tuning Models Hard? 2. Common Tuning Methods 3. Deep Learning Example 4. Tuning Multiple Metrics 5. Multi-metric Optimization Examples
  3. 3. Algorithmic Trading and AI / ML are extremely powerful Tuning these systems is extremely non-intuitive
  4. 4. Photo: Joe Ross
  5. 5. TUNABLE PARAMETERS IN DEEP LEARNING
  6. 6. TUNABLE PARAMETERS IN DEEP LEARNING
  7. 7. TUNABLE PARAMETERS IN DEEP LEARNING
  8. 8. TUNABLE PARAMETERS IN DEEP LEARNING
  9. 9. TUNABLE PARAMETERS IN DEEP LEARNING
  10. 10. Photo: Tammy Strobel
  11. 11. STANDARD METHODS FOR PARAMETER SEARCH
  12. 12. STANDARD TUNING METHODS Trading Models Data Backtest / Simulation Parameter Configuration ? Grid Search Random Search Manual Search - Weights - Thresholds - Window sizes - Transformations Domain Expertise
  13. 13. OPTIMIZATION FEEDBACK LOOP Objective Metric Better Results REST API New configurations Trading Models Data Backtest / Simulation Domain Expertise
  14. 14. ● Create a strategy to trade Select Sector SPDR ETFs ○ XLV, XLF, XLP, XLE, XLK, XLB, XLU, XLI ● Trade on common signals ○ Relative Strength Interest (RSI) ○ Rate of Change (ROC) ● Maximize Sharpe Ratio PROBLEM https://blog.quantopian.com/bayesian-optimization-of-a-technical-trading-algorithm-with-ziplinesigopt-2/
  15. 15. TUNABLE PARAMETERS IN ALGO TRADING ● Relative Strength Interest (RSI) ○ Lookback window for # of prices used in the RSI calculation ○ Lower_bound value defining the trade entry condition ○ Range_width, which will be added to the Lower-bound ■ Lower_bound + Range_width is the range of values over which our RSI signal will be considered True ● Rate of Change (ROC) ○ Lookback window for # of prices used in the ROC calculation ○ Lower_bound value defining the trade entry condition ○ Range_width, which will be added to the Lower-bound ■ Lower_bound + Range_width is the range of values over which our ROC signal will be considered True ● Signal evaluation frequency ○ Number of days between evaluation of if our signals ■ Do we evaluate them every day, every week, every month, etc.
  16. 16. COMBINATORIAL EXPLOSION ● RSI lookback window: 115 values (5 to 120) ● RSI lower bound: 90 values (0 to 90) ● RSI range width: 20 values (10 to 30) ● ROC lookback window: 61 values (2 to 63) ● ROC lower bound: 30 values (0 to 30) ● ROC range width: 195 values (5 to 200) ● Evaluation frequency: 18 values (3 to 21) = 1,329,623,100,000 possible configurations
  17. 17. COMPARATIVE PERFORMANCE Grid Search Expert Grid ● Better: 200% Higher model returns than manual search ● Faster/Cheaper: 10x fewer evaluations vs standard methods BacktestPortfolioValue Time (2004-2012) Blog Post
  18. 18. COMPARATIVE PERFORMANCE https://papers.ssrn.com/sol3/paper s.cfm?abstract_id=2745220 ● Out of sample performance is terrible ● We need better metrics
  19. 19. TUNING MULTIPLE METRICS What if we want to optimize multiple competing metrics? ● Trading Tradeoffs ○ Sharpe Ratio vs Drawdown ○ Backtest Alpha vs Uncertainty ○ Quality vs Robustness ● Complexity Tradeoffs ○ Accuracy vs Training Time ○ Accuracy vs Inference Time
  20. 20. PARETO OPTIMAL What does it mean to optimize two metrics simultaneously? Pareto efficiency or Pareto optimality is a state of allocation of resources from which it is impossible to reallocate so as to make any one individual or preference criterion better off without making at least one individual or preference criterion worse off.
  21. 21. PARETO OPTIMAL What does it mean to optimize two metrics simultaneously? The red points are on the Pareto Efficient Frontier, they strictly dominate all of the grey points. You can do no better in one metric without sacrificing performance in the other. Point N is Pareto Optimal compared to Point K.
  22. 22. PARETO EFFICIENT FRONTIER Goal is to have best set of feasible solutions to select from After optimization the expert picks one or more of the red points from the Pareto Efficient Frontier to further study or put into production.
  23. 23. TOY EXAMPLE
  24. 24. MULTI-METRIC OPTIMIZATION
  25. 25. DEEP LEARNING EXAMPLES
  26. 26. MULTI-METRIC OPT IN DEEP LEARNING https://devblogs.nvidia.com/sigopt-deep-learning-hyperparameter-optimization/
  27. 27. DEEP LEARNING TRADEOFFS ● Deep Learning pipelines are time consuming and expensive to run ● Application and deployment conditions may make certain configurations less desirable ● Tuning for both accuracy and complexity metrics like training or inference time allows expert to make best decision for production
  28. 28. ● Comparison of several RMSProp SGD parametrizations ● Different configurations converge differently STOCHASTIC GRADIENT DESCENT
  29. 29. TEXT CLASSIFICATION PIPELINE ML / AI Model (MXNet) Testing Text Validation Accuracy Better Results REST API Hyperparameter Configurations and Feature Transformations Training Text Training Time
  30. 30. FINDING THE FRONTIER
  31. 31. SEQUENCE CLASSIFICATION PIPELINE ML / AI Model (Tensorflow) Testing Sequences Validation Accuracy Better Results REST API Hyperparameter Configurations and Feature Transformations Training Sequences Inference Time
  32. 32. TEXT CLASSIFICATION PIPELINE
  33. 33. FINDING THE FRONTIER
  34. 34. FINDING THE FRONTIER
  35. 35. LOAN CLASSIFICATION PIPELINE ML / AI Model (LightGBM) Testing Data Validation AUCPR Better Results REST API Hyperparameter Configurations and Feature Transformations Training Data Avg $ Lost
  36. 36. GRID SEARCH CAN MISLEAD ● Best grid search point (wrt accuracy) loses >$35 / transaction ● Best grid search point (wrt loss) has 70% accuracy ● Points of the Pareto Frontier give user more information about what is possible and more control of trade-offs
  37. 37. TAKEAWAYS One metric may not paint the whole picture - Think about metric trade-offs in your model pipelines - Optimizing for the wrong thing can be very expensive Not all optimization strategies are equal - Pick an optimization strategy that gives the most flexibility - Different tools enable you to tackle new problems
  38. 38. Questions? contact@sigopt.com https://sigopt.com @SigOpt

×