Modeling at Scale: SigOpt at TWIMLcon 2019

1. Accelerate and amplify the impact of modelers everywhere

2. SigOpt. Conﬁdential.3 How I got here: 10+ years of tuning models

3. Diﬀerentiated Models Tailored Models 10,000x Analytics 2.0 Models 100x 1x Modelers by Segment Value per Model Enterprise AI Goals: Diﬀerentiate Products Generate Revenue Requirements: Modelers with Expertise Best-in-Class Solutions

4. SigOpt. Confidential. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Your data and models stay private Iterative, automated optimization Integrates with any modeling stack

5. $300B+ in assets under management Current SigOpt algorithmic trading customers represent $500B+ in market capitalization Current SigOpt enterprise customers across six industries represent

6. SigOpt. Conﬁdential. “We’ve integrated SigOpt’s optimization service and are now able to get better results faster and cheaper than any solution we’ve seen before.” Matt Adereth Managing Director

7. Lessons 1. Balance ﬂexibility with standardization 2. Maximize resource utilization 3. Unlock new modeling capabilities

8. Balance ﬂexibility with standardization1

9. Simulations & Evaluation Modeling FrameworkData Management Data access Data pipelines Data labeling Feature repo. Feature prov. Backﬁlls, versioning Compute Environment Model Execution Model Serving Instrumentation Data validation Phased deploys Online A/B tests Batch scoring On-Premise Optimization & Experimentation Hybrid Multi-Cloud Tracking, Visualization, Version Control Automated Hyperparameter Optimization Distributed Tuning & Job Scheduling Workﬂows Auto Feature & AutoML Coding Environment Framework Support Library Backtests Metric Iteration Model Evaluation Portfolio Optimization

10. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiate Optimization & Experimentation Strategy: Standardize and scale

11. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiateOptimization & Experimentation Strategy: Standardize and scale

14. Data Management Data access Data pipelines Data labeling Feature repo. Feature prov. Backﬁlls, versioning Simulation & Evaluation Modeling Framework Hardware Environment On-Premise Hybrid Multi-Cloud Optimization & Experimentation Insights, Tracking, Collaboration Model Search, Hyperparameter Tuning Resource Scheduler, Management Backtests Metric Iteration Model Evaluation Portfolio Optimization Model Execution Model Serving Instrumentation Data validation Phased deploys Online A/B tests Batch scoring Flexibility + Standardization

15. SigOpt. Conﬁdential. Beneﬁts of Robust Experimentation & Optimization Learn fast, fail fast Give yourself the best chance at finding good use cases while avoiding false negatives Connect outputs to outcomes Define, select and iterate on your metrics with end-to-end evaluation Find the global maximum Early non-optimized decisions in the process limit your ability to maximize performance Boost productivity Automate modeling tasks so modelers spend more time applying their expertise

16. Maximize resource utilization2

17. Better, Faster, Cheaper

18. SigOpt. Conﬁdential.19

19. SigOpt. Conﬁdential. Hyperparameter Optimization Model Tuning Grid Search Random Search Bayesian Optimization Training & Tuning Evolutionary Algorithms Deep Learning Architecture Search Hyperparameter Search

20. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize

21. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize

22. SigOpt. Conﬁdential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize

25. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 100x Asynchronous Parallelization

26. SigOpt. Conﬁdential. 24 Days to optimize a model without SigOpt 3 Days to optimize a model with SigOpt Learn more: https://twosigma.com/news/article/why-two-sigma-is-using-sigopt-for-automated-parameter-tuning/

27. Unlock new modeling capabilities3

28. Better, Faster, Cheaper

29. SigOpt. Confidential. Unlock new modeling capabilities 30 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case

30. SigOpt. Confidential. Train and tune expensive models 31 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Combine the intelligence of Bayesian optimization with the efficiency of early termination techniques

31. SigOpt. Confidential. Compare and select the right metric 32 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Optimize multiple metrics at the same time to inform your metric definition and selection process

32. SigOpt. Confidential. Identify the right tool for the job 33 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Take into account the conditionality of certain parameter types in the optimization process

33. Lessons 1. Balance ﬂexibility with standardization 2. Maximize resource utilization 3. Unlock new modeling capabilities

34. Try our solution Our team is here to help. Find our table at Booth 02, and sign up to demo SigOpt today. Join our webinar on modeling platforms https://sigopt.com/company /events/webinar-capabilities -ml-platforms/ Listen to our recent podcast with Two Sigma https://twimlai.com/twiml-talk-273-supporting -rapid-model-development-at-two-sigma- with-matt-adereth-scott-clark/ Download TWIML’s Modeling Platforms ebook https://sigopt.com/guide-for- machine-learning-platforms/