Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modeling at Scale: SigOpt at TWIMLcon 2019

292 views

Published on

SigOpt founder and CEO, Scott Clark, PhD, explains the tradeoffs you'll want to consider when designing your modeling platform and integrating hyperparameter optimization to enhance data scientist productivity.

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Modeling at Scale: SigOpt at TWIMLcon 2019

  1. 1. Accelerate and amplify the impact of modelers everywhere
  2. 2. SigOpt. Confidential.3 How I got here: 10+ years of tuning models
  3. 3. Differentiated Models Tailored Models 10,000x Analytics 2.0 Models 100x 1x Modelers by Segment Value per Model Enterprise AI Goals: Differentiate Products Generate Revenue Requirements: Modelers with Expertise Best-in-Class Solutions
  4. 4. SigOpt. Confidential. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters Your data and models stay private Iterative, automated optimization Integrates with any modeling stack
  5. 5. $300B+ in assets under management Current SigOpt algorithmic trading customers represent $500B+ in market capitalization Current SigOpt enterprise customers across six industries represent
  6. 6. SigOpt. Confidential. “We’ve integrated SigOpt’s optimization service and are now able to get better results faster and cheaper than any solution we’ve seen before.” Matt Adereth Managing Director
  7. 7. Lessons 1. Balance flexibility with standardization 2. Maximize resource utilization 3. Unlock new modeling capabilities
  8. 8. Balance flexibility with standardization1
  9. 9. Simulations & Evaluation Modeling FrameworkData Management Data access Data pipelines Data labeling Feature repo. Feature prov. Backfills, versioning Compute Environment Model Execution Model Serving Instrumentation Data validation Phased deploys Online A/B tests Batch scoring On-Premise Optimization & Experimentation Hybrid Multi-Cloud Tracking, Visualization, Version Control Automated Hyperparameter Optimization Distributed Tuning & Job Scheduling Workflows Auto Feature & AutoML Coding Environment Framework Support Library Backtests Metric Iteration Model Evaluation Portfolio Optimization
  10. 10. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiate Optimization & Experimentation Strategy: Standardize and scale
  11. 11. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiateOptimization & Experimentation Strategy: Standardize and scale
  12. 12. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiate Optimization & Experimentation Strategy: Standardize and scale
  13. 13. Simulations & Evaluation Strategy: Customize and differentiate Modeling Framework Strategy: Modeler choice and flexibility Data Management Strategy: Customize and differentiate Compute Environment Strategy: Modeler choice and flexibility Model Execution Strategy: Customize and differentiate Optimization & Experimentation Strategy: Standardize and scale
  14. 14. Data Management Data access Data pipelines Data labeling Feature repo. Feature prov. Backfills, versioning Simulation & Evaluation Modeling Framework Hardware Environment On-Premise Hybrid Multi-Cloud Optimization & Experimentation Insights, Tracking, Collaboration Model Search, Hyperparameter Tuning Resource Scheduler, Management Backtests Metric Iteration Model Evaluation Portfolio Optimization Model Execution Model Serving Instrumentation Data validation Phased deploys Online A/B tests Batch scoring Flexibility + Standardization
  15. 15. SigOpt. Confidential. Benefits of Robust Experimentation & Optimization Learn fast, fail fast Give yourself the best chance at finding good use cases while avoiding false negatives Connect outputs to outcomes Define, select and iterate on your metrics with end-to-end evaluation Find the global maximum Early non-optimized decisions in the process limit your ability to maximize performance Boost productivity Automate modeling tasks so modelers spend more time applying their expertise
  16. 16. Maximize resource utilization2
  17. 17. Better, Faster, Cheaper
  18. 18. SigOpt. Confidential.19
  19. 19. SigOpt. Confidential. Hyperparameter Optimization Model Tuning Grid Search Random Search Bayesian Optimization Training & Tuning Evolutionary Algorithms Deep Learning Architecture Search Hyperparameter Search
  20. 20. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  21. 21. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  22. 22. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  23. 23. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  24. 24. SigOpt. Confidential. Pro Con Manual Search Leverages expertise Not scalable, inconsistent Grid Search Simple to implement Not scalable, often infeasible Random Search Scalable Inefficient Evolutionary Algorithms Effective at architecture search Very resource intensive Bayesian Optimization Efficient, effective Can be tough to parallelize
  25. 25. Your firewall Training Data AI, ML, DL, Simulation Model Model Evaluation or Backtest Testing Data New Configurations Objective Metric Better Results EXPERIMENT INSIGHTS Track, organize, analyze and reproduce any model ENTERPRISE PLATFORM Built to fit any stack and scale with your needs OPTIMIZATION ENGINE Explore and exploit with a variety of techniques RESTAPI Configuration Parameters or Hyperparameters 100x Asynchronous Parallelization
  26. 26. SigOpt. Confidential. 24 Days to optimize a model without SigOpt 3 Days to optimize a model with SigOpt Learn more: https://twosigma.com/news/article/why-two-sigma-is-using-sigopt-for-automated-parameter-tuning/
  27. 27. Unlock new modeling capabilities3
  28. 28. Better, Faster, Cheaper
  29. 29. SigOpt. Confidential. Unlock new modeling capabilities 30 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case
  30. 30. SigOpt. Confidential. Train and tune expensive models 31 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Combine the intelligence of Bayesian optimization with the efficiency of early termination techniques
  31. 31. SigOpt. Confidential. Compare and select the right metric 32 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Optimize multiple metrics at the same time to inform your metric definition and selection process
  32. 32. SigOpt. Confidential. Identify the right tool for the job 33 Accelerate Model Optimization ● Multitask optimization efficiently optimizes “expensive” models ● Relevant docs ● Image classification use case Solve for Competing Business Objectives ● Multimetric optimization empowers you to evaluate trade-offs ● Relevant docs ● Sequence classification use case Explore Model Architectures ● Conditional parameters result in more efficient architecture search ● Relevant docs ● NLP use case Take into account the conditionality of certain parameter types in the optimization process
  33. 33. Lessons 1. Balance flexibility with standardization 2. Maximize resource utilization 3. Unlock new modeling capabilities
  34. 34. Try our solution Our team is here to help. Find our table at Booth 02, and sign up to demo SigOpt today. Join our webinar on modeling platforms https://sigopt.com/company /events/webinar-capabilities -ml-platforms/ Listen to our recent podcast with Two Sigma https://twimlai.com/twiml-talk-273-supporting -rapid-model-development-at-two-sigma- with-matt-adereth-scott-clark/ Download TWIML’s Modeling Platforms ebook https://sigopt.com/guide-for- machine-learning-platforms/

×