Machine learning infrastructure solve data scientists' problems using infrastructure tools. This talk shows the case study of building SigOpt Orchestrate, an ML infrastructure tool. The talk highlights how data scientists' concerns as user mapped to solutions with some of today's most popular infrastructure tools. To learn more about SigOpt Orchestrate: https://sigopt.com/orchestrate Originally given as a talk for UC Berkeley's Women in Electrical Engineering and Computer Science group on January 24, 2019.

1. ML Infrastructure
alexandra@sigopt.com | @alexandraj777

2. About Me
● Alum of Carnegie Mellon SCS
● Joined SigOpt in 2015
● Tech lead for the Platform Team, handling
frontend, backend, infrastructure and
testing
● Recent project in ML Infrastructure: SigOpt
Orchestrate
● Co-organizer for Bay Area chapter of
Women in Machine Learning and Data
Science (join us!)

3. ML Infrastructure
solves data scientists'
problems using
infrastructure tools

4. Challenge:
● Data scientists want to maximize the
performance of their models
● SigOpt provides an API for
hyperparameter optimization (HPO)
● SigOpt HPO helps data scientists
maximize the performance of their models!
● Data scientists need to use clusters to
properly perform HPO
Machine Learning Infrastructure

5. Challenge:
● Data scientists want to maximize the
performance of their models
● SigOpt provides an API for
hyperparameter optimization (HPO)
● SigOpt HPO helps data scientists
maximize the performance of their models!
● Data scientists need to use clusters to
properly perform HPO
Machine Learning Infrastructure
Data scientists specialize in:
● Gathering data
● Building models
● Extracting business insights
Infrastructure engineers specialize in:
● Building shared tools
● Application scalability and performance
● Keeping track of interactions between
large distributed systems

6. Case Study: Building SigOpt Orchestrate
● Project started in 2018 to bridge ML and
infrastructure
● What problems did our customers ask us to
solve?
● How did a challenge for the user turn into a
technical problem?
● Which tools / technologies did we use?

7. Challenge #1: Can't Train Model on Laptop
Problem: Setup each remote machine
Initial Solution:
● Write a setup script to install
dependencies
● SCP data, code, and setup script to every
remote machine

8. Solution #1: Containerize!
Problem: Setup each remote machine
New Solution:
● Containerize code and dependencies on
the user's local environment
● Push the container to a registry
● Each machine pulls the container from a
registry

9. Challenge #2: Start Training in Parallel
Problem: Kick off the hyperparameter
optimization job on six machines at once
Initial Solution:
● Open a tmux window on every remote
instance
● SSH over command to run setup script
into each tmux window
● SSH over command to train model into
each tmux window

10. Solution #2: Kubernetes!
Problem: Kick off the hyperparameter
optimization job on six machines at once
New Solution:
● Spin up AWS EKS (Kubernetes) cluster
● Create a job spec
○ "run 6 copies of this container at the same
time"
● Submit job spec to Kubernetes API
● Kubernetes starts the job on the cluster

11. Challenge #3: View Progress and Debug
Problem: View the status of a hyperparameter
optimization job at a glance
Initial Solution:
● Save hostname and error information as
metadata in calls to external API
● SSH into machines and view the logs
directly (pre-Kubernetes)
● Use Kubernetes CLI to view logs

12. Solution #3: Build a CLI!
Problem: View the status of a hyperparameter
optimization job at a glance
New Solution:
● Write an interface for the data scientist to
interact with the infrastructure tool
● We chose a command line interface
● Serves as an abstraction on top of
Kubernetes APIs + externals APIs
● Screenshots (top and bottom)
○ sigopt logs <experiment_id>
○ sigopt status <experiment_id>

13. Final Thoughts...
Paper: Orchestrate: Infrastructure for Enabling
Parallelism during Hyperparameter Optimization,
Alexandra Johnson and Michael McCourt
SigOpt is free for academics!
We're hiring research engineers/interns and
software engineers/interns!

14. Thank You! Questions?