This document discusses how to build consistent and scalable workspaces for data science teams. It recommends identifying system requirements, stabilizing dependencies, increasing test coverage, and using continuous integration to ensure resources are available. It also suggests creating a pool of worker machines and asynchronous task queue to scale workloads. This allows tasks to run in isolated, identical environments and provides flexible use of cloud computing resources. Benefits include guaranteed task environments, extensibility, and a reusable command line interface. Examples of use cases provided are quality assurance testing and parallelizable data and model tasks.

1. How to build consistent, scalable
workspaces for data science teams
Elaine Lee

2. Data science is hard.
Doing data science is even harder.
Ensuring enough resourcesManaging dependencies
http://www.seriouseats.com/assets_c/2014/06/20140525-294370-best-deep-dish-pizza-art-of-pizza-
primary-thumb-1500xauto-404176.jpghttps://s-media-cache-ak0.pinimg.com/736x/91/6b/f0/916bf0f23660fc7019353800668060af.jpg

3. Nail it down
Identify system requirements for base Docker image
Stabilize dependencies for data science work environment
Increase test coverage
Get continuous integration (CI) platform on the same page

4. Scale it up
Create a pool of worker machines ready to accept jobs
Set up an asynchronous task queue
Provide a simple command line interface for data scientists

5. Putting it all together
Pull changes Start Docker
container
Run test suite Report Pass/Fail Export image for
commit
Commit pushed
to Github
Report resultGet image for
commit
Start container
from image
Run task
Request arrives
in queue
workers
123abc…123abc…
123abc…123abc…
s3

6. Benefits
Flexible to any
composition of EC2
instances
-Extensible to EMR
Task environment
guaranteed
-Isolated from other tasks
-Identical to conditions at
time of development
One-time configuration
-EC2 AMI
Extensible command line
interface
-R interface
-Cluster management
-Job monitoring

7. Use case: Quality assurance
CI testing
Other tests
- Data validation
- Model consistency
http://img.pandawhale.com/post-52368-thanks-obama-making-sandwich-m-whnc.jpeg

8. Use case: Parallelizable tasks
Data manipulation
- Feature engineering
Model builds
- Advanced machine learning algorithms
- Hyperparameter search
https://pbs.twimg.com/media/Buw8Bz6IIAAxgxg.png

9. Elaine Lee
Data Engineer
elaine@elaineklee.com
@elaineklee
avant.com

10. Elaine Lee
Data Engineer
elaine@elaineklee.com
@elaineklee
avant.com