The document discusses the compute infrastructure at Stitch Fix, emphasizing the integration of algorithms and data science in their operations since 2011. It outlines the components of the data warehouse, the utilization of Spark and Genie for managing jobs and workloads, and the training needs for data scientists. Additionally, it highlights the importance of tools for data access and processing, such as Python and R clients, and the centralized Spark history server called Spaceman.

1. A compute infrastructure for
Data Scientists
Neelesh Salian
Stitch Fix

2. About Me

3. This talk..
● Data @ Stitch Fix
● Data Warehouse
● Compute Infrastructure
● Questions

4. Data @ Stitch Fix

5. Stitch Fix
● Personal styling service. We serve men & women, plus sizes.
● Founded in 2011, Led by CEO & Founder, Katrina Lake
● Employ more than 5,800 nationwide (USA)
● Algorithms + Humans
● Data Science is our culture

7. Algorithms Team
1. Data Science
○ Vertical teams based on functionality
○ Enablers of Data for the business
○ Cross-functional collaboration
2. Data Platform
○ Building a Platform for Data Science
○ Responsible for Access, Ingestion and Tooling
Read More
https://multithreaded.stitchfix.com/
http://algorithms-tour.stitchfix.com/

8. Challenges - Usability is Key
● Bringing SQL into Spark
● Training Data Scientists’ with Spark basics
● Assistance in Tuning recommendations to optimize
● Maintain latest releases for features and fixes

9. Data Warehouse

11. Data Warehouse Details
● S3 is the source of truth
● HiveMetastore used to store tabular information - schema, metadata
● Python and R based Readers and Writers
● Presto is used to read i.e. ad-hoc queries
● Spark is used for testing/ production complex applications

12. Compute Infrastructure

13. Spark

15. Genie

16. Genie
● Redirection Layer
● From Netflix
● Open Source - 2.x
● Used for Spark jobs - Load balance, Maintain Versions, Maintaining
Multi-Tenant environment

17. Genie Data Model - Applications
● Custom Builds of Spark
● Spark 2.x may refer 2.1.0 or 2.2.0
Applications

18. Genie Data Model - Commands
● Equivalent of spark-submit for each version
● Tags (aliases)
● Test commands with different configurations
Commands

19. Genie Data Model - Clusters
● Red/ Black Cluster deployment
● No disruption to Data Scientists’ workflow
Commands

20. Genie Data Model
Applications
Commands Clusters
Jobs

21. Babylon

22. Sheriff of Babylon
● Job Submissions
● Genie client library
● Overrides to Spark default values
● Run Commands
1. run_spark
2. run_sql
3. run_with_json

23. Additional Tools
● Custom SparkContext
● Metastore Library
● Table Statistics
● Python Query API
● Diagnostic Tool

24. Spaceman - Centralized Spark History
Server

25. Spaceman
● Archiving Log Data from all clusters
● Singular entity to access Spark logs

26. Spaceman - Features
● Runs for each cluster - easy to add new clusters
● Checks for completed applications at intervals
● Stores History of Apps from all clusters for a month

27. Readers and Writers

28. Readers and Writers
● Python and R based clients
● Uses pyarrow while using pandas dataframe for reading and writing tabular
data
● Backed by Livy - reduces bootstrapping with Genie
● Interfaces with the HiveMetastore

30. We are Hiring
https://multithreaded.stitchfix.com/careers/

31. Thank you