Stitch Fix aspires to help you find the style that you will love. Data, the backbone of the business, is used to help with styling recommendations, demand modeling, user acquisition, and merchandise planning and also to influence business decisions throughout the organization. These decisions are backed by algorithms and data collected and interpreted based on client preferences. This talk offers an overview of the compute infrastructure used by the data science team at Stitch Fix, covering the architecture, tools within the larger ecosystem, and the challenges that the team overcame along the way. Apache Spark plays an important role in Stitch Fix’s data platform, and the company’s data scientists use Spark for their ETL and Presto for their ad hoc queries. The goal for the team running the compute infrastructure is to understand and make the data scientists’ lives easier, particularly in terms of usability of Spark, by building tools that make it easier to get started with Spark and transition themselves to a daily workflow. The compute infrastructure is a part of the data platform that is responsible for all the needs of data scientists as Stitch Fix. In this talk, we look at Stitch Fix’s journey, exploring its Spark setup, in-house tools and how they work in synergy with open source frameworks in a cloud environment. There are additional improvements to the infrastructure that help persist information for future use and optimization and we look at how the implementation of Amazon’s EMR FS has helped make it easier for us to read from the S3 source.

1. A compute infrastructure for
Data Scientists
Neelesh Salian
Stitch Fix

2. About Me

3. This talk..
● Data @ Stitch Fix
● Data Warehouse
● Compute Infrastructure
● Questions

4. Data @ Stitch Fix

5. Stitch Fix
● Personal styling service. We serve men & women, plus sizes.
● Founded in 2011, Led by CEO & Founder, Katrina Lake
● Employ more than 5,800 nationwide (USA)
● Algorithms + Humans
● Data Science is our culture

7. Algorithms Team
1. Data Science
○ Vertical teams based on functionality
○ Enablers of Data for the business
○ Cross-functional collaboration
2. Data Platform
○ Building a Platform for Data Science
○ Responsible for Access, Ingestion and Tooling
Read More
https://multithreaded.stitchfix.com/
http://algorithms-tour.stitchfix.com/

8. Challenges - Usability is Key
● Bringing SQL into Spark
● Training Data Scientists’ with Spark basics
● Assistance in Tuning recommendations to optimize
● Maintain latest releases for features and fixes

9. Data Warehouse

11. Data Warehouse Details
● S3 is the source of truth
● HiveMetastore used to store tabular information - schema, metadata
● Python and R based Readers and Writers
● Presto is used to read i.e. ad-hoc queries
● Spark is used for testing/ production complex applications

12. Compute Infrastructure

13. Spark

15. Genie

16. Genie
● Redirection Layer
● From Netflix
● Open Source - 2.x
● Used for Spark jobs - Load balance, Maintain Versions, Maintaining
Multi-Tenant environment

17. Genie Data Model - Applications
● Custom Builds of Spark
● Spark 2.x may refer 2.1.0 or 2.2.0
Applications

18. Genie Data Model - Commands
● Equivalent of spark-submit for each version
● Tags (aliases)
● Test commands with different configurations
Commands

19. Genie Data Model - Clusters
● Red/ Black Cluster deployment
● No disruption to Data Scientists’ workflow
Commands

20. Genie Data Model
Applications
Commands Clusters
Jobs

21. Babylon

22. Sheriff of Babylon
● Job Submissions
● Genie client library
● Overrides to Spark default values
● Run Commands
1. run_spark
2. run_sql
3. run_with_json

23. Additional Tools
● Custom SparkContext
● Metastore Library
● Table Statistics
● Python Query API
● Diagnostic Tool

24. Spaceman - Centralized Spark History
Server

25. Spaceman
● Archiving Log Data from all clusters
● Singular entity to access Spark logs

26. Spaceman - Features
● Runs for each cluster - easy to add new clusters
● Checks for completed applications at intervals
● Stores History of Apps from all clusters for a month

27. Readers and Writers

28. Readers and Writers
● Python and R based clients
● Uses pyarrow while using pandas dataframe for reading and writing tabular
data
● Backed by Livy - reduces bootstrapping with Genie
● Interfaces with the HiveMetastore

30. We are Hiring
https://multithreaded.stitchfix.com/careers/

31. Thank you