Enabling full stack data scientists

•

0 likes•169 views

Autonomy and ownership are core to working at Stitch Fix, particularly on the Algorithms team. Data Scientists are expected to build their systems end to end and maintain them in the long run. We rely on automation, documentation, and collaboration to enable data scientist to build and maintain production services. In this talk I will discuss what we have built and how we communicate about these tools with our data scientists.

Technology

Enabling
Full Stack
Data Scientists
at Stitch Fix
Juliet Hougland
@j_houg
April 2018

Agenda
@j_houg 2
WHAT WE DO
HOW WE OPERATE
TRANSFORMING THE WAY
DATA SCIENTISTS
DO WHAT THEY LOVE

@j_houg
The Business
The Customer Experience
What We Do

@j_houg
Transforming The
Way People Find
What They Love

@j_houg
The Algorithms Org
Operating Principles
Data Platform
How We Operate

@j_houg
Algorithms
Data Platform Data Science
ClientStyling Operations Horizontal

@j_houg
Typical Data Science Department
Data
Engineers
Data
Scientists
Software
Engineers
Data Infrastructure Engineers
provide
resources &
support

@j_houg
Typical Data Science Department
Data Driven Capability
Data Engineers
Data Scientists
Software Engineers
Infrastructure Engineers

@j_houg
Another View
Capability A Capability B Capability C Capability D
Data Engineers
Data Scientists
Software Engineers
Infrastructure Engineers
specialized by function

@j_houg
High Coordination Cost
Capability A Capability B Capability C Capability D
Data Engineers
Data Scientists
Software Engineers
Infrastructure Engineers
specialization by function
tightly
coupled,
loosely
aligned

@j_houg
A Different Way to Organize
Data Driven
Capability
DataScientists

@j_houg
Organized by Capability
specialized by capability
Capability A Capability B Capability C Capability D
DataScientists
DataScientists
DataScientists
DataScientists

@j_houg
Focus on Capabilities
Data Driven
Capability
DataScientists
Full Stack Data
Scientists:
" Full Stack Data Scientists Own:
" Implementation
" Coordination with the business
" Production support (sort of)

@j_houg
Focus on Capabilities
Data Platform Engineers
loosely
coupled,
tightly
aligned
specialized by function
Capability A Capability B Capability C Capability D
DataScientists
specialized by capability
DataScientists
DataScientists
DataScientists

@j_houg
Capabilities
Data Platform Engineers
specialized by function
Capability A Capability B Capability C Capability D
DataScientists
specialized by capability
DataScientists
DataScientists
DataScientists

@j_houg
Analysis &
Model Building
Hard:
•Reproducible
•Accurate
•Collaborative
Opinionated Analysis Development —
Hilary Parker
https://peerj.com/preprints/3210/

@j_houg
Interested in
what you heard?
We are hiring!

Similar to Enabling full stack data scientists

[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...

Rory Hope

Data Discoverability with DataHub

Gleb Mezhanskiy

http://www.callandcontactcentreexpo.co.uk/speakers/margriet-groenendijk/ In order to move past the hype and achieve the full potential of machine learning, data scientists and software developers need to work more closely together towards their common goal of delivering well-architected, data-driven applications. Every industry is in the process of being transformed by software and data. It is in the collaboration between data scientists and software developers where the real value can be found by creating integrated data workflows that benefit from the unique knowledge and skillsets of each discipline.

The Convergence of Data Science and Software Development

Margriet Groenendijk

Big data careers

Mohammad Hassan Adjigol

Data Science

AnushkaTripathi14

Getting Content Out The Door Quickly with Scraping, Outsourcing and Team Work...

John-Henry Scherck

Nonprofits + Data: Pathway to Innovation

Tim Sarrantonio

Analytics Organizations & The New Emerging Roles

Vandana Thakur

IMGS 2015 - Ordnance Survey Ireland - Hugh Mangan

IMGS

IRJET- Youtube Data Sensitivity and Analysis using Hadoop Framework

IRJET Journal

Watch full webinar here: https://bit.ly/3offv7G Presented at AI Live APAC Advanced data science techniques, like machine learning, have proven an extremely useful tool to derive valuable insights from existing data. Platforms like Spark, and complex libraries for R, Python and Scala put advanced techniques at the fingertips of the data scientists. However, these data scientists spend most of their time looking for the right data and massaging it into a usable format. Data virtualization offers a new alternative to address these issues in a more efficient and agile way. Watch this on-demand session to learn how companies can use data virtualization to: - Create a logical architecture to make all enterprise data available for advanced analytics exercise - Accelerate data acquisition and massaging, providing the data scientist with a powerful tool to complement their practice - Integrate popular tools from the data science ecosystem: Spark, Python, Zeppelin, Jupyter, etc.

How Data Virtualization Puts Enterprise Machine Learning Programs into Produc...

Denodo

AI Deep Dive_ A Journey through Heroku_OpenAI Integration.pdf

mattmeyers15

Join our #DataTalk on Thursdays at 5 p.m. ET. This week, we learned from DataKind – Harnessing the Power of Data Science in the Service of humanity, Real Impact Analytics, Elissa Redmiles, a Data Science for Social Good Summer Fellow at the University of Chicago, Nick Eng, Data Scientist at the Center for Data Science and Public Policy at the University of Chicago, Kevin Chen, the Chief Scientist at the Experian, North America Data Lab, and others.

How to Use Data for Good

Experian_US

Date: 13th November 2018 Location: Self-Service Analytics Theatre Time: 12:30 - 13:00 Speaker: Joel McKelvey Organisation: Looker About: Companies that use data well are more efficient, effective, and profitable. Unfortunately, most organizations struggle to keep up with the changing supply of data — and the growing business demands for that data. The key is to connect data supply to data users in a way that scales, supports existing workflows, and serves as a foundation for the future. This session will explore how to bring data to users where and when they need it without sacrificing data governance or unified metrics. This session will also present proven ways to build a data foundation for your organisation that can support future changes in both data supply and data demand. Specifically, attendees will discover: • The key considerations to driving the most value from data, including: self-service, governance, custom interfaces, modeling, and connections to existing business systems. • How to provide users access to data in a way that naturally fits in their existing workflows and allows users to take immediate action. • How companies like Deliveroo and King extract critical business insights from growing data and deliver those insights to their business users.

Big Data LDN 2018: FIGHTING DATA CHAOS: CONNECTING USERS TO DATA AT SCALE

Matt Stubbs

Strata Conference NYC 2013

Taewook Eom

Building Data Science Pipelines in Python using Luigi

Shwet Kamal Mishra

Data Science Pipelines in Python using Luigi

Shivam Bansal

Come diventare data scientist - Paolo Pellegrini

Donatella Cambosu

Transition to a modern data platform

Michael Ghen

DevTalks Keynote Powering interactive data analysis with Google BigQuery

Márton Kodok

Similar to Enabling full stack data scientists (20)

[Brighton SEO] Audience Intelligence & SEO: How to integrate data sources to ...

Data Discoverability with DataHub

The Convergence of Data Science and Software Development

Big data careers

Data Science

Getting Content Out The Door Quickly with Scraping, Outsourcing and Team Work...

Nonprofits + Data: Pathway to Innovation

Analytics Organizations & The New Emerging Roles

IMGS 2015 - Ordnance Survey Ireland - Hugh Mangan

IRJET- Youtube Data Sensitivity and Analysis using Hadoop Framework

How Data Virtualization Puts Enterprise Machine Learning Programs into Produc...

AI Deep Dive_ A Journey through Heroku_OpenAI Integration.pdf

How to Use Data for Good

Big Data LDN 2018: FIGHTING DATA CHAOS: CONNECTING USERS TO DATA AT SCALE

Strata Conference NYC 2013

Building Data Science Pipelines in Python using Luigi

Data Science Pipelines in Python using Luigi

Come diventare data scientist - Paolo Pellegrini

Transition to a modern data platform

DevTalks Keynote Powering interactive data analysis with Google BigQuery

More from Stitch Fix Algorithms

T-tests are the industry de facto method for analyzing A/B tests. Regression is a more general approach that allows you to also control for covariates, potentially increasing your power. This can lead to reduced run times, the ability to detect smaller changes, and higher testing velocity. Come learn why Stitch Fix uses regression-based analysis for experiments. We'll also share practical insights on how we’re enabling this in an automated platform at scale at Stitch Fix.

Progression by Regression: How to increase your A/B Test Velocity

Stitch Fix Algorithms

Deep recommendations in PyTorch

Stitch Fix Algorithms

Personalization allows Stitch Fix to style its clients and provide recommendations to help them find what they love. To do this, the company gathers information about a client’s preferences up front when they sign up from the service and learns more about them as they become longer-term customers. This information is important for making recommendations but also must be protected and managed with care. The data science team at Stitch Fix is the primary owner of the recommendation systems. Backing them up is the data platform team, who maintain the data infrastructure, data warehouse, and supporting tools and services. This data warehouse has several different data sources that read and write into it. This includes a logging pipeline for events, every Spark-based ETL, and daily snapshots of structured data from Stitch Fix applications. Neelesh Srinivas Salian explains Stitch Fix’s process to better understand the movement and evolution of data within its data warehouse, from the initial ingestion from outside sources through all of its ETLs. Neelesh also details how Stitch Fix built a service that helps the company understand the lineage information that is associated with each table in the data warehouse. This service helps the company understand the source, parentage, and journey of all data in the warehouse. Although Stitch Fix makes sure to anonymize and filter out sensitive information from this data, the company needs a more flexible long-term solution as the business expands.

Tracking data lineage at Stitch Fix

Stitch Fix Algorithms

Stitch Fix aspires to help you find the style that you will love. Data, the backbone of the business, is used to help with styling recommendations, demand modeling, user acquisition, and merchandise planning and also to influence business decisions throughout the organization. These decisions are backed by algorithms and data collected and interpreted based on client preferences. Neelesh Srinivas Salian offers an overview of the compute infrastructure used by the data science team at Stitch Fix, covering the architecture, tools within the larger ecosystem, and the challenges that the team overcame along the way. Apache Spark plays an important role in Stitch Fix’s data platform, and the company’s data scientists use Spark for their ETL and Presto for their ad hoc queries. The goal for the team running the compute infrastructure is to understand and make the data scientists’ lives easier, particularly in terms of usability of Spark, by building tools that expedite the process of getting started with Spark and transitioning from an ad hoc to a production workflow. The compute infrastructure is a part of the data platform that is responsible for all the needs of data scientists as Stitch Fix. Neelesh shares Stitch Fix’s journey, exploring its ad hoc and production infrastructure and detailing its in-house tools and how they work in synergy with open source frameworks in a cloud environment. Neelesh also discusses the additional improvements to the infrastructure that help persist information for future use and optimization and explains how the implementation of Amazon’s EMR FS has helped make it easier to read from the S3 source.

Improving ad hoc and production workflows at Stitch Fix

Stitch Fix Algorithms

Stitch Fix aspires to help you find the style that you will love. Data, the backbone of the business, is used to help with styling recommendations, demand modeling, user acquisition, and merchandise planning and also to influence business decisions throughout the organization. These decisions are backed by algorithms and data collected and interpreted based on client preferences. This talk offers an overview of the compute infrastructure used by the data science team at Stitch Fix, covering the architecture, tools within the larger ecosystem, and the challenges that the team overcame along the way. Apache Spark plays an important role in Stitch Fix’s data platform, and the company’s data scientists use Spark for their ETL and Presto for their ad hoc queries. The goal for the team running the compute infrastructure is to understand and make the data scientists’ lives easier, particularly in terms of usability of Spark, by building tools that make it easier to get started with Spark and transition themselves to a daily workflow. The compute infrastructure is a part of the data platform that is responsible for all the needs of data scientists as Stitch Fix. In this talk, we look at Stitch Fix’s journey, exploring its Spark setup, in-house tools and how they work in synergy with open source frameworks in a cloud environment. There are additional improvements to the infrastructure that help persist information for future use and optimization and we look at how the implementation of Amazon’s EMR FS has helped make it easier for us to read from the S3 source.

A compute infrastructure for data scientists

Stitch Fix Algorithms

At Stitch Fix, hierarchical models are one of the core machine learning frameworks used in our recommender systems technology. Hierarchical models allow for estimation on clustered data, when classical assumptions of identically distributed random variables break down. Traditional likelihood-based methods for fitting hierarchical models often struggle with the scale of data found in industry, which has prompted recent research into moment-based procedures for parameter estimation. Spark doesn't have a native library for fitting these models, and to our knowledge, no moment-based estimation software has been developed previously utilizing a distributed computational system. This talk will review our development of Spark software utilizing these new estimation methods, detail the theory behind the approach, and compare our software to similar open source packages in Spark and other popular languages.

Moment-based estimation for hierarchical models in Apache Spark

Stitch Fix Algorithms

I'll provide guidelines for thinking about empirical performance evaluation of parallel programs in general and of Spark jobs in particular. It's easier to be systematic about this if you think in terms of "what's the effective network bandwidth we're getting?" instead of "How fast does this particular job run?" In addition, the figure of merit for parallel performance isn't necessarily obvious. If you want to minimize your AWS bill you should almost certainly run on a single node (but your job may take six months to finish). You may think you want answers as quickly as possible, but if you could make a job finish in 55 minutes instead 60 minutes while doubling your AWS bill, would you do it? No? Then what exactly is the metric that you should optimize?

Optimizing Spark

Stitch Fix Algorithms

The data platform at Stitch Fix runs thousands of jobs a day to feed data products that provide algorithmic capabilities to power nearly all aspects of the business, from merchandising to operations to styling recommendations. Many of these jobs are distributed across Spark clusters, while many others are scheduled as isolated single-node tasks in containers running Python, R, or Scala. Pipelines are often comprised of a mix of task types and containers. This talk will cover thoughts and guidelines on how we develop, schedule, and maintain these pipelines at Stitch Fix. We’ll discuss guidelines on how we think about which portions of the pipelines we develop to run on what platforms (e.g. what is important to run distributed across Spark clusters vs run in stand-alone containers) and how we get them to play well together. We’ll also provide an overview of tools and abstractions that have been developed at Stitch Fix to facilitate the process from development, to deployment, to monitoring them in production.

When We Spark and When We Don’t: Developing Data and ML Pipelines

Stitch Fix Algorithms

When marketing teams spend money on a paid acquisitions program it is crucial to understand the effect of that ad spend. In this talk, we will outline incrementality as a way to measure the causal impact that ad spend has on acquiring new customers and its advantages over more traditional metrics. We will walk through several ad measurement products available today and give examples of how to apply them to your business.

Incrementality

Stitch Fix Algorithms

Apache Spark & ML Workflows

Stitch Fix Algorithms

More from Stitch Fix Algorithms (10)

Progression by Regression: How to increase your A/B Test Velocity

Deep recommendations in PyTorch

Tracking data lineage at Stitch Fix

Improving ad hoc and production workflows at Stitch Fix

A compute infrastructure for data scientists

Moment-based estimation for hierarchical models in Apache Spark

Optimizing Spark

When We Spark and When We Don’t: Developing Data and ML Pipelines

Incrementality

Apache Spark & ML Workflows

Recently uploaded

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Heather Hedden, Senior Consultant at Enterprise Knowledge, presented “The Role of Taxonomy and Ontology in Semantic Layers” at a webinar hosted by Progress Semaphore on April 16, 2024. Taxonomies at their core enable effective tagging and retrieval of content, and combined with ontologies they extend to the management and understanding of related data. There are even greater benefits of taxonomies and ontologies to enhance your enterprise information architecture when applying them to a semantic layer. A survey by DBP-Institute found that enterprises using a semantic layer see their business outcomes improve by four times, while reducing their data and analytics costs. Extending taxonomies to a semantic layer can be a game-changing solution, allowing you to connect information silos, alleviate knowledge gaps, and derive new insights. Hedden, who specializes in taxonomy design and implementation, presented how the value of taxonomies shouldn’t reside in silos but be integrated with ontologies into a semantic layer. Learn about: - The essence and purpose of taxonomies and ontologies in information and knowledge management; - Advantages of semantic layers leveraging organizational taxonomies; and - Components and approaches to creating a semantic layer, including the integration of taxonomies and ontologies

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Enterprise Knowledge

Abhishek Deb(1), Mr Abdul Kalam(2) M. Des (UX) , School of Design, DIT University , Dehradun. This paper explores the future potential of AI-enabled smartphone processors, aiming to investigate the advancements, capabilities, and implications of integrating artificial intelligence (AI) into smartphone technology. The research study goals consist of evaluating the development of AI in mobile phone processors, analyzing the existing state as well as abilities of AI-enabled cpus determining future patterns as well as chances together with reviewing obstacles as well as factors to consider for more growth.

Exploring the Future Potential of AI-Enabled Smartphone Processors

debabhi2

The 7 Things I Know About Cyber Security After 25 Years | April 2024

Rafal Los

BooK Now Call us at +918448380779 to hire a gorgeous and seductive call girl for sex. Take a Delhi Escort Service. The help of our escort agency is mostly meant for men who want sexual Indian Escorts In Delhi NCR. It should be noted that any impersonator will get 100 attention from our Young Girls Escorts in Delhi. They will assume the position of reliable allies. VIP Call Girl With Original Photos Book Tonight +918448380779 Our Cheap Price 1 Hour not available 2 Hours 5000 Full Night 8000 TAG: Call Girls in Delhi, Noida, Gurgaon, Ghaziabad, Connaught Place, Greater Kailash Delhi, Lajpat Nagar Delhi, Mayur Vihar Delhi, Chanakyapuri Delhi, New Friends Colony Delhi, Majnu Ka Tilla, Karol Bagh, Malviya Nagar, Saket, Khan Market, Noida Sector 18, Noida Sector 76, Noida Sector 51, Gurgaon Mg Road, Iffco Chowk Gurgaon, Rajiv Chowk Gurgaon All Delhi Ncr Free Home Deliver

08448380779 Call Girls In Civil Lines Women Seeking Men

Delhi Call girls

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Delhi Call girls

Presentation on how to chat with PDF using ChatGPT code interpreter

naman860154

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

How to convert PDF to text with Nanonets

naman860154

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

How to Troubleshoot Apps for the Modern Connected Worker

ThousandEyes

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Imagine a world where information flows as swiftly as thought itself, making decision-making as fluid as the data driving it. Every moment is critical, and the right tools can significantly boost your organization’s performance. The power of real-time data automation through FME can turn this vision into reality. Aimed at professionals eager to leverage real-time data for enhanced decision-making and efficiency, this webinar will cover the essentials of real-time data and its significance. We’ll explore: FME’s role in real-time event processing, from data intake and analysis to transformation and reporting An overview of leveraging streams vs. automations FME’s impact across various industries highlighted by real-life case studies Live demonstrations on setting up FME workflows for real-time data Practical advice on getting started, best practices, and tips for effective implementation Join us to enhance your skills in real-time data automation with FME, and take your operational capabilities to the next level.

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Safe Software

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Histor y of HAM Radio presentation slide

vu2urc

Recently uploaded (20)

A Domino Admins Adventures (Engage 2024)

Axa Assurance Maroc - Insurer Innovation Award 2024

The Role of Taxonomy and Ontology in Semantic Layers - Heather Hedden.pdf

Exploring the Future Potential of AI-Enabled Smartphone Processors

The 7 Things I Know About Cyber Security After 25 Years | April 2024

08448380779 Call Girls In Civil Lines Women Seeking Men

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

08448380779 Call Girls In Greater Kailash - I Women Seeking Men

Presentation on how to chat with PDF using ChatGPT code interpreter

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

How to convert PDF to text with Nanonets

GenAI Risks & Security Meetup 01052024.pdf

Partners Life - Insurer Innovation Award 2024

How to Troubleshoot Apps for the Modern Connected Worker

Strategies for Landing an Oracle DBA Job as a Fresher

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

From Event to Action: Accelerate Your Decision Making with Real-Time Automation

Scaling API-first – The story of a global engineering organization

Automating Google Workspace (GWS) & more with Apps Script

Histor y of HAM Radio presentation slide

Enabling full stack data scientists

1. Enabling Full Stack Data Scientists at Stitch Fix Juliet Hougland @j_houg April 2018

2. Agenda @j_houg 2 WHAT WE DO HOW WE OPERATE TRANSFORMING THE WAY DATA SCIENTISTS DO WHAT THEY LOVE

3. @j_houg The Business The Customer Experience What We Do

4. @j_houg

5. @j_houg Transforming The Way People Find What They Love

6. @j_houg

7. @j_houg The Algorithms Org Operating Principles Data Platform How We Operate

8. @j_houg Algorithms Data Platform Data Science ClientStyling Operations Horizontal

9. @j_houg Autonomy & Ownership

10. @j_houg Typical Data Science Department Data Engineers Data Scientists Software Engineers Data Infrastructure Engineers provide resources & support

11. @j_houg Typical Data Science Department Data Driven Capability Data Engineers Data Scientists Software Engineers Infrastructure Engineers

12. @j_houg Another View Capability A Capability B Capability C Capability D Data Engineers Data Scientists Software Engineers Infrastructure Engineers specialized by function

13. @j_houg High Coordination Cost Capability A Capability B Capability C Capability D Data Engineers Data Scientists Software Engineers Infrastructure Engineers specialization by function tightly coupled, loosely aligned

14. @j_houg A Different Way to Organize Data Driven Capability DataScientists

15. @j_houg Organized by Capability specialized by capability Capability A Capability B Capability C Capability D DataScientists DataScientists DataScientists DataScientists

16. @j_houg Focus on Capabilities Data Driven Capability DataScientists Full Stack Data Scientists: " Full Stack Data Scientists Own: " Implementation " Coordination with the business " Production support (sort of)

17. @j_houg Where did the engineers go?

18. @j_houg Focus on Capabilities Data Platform Engineers loosely coupled, tightly aligned specialized by function Capability A Capability B Capability C Capability D DataScientists specialized by capability DataScientists DataScientists DataScientists

19. @j_houg Capabilities Data Platform Engineers specialized by function Capability A Capability B Capability C Capability D DataScientists specialized by capability DataScientists DataScientists DataScientists

20. @j_houg Infrastructure

21. @j_houg Black Box Data Scientists

22. @j_houg Data Transport

23. @j_houg The Black Box

24. @j_houg Analysis & Model Building

25. @j_houg Analysis & Model Building

26. @j_houg Analysis & Model Building Hard: •Reproducible •Accurate •Collaborative Opinionated Analysis Development — Hilary Parker https://peerj.com/preprints/3210/

27. @j_houg Analysis & Model Building

28. @j_houg Analysis & Model Building

29. @j_houg Dashboards

30. @j_houg Services Model Application Code

31. @j_houg Services