This document discusses setting up data science projects for success by focusing on the importance of data preparation. It notes that 76% of data scientists view data preparation as the least enjoyable part of their work. The document outlines various facets of data preparation, including collecting, understanding, cleaning, and reshaping data. It emphasizes that data quality is important and a shared responsibility across data engineering, data science, and business intelligence teams. It recommends creating a single source of truth for data through techniques like data dictionaries to define data for all teams.

2. Setting up Data Science
for Success: The Data
Layer

3. WE ARE HIRING!
https://www.weightwatchers.com/us/corporate-careers

5. 80% 10% 10%
Data
Preparation
Modeling Interpretation
& Presentation

6. 76% of data scientists view data preparation as the least
enjoyable part of their work (Crowdflower / Figure8, 2016)

7. Josh Wills (Slack):
”I’m a data janitor. That’s the
sexiest job of the 21st century.”

9. 80% 10% 10%
A B C
time

10. ● Collecting: searching, API, web scraping
● Understanding structure: entities, relations, foreign keys, UUIDs
● Understanding data: what fields / metrics mean, categorical variables
● Profiling: EDA, ranges, distributions
● Cleaning & normalizing: date formats, encodings, data types
● Reshaping and re-formatting data
● Filter, aggregate
● Deal with scaling issues, waiting for queries, set indexes & partitions

11. Various facets of data quality:
● Accurate: right types/format
● Coherent: referential integrity, no dupes
● Complete: no missing records & values
● Timely: in order, not late
● Defined: data dictionary, no field stuffing
● ....

12. Data
Engineering
Data
Science
Business
Intelligence
Internal Apps
/ Services
External Data /
Services

13. Not my Job!
Data Quality has
to be a shared
responsibility

14. Upstream, defensive programming

15. ● UTC
● Stable schema
● DB constraints: no dupes
● Audit table / change log
● “Our eventing has always been ‘fire and
forget’ with no guarantee of delivery.”

16. Validate with known answers

17. ● Monitoring / Alerts
● Retries
● Data type & range checks
● DB constraints
● Audit logs
● Schema on write

18. Own:
● Single Source of Truth
● Metric implementation
● Monitoring / Alerts
● Data Dictionary

19. “I found the right table and have this field but I don’t
know what it means”

20. https://medium.com/@leapingllamas: Data Dictionary: a how to and best practices
“How do we define ARPU?”
“This is my understanding of active user”

21. ● Data quality is important and a shared responsibility
● Create a single source of truth
● Create table-level and company-level data dictionaries
Your data scientists will thank you!

22. Thank you!
https://www.weightwatchers.com/us/corporate-careers