Downloaded 69 times
Uploaded byDataWorks Summit
Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
The document discusses data quality at Netflix, highlighting the company's data engineering and analytics framework used to support over 100 million members and 60+ petabytes of data. It emphasizes the use of a federated metastore and the QUINTO data quality service to ensure high-quality data through audit, publish, and ETL (Extract, Transform, Load) patterns. The presentation outlines key metrics for data evaluation, tips for implementation, and future directions for enhancing data management strategies.
More Related Content
Similar to Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
More from DataWorks Summit
Whoops, The Numbers Are Wrong! Scaling Data Quality @ Netflix
- 1. WHOOPS, THE NUMBERS AREWRONG! SCALING DATA QUALITY @ MICHELLE UFFORD DATA ENGINEERING & ANALYTICS, NETFLIX HADOOP SUMMIT 2017
- 2.
- 3. The business. 20170612 100+ million members $6billion on content 125+ million hours watched launched in 1997 every. day. $
- 4. Anytime. Anywhere.* 20170612 * Well,almost anywhere.
- 5.
- 6. 300 terabyte DW writes 5petabyte DW reads The data. 20170612 60+ petabyte data warehouse 700+ billion events written
- 7. 300 terabyte DW writes 5petabyte DW reads The data. 20170612 60+ petabyte data warehouse 700+ billion events written
- 8. 20170612 data access AWS S3 Amazon Redshift data processing faststorage data viz METACA T data services events data operational data elastic storage Apache Pig Big Data Platform
- 9.
- 11. Federated metastore & extensibledata catalog
- 12.
- 13.
- 14.
- 15.
- 16. 20170612 Extended table attributes ●primary key(s) ● column types ● lifecycle ● audience ● “valid-thru” timestamp ● … and much more Metacat Federated Metastore
- 17.
- 18.
- 19.
- 20.
- 21.
- 22. Write - Audit- Publish ETL pattern for high-quality big data jobs
- 23.
- 24.
- 25.
- 26.
- 27. 20170612 s3://…/utc_date=20170101/batchid=1483229855 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount:21 ... Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... ETL Pattern
- 28. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric evalbehavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job Quinto configuration utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... ETL Pattern
- 29. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric evalbehavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 30. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric evalbehavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 31. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 32. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 33. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 34. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 35. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 36. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 37. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job fail Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 38. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job fail Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 39.
- 40.
- 41.
- 43. 20170612 Quinto evaluations ● intelligentrecommendations ● multiple tiers of coverage ● configurable rules Jumpstarter. Python Library
- 44. WAP. Python Library Minimal requirements ●parameterized destination table
- 45.
- 47. 20170612 What’s Next. ● additionalMetacat statistics ● robust anomaly detection (RAD) ● complete migration for all prod tables
- 48. 20170612 Tips & LessonsLearned. ● Query-based solution may be “good enough” for many. ● Not all tables need quality coverage. ● One size rarely fits all tables. ● Build components, not “all-or-nothing” frameworks.
- 49.
Editor's Notes
- #2 https://dataworkssummit.com/san-jose-2017/sessions/whoops-the-numbers-are-wrong-scaling-data-quality-netflix/ WHOOPS, THE NUMBERS ARE WRONG! SCALING DATA QUALITY @ NETFLIX Netflix is a famously data-driven company. Data is used to make informed decisions on everything from content acquisition to content delivery, and everything in-between. As with any data-driven company, it’s critical that data used by the business is accurate. Or, at worst, that the business has visibility into potential quality issues as soon as they arise. But even in the most mature data warehouses, data quality can be hard. How can we ensure high quality in a cloud-based, internet-scale, modern big data warehouse employing a variety of data engineering technologies? In this talk, Michelle Ufford will share how the Data Engineering & Analytics team at Netflix is doing exactly that. We’ll kick things off with a quick overview of Netflix’s analytics environment, then dig into details of our data quality solution. We’ll cover what worked, what didn’t work so well, and what we plan to work on next. We’ll conclude with some tips and lessons learned for ensuring data quality on big data. DETAILS This session is a (Intermediate) talk in our Data Processing and Warehousing track. It focuses on Apache Hadoop, Apache Hive, Apache Pig, Apache Spark and is geared towards Architect, Data Analyst, Developer / Engineer, Operations / IT audiences.
- #6 1500+ devices as of Q1 2017
- #11 Goal is to provide behind-the-scenes look at how we’re approaching DQ. We’re sharing ideas, not code – no open-source announcement.
- #43 That’s cool but sounds like a lot of work I need to: know what stats are available in Metacat know what quality templates exist figure out which ones I should use figure out a good configuration for each and do everything we just walked through in WAP
- #46 Takes ~5 minutes to enable WAP with 108 audits on a new Spark job. Added 42 seconds to hourly processing time.
- #47 It’s a combination of these solutions that allow us to scale not only the processing time but the engineering time too. Takes ~5 minutes to enable WAP with 108 audits on a new Spark job. Added 42 seconds to hourly processing time.
- #48 Stats Cardinality Histograms Map keys RAD High cardinality dimensions Seasonality beyond week-over-week Atypical data distributions Reduce false positives
- #49 Query-based solution Not efficient but much less complicated to implement Good place to start Works well for small-to-medium datasets and/or nightly batch ETL Transient, experimental, single-user Content vs. Streaming
- #51 2 main motivations Confidence Notify users when quality issues arise Only make data available after some basic validation Increase confidence for data consumers that data is good to use Efficiency Catch issues faster Less business impact Much easier to simply not update downstream dependencies than to fix it after-the-fact