Uploaded byMichelle Ufford
Scaling Data Quality @ Netflix
Michelle Ufford of Netflix presented on their approach to data quality. They developed Quinto, a data quality service that implements a Write-Audit-Publish pattern for ETL jobs. It audits metrics after data is written to check for issues like row counts being too high/low. Configurable rules determine if issues warrant failing or warning on a job. Future work includes expanding metadata tracking and anomaly detection. The presentation emphasized building modular components over monolithic frameworks and only implementing quality checks where needed.
More Related Content
Similar to Scaling Data Quality @ Netflix
Scaling Data Quality @ Netflix
- 1. WHOOPS, THE NUMBERS AREWRONG! SCALING DATA QUALITY @ MICHELLE UFFORD DATA ENGINEERING & ANALYTICS, NETFLIX HADOOP SUMMIT 2017
- 2.
- 3. The business. 20170612 100+ million members $6billion on content 125+ million hours watched launched in 1997 every. day. $
- 4. Anytime. Anywhere.* 20170612 * Well,almost anywhere.
- 5.
- 6. 300 terabyte DW writes 5petabyte DW reads The data. 20170612 60+ petabyte data warehouse 700+ billion events written
- 7. 300 terabyte DW writes 5petabyte DW reads The data. 20170612 60+ petabyte data warehouse 700+ billion events written
- 8. 20170612 data access AWS S3 Amazon Redshift data processing faststorage data viz METACA T data services events data operational data elastic storage Apache Pig Big Data Platform
- 9.
- 11. Federated metastore & extensibledata catalog
- 12.
- 13.
- 14.
- 15.
- 16. 20170612 Extended table attributes ●primary key(s) ● column types ● lifecycle ● audience ● “valid-thru” timestamp ● … and much more Metacat Federated Metastore
- 17.
- 18.
- 19.
- 20.
- 21.
- 22. Write - Audit- Publish ETL pattern for high-quality big data jobs
- 23.
- 24.
- 25.
- 26.
- 27. 20170612 s3://…/utc_date=20170101/batchid=1483229855 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount:21 ... Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... ETL Pattern
- 28. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric evalbehavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job Quinto configuration utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... ETL Pattern
- 29. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric evalbehavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 30. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_f Quint o metric evalbehavior result -------------------------------------------------- RowCount >= zero fail job RowCount >= prior value fail job NullCount normal dist warn job utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 31. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 32. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 33. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 34. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 35. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 36. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 37. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job fail Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 38. 20170612 … WAPStage-2: Audit audit.my_table_f_1497312000dw.my_table_fmetric evalbehavior result -------------------------------------------------- RowCount >= zero fail job pass RowCount >= prior value fail job pass NullCount normal dist warn job fail Quint o utc_date=20170612 com.netflix.dse.mds.metric.RowCount: 17240 com.netflix.dse.mds.metric.NullCount: 17240 ... utc_date=20170611 com.netflix.dse.mds.metric.RowCount: 16135 com.netflix.dse.mds.metric.NullCount: 21 ... Quinto configuration ETL Pattern
- 39.
- 40.
- 41.
- 43. 20170612 Quinto evaluations ● intelligentrecommendations ● multiple tiers of coverage ● configurable rules Jumpstarter. Python Library
- 44. WAP. Python Library Minimal requirements ●parameterized destination table
- 45.
- 47. 20170612 What’s Next. ● additionalMetacat statistics ● robust anomaly detection (RAD) ● complete migration for all prod tables
- 48. 20170612 Tips & LessonsLearned. ● Query-based solution may be “good enough” for many. ● Not all tables need quality coverage. ● One size rarely fits all tables. ● Build components, not “all-or-nothing” frameworks.
- 49.
Editor's Notes
- #2 https://dataworkssummit.com/san-jose-2017/sessions/whoops-the-numbers-are-wrong-scaling-data-quality-netflix/ WHOOPS, THE NUMBERS ARE WRONG! SCALING DATA QUALITY @ NETFLIX Netflix is a famously data-driven company. Data is used to make informed decisions on everything from content acquisition to content delivery, and everything in-between. As with any data-driven company, it’s critical that data used by the business is accurate. Or, at worst, that the business has visibility into potential quality issues as soon as they arise. But even in the most mature data warehouses, data quality can be hard. How can we ensure high quality in a cloud-based, internet-scale, modern big data warehouse employing a variety of data engineering technologies? In this talk, Michelle Ufford will share how the Data Engineering & Analytics team at Netflix is doing exactly that. We’ll kick things off with a quick overview of Netflix’s analytics environment, then dig into details of our data quality solution. We’ll cover what worked, what didn’t work so well, and what we plan to work on next. We’ll conclude with some tips and lessons learned for ensuring data quality on big data. DETAILS This session is a (Intermediate) talk in our Data Processing and Warehousing track. It focuses on Apache Hadoop, Apache Hive, Apache Pig, Apache Spark and is geared towards Architect, Data Analyst, Developer / Engineer, Operations / IT audiences.