Netflix Data Engineering @ Uber Engineering Meetup
•
22 likes•4,765 views
People, Platform, Projects: these slides overview how Netflix works with Big Data. I share how our teams are organized, the roles we typically have on the teams, an overview of our Big Data Platform, and two example projects.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Netflix Data Engineering @ Uber Engineering Meetup
Similar to Netflix Data Engineering @ Uber Engineering Meetup (20)
Recently uploaded
Recently uploaded (20)
Netflix Data Engineering @ Uber Engineering Meetup
- 1. NETFLIX DATA ENGINEERING & ANALYTICS BLAKE IRVINE DATA ENGINEERING & ANALYTICS 2016.09.22
- 2. TOPICS PEOPLE WHO WE ARE PLATFORM HOW WE WORK PROJECTS WHAT WE DO
- 4. PEOPLE Analytic Groups @ NFLX Data Engineering & Analytics Science & Algorithms FP&A
- 5. PEOPLE (Data) Engineering Groups @ NFLX Data Engineering & Analytics Cloud Platform Engineering Product Engineering
- 7. PEOPLE Aligned Teams = Maximum Context Marketing Product Playback Content FinOps
- 8. PEOPLE Roles + Freedom & Responsibility Data Analysts Viz Engineers Analytics Engineers Data Engineers
- 9. PEOPLE Role Core Functions Data Analysts Viz Engineers Analytics Engineers Data Engineers PipelinesDashboardsJS / VizAnalytics
- 10. QUESTION BREAK ?
- 12. PLATFORM Another talented, aligned team
- 13. PEOPLE What is the Netflix data platform?
- 15. PLATFORM Event Pipeline 〉 83 Million members 〉 1000+ Devices 〉 Up to 1 Trillion daily events 〉 ~1m traversal time 〉 S3 warehouse is 60 PB
- 16. PLATFORM Query Engines 〉 Hadoop (EMR) is primarily batch 〉 Presto is primarily ad-hoc 〉 Spark is emergent batch + ad-hoc
- 17. 〉 S3 is our source-of-truth 〉 We copy forward to Redshift 〉 Indexed data loaded to Druid 〉 Data access via Big Data API PLATFORM Data Stores
- 18. 〉 Big Data Portal 〉 Tableau 〉 JS / React / Node Apps Analytic Tools PLATFORM Big Data Portal
- 22. PLATFORM My Team’s Analytic Tools Usage
- 23. QUESTION BREAK ?
- 25. 〉 Big fast data 〉 Efficient compute PROJECTS WHAT WE DO
- 26. 〉 Partner Ecosystem Dashboard PROJECTS Big Fast Data
- 27. PROJECTS Partner Example - Trip to Manhattan
- 28. PROJECTS Partner Example - NYC Marriott
- 29. PROJECTS Partner Example - Work & Home Again
- 30. PROJECTS Partner Example - Multiple Partners
- 31. 〉 This dashboard’s dimensional grain >1B rows PROJECTS Partner Ecosystem Dashboard 〉 Current solution is 〉 ETL in Pig 〉 Prepare Druid indexed dataset with Hadoop job 〉 Load to 100 historical nodes 〉 Queries are typically <2 seconds 〉 Solution v1 was Tableau + Redshift SSD 〉 Some views rendered in <10 seconds 〉 But multiple views with filters took >1 minute 〉 Future solution is...
- 32. 〉 Counting things 〉 Measuring time PROJECTS Efficient compute
- 33. 〉 Easy: playback events, units of time, other discrete events 〉 Hard: uniques, especially over windows or groups PROJECTS Efficient compute - Counting 〉 Solution: estimates, i.e. HyperLogLog(++) 〉 Where we’ll implement: 〉 Distribution arrays in staging tables 〉 In Druid data sources 〉 Merge functions in query layer, so UDFs for Pig/Hive/Presto/JS
- 34. 〉 Easy: average, but not accurate 〉 Hard: percentiles PROJECTS Efficient compute - Measuring time 〉 Solution: estimates, most likely T-digest 〉 Where we’ll implement: 〉 Digests in staging tables 〉 In Druid data sources 〉 Merge functions in query layer, so UDFs for Pig/Hive/Presto/JS
- 35. THANK YOU PEOPLE WHO WE ARE PLATFORM HOW WE WORK PROJECTS WHAT WE DO