A few years ago, Netflix had a fairly classic business intelligence tech stack. Now, things have changed. Netflix is a heavy user of AWS for much of its ongoing operations, and Data Science & Engineering (DSE) is no exception. In this talk, we dive into the Netflix DSE architecture: what and why. Key topics include their use of Big Data technologies (Cassandra, Hadoop, Pig + Python, and Hive); their Amazon S3 central data hub; their multiple persistent Amazon EMR clusters; how they benefit from AWS elasticity; their data science-as-a-service approach, how they made a hybrid AWS/data center setup work well, their open-source Hadoop-related software, and more.

1. Data Science at Netflix
with Amazon EMR
Kurt Brown, Director of Data Platform, Netflix
November 13, 2013
© 2013 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

3. Data Platform

6. Data Platform
Suro
S3

8. Aegisthus

10. Data Platform
Suro
Aegisthus
S3

11. Sting

14. Data Platform
Sting
Suro
Aegisthus
S3

17. Data Platform
Sting
Suro
Aegisthus
S3

18. Data Platform
Sting
Suro
Aegisthus
S3

19. S3

20. S3

21. 99.999999999%

23. S3

24. S3
High SLA
Query

26. HDFS
?

27. Eventual
Consistency

28. S3mper

29. “Data as a Service”
• Execution Service
• Event Service
• Metadata Service

38. High SLA Cluster Job
High SLA
Query Cluster Job
Query
S3

39. High SLA
Query Cluster Job
Query
S3

40. High SLA Cluster Job
High SLA
Query Cluster Job
Query
S3

41. Bonus Cluster Job
Bonus
High SLA Cluster Job
High SLA
Query Cluster Job
Query
S3

42. Bonus Cluster Job
High SLA Cluster Job
High SLA
Query Cluster Job
Query
S3

47. Tez

48. Suro

49. Aegisthus

52. Questions?
kurtbrown@netflix.com
http://jobs.netflix.com

53. Please give us your feedback on this
presentation
BDT306
As a thank you, we will select prize
winners daily for completed surveys!