The document summarizes the evolution of Netflix's big data platform to meet the challenges of their growing scale. Key points include: - Netflix now has over 65 million members in over 50 countries and supports over 1000 devices. They stream over 10 billion hours of content per quarter. - Their traditional business intelligence stack could no longer meet the demands of scale. They transitioned to using AWS services like S3 for storage and open source tools like Kafka, Cassandra, and Parquet to enable real-time analytics and machine learning on their massive data volumes. - Netflix has adopted an open source-first strategy and contributes back to the community as their own tools evolve to meet processing needs and achieve the necessary scale to

1. The Evolution of Big Data Platform
@
Netflix
Eva Tse
July 22, 2015

6. Our biggest challenge is scale

7. Netflix Key Business Metrics
65+ million
members
50 countries 1000+ devices
supported
10 billion
hours / quarter

8. Global Expansion
200 countries by end of 2016

9. Big Data Size
Total ~20 PB DW on S3
Read ~10% DW daily
Write ~10% of read data daily
~ 500 billion events daily
~ 350 active users

10. Our traditional BI stack is
our competition

11. How do we meet the functionality bar and yet
make it scale?
How do we make big data bite-size again?

12. Our North Star

13. Cloud
apps
Suro/Kaf
ka
Ursula
Cassandra
Aegisthus
Dimension Data
Event Data
15 min
Daily
AWS
S3
SS Tables

14. Storage Compute Service Tools
AWS
S3

15. Analytics
ETL
Interactive data exploration
Interactive slice & dice
RT analytics & iterative/ML algo
Evolving Big Data Processing Needs

16. Service Tools
Big Data
Portal
API Portal
Big Data
API
Evolving Services/Tools Ecosystem

17. AWS S3 as our DW Storage

18. Evolution of Big Data Processing Systems

24. Parquet

27. Service Tools
Big Data
Portal
API Portal
Big Data
APIEvolution of Services/Tools
Ecosystem

29. Metacat

30. Service Tools
Big Data
Portal
API Portal
Big Data
API dd

36. Big Data API

39. Big Data Portal

44. Open source is an integral part of our
strategy to achieve scale

45. Big Data Processing
Systems
Services/Tools
Ecosystem

46. Why use Open Source?

47. Why contribute back?

48. Why contribute our own tool?

50. Is open source right for you?

52. Measuring big data - understanding data by usage
By Charles Smith, Netflix
Tomorrow @ 1:40-2:20pm

53. Eva Tse
etse@netflix.com
jobs.netflix.com