Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The evolution of the big data platform @ Netflix (OSCON 2015)

20,099 views

Published on

At Netflix, the big data platform is the foundation for analytics that drive all product decisions that directly impact our customer experience. As for scale, it is one of the top three largest services running at Netflix, in terms of compute power and data size.

In this talk, we will take the audience through a journey to understand how we scale the platform to handle the increasing amount of data (over 500 billion events generated daily), the increasing demand of analytics (which translates to compute power), and the increasing number of users dependent on our platform to make business decisions.

Published in: Engineering
  • Be the first to comment

The evolution of the big data platform @ Netflix (OSCON 2015)

  1. 1. The Evolution of Big Data Platform @ Netflix Eva Tse July 22, 2015
  2. 2. Our biggest challenge is scale
  3. 3. Netflix Key Business Metrics 65+ million members 50 countries 1000+ devices supported 10 billion hours / quarter
  4. 4. Global Expansion 200 countries by end of 2016
  5. 5. Big Data Size Total ~20 PB DW on S3 Read ~10% DW daily Write ~10% of read data daily ~ 500 billion events daily ~ 350 active users
  6. 6. Our traditional BI stack is our competition
  7. 7. How do we meet the functionality bar and yet make it scale? How do we make big data bite-size again?
  8. 8. Our North Star
  9. 9. Cloud apps Suro/Kaf ka Ursula Cassandra Aegisthus Dimension Data Event Data 15 min Daily AWS S3 SS Tables
  10. 10. Storage Compute Service Tools AWS S3
  11. 11. Analytics ETL Interactive data exploration Interactive slice & dice RT analytics & iterative/ML algo Evolving Big Data Processing Needs
  12. 12. Service Tools Big Data Portal API Portal Big Data API Evolving Services/Tools Ecosystem
  13. 13. AWS S3 as our DW Storage
  14. 14. Evolution of Big Data Processing Systems
  15. 15. Parquet
  16. 16. Service Tools Big Data Portal API Portal Big Data APIEvolution of Services/Tools Ecosystem
  17. 17. Metacat
  18. 18. Service Tools Big Data Portal API Portal Big Data API dd
  19. 19. Big Data API
  20. 20. Big Data Portal
  21. 21. Open source is an integral part of our strategy to achieve scale
  22. 22. Big Data Processing Systems Services/Tools Ecosystem
  23. 23. Why use Open Source?
  24. 24. Why contribute back?
  25. 25. Why contribute our own tool?
  26. 26. Is open source right for you?
  27. 27. Measuring big data - understanding data by usage By Charles Smith, Netflix Tomorrow @ 1:40-2:20pm
  28. 28. Eva Tse etse@netflix.com jobs.netflix.com

×