Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Zero to Insights Real time analytics with Kafka, C*, and Spark Peter Bakas
Peter Bakas | @peter_bakas @ Netflix : Cloud Platform Engineering - Event and Data Pipelines @ Ooyala : Analytics, Discove...
Let’s get down to business
Netflix is a logging company
that occasionally streams video
● 450 billion events per day ● 8 million events & 17 GB per second during peak ● Hundreds of event types By the Numbers
Publish, Collect, Process, Aggregate & Move Data
@ Cloud Scale
What does it take to run @ Cloud Scale?
How did we get here?
EMR Event Producer Chukwa
What are we supposed to do?
Event Producer Druid EMR Stream Consumers Kafka Router Suro Event Producer
Event Producer Druid EMR Stream Consumers Kafka Router Suro Event Producer
Where are we going?
Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
Routing Service ++
Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
Consumer Kafka Custom Apps Real time processing
Consumer Kafka Custom Apps Real time processing
Consumer Kafka Custom Apps Real time processing
Consumer Kafka Custom Apps Real time processing
Consumer Kafka Custom Apps Real time processing Fronting Kafka
Ooyala’s experience
About Ooyala Powering personalized video experiences across all screens.
● 5 billion events per day ● 1 billion videos per month ● 200 million unique users per month ● 130 countries ● 25% of U.S....
Where did it all start?
Precomputed Aggregates
What if we need more dynamic queries?
Why not just use C*?
What were the options? 100% Precomputation 100% Dynamic
Where we wanted to be 100% Precomputation 100% Dynamic Partly dynamic
Our solution
API loggersloggersloggersloggersloggers loggersloggersloggersloggersingest loggersloggersloggersloggersjob server Delphi -...
Challenges
● Hiring ● Rapidly evolving ecosystem ● Enterprise Service for Enterprise Software Challenges
Q&A time!
Obligatory... Everyone is hiring pbakas@netflix.com
Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015
Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015
Upcoming SlideShare
Loading in …5
×

Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015

1,363 views

Published on

In this talk, Peter will cover his experience using Spark, Cassandra & Kafka to build a real time analytics platform that processed billions events a day. He will cover the challenges in how to turn all those raw events into actionable insights. He will also cover scaling the platform across multiple regions, as well as across multiple cloud environments.

Published in: Data & Analytics
no profile picture user

  • Login to see the comments

  • Be the first to like this

Peter Bakas - Zero to Insights - Real time analytics with Kafka, C*, and Spark - NoSQL matters Dublin 2015

  1. 1. Zero to Insights Real time analytics with Kafka, C*, and Spark Peter Bakas
  2. 2. Peter Bakas | @peter_bakas @ Netflix : Cloud Platform Engineering - Event and Data Pipelines @ Ooyala : Analytics, Discovery, Platform Engineering & Infrastructure @ Yahoo : Display Advertising, Behavioral Targeting, Payments @ PayPal : Site Engineering and Architecture @ Play : Advisor to various Startups (Data, Security, Containers) Who is this guy?
  3. 3. Let’s get down to business
  4. 4. Netflix is a logging company
  5. 5. that occasionally streams video
  6. 6. ● 450 billion events per day ● 8 million events & 17 GB per second during peak ● Hundreds of event types By the Numbers
  7. 7. Publish, Collect, Process, Aggregate & Move Data
  8. 8. @ Cloud Scale
  9. 9. What does it take to run @ Cloud Scale?
  10. 10. How did we get here?
  11. 11. EMR Event Producer Chukwa
  12. 12. What are we supposed to do?
  13. 13. Event Producer Druid EMR Stream Consumers Kafka Router Suro Event Producer
  14. 14. Event Producer Druid EMR Stream Consumers Kafka Router Suro Event Producer
  15. 15. Where are we going?
  16. 16. Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
  17. 17. Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
  18. 18. Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
  19. 19. Routing Service ++
  20. 20. Stream Consumers Router EMR Fronting Kafka Event Producer Druid Consumer Kafka Keystone
  21. 21. Consumer Kafka Custom Apps Real time processing
  22. 22. Consumer Kafka Custom Apps Real time processing
  23. 23. Consumer Kafka Custom Apps Real time processing
  24. 24. Consumer Kafka Custom Apps Real time processing
  25. 25. Consumer Kafka Custom Apps Real time processing Fronting Kafka
  26. 26. Ooyala’s experience
  27. 27. About Ooyala Powering personalized video experiences across all screens.
  28. 28. ● 5 billion events per day ● 1 billion videos per month ● 200 million unique users per month ● 130 countries ● 25% of U.S. online viewers watch video powered by Ooyala By the Numbers
  29. 29. Where did it all start?
  30. 30. Precomputed Aggregates
  31. 31. What if we need more dynamic queries?
  32. 32. Why not just use C*?
  33. 33. What were the options? 100% Precomputation 100% Dynamic
  34. 34. Where we wanted to be 100% Precomputation 100% Dynamic Partly dynamic
  35. 35. Our solution
  36. 36. API loggersloggersloggersloggersloggers loggersloggersloggersloggersingest loggersloggersloggersloggersjob server Delphi - Real time Analytics Kafka
  37. 37. Challenges
  38. 38. ● Hiring ● Rapidly evolving ecosystem ● Enterprise Service for Enterprise Software Challenges
  39. 39. Q&A time!
  40. 40. Obligatory... Everyone is hiring pbakas@netflix.com

×