Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Spark and Spark
Streaming @ Netflix
Kedar Sadekar & Monal Daxini
Mission
•  Enable rapid pace of innovation for
Algorithm Engineers
•  Business Value – More A/B tests
Experiments
Users with plays Feature selection Large sample size
Turn back time Multiple ideas
Use Cases
Feature
Selection
Feature
Generation
Model
Training
Metric
Evaluation
Use Cases
More
Users
Faster
Iteration
Interactive
Turn back
time
Solution – Netflix BDAS
Netflix BDAS
Notebooks
• Zeppelin /
iPython
• Inline Graphs /
REPL
Prana Sidecar
• Netflix
Ecosystem...
Netflix BDAS - Features
•  Simplicity
–  Individual cluster
•  Prana - Netflix ecosystem
–  Automatic configuration
–  Cla...
Netflix BDAS - Features
•  Ad-hoc experimentation
•  Time machine functionality
•  Access to Hive data and micro services ...
Netflix BDAS – Sample Deployment
Wins
•  8X the number of users
•  5x - 9x faster
•  Interactive
•  Turn back time
Learnings
•  Easy to bring down an online system
•  Almost killed 1000’s of ETL jobs
- hive metastore update
•  Too many s...
Increased Adoption
Adoption increasing amongst teams
•  Multiple Algorithmic Eng. teams
•  Personalization Infrastructure
...
Looking Ahead
•  Spark-R / Dataframes support
•  Multi-tenancy
–  Job specific configurations
•  Debuggability
•  Newer no...
Netflix Streaming
Event Data Pipeline
Monal Daxini
Netflix Event Data Pipeline
Event Streams
Stream processing
Publish
Collect
Move
Process
Events @ Cloud Scale
450 Billion Events per Day
8 Million (17 GB) per sec peak
S3
EMR
Event
Producer
Fronting
Kafka
Consumer
Kafka
At least Once
Processing
Stream
Consumers
Mantis
450 Billion events / ...
S3
EMR
Event
Producer
Fronting
Kafka
Consumer
Kafka
At least Once
Processing
Stream
Consumers
Mantis
450 Billion events / ...
S3
EMR
Event
Producer
Fronting
Kafka
Consumer
Kafka
At least Once
Processing
Stream
Consumers
Mantis
450 Billion events / ...
S3
EMR
Event
Producer
Fronting
Kafka
Consumer
Kafka
Stream
Consumers
Mantis
450 Billion events / day
At least Once
Process...
Backpressure
Direct API for Kafka
Improve Cloud Multi-tenancy
What’s Missing?
+
ì
Backpressure+
JobScheduler JobGenerator DStreamGraph
Driver
00 : 0000 : 0100 : 0200 : 0300 : 0400 : 0500 : 0600 : 0700 : 0...
Backpressure
SPARK-7398 – Add backpressure to Spark streaming
SPARK-6691 – Add dynamic rate limiter to Spark streaming
+
Backpressure+
Backpressure implementation slated for
Spark 1.5 release
Direct API for Kafka
Spark 1.3
Kafka Integration
Spark 1.2
Receiver based
Kafka Integration
2x
Faster
Direct API for Kafka
Prefetch messages
Connection reuse (pooling)
Enhance
Cloud Multi-tenancyì
Cloud
Scheduler
Mesos Framework
Mesos Slave
Docker
Executor
Task Task
Docker
Executor
Task Task
Meso...
Next?
Measuring Spark Streaming Latencies
Spark Streaming Cloud Multi-tenancy
Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)
Upcoming SlideShare
Loading in …5
×

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)

9,379 views

Published on

Presentation at Spark Summit 2015

Published in: Data & Analytics
  • Hello! High Quality And Affordable Essays For You. Starting at $4.99 per page - Check our website! https://vk.cc/82gJD2
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Spark and Spark Streaming at Netfix-(Kedar Sedekar and Monal Daxini, Netflix)

  1. 1. Spark and Spark Streaming @ Netflix Kedar Sadekar & Monal Daxini
  2. 2. Mission •  Enable rapid pace of innovation for Algorithm Engineers •  Business Value – More A/B tests
  3. 3. Experiments Users with plays Feature selection Large sample size Turn back time Multiple ideas
  4. 4. Use Cases Feature Selection Feature Generation Model Training Metric Evaluation
  5. 5. Use Cases More Users Faster Iteration Interactive Turn back time
  6. 6. Solution – Netflix BDAS Netflix BDAS Notebooks • Zeppelin / iPython • Inline Graphs / REPL Prana Sidecar • Netflix Ecosystem Berkeley BDAS • Faster Compute • Scale Users • Access to S3 / Hive
  7. 7. Netflix BDAS - Features •  Simplicity –  Individual cluster •  Prana - Netflix ecosystem –  Automatic configuration –  Classloader isolation –  Discovery & healthcheck
  8. 8. Netflix BDAS - Features •  Ad-hoc experimentation •  Time machine functionality •  Access to Hive data and micro services from single place –  Access to multiple AWS buckets (S3)
  9. 9. Netflix BDAS – Sample Deployment
  10. 10. Wins •  8X the number of users •  5x - 9x faster •  Interactive •  Turn back time
  11. 11. Learnings •  Easy to bring down an online system •  Almost killed 1000’s of ETL jobs - hive metastore update •  Too many systems and configuration •  Playing catch up with libraries and tools - Hadoop, iPython, Zeppelin •  Scala / Spark learning curve •  Debugging - files open, no resources etc.
  12. 12. Increased Adoption Adoption increasing amongst teams •  Multiple Algorithmic Eng. teams •  Personalization Infrastructure •  Marketing •  Security •  A/B Test Engineering
  13. 13. Looking Ahead •  Spark-R / Dataframes support •  Multi-tenancy –  Job specific configurations •  Debuggability •  Newer notebooks •  Spark Streaming –  Lambda Architecture –  Real-time algorithms (trending now)
  14. 14. Netflix Streaming Event Data Pipeline Monal Daxini
  15. 15. Netflix Event Data Pipeline Event Streams Stream processing
  16. 16. Publish Collect Move Process Events @ Cloud Scale
  17. 17. 450 Billion Events per Day 8 Million (17 GB) per sec peak
  18. 18. S3 EMR Event Producer Fronting Kafka Consumer Kafka At least Once Processing Stream Consumers Mantis 450 Billion events / day
  19. 19. S3 EMR Event Producer Fronting Kafka Consumer Kafka At least Once Processing Stream Consumers Mantis 450 Billion events / day
  20. 20. S3 EMR Event Producer Fronting Kafka Consumer Kafka At least Once Processing Stream Consumers Mantis 450 Billion events / day
  21. 21. S3 EMR Event Producer Fronting Kafka Consumer Kafka Stream Consumers Mantis 450 Billion events / day At least Once Processing
  22. 22. Backpressure Direct API for Kafka Improve Cloud Multi-tenancy What’s Missing? + ì
  23. 23. Backpressure+ JobScheduler JobGenerator DStreamGraph Driver 00 : 0000 : 0100 : 0200 : 0300 : 0400 : 0500 : 0600 : 0700 : 0800 : 0900 : 10 Unbounded … Ops Queue
  24. 24. Backpressure SPARK-7398 – Add backpressure to Spark streaming SPARK-6691 – Add dynamic rate limiter to Spark streaming +
  25. 25. Backpressure+ Backpressure implementation slated for Spark 1.5 release
  26. 26. Direct API for Kafka Spark 1.3 Kafka Integration Spark 1.2 Receiver based Kafka Integration 2x Faster
  27. 27. Direct API for Kafka Prefetch messages Connection reuse (pooling) Enhance
  28. 28. Cloud Multi-tenancyì Cloud Scheduler Mesos Framework Mesos Slave Docker Executor Task Task Docker Executor Task Task Mesos Slave Docker Spark Driver Improve
  29. 29. Next? Measuring Spark Streaming Latencies Spark Streaming Cloud Multi-tenancy

×