Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerard Maas
•
17 likes•4,188 views
This document discusses Spark Streaming and how it can push throughput limits in a reactive way. It describes how Spark Streaming works by breaking streams into micro-batches and processing them through Spark. It also discusses how Spark Streaming can be made more reactive by incorporating principles from Reactive Streams, including composable back pressure. The document concludes by discussing challenges like data locality and providing resources for further information.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerard Maas
Similar to Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerard Maas (20)
More from Spark Summit
More from Spark Summit (20)
Recently uploaded
Recently uploaded (20)
Spark Streaming: Pushing the throughput limits by Francois Garillot and Gerard Maas
- 1. SPARK STREAMING: PUSHING THE THROUGHPUT LIMITS, THE REACTIVE WAY François Garillot, Gerard Maas
- 2. Who Are We ? Gerard Maas Data Processing Team Lead François Garillot work done at Spark Streaming at
- 3. @maasg @huitseeker Spark Streaming (Refresher)
- 4. @maasg @huitseeker Spark Streaming (Refresher) DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1
- 5. @maasg @huitseeker Spark Streaming (Refresher) DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Transformations
- 6. @maasg @huitseeker Spark Streaming (Refresher) DStream[T] RDD[T] RDD[T] RDD[T] RDD[T] RDD[T] t0 t1 t2 t3 ti ti+1 RDD[U] RDD[U] RDD[U] RDD[U] RDD[U] Actions Transformations
- 7. @maasg @huitseeker Spark Streaming (Refresher) Spark API for Streams Fault-tolerant High Throughput Scalable
- 8. @maasg @huitseeker Streaming Spark t0 t1 t2 #0 Consumer Consumer Consumer Scheduling
- 9. @maasg @huitseeker Streaming Spark t0 t1 t2 #1 Consumer Consumer Consumer #0 Scheduling Process Time < Batch Interval
- 10. @maasg @huitseeker Streaming Spark t0 t1 t2 #2 Consumer Consumer Consumer #0 #1 #3 Scheduling Scheduling Delay
- 11. @maasg @huitseeker From Streams to μbatches Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark #partitions = receivers x batchInterval / blockInterval
- 12. @maasg @huitseeker From Streams to μbatches #0 RDD Partitions Spark Spark Executors Spark Streaming
- 13. @maasg @huitseeker From Streams to μbatches #0 RDD Spark Spark Executors Spark Streaming
- 14. @maasg @huitseeker From Streams to μbatches #0 RDD Spark Spark Executors Spark Streaming
- 15. @maasg @huitseeker From Streams to μbatches #0 RDD Spark Spark Executors Spark Streaming
- 18. @maasg @huitseeker From Streams to μbatches Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark #partitions = receivers x batchInterval / blockInterval
- 19. @maasg @huitseeker From Streams to μbatches Consumer #0 #1 batchInterval blockInterval Spark Streaming Spark spark.streaming.blockInterval = batchInterval x receivers / (partitionFactor x sparkCores)
- 20. @maasg @huitseeker The Importance of Caching dstream.foreachRDD { rdd => rdd.cache() // cache the RDD before iterating! keys.foreach{ key => rdd.filter(elem=> key(elem) == key).saveAsFooBar(...) } rdd.unpersist() }
- 21. @maasg @huitseeker Intervals (Read TD’s Adaptive Stream Processing using Dynamic Batch Sizing before drawing any conclusions !) O(n²) O(n²) O(n) O(n)
- 22. @maasg @huitseeker The Receiver model spark.streaming.receiver.maxRate Fault tolerance ? WAL
- 23. @maasg @huitseeker Direct Kafka Stream compute(offsets)
- 24. Kafka:The Receiver-less model Simplified Parallelism Efficiency Exactly-once semantics Less degrees of freedom val directKafkaStream = KafkaUtils. createDirectStream[ [key class], [value class], [key decoder class], [value decoder class] ]( streamingContext, [map of Kafka parameters], [set of topics to consume] ) spark.streaming.kafka.maxRatePerPartition
- 26. @maasg @huitseeker Reactive Principles Reactive Streams : composable back-pressure
- 27. @maasg @huitseeker Spark Streaming made Reactive
- 28. @maasg @huitseeker Spark Streaming made Reactive
- 29. @maasg @huitseeker Spark Streaming made Reactive
- 30. @maasg @huitseeker Spark Streaming Made Reactive
- 33. @maasg @huitseeker Pain point : Data Locality - Where is your job getting executed ? spark.locality.wait & spark.streaming.blockInterval - On Mesos, it’s worse (SPARK-4940)
- 34. @maasg @huitseeker Resources Backpressure in Spark Streaming: http://blog.garillot.net/post/121183250481/a-quick-update-on-spark-streaming-work-since-i The Virdata’s Spark Streaming tuning guide: http://www.virdata.com/tuning-spark/ TD’s paper on dynamic batch sizing : http://dl.acm.org/citation.cfm?id=2670995 Diving into Spark Streaming Execution Model: https://databricks.com/blog/2015/07/30/diving-into-spark-streamings-execution-model.html Spark Streaming / Storm Trident numbered comparison: https://www.cs.utoronto.ca/~patricio/docs/Analysis_of_Real_Time_Stream_Processing_Systems_Considering_Latency.pdf Kafka direct approach: https://github.com/koeninger/kafka-exactly-once/blob/master/blogpost.md