Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (6)
Similar to Apache spark? if only it worked
Similar to Apache spark? if only it worked (20)
Recently uploaded
Recently uploaded (20)
Apache spark? if only it worked
- 1. @mszymani#Devoxx #TantusData #spark Apache Spark - if only it worked Marcin Szymaniuk TantusData
- 3. @mszymani#Devoxx #TantusData #spark Agenda • Execution model • Sizing executors • Skewed data • Locality • Caching • Debugging tools • Local run/tests • Challenge and a draw!
- 4. @mszymani#Devoxx #TantusData #spark What is Spark? • Engine for distributed data processing • Java, Scala, R, Python API • SQL, Streaming, Machine Learning
- 5. @mszymani#Devoxx #TantusData #spark … … … … … … … Stage 1 Stage N … … … … … … … Stage 2 … Shuffle Shuffle RDD
- 6. @mszymani#Devoxx #TantusData #spark … … … … … … … Stage 1 Stage N … … … … … … … Stage 2 … Shuffle Shuffle
- 7. @mszymani#Devoxx #TantusData #spark … … … … … … … Stage 1 Stage N … … … … … … … Stage 2 … Shuffle Shuffle
- 8. @mszymani#Devoxx #TantusData #spark … … … … … … … Stage 1 Stage N … … … … … … … Stage 2 … Shuffle Shuffle
- 9. @mszymani#Devoxx #TantusData #spark .reduceByKey{case (x, y) => x + y} .saveAsTextFile(output) Block 1 Block 2 … Task 1 sc.textFile(“/input/text/“) .flatMap(line=>line.split(" ")) .map(word => (word, 1)) Stage 1 Stage 2 … … … … … … … … … … … Stage 2 … … Stage 1 ………
- 10. @mszymani#Devoxx #TantusData #spark .reduceByKey{case (x, y) => x + y} .saveAsTextFile(output) Block 1 Block 2 … Task 1 sc.textFile(“/input/text/“) .flatMap(line=>line.split(" ")) .map(word => (word, 1)) Stage 1 Stage 2 … … … … … … … … … … … Stage 2 … … Stage 1 ………
- 11. @mszymani#Devoxx #TantusData #spark Sizing executors Executor - JVM process able to run one or more tasks … Executor Example config: --executor-cores 3 --executor-memory 10g … Executor … … Executor … Pending Tasks Complete Tasks … … Executor Job resources
- 12. @mszymani#Devoxx #TantusData #spark Executor - JVM process able to run one or more tasks … Executor Example config: --executor-cores 3 --executor-memory 10g … Executor … … Executor … Pending Tasks Complete Tasks … … Executor Job resources Driver Sizing executors
- 13. @mszymani#Devoxx #TantusData #spark Sizing executors Executor - JVM process able to run one or more tasks … Executor Example config: --executor-cores 3 --executor-memory 10g Executor Executor Executor Executor Node … Executor Executor Node … Node … Executor Executor Executor Executor Node Executor … vs vs vs vs…
- 14. @mszymani#Devoxx #TantusData #spark Sizing executors • Spark can benefit from running multiple tasks in the same JVM • Many cores leads to problems: HDFS I/O, GC • 1-4 CPUs should be good for start
- 15. @mszymani#Devoxx #TantusData #spark Sizing executors Executor heap Container Memory overhead spark.executor.memory spark.yarn.executor.memoryOverhead spark.shuffle.memoryFraction spark.storage.memoryFraction Executor heap Cache Shuffle User program
- 16. @mszymani#Devoxx #TantusData #spark Sizing executors • Keep in mind memory overhead • Keep some resources for OS • Determining memory consumption - cache an RDD • Consider dynamic resource allocation
- 17. @mszymani#Devoxx #TantusData #spark Shuffle zoom-in Task 1 Task 2 Task 3 Stage 1 Stage 2
- 18. @mszymani#Devoxx #TantusData #spark Task 1 Task 2 Task 3 Stage 1 Stage 2 Shuffle zoom-in
- 19. @mszymani#Devoxx #TantusData #spark Task 1 Task 2 Task 3 Stage 1 Stage 2 BOOM!BOOM! Shuffle zoom-in
- 20. @mszymani#Devoxx #TantusData #spark We are up & running • 2G block limit • Timeouts • GC overhead limit exceeded • OOM • ExecutorLostFailure
- 21. @mszymani#Devoxx #TantusData #spark We are up & running What to watch out for: • Failing tasks • GC heavy tasks • Shuffle read/write sizes • Long running tasks
- 22. @mszymani#Devoxx #TantusData #spark We are up & running • Too much data per task? • Is parallelism level large enough? • More memory for executor? def groupByKey(numPartitions: Int) def repartition(numPartitions: Int)
- 23. @mszymani#Devoxx #TantusData #spark Skewed data … … … … … … … … … … … … … … … … … … … … … … You want: NOT: FOO V1 FOO V2 FOO V3 FOO_1 V1 FOO_1 OUT1 SALT EXEC FOO OUTMERGE V4BAR FOO_2 V2 FOO_1 V3 BAR_1 V4 FOO_2 OUT2 BAR_1 OUT3 BAR OUT
- 24. @mszymani#Devoxx #TantusData #spark Locality Executor Node 1 HDFS Block 1 Executor Node 2 HDFS Block 2 Executor Node 3 HDFS Block 3 Driver sparkContext.textFile(“hdfs://…”)
- 25. @mszymani#Devoxx #TantusData #spark Locality • Increase number of executors • For small jobs better to leave as is • spark.locality.wait parameter
- 26. @mszymani#Devoxx #TantusData #spark Caching val rdd1=calculate1() rdd1. … saveAsTextFile(…) rdd1. … saveAsTextFile(…) Executed twice!
- 27. @mszymani#Devoxx #TantusData #spark val rdd1=calculate1() rdd1. … saveAsTextFile(…) rdd1. … saveAsTextFile(…) Executed twice! Caching
- 28. @mszymani#Devoxx #TantusData #spark Caching • Branch in execution plan is a candidate for caching • Spark UI shows to see how much memory an RDD is taking • You cannot control priority - it's LRU • Don't cache to disk if computation is cheap • Caching with RF - only when recreation is extremely costly • Checkpointing vs caching • Shuffle data is automatically persisted
- 29. @mszymani#Devoxx #TantusData #spark Join zoom-in Task 1 Task 2 Stage 1 Task 1 Task 2 Stage 2 Stage 3
- 30. @mszymani#Devoxx #TantusData #spark Optimize shuffle Use the same number of partition Change map to mapValues
- 32. @mszymani#Devoxx #TantusData #spark Broadcast Variable Executor RDD1_0 RDD2 RDD2 Join Executor RDD1_1 RDD2 Join Executor RDD1_2 RDD2 Join Executor RDD1_3 RDD2 Join Broadcast
- 33. @mszymani#Devoxx #TantusData #spark Optimize shuffle - recap • Control number of partitions • Use mapValues instead of map if you can • Broadcast variables • Filter before shuffle • Avoid groupByKey, use reduceByKey
- 34. @mszymani#Devoxx #TantusData #spark Debugging tools • Spark UI • HDFS monitoring • Aggregate your logs • Extra Java options - observe your GC • Run and test locally
- 35. @mszymani#Devoxx #TantusData #spark Challenge time! goo.gl/7eTtvH
- 36. @mszymani#Devoxx #TantusData #spark Use case UUID UUID UUID UUID UUID UUID goo.gl/7eTtvH
- 37. @mszymani#Devoxx #TantusData #spark General notes • Tests vs no tests? - Test your code! • You are probably not the only user of the cluster • What are you optimizing for? • Share the knowledge • Spark actually works :) goo.gl/7eTtvH
- 39. @mszymani#Devoxx #TantusData #spark Thank you! @mszymani marcin@tantusdata.com