Download as PDF, PPTX
![Grappe
Clic
Nav
AT
DataLake
Avis
Evts.
Supports
Comptes
Spark Job
DataMart
Contribution
Visiteur
Session
Recherche
Bandeau
[LR]
[FD]
[Clic]
Project Context](https://image.slidesharecdn.com/sparkmesosclusteroptimization-160429211934/85/Spark-Mesos-Cluster-Optimization-2-320.jpg)
![SHUFFLING OPTIMIZATION 1/2
(k1, v1, w1)
(k2, v2, w2)
(k3, v3, w3)
(k1, v4 w4)
(k1, v5, w5)
(k3,v6,w6)
(k2, v7, w7)
(k2 v8, w8)
(k3,v9,w9)
(k1, v1)
(k2, v2)
(k3,v3)
(k1, v4)
(k1, v5)
(k3,v6)
(k2, v7)
(k2 v8)
(k3,v9)
map map map
(k1, [v1, v4, v5]) (k3, [v3, v6, v9]) (k2, [v2, v7, v8])
groupByKey groupByKey groupByKey
(k1,v1+v4+v5) (k3,v3+v6+v9) (k2, v2+v7+v8])
Shuffling
Bonus : OOM](https://image.slidesharecdn.com/sparkmesosclusteroptimization-160429211934/85/Spark-Mesos-Cluster-Optimization-17-320.jpg)
Uploaded byebiznext
Spark / Mesos Cluster Optimization
Spark and Mesos cluster optimization was discussed. The key points were: 1. Spark concepts like stages, tasks, and partitions were explained to understand application behavior and optimization opportunities around shuffling. 2. Application optimization focused on reducing shuffling through techniques like partitioning, reducing object sizes, and optimizing closures. 3. Memory tuning in Spark involved configuring storage and shuffling fractions to control memory usage between user data and Spark's internal data. 4. When running Spark on Mesos, coarse-grained and fine-grained allocation modes were described along with solutions like using Mesos roles to control resource allocation and dynamic allocation in coarse-grained mode.
More Related Content
![[245] presto 내부구조 파헤치기](https://cdn.slidesharecdn.com/ss_thumbnails/245presto-150915054242-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
Similar to Spark / Mesos Cluster Optimization
![[DSC Europe 25] Mijat Kustudic - Building Financial Intelligence with AI Agen...](https://cdn.slidesharecdn.com/ss_thumbnails/38y2lb5lse6wstegtvas-3-mijat-kustudic-building-financial-intelligence-with-ai-agents-260114111931-1a4783ce-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Srba Markovic - From Pilot to Production: Overcoming AI Deplo...](https://cdn.slidesharecdn.com/ss_thumbnails/yjjmrtytmwbalxlba7px-4-srba-markovic-from-pilot-to-production-overcoming-ai-deployment-blockers-with-260114111931-4a892d44-thumbnail.jpg?width=640&height=640&fit=bounds)
Spark / Mesos Cluster Optimization
- 1. HAYSSAM SALEH Spark / Mesos Cluster Optimization Paris Spark MeetupApril 28th 2016 @Criteo
- 2.
- 3.
- 4.
- 5. OBJECTIVES — 100 Gb of data / day => 50 million requests — 40To / year —How we turned from a 4 hours job on : ¢ 6 nodes ¢ 8 cores & 32 Gb per node ¢ To a 20 minutes Job on : ¢ 4 nodes, 8 cores & 8 Gb per node
- 6. SUMMARY ¢ Spark concepts ¢ Spark UI offline ¢Application Optimization — Shuffling — Partitioning — Closures ¢ Parameters Optimization — Spark Shuffling — Mesos application distribution ¢ Elasticsearch Optimization — Google “elasticsearch performance tuning” -> blog.ebiznext.com
- 7. SPARK : LES CONCEPTS ¢Application — Main application ¢ Job — Roundtrip Driver -> Cluster ¢ Stage — Shuffle Boundary ¢ Task — Thread working on a single RDD partition ¢ Partition — RDD are split into partitions — Partition is unit of work for each task ¢ Executor — System process Application Job Stage Task 1 n n n 1 1 Driver code Spark Action Shuffle Boundary One task / partition RDD Partition
- 8.
- 9.
- 10.
- 11.
- 12.
- 13.
- 14.
- 15. WHY OPTIMIZE SHUFFLING Distance betweenData & CPU Duaration (scaled) Cache L1 1 seconde RAM 3 minutes Node to node communication 3 jours
- 16. TRANSFORMATIONS LEADING TOSHUFFLING ¢ repartition ¢ cogroup ¢ ...join ¢ ...ByKey ¢ distinct
- 17. SHUFFLING OPTIMIZATION 1/2 (k1, v1, w1) (k2, v2, w2) (k3, v3, w3) (k1, v4 w4) (k1, v5, w5) (k3,v6,w6) (k2, v7, w7) (k2 v8, w8) (k3,v9,w9) (k1, v1) (k2, v2) (k3,v3) (k1, v4) (k1, v5) (k3,v6) (k2, v7) (k2 v8) (k3,v9) mapmap map (k1, [v1, v4, v5]) (k3, [v3, v6, v9]) (k2, [v2, v7, v8]) groupByKey groupByKey groupByKey (k1,v1+v4+v5) (k3,v3+v6+v9) (k2, v2+v7+v8]) Shuffling Bonus : OOM
- 18. SHUFFLING OPTIMIZATION 2/2 (k1, v1, w1) (k2, v2, w2) (k3, v3, w3) (k1, v4 w4) (k1, v5, w5) (k3,v6,w6) (k2, v7, w7) (k2 v8, w8) (k3,v9,w9) (k1, v1) (k2, v2) (k3,v3) (k1, v4) (k1, v5) (k3,v6) (k2, v7) (k2 v8) (k3,v9) mapmap map (k1, v1+v45)) (k3,v3+v6+v9) (k2, v2+v78) reduceByKey (2/2) reduceByKey (2/2) reduceByKey (2/2) (k1, v1) (k2, v2) (k3,v3) (k1, v4+v5) (k3,v6) (k2, v7+v8) (k3,v9) reduceByKey (1/2) reduceByKey (1/2) reduceByKey (1/2) Less shuffling
- 19. OPTIMIZATIONS BROUGHT BYSPARK SQL L Weak typing L Limited Java Support L Schema inference not supported • requires scala.Product
- 20. LES DATASETS –EXPERIMENTAL - Scala Java J API similar to RDD, strongly typed
- 21.
- 22.
- 23.
- 24.
- 25. PARTITIONING ¢ Symptoms — Important number of short lived tasks ¢Solutions — Review your implementation — Define custom partitionner — Control the number of partitions with ¢ coalesce et repartition
- 26. CHOOSE THE RIGHTSERIALIZATION ALGORITHM
- 27. REDUCE OBJECT SIZE ¢Kryo Serialization — Reduce space – up to 10X — Perform gain – 2X to 3X 1. Require Spark to use Kryo serialization 2. Optimize class naming => reduce object size 3. Force all serialized classes to register Not applicable to the Dataset API that use Encoders instead
- 28.
- 29.
- 30. SPARK MEMORY MANAGEMENT–SPARK 1.6.X Shuffle fraction Memory reserved to Spark (300Mb) Memory assigned to user data 1 - spark.memory.fraction Memory assigned to spark data spark.memory.fraction = 0.75 spark.memory.storageFraction = 0.5 16Go 11.78Go 3.92Go
- 31. SPARK MEMORY MANAGEMENT– SPARK 1.6.X ¢ Storage Fraction — Host cached RDDs — Host broadcast variables — Data in this fraction are subject to eviction ¢ Shuffling Fraction — Hold Intermediate Data — Spill to disk when full — Data this fraction cannot be evicted by other threads Memory reserved to Spark (300Mb) Memory assigned to user data 1 - spark.memory.fraction Memory assigned to spark data spark.memory.fraction = 0.75 spark.memory.storageFraction = 0.5 16Go 11.78Go 3.92Go
- 32. SPARK MEMORY MANAGEMENT– SPARK 1.6.X ¢ Shuffle & Storage fractions may borrow memory from each other under certain conditions: — The shuffling fraction cannot extend beyond its defined size if storage fraction uses all its memory — Storage fraction cannot evict data from the shuffling fraction even if it expanded beyond its defined size. Memory reserved to Spark (300Mb) Memory assigned to user data 1 - spark.memory.fraction Memory assigned to spark data spark.memory.fraction = 0.75 spark.memory.storageFraction = 0.5 16Go 11.78Go 3.92Go
- 33.
- 34. HASH SHUFFLE MANAGER Executor Partition Partition Partition Partition Partition Task Task Task Numberof tasks = spark.executor.cores / spark.task.cpus(1) YARN & standalone only Local Filesystem spark.local.dir File 1 File 2 File … …
- 35. SORT SHUFFLE MANAGER Executor Partition Partition Partition Partition Partition Task Task Task Numberof tasks = spark.executor.cores / spark.task.cpus(1) YARN & standalone only Filesystem local spark.local.dir Single fileMap AppendOnly Sort & Spill spark.shuffle.spill = true id1 id2 id3 id.. sorted by reducer id
- 36. SHUFFLE MANAGER ¢ spark.shuffle.manager= hash —Perform better when the number of mapper/reducer is small (generation of M * R files) ¢ spark.shuffle.manager= sort — Perform better for an important number of mapper/reducer ¢ Best of both world — spark.shuffle.sort.bypassMergeThreshold ¢ spark.shuffle.manager= tungsten-sort ??? — Similar to sort with the following benefits : ¢ Off-Heap allocation ¢ Work directly on binary object serialization (much smaller than JVM objects) – aka memcopy J ¢ Use 8 bytes / record pour les sort => take advantage on L* cache
- 37.
- 38. COARSE GRAINED MODE JStatic resource allocation Spark Driver CoarseMesosSchedulerBackend Mesos Master Mesos Worker CoarseGrainedExecutorBackend Mesos Worker CoarseGrainedExecutorBackend Mesos Worker CoarseGrainedExecutorBackend Mesos Worker CoarseGrainedExecutorBackend Executor Task Task Task Task Task Task …
- 39. COARSE GRAINED MODE LOnly one executor / node L No control over the number of executors 8 cores 16g 8 cores 16g 8 cores 16g 8 cores 16g 16 cores 16g 16 cores 16g 32 cores / 64Go 32 cores / 32Go
- 40. FINE GRAINED MODE LOnly one executor / node L No guaranty of resource availability L No control over the number of executors Spark Driver MesosSchedulerBackend Mesos Master Mesos Worker CoarseGrainedExecutorBackend Mesos Worker CoarseGrainedExecutorBackend Mesos Worker CoarseGrainedExecutorBackend Mesos Worker MesosExecutorBackend Executor Task … Executor Task Executor Task Executor Task spark.mesos.mesosExecutor.cores (1)
- 41. SOLUTION 1 : USE MESOSROLES ¢ Limit the number of cores / node on each mesos worker L Must be configured for each Spark application • Assign a Mesos role to your Spark job 8 cores 16g 8 cores 16g 8 cores 16g 8 cores 16g
- 42. SOLUTION 2 : DYNAMIC ALLOCATION(COARSED GRAINED MODE ONLY) ¢ Principles — Dynamically add/remove Spark executors ¢ Requires a dedicated process to move shuffled data (spawned on each Mesos worker through Marathon)
- 43.