Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

of

Apache Spark Core – Practical Optimization Slide 1 Apache Spark Core – Practical Optimization Slide 2 Apache Spark Core – Practical Optimization Slide 3 Apache Spark Core – Practical Optimization Slide 4 Apache Spark Core – Practical Optimization Slide 5 Apache Spark Core – Practical Optimization Slide 6 Apache Spark Core – Practical Optimization Slide 7 Apache Spark Core – Practical Optimization Slide 8 Apache Spark Core – Practical Optimization Slide 9 Apache Spark Core – Practical Optimization Slide 10 Apache Spark Core – Practical Optimization Slide 11 Apache Spark Core – Practical Optimization Slide 12 Apache Spark Core – Practical Optimization Slide 13 Apache Spark Core – Practical Optimization Slide 14 Apache Spark Core – Practical Optimization Slide 15 Apache Spark Core – Practical Optimization Slide 16 Apache Spark Core – Practical Optimization Slide 17 Apache Spark Core – Practical Optimization Slide 18 Apache Spark Core – Practical Optimization Slide 19 Apache Spark Core – Practical Optimization Slide 20 Apache Spark Core – Practical Optimization Slide 21 Apache Spark Core – Practical Optimization Slide 22 Apache Spark Core – Practical Optimization Slide 23 Apache Spark Core – Practical Optimization Slide 24 Apache Spark Core – Practical Optimization Slide 25 Apache Spark Core – Practical Optimization Slide 26 Apache Spark Core – Practical Optimization Slide 27 Apache Spark Core – Practical Optimization Slide 28 Apache Spark Core – Practical Optimization Slide 29 Apache Spark Core – Practical Optimization Slide 30 Apache Spark Core – Practical Optimization Slide 31 Apache Spark Core – Practical Optimization Slide 32 Apache Spark Core – Practical Optimization Slide 33 Apache Spark Core – Practical Optimization Slide 34 Apache Spark Core – Practical Optimization Slide 35 Apache Spark Core – Practical Optimization Slide 36 Apache Spark Core – Practical Optimization Slide 37 Apache Spark Core – Practical Optimization Slide 38 Apache Spark Core – Practical Optimization Slide 39 Apache Spark Core – Practical Optimization Slide 40
Upcoming SlideShare
What to Upload to SlideShare
Next
Download to read offline and view in fullscreen.

6 Likes

Share

Download to read offline

Apache Spark Core – Practical Optimization

Download to read offline

Properly shaping partitions and your jobs to enable powerful optimizations, eliminate skew and maximize cluster utilization. We will explore various Spark Partition shaping methods along with several optimization strategies including join optimizations, aggregate optimizations, salting and multi-dimensional parallelism.

Apache Spark Core – Practical Optimization

  1. 1. WIFI SSID:Spark+AISummit | Password: UnifiedDataAnalytics
  2. 2. Daniel Tomes, Databricks Spark Core – Proper Optimization #UnifiedAnalytics #SparkAISummit
  3. 3. TEMPORARY – NOTE FOR REVIEW I gave this talk at Summit US 2019. I will be adding supplemental slides for SparkUI deep dive and delta optimization depending on how many people in the crowd has seen this presentation on YT. If none, I will go with same presentation, if many, I will do the first half as review and second half with new info. New info to come soon. 3#UnifiedAnalytics #SparkAISummit
  4. 4. Talking Points • Spark Hierarchy • The Spark UI • Understanding Partitions • Common Opportunities For Optimization 4#UnifiedAnalytics #SparkAISummit
  5. 5. Spark Hierarchy 5#UnifiedAnalytics #SparkAISummit
  6. 6. Spark Hierarchy • Actions are eager – Made of transformations (lazy) • narrow • wide (requires shuffle) – Spawn jobs • Spawn Stages – Spawn Tasks » Do work & utilize hardware 6#UnifiedAnalytics #SparkAISummit
  7. 7. Minimize Data Movement Less Jobs Less Stages More Tasks More Ops/Task 7#UnifiedAnalytics #SparkAISummit
  8. 8. Navigating The Spark UI DEMO 8#UnifiedAnalytics #SparkAISummit
  9. 9. Get A Baseline • Is your action efficient? – Spills? • CPU Utilization – GANGLIA / YARN / Etc – Tails 9#UnifiedAnalytics #SparkAISummit Goal
  10. 10. Understand Your Hardware • Core Count & Speed • Memory Per Core (Working & Storage) • Local Disk Type, Count, Size, & Speed • Network Speed & Topology • Data Lake Properties (rate limits) • Cost / Core / Hour – Financial For Cloud – Opportunity for Shared & On Prem 10#UnifiedAnalytics #SparkAISummit
  11. 11. Minimize Data Scans (Lazy Load) • Data Skipping – HIVE Partitions – Bucketing • Only Experts – Nearly Impossible to Maintain – Databricks Delta Z-Ordering • What is It • How To Do It 11#UnifiedAnalytics #SparkAISummit
  12. 12. 12#UnifiedAnalytics #SparkAISummit Without Partition Filter With Partition Filter Shrink Partition Range Using a Filter on Partitioned Column
  13. 13. 13#UnifiedAnalytics #SparkAISummit Simple Extra Shuffle Partitions With Broadcast
  14. 14. Partitions – Definition Each of a number of portions into which some operating systems divide memory or storage 14#UnifiedAnalytics #SparkAISummit HIVE PARTITION == SPARK PARTITION
  15. 15. Spark Partitions – Types • Input – Controls • spark.default.parallelism (don’t use) • spark.sql.files.maxPartitionBytes (mutable) – assuming source has sufficient partitions • Shuffle – Control = partition count • spark.sql.shuffle.partitions • Output – Control = stage partition count split by max records per file • Coalesce(n) to shrink • Repartition(n) to increase and/or balance (shuffle) • df.write.option(“maxRecordsPerFile”, N) 15#UnifiedAnalytics #SparkAISummit
  16. 16. Partitions – Right Sizing – Shuffle – Master Equation • Largest Shuffle Stage – Target Size <= 200 MB/partition • Partition Count = Stage Input Data / Target Size – Solve for Partition Count EXAMPLE Shuffle Stage Input = 210GB x = 210000MB / 200MB = 1050 spark.conf.set(“spark.sql.shuffle.partitions”, 1050) BUT -> If cluster has 2000 cores spark.conf.set(“spark.sql.shuffle.partitions”, 2000) 16#UnifiedAnalytics #SparkAISummit
  17. 17. 17#UnifiedAnalytics #SparkAISummit Stage 21 -> Shuffle Fed By Stage 19 & 20 THUS Stage 21 Shuffle Input = 45.4g + 8.6g == 54g Default Shuffle Partition == 200 == 54000mb/200parts =~ 270mb/shuffle part Cluster Spec 96 cores @ 7.625g/core 3.8125g Working Mem 3.8125g Storage Mem
  18. 18. 18#UnifiedAnalytics #SparkAISummit Cluster Spec 96 cores @ 7.625g/core 3.8125g Working Mem 3.8125g Storage Mem 480 shuffle partitions – WHY? Target shuffle part size == 100m p = 54g / 100m == 540 540p / 96 cores == 5.625 96 * 5 == 480 If p == 540 another 60p have to be loaded and processed after first cycle is complete NO SPILL
  19. 19. 19#UnifiedAnalytics #SparkAISummit
  20. 20. Partitions – Right Sizing (input) • Use Spark Defaults (128MB) unless… – Source Structure is not optimal (upstream) – Remove Spills – Increase Parallelism – Heavily Nested/Repetitive Data – UDFs 20#UnifiedAnalytics #SparkAISummit Get DF Partitions df.rdd.partitions.size
  21. 21. 21#UnifiedAnalytics #SparkAISummit 14.7g/452part == 32.3mb/part spark.sql.files.maxPartitionBytes == 128MB Parquet Consideration: Source must have sufficient row blocks sc.hadoopConfiguration.setInt("parquet.block.size", 1024 * 1024 * 16) df.write… FIX THIS SLIDE
  22. 22. Partitions – Right Sizing (output) • Write Once -> Read Many – More Time to Write but Faster to Read • Perfect writes limit parallelism – Compactions (minor & major) Write Data Size = 14.7GB Desired File Size = 1500MB Max stage parallelism = 10 96 – 10 == 86 cores idle during write 22#UnifiedAnalytics #SparkAISummit
  23. 23. 23#UnifiedAnalytics #SparkAISummit Only 10 Cores Used All 96 Cores Used Average File Size == 1.5g Average File Size == 0.16g
  24. 24. Partitions – Why So Serious? • Avoid The Spill – If (Partition Size > Working Memory Size) Spill – If (Storage Memory Available) Spill to Memory – If (Storage Memory Exhausted) Spill to Disk – If (Local Disk Exhausted) Fail Job • Maximize Parallelism – Utilize All Cores – Provision only the cores you need 24#UnifiedAnalytics #SparkAISummit
  25. 25. Balance • Maximizing Resources Requires Balance – Task Duration – Partition Size • SKEW – When some partitions are significantly larger than most 25#UnifiedAnalytics #SparkAISummit Input Partitions Shuffle Partitions Output Files Spills GC Times Straggling Tasks
  26. 26. 26#UnifiedAnalytics #SparkAISummit 75th percentile ~ 2m recs max ~ 45m recs stragglers take > 22X longer IF no spillage With spillage, 100Xs longer
  27. 27. Skew Join Optimization • OSS Fix – Add Column to each side with random int between 0 and spark.sql.shuffle.partitions – 1 to both sides – Add join clause to include join on generated column above – Drop temp columns from result • Databricks Fix (Skew Join) val skewedKeys = List(”id1”, “id200”, ”id-99”) df.join( skewDF.hint(“tblA”, “skewKey”, skewedKeys), Seq(keyCol), “inner”) 27#UnifiedAnalytics #SparkAISummit
  28. 28. Minimize Data Scans (Persistence) • Persistence – Not Free • Repetition – SQL Plan 28#UnifiedAnalytics #SparkAISummit df.cache == df.persist(StorageLevel.MEMORY_AND_DISK) • Types – Default (MEMORY_AND_DISK) • Deserialized – Deserialized = Faster = Bigger – Serialized = Slower = Smaller – _2 = Safety = 2X bigger – MEMORY_ONLY – DISK_ONLY Don’t Forget To Cleanup! df.unpersist
  29. 29. 29#UnifiedAnalytics #SparkAISummit
  30. 30. 30#UnifiedAnalytics #SparkAISummit TPCDS Query 4
  31. 31. Minimize Data Scans (Delta Cache) CACHE SELECT column_name[, column_name, ...] FROM [db_name.]table_name [ WHERE boolean_expression ] 31#UnifiedAnalytics #SparkAISummit Why So Fast? HOW TO USE AWS - i3s – On By Default AZURE – Ls-series – On By Default spark.databricks.io.cache.enabled true spark.databricks.io.cache.maxDiskUsage 50g spark.databricks.io.cache.maxMetaDataCache 1g spark.databricks.io.cache.compression.enabled false Hot Data Auto Cached Super Fast Relieves Memory Pressure
  32. 32. Join Optimization • Default = SortMergeJoin • Broadcast Joins – Automatic If: (one side < spark.sql.autoBroadcastJoinThreshold) (default 10m) – Risks • Not Enough Driver Memory • DF > spark.driver.maxResultSize • DF > Single Executor Available Working Memory – Prod – Mitigate The Risks • Checker Functions 32#UnifiedAnalytics #SparkAISummit
  33. 33. Persistence Vs. Broadcast 33#UnifiedAnalytics #SparkAISummit Attempt to send compute to the data Data availability guaranteed -> each executor has entire dataset
  34. 34. 34#UnifiedAnalytics #SparkAISummit 126MB 270MB
  35. 35. Range Join Optimization • Range Joins Types – Point In Interval Range Join • Predicate specifies value in one relation that is between two values from the other relation – Interval Overlap Range Join • Predicate specifies an overlap of intervals between two values from each relation REFERENCE 35#UnifiedAnalytics #SparkAISummit
  36. 36. Omit Expensive Ops • Repartition – Use Coalesce or Shuffle Partition Count • Count – Do you really need it? • DistinctCount – use approxCountDistinct() • If distincts are required, put them in the right place – Use dropDuplicates – dropDuplicates BEFORE the join – dropDuplicates BEFORE the groupBy 36#UnifiedAnalytics #SparkAISummit
  37. 37. UDF Penalties • Traditional UDFs cannot use Tungsten – Use org.apache.spark.sql.functions – Use PandasUDFs • Utilizes Apache Arrow – Use SparkR UDFs 37#UnifiedAnalytics #SparkAISummit
  38. 38. Source Data Format • File Size • Splittable • Compressed • HIVE Partitioned 38#UnifiedAnalytics #SparkAISummit
  39. 39. QUESTIONS 39#UnifiedAnalytics #SparkAISummit
  40. 40. DON’T FORGET TO RATE AND REVIEW THE SESSIONS SEARCH SPARK + AI SUMMIT
  • prismalytics

    Jul. 7, 2021
  • TadasBI

    Jun. 21, 2021
  • MarcosColebrookSantamaria

    Nov. 20, 2019
  • KonstantinUseinov

    Nov. 17, 2019
  • kaidataLee

    Nov. 8, 2019
  • FeiWang79

    Nov. 5, 2019

Properly shaping partitions and your jobs to enable powerful optimizations, eliminate skew and maximize cluster utilization. We will explore various Spark Partition shaping methods along with several optimization strategies including join optimizations, aggregate optimizations, salting and multi-dimensional parallelism.

Views

Total views

1,745

On Slideshare

0

From embeds

0

Number of embeds

24

Actions

Downloads

184

Shares

0

Comments

0

Likes

6

×