Apache Spark Internals - Part 2
•
2 likes•451 views
The idea of this presentation is to understand more about Apache Spark internals. How it deals with resilience for each component, how Shard allocation works using RDD and how it abstract data partitioning and cluster distribution complexity.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Apache Spark Internals - Part 2
Similar to Apache Spark Internals - Part 2 (20)
More from Jéferson Machado
More from Jéferson Machado (20)
Recently uploaded
Recently uploaded (20)
Apache Spark Internals - Part 2
- 2. Resilience Worker Executor Task Task Worker Executor Task Task Worker Executor Task Task Driver Master (Active) Job Job
- 3. Resilience Driver Worker Executor Task Task Worker Executor Task Task Worker Executor Task Task Driver Master (Active) Job Job ./spark-submit --deploy-mode "cluster" --supervise
- 4. Resilience Driver Worker Executor Task Task Worker Executor Task Task Worker Executor Task Task Driver Master (Active) Job Job Driver runs in the worker
- 5. Resilience Driver Worker Executor Task Task Worker Executor Task Task Worker Executor Task Task Driver Master (Active) Job Job Driver is started in a new worker
- 6. Resilience Master Master (Active) Job Job Zookeeper Master (Standby) Job Job Worker Executor Task Task Worker Executor Task Task Worker Executor Task Task Driver
- 7. Master (Active) Resilience Master Zookeeper Master (Standby) Job Job Job Job Driver Worker Executor Task Task Worker Executor Task Task Worker Executor Task Task
- 8. Master (Active) Resilience Worker Zookeeper Master (Standby) Job Job Job Job Driver Driver and Executor are also killed Worker Executor Task Task Worker Executor Task Task Worker Executor Task Task
- 9. Master (Active) Resilience Worker Zookeeper Master (Standby) Job Job Job Job Driver Worker is relaunched Driver and executor are also relaunched Worker Executor Task Task Worker Executor Task Task Worker Executor Task Task
- 10. Resilience RDD ● An RDD is an immutable, deterministically re-computable, distributed dataset. ● Each RDD remembers the lineage of deterministic operations that were used on a fault-tolerant input dataset to create it. ● If any partition of an RDD is lost due to a worker node failure, then that partition can be re-computed from the original fault-tolerant dataset using the lineage of operations. ● Assuming that all of the RDD transformations are deterministic, the data in the final transformed RDD will always be the same irrespective of failures in the Spark cluster.
- 11. cache logLinesRDD cleanedRDD collect() errosRDD Error, ts, msg1, ts, msg3, ts Error, ts, msg4, ts, msg1 Error, ts, msg1, ts Error, ts, ts, msg1 filter(fx) errorMsg1RDD count() saveToCassandra() Resilience RDD filter(fx) coalesce(2) If partition is damaged, it can recompute from his parent, if parents aren't in memory anymore, it'll reprocess from disk
- 12. RDD Shard allocation RDD - Resilient Distributed Dataset Error, ts, msg1, warn, ts, msg2, Error info, ts, msg8, info, ts, msg3, info Error, ts, msg5, ts, info Error, ts, info, msg9, ts, info, Error File (hdfs, s3, etc) partitions Default Algorithm: Hash partition RDD = Data abstraction It hides data partitioning and distribution complexity
- 13. Worker Executor Task Worker Executor Task Worker Executor TaskTask RDD Shard allocation RDD - Resilient Distributed Dataset Error, ts, msg1, warn, ts, msg2, Error info, ts, msg8, info, ts, msg3, info Error, ts, msg5, ts, info Error, ts, info, msg9, ts, info, Error File (hdfs, s3, etc) Default Algorithm: Hash partition partitions
- 14. Shard allocation Partition configuration - numbers of partition Specifying number of partition By default it create one partition for each processor core
- 15. Default settings: ● mapreduce.input.fileinputformat.split.minsize = 1 byte (minSize) ● dfs.block.size = 128 MB (cluster) / fs.local.block.size = 32 MB (local) (blockSize) Calculating goal size: e.g.: ● Total size of input files = T = 599 MB ● Desired number of partitions = P = 30 (parametrized) ● Partition Goal size = PGS = T / P = 599 / 30 = 19 MB Result: Math.max(1, Math.min(19, 32)) == 19 MB Shard allocation Partition configuration - defining partition size
- 16. Fewer partitions ● more data in each partition ● less network and disk i/o ● fast access to data ● increase memory pressure ● don't make use of parallelism More partitions ● increase parallelism processing ● less data in each partition ● more network and disk i/o Shard allocation Trade offs
- 17. Shard allocation Example - Cases - auxiliary function
- 18. Shard allocation Example - Case 1 Correctly distributed between 8 partitions
- 19. Shard allocation Example - Case 2 Inefficient use of resources - 8 cores, 4 idles
- 20. Shard allocation Example - Case 1 - explanation val = 2.000.000 / 8 = 250.000 Range partition: [0] -> 2 - 250.000 [1] -> 250.001 - 500.000 [2] -> 500.001 - 750.000 [3] -> 750.001 - 1.000.000 [4] -> 1.000.001 - 1.025.000 [5] -> 1.025.001 - 1.050,000 [6] -> 1.050.001 - 1.075.000 [7] -> 1.075.001 - 2.000.000
- 21. Shard allocation Example - Case 2 - explanation val = 2.000.000 map() turned into (key,value), where: Each value was a list of all integers we needed to multiply the key by to find the multiples up to 2 million. For half of them (all keys greater than 1 million) this meant that the value was an empty list E.g.: (2, Range(2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159,... ... (200013,Range(2, 3, 4, 5, 6, 7, 8, 9))
- 22. Shard allocation Example - Case 3 - fixing it using repartition Correctly distributed between 8 partitions Shuffle partitions