Alexey Grishchenko, profile picture
Uploaded byAlexey Grishchenko
83,051 views

Apache Spark Architecture

The document provides an overview of Apache Spark, focusing on its architectural features, motivations for use over Hadoop, and key concepts like Resilient Distributed Datasets (RDDs) and Directed Acyclic Graphs (DAGs). It emphasizes Spark's advantages, such as lazy computation, in-memory data caching, and efficient data processing metrics compared to traditional MapReduce. Additionally, the document discusses the structure of Spark clusters, drivers, executors, and applications, highlighting how these components work together to handle large-scale data operations.

Embed presentation

1 / 114
2 / 114
3 / 114
4 / 114
5 / 114
6 / 114
7 / 114
8 / 114
9 / 114
10 / 114
11 / 114
12 / 114
13 / 114
14 / 114
15 / 114
16 / 114
17 / 114
18 / 114
19 / 114
20 / 114
21 / 114
22 / 114
23 / 114
24 / 114
25 / 114
26 / 114
27 / 114
28 / 114
29 / 114
30 / 114
31 / 114
32 / 114
33 / 114
34 / 114
35 / 114
36 / 114
37 / 114
38 / 114
39 / 114
40 / 114
41 / 114
42 / 114
43 / 114
44 / 114
45 / 114
46 / 114
47 / 114
48 / 114
49 / 114
50 / 114
51 / 114
52 / 114
53 / 114
54 / 114
55 / 114
56 / 114
57 / 114
58 / 114
59 / 114
60 / 114
61 / 114
Most read
62 / 114
63 / 114
64 / 114
65 / 114
66 / 114
67 / 114
68 / 114
69 / 114
70 / 114
71 / 114
Most read
72 / 114
73 / 114
74 / 114
75 / 114
76 / 114
77 / 114
78 / 114
79 / 114
80 / 114
81 / 114
82 / 114
83 / 114
84 / 114
85 / 114
86 / 114
87 / 114
88 / 114
89 / 114
90 / 114
91 / 114
92 / 114
93 / 114
94 / 114
95 / 114
96 / 114
97 / 114
98 / 114
99 / 114
100 / 114
101 / 114
102 / 114
103 / 114
104 / 114
105 / 114
106 / 114
107 / 114
108 / 114
109 / 114
110 / 114
Most read
111 / 114
112 / 114
113 / 114
114 / 114
1Pivotal Confidential–Internal Use Only 1Pivotal Confidential–Internal Use Only Spark Architecture A.Grishchenko
About me Enterprise Architect @ Pivotal  7 years in data processing  5 years with MPP  4 years with Hadoop  Spark contributor  http://0x0fff.com
Outline  Spark Motivation  Spark Pillars  Spark Architecture  Spark Shuffle  Spark DataFrame
Outline  Spark Motivation  Spark Pillars  Spark Architecture  Spark Shuffle  Spark DataFrame
Spark Motivation  Difficultly of programming directly in Hadoop MapReduce
Spark Motivation  Difficultly of programming directly in Hadoop MapReduce  Performance bottlenecks, or batch not fitting use cases
Spark Motivation  Difficultly of programming directly in Hadoop MapReduce  Performance bottlenecks, or batch not fitting use cases  Better support iterative jobs typical for machine learning
Difficulty of Programming in MR Word Count implementations  Hadoop MR – 61 lines in Java  Spark – 1 line in interactive shell sc.textFile('...').flatMap(lambda x: x.split()) .map(lambda x: (x, 1)).reduceByKey(lambda x, y: x+y) .saveAsTextFile('...') VS
Performance Bottlenecks How many times the data is put to the HDD during a single MapReduce Job?  One  Two  Three  More
Performance Bottlenecks How many times the data is put to the HDD during a single MapReduce Job?  One  Two  Three  More
Performance Bottlenecks Consider Hive as main SQL tool
Performance Bottlenecks Consider Hive as main SQL tool  Typical Hive query is translated to 3-5 MR jobs
Performance Bottlenecks Consider Hive as main SQL tool  Typical Hive query is translated to 3-5 MR jobs  Each MR would scan put data to HDD 3+ times
Performance Bottlenecks Consider Hive as main SQL tool  Typical Hive query is translated to 3-5 MR jobs  Each MR would scan put data to HDD 3+ times  Each put to HDD – write followed by read
Performance Bottlenecks Consider Hive as main SQL tool  Typical Hive query is translated to 3-5 MR jobs  Each MR would scan put data to HDD 3+ times  Each put to HDD – write followed by read  Sums up to 18-30 scans of data during a single Hive query
Performance Bottlenecks Spark offers you  Lazy Computations – Optimize the job before executing
Performance Bottlenecks Spark offers you  Lazy Computations – Optimize the job before executing  In-memory data caching – Scan HDD only once, then scan your RAM
Performance Bottlenecks Spark offers you  Lazy Computations – Optimize the job before executing  In-memory data caching – Scan HDD only once, then scan your RAM  Efficient pipelining – Avoids the data hitting the HDD by all means
Outline  Spark Motivation  Spark Pillars  Spark Architecture  Spark Shuffle  Spark DataFrame
Spark Pillars Two main abstractions of Spark
Spark Pillars Two main abstractions of Spark  RDD – Resilient Distributed Dataset
Spark Pillars Two main abstractions of Spark  RDD – Resilient Distributed Dataset  DAG – Direct Acyclic Graph
RDD  Simple view – RDD is collection of data items split into partitions and stored in memory on worker nodes of the cluster
RDD  Simple view – RDD is collection of data items split into partitions and stored in memory on worker nodes of the cluster  Complex view – RDD is an interface for data transformation
RDD  Simple view – RDD is collection of data items split into partitions and stored in memory on worker nodes of the cluster  Complex view – RDD is an interface for data transformation – RDD refers to the data stored either in persisted store (HDFS, Cassandra, HBase, etc.) or in cache (memory, memory+disks, disk only, etc.) or in another RDD
RDD  Complex view (cont’d) – Partitions are recomputed on failure or cache eviction
RDD  Complex view (cont’d) – Partitions are recomputed on failure or cache eviction – Metadata stored for interface ▪ Partitions – set of data splits associated with this RDD
RDD  Complex view (cont’d) – Partitions are recomputed on failure or cache eviction – Metadata stored for interface ▪ Partitions – set of data splits associated with this RDD ▪ Dependencies – list of parent RDDs involved in computation
RDD  Complex view (cont’d) – Partitions are recomputed on failure or cache eviction – Metadata stored for interface ▪ Partitions – set of data splits associated with this RDD ▪ Dependencies – list of parent RDDs involved in computation ▪ Compute – function to compute partition of the RDD given the parent partitions from the Dependencies
RDD  Complex view (cont’d) – Partitions are recomputed on failure or cache eviction – Metadata stored for interface ▪ Partitions – set of data splits associated with this RDD ▪ Dependencies – list of parent RDDs involved in computation ▪ Compute – function to compute partition of the RDD given the parent partitions from the Dependencies ▪ Preferred Locations – where is the best place to put computations on this partition (data locality)
RDD  Complex view (cont’d) – Partitions are recomputed on failure or cache eviction – Metadata stored for interface ▪ Partitions – set of data splits associated with this RDD ▪ Dependencies – list of parent RDDs involved in computation ▪ Compute – function to compute partition of the RDD given the parent partitions from the Dependencies ▪ Preferred Locations – where is the best place to put computations on this partition (data locality) ▪ Partitioner – how the data is split into partitions
RDD  RDD is the main and only tool for data manipulation in Spark  Two classes of operations – Transformations – Actions
RDD Lazy computations model Transformation cause only metadata change
DAG Direct Acyclic Graph – sequence of computations performed on data
DAG Direct Acyclic Graph – sequence of computations performed on data  Node – RDD partition
DAG Direct Acyclic Graph – sequence of computations performed on data  Node – RDD partition  Edge – transformation on top of data
DAG Direct Acyclic Graph – sequence of computations performed on data  Node – RDD partition  Edge – transformation on top of data  Acyclic – graph cannot return to the older partition
DAG Direct Acyclic Graph – sequence of computations performed on data  Node – RDD partition  Edge – transformation on top of data  Acyclic – graph cannot return to the older partition  Direct – transformation is an action that transitions data partition state (from A to B)
DAG WordCount example
DAG WordCount example HDFSInputSplits HDFS RDDPartitions RDD RDD RDD RDD RDD sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Outline  Spark Motivation  Spark Pillars  Spark Architecture  Spark Shuffle  Spark DataFrames
Spark Cluster Driver Node … Worker Node Worker Node Worker Node
Spark Cluster Driver Node Driver … Worker Node … Executor Executor Worker Node … Executor Executor Worker Node … Executor Executor
Spark Cluster Driver Node Driver Spark Context … Worker Node … Executor Cache Executor Cache Worker Node … Executor Cache Executor Cache Worker Node … Executor Cache Executor Cache
Spark Cluster Driver Node Driver Spark Context … Worker Node … Executor Cache Task … Task Task Executor Cache Task … Task Task Worker Node … Executor Cache Task … Task Task Executor Cache Task … Task Task Worker Node … Executor Cache Task … Task Task Executor Cache Task … Task Task
Spark Cluster  Driver – Entry point of the Spark Shell (Scala, Python, R)
Spark Cluster  Driver – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext is created
Spark Cluster  Driver – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext is created – Translates RDD into the execution graph
Spark Cluster  Driver – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext is created – Translates RDD into the execution graph – Splits graph into stages
Spark Cluster  Driver – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext is created – Translates RDD into the execution graph – Splits graph into stages – Schedules tasks and controls their execution
Spark Cluster  Driver – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext is created – Translates RDD into the execution graph – Splits graph into stages – Schedules tasks and controls their execution – Stores metadata about all the RDDs and their partitions
Spark Cluster  Driver – Entry point of the Spark Shell (Scala, Python, R) – The place where SparkContext is created – Translates RDD into the execution graph – Splits graph into stages – Schedules tasks and controls their execution – Stores metadata about all the RDDs and their partitions – Brings up Spark WebUI with job information
Spark Cluster  Executor – Stores the data in cache in JVM heap or on HDDs
Spark Cluster  Executor – Stores the data in cache in JVM heap or on HDDs – Reads data from external sources
Spark Cluster  Executor – Stores the data in cache in JVM heap or on HDDs – Reads data from external sources – Writes data to external sources
Spark Cluster  Executor – Stores the data in cache in JVM heap or on HDDs – Reads data from external sources – Writes data to external sources – Performs all the data processing
Executor Memory
Spark Cluster – Detailed
Spark Cluster – PySpark
Application Decomposition  Application – Single instance of SparkContext that stores some data processing logic and can schedule series of jobs, sequentially or in parallel (SparkContext is thread-safe)
Application Decomposition  Application – Single instance of SparkContext that stores some data processing logic and can schedule series of jobs, sequentially or in parallel (SparkContext is thread-safe)  Job – Complete set of transformations on RDD that finishes with action or data saving, triggered by the driver application
Application Decomposition  Stage – Set of transformations that can be pipelined and executed by a single independent worker. Usually it is app the transformations between “read”, “shuffle”, “action”, “save”
Application Decomposition  Stage – Set of transformations that can be pipelined and executed by a single independent worker. Usually it is app the transformations between “read”, “shuffle”, “action”, “save”  Task – Execution of the stage on a single data partition. Basic unit of scheduling
WordCount ExampleHDFSInputSplits HDFS RDDPartitions RDD RDD RDD RDD RDD sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Stage 1 WordCount ExampleHDFSInputSplits HDFS RDDPartitions RDD RDD RDD RDD RDD sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Stage 2Stage 1 WordCount ExampleHDFSInputSplits HDFS RDDPartitions RDD RDD RDD RDD RDD sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Stage 2Stage 1 WordCount ExampleHDFSInputSplits HDFS RDDPartitions RDD RDD RDD RDD RDD sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach Task 1 Task 2 Task 3 Task 4
Stage 2Stage 1 WordCount ExampleHDFSInputSplits HDFS RDDPartitions RDD RDD RDD RDD RDD sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8
Stage 2Stage 1 WordCount ExampleHDFSInputSplits HDFS pipeline Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 partition shuffle pipeline
Persistence in Spark Persistence Level Description MEMORY_ONLY Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, some partitions will not be cached and will be recomputed on the fly each time they're needed. This is the default level. MEMORY_AND_DISK Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed. MEMORY_ONLY_SER Store RDD as serialized Java objects (one byte array per partition). This is generally more space-efficient than deserialized objects, especially when using a fast serializer, but more CPU-intensive to read. MEMORY_AND_DISK_SER Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed. DISK_ONLY Store the RDD partitions only on disk. MEMORY_ONLY_2, DISK_ONLY_2, etc. Same as the levels above, but replicate each partition on two cluster nodes.
Persistence in Spark  Spark considers memory as a cache with LRU eviction rules  If “Disk” is involved, data is evicted to disks rdd = sc.parallelize(xrange(1000)) rdd.cache().count() rdd.persist(StorageLevel.MEMORY_AND_DISK_SER).count() rdd.unpersist()
Outline  Spark Motivation  Spark Pillars  Spark Architecture  Spark Shuffle  Spark DataFrame
Shuffles in Spark  Hash Shuffle – default prior to 1.2.0
Shuffles in Spark  Hash Shuffle – default prior to 1.2.0  Sort Shuffle – default now
Shuffles in Spark  Hash Shuffle – default prior to 1.2.0  Sort Shuffle – default now  Tungsten Sort – new optimized one!
Hash Shuffle
Executor JVM Partition Partition Partition Partition Partition Partition Partition Partition Hash Shuffle
Executor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task spark.executor.cores/ spark.task.cpus … Hash Shuffle
Local DirectoryExecutor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task spark.executor.cores/ spark.task.cpus … Output File Output File Output File … Numberof “Reducers” spark.local.dir Hash Shuffle
Local DirectoryExecutor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task spark.executor.cores/ spark.task.cpus … Output File Output File Output File … Numberof “Reducers” Output File Output File Output File … Numberof “Reducers” … spark.local.dir Hash Shuffle Numberof“map”tasks executedbythisexecutor
Hash Shuffle with Consolidation
Executor JVM Partition Partition Partition Partition Partition Partition Partition Partition Hash Shuffle With Consolidation
Local DirectoryExecutor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task … Numberof “Reducers” Output File Output File Output File spark.local.dir Hash Shuffle With Consolidationspark.executor.cores/ spark.task.cpus
Local DirectoryExecutor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … … … Numberof “Reducers” spark.local.dir Hash Shuffle With Consolidationspark.executor.cores/ spark.task.cpus spark.executor.cores/ spark.task.cpus … Numberof “Reducers” Output File Output File Output File Output File Output File Output File
Local DirectoryExecutor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task … … … Numberof “Reducers” spark.local.dir Hash Shuffle With Consolidationspark.executor.cores/ spark.task.cpus spark.executor.cores/ spark.task.cpus … Numberof “Reducers” Output File Output File Output File Output File Output File Output File
Local DirectoryExecutor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … … … Numberof “Reducers” spark.local.dir Hash Shuffle With Consolidationspark.executor.cores/ spark.task.cpus spark.executor.cores/ spark.task.cpus Output File Output File Output File … Numberof “Reducers” Output File Output File Output File
Local DirectoryExecutor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task … … … Numberof “Reducers” spark.local.dir Hash Shuffle With Consolidationspark.executor.cores/ spark.task.cpus spark.executor.cores/ spark.task.cpus Output File Output File Output File … Numberof “Reducers” Output File Output File Output File
Local DirectoryExecutor JVM Partition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … … … Numberof “Reducers” spark.local.dir Hash Shuffle With Consolidationspark.executor.cores/ spark.task.cpus spark.executor.cores/ spark.task.cpus … Numberof “Reducers” Output File Output File Output File Output File Output File Output File
Sort Shuffle
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition Sort Shuffle spark.storage.[safetyFraction * memoryFraction] Partition Partition
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Sort Shuffle spark.storage.[safetyFraction * memoryFraction] Partition Partition spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition AppendOnlyMap … AppendOnlyMap spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition AppendOnlyMap … AppendOnlyMap sort & spill spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition AppendOnlyMap … AppendOnlyMap sort & spill Local Directory Output File index spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition AppendOnlyMap … AppendOnlyMap sort & spill sort & spill Local Directory Output File index Output File index spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition AppendOnlyMap … AppendOnlyMap sort & spill sort & spill sort & spill … Local Directory … Output File index Output File index Output File index spark.executor.cores/spark.task.cpus
MinHeap Merge MinHeap Merge Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition AppendOnlyMap … AppendOnlyMap sort & spill sort & spill sort & spill … Local Directory … “reduce”task“reduce”task …Output File index Output File index Output File index spark.executor.cores/spark.task.cpus
Tungsten Sort Shuffle
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] Partition Partition
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] Partition Partition spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition Serialized Data LinkedList<MemoryBlock> Array of data pointers and Partition IDs, long[] spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition Serialized Data LinkedList<MemoryBlock> Array of data pointers and Partition IDs, long[] sort & spill spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition Serialized Data LinkedList<MemoryBlock> Local Directory Array of data pointers and Partition IDs, long[] sort & spill spark.local.dir Output File partition partition partition partition index spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition Serialized Data LinkedList<MemoryBlock> Local Directory Array of data pointers and Partition IDs, long[] sort & spill sort & spill … spark.local.dir Output File partition partition partition partition index Output File partition partition partition partition index spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition Serialized Data LinkedList<MemoryBlock> Local Directory Array of data pointers and Partition IDs, long[] sort & spill sort & spill … spark.local.dir Output File partition partition partition partition index Output File partition partition partition partition index spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition Serialized Data LinkedList<MemoryBlock> … Local Directory Array of data pointers and Partition IDs, long[] Serialized Data LinkedList<MemoryBlock> Array of data pointers and Partition IDs, long[] sort & spill sort & spill … spark.local.dir Output File partition partition partition partition index Output File partition partition partition partition index spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition Serialized Data LinkedList<MemoryBlock> … Local Directory … Array of data pointers and Partition IDs, long[] Serialized Data LinkedList<MemoryBlock> Array of data pointers and Partition IDs, long[] sort & spill sort & spill … sort & spill spark.local.dir Output File partition partition partition partition index Output File partition partition partition partition index Output File partition partition partition partition index spark.executor.cores/spark.task.cpus
Executor JVMPartition Partition Partition Partition Partition Partition Partition Partition “map” task “map” task … Tungsten Sort Shuffle spark.storage.[safetyFraction * memoryFraction] spark.shuffle. [safetyFraction * memoryFraction] Partition Partition Serialized Data LinkedList<MemoryBlock> … Local Directory … Array of data pointers and Partition IDs, long[] Serialized Data LinkedList<MemoryBlock> Array of data pointers and Partition IDs, long[] sort & spill sort & spill … sort & spill spark.local.dir Output File partition partition partition partition index Output File partition partition partition partition index Output File partition partition partition partition index Output File partition partition partition partition index merge spark.executor.cores/spark.task.cpus
Outline  Spark Motivation  Spark Pillars  Spark Architecture  Spark Shuffle  Spark DataFrame
DataFrame Idea
DataFrame Implementation  Interface – DataFrame is an RDD with schema – field names, field data types and statistics – Unified transformation interface in all languages, all the transformations are passed to JVM – Can be accessed as RDD, in this case transformed to the RDD of Row objects
DataFrame Implementation  Internals – Internally it is the same RDD – Data is stored in row-columnar format, row chunk size is set by spark.sql.inMemoryColumnarStorage.batchSize – Each column in each partition stores min-max values for partition pruning – Allows better compression ratio than standard RDD – Delivers faster performance for small subsets of columns
113Pivotal Confidential–Internal Use Only 113Pivotal Confidential–Internal Use Only Questions?
BUILT FOR THE SPEED OF BUSINESS

More Related Content

PDF
Introduction to Spark Internals
byPietro Michiardi
 
PDF
Apache Spark Overview
byVadim Y. Bichutskiy
 
PPTX
Intro to Apache Spark
byRobert Sanders
 
PPTX
Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spar...
bySimplilearn
 
PDF
Apache Spark in Depth: Core Concepts, Architecture & Internals
byAnton Kirillov
 
PPTX
Apache Spark overview
byDataArt
 
PDF
Introduction to apache spark
byAakashdata
 
PDF
Intro to HBase
byalexbaranau
 
Introduction to Spark Internals
byPietro Michiardi
 
Apache Spark Overview
byVadim Y. Bichutskiy
 
Intro to Apache Spark
byRobert Sanders
 
Apache Spark Architecture | Apache Spark Architecture Explained | Apache Spar...
bySimplilearn
 
Apache Spark in Depth: Core Concepts, Architecture & Internals
byAnton Kirillov
 
Apache Spark overview
byDataArt
 
Introduction to apache spark
byAakashdata
 
Intro to HBase
byalexbaranau
 

What's hot

PDF
Introduction to Redis
byDvir Volk
 
PDF
Cassandra Introduction & Features
byDataStax Academy
 
PPTX
Introduction to Storm
byChandler Huang
 
KEY
Introduction to memcached
byJurriaan Persyn
 
PPTX
Spark introduction and architecture
bySohil Jain
 
PPTX
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
byBo Yang
 
PDF
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
bymumrah
 
PDF
Making Apache Spark Better with Delta Lake
byDatabricks
 
PPTX
Spark
byHeena Madan
 
PDF
Deep Dive: Memory Management in Apache Spark
byDatabricks
 
PPTX
Hive + Tez: A Performance Deep Dive
byDataWorks Summit
 
PPTX
Spark architecture
byGauravBiswas9
 
PDF
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
byDatabricks
 
PPTX
Introduction to Apache Spark
byRahul Jain
 
PPTX
From cache to in-memory data grid. Introduction to Hazelcast.
byTaras Matyashovsky
 
PDF
Simplifying Big Data Analytics with Apache Spark
byDatabricks
 
PDF
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
byNoritaka Sekiyama
 
PDF
Spark shuffle introduction
bycolorant
 
PDF
Spark SQL
byJoud Khattab
 
PDF
Apache Spark Introduction
bysudhakara st
 
Introduction to Redis
byDvir Volk
 
Cassandra Introduction & Features
byDataStax Academy
 
Introduction to Storm
byChandler Huang
 
Introduction to memcached
byJurriaan Persyn
 
Spark introduction and architecture
bySohil Jain
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
byBo Yang
 
Introduction and Overview of Apache Kafka, TriHUG July 23, 2013
bymumrah
 
Making Apache Spark Better with Delta Lake
byDatabricks
 
Spark
byHeena Madan
 
Deep Dive: Memory Management in Apache Spark
byDatabricks
 
Hive + Tez: A Performance Deep Dive
byDataWorks Summit
 
Spark architecture
byGauravBiswas9
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
byDatabricks
 
Introduction to Apache Spark
byRahul Jain
 
From cache to in-memory data grid. Introduction to Hazelcast.
byTaras Matyashovsky
 
Simplifying Big Data Analytics with Apache Spark
byDatabricks
 
Amazon S3 Best Practice and Tuning for Hadoop/Spark in the Cloud
byNoritaka Sekiyama
 
Spark shuffle introduction
bycolorant
 
Spark SQL
byJoud Khattab
 
Apache Spark Introduction
bysudhakara st
 

Viewers also liked

PPTX
Modern Data Architecture
byAlexey Grishchenko
 
PDF
Apache Spark 2.0: Faster, Easier, and Smarter
byDatabricks
 
PPTX
Apache HAWQ Architecture
byAlexey Grishchenko
 
PDF
MPP vs Hadoop
byAlexey Grishchenko
 
PPTX
Introduction to Apache Spark Developer Training
byCloudera, Inc.
 
PDF
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
bySpark Summit
 
PDF
SQL to Hive Cheat Sheet
byHortonworks
 
PPTX
How to understand and analyze Apache Hive query execution plan for performanc...
byDataWorks Summit/Hadoop Summit
 
PDF
MapR Tutorial Series
byselvaraaju
 
PDF
Apache Spark & Hadoop
byMapR Technologies
 
PPTX
MapR and Cisco Make IT Better
byMapR Technologies
 
PPTX
Deep Learning for Fraud Detection
byDataWorks Summit/Hadoop Summit
 
PDF
MapR M7: Providing an enterprise quality Apache HBase API
bymcsrivas
 
PDF
Hands on MapR -- Viadea
byviadea
 
PDF
Architectural Overview of MapR's Apache Hadoop Distribution
bymcsrivas
 
PPT
Hadoop MapReduce Fundamentals
byLynn Langit
 
PDF
MapR Data Analyst
byselvaraaju
 
Modern Data Architecture
byAlexey Grishchenko
 
Apache Spark 2.0: Faster, Easier, and Smarter
byDatabricks
 
Apache HAWQ Architecture
byAlexey Grishchenko
 
MPP vs Hadoop
byAlexey Grishchenko
 
Introduction to Apache Spark Developer Training
byCloudera, Inc.
 
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
bySpark Summit
 
SQL to Hive Cheat Sheet
byHortonworks
 
How to understand and analyze Apache Hive query execution plan for performanc...
byDataWorks Summit/Hadoop Summit
 
MapR Tutorial Series
byselvaraaju
 
Apache Spark & Hadoop
byMapR Technologies
 
MapR and Cisco Make IT Better
byMapR Technologies
 
Deep Learning for Fraud Detection
byDataWorks Summit/Hadoop Summit
 
MapR M7: Providing an enterprise quality Apache HBase API
bymcsrivas
 
Hands on MapR -- Viadea
byviadea
 
Architectural Overview of MapR's Apache Hadoop Distribution
bymcsrivas
 
Hadoop MapReduce Fundamentals
byLynn Langit
 
MapR Data Analyst
byselvaraaju
 

Similar to Apache Spark Architecture

PPTX
Ten tools for ten big data areas 03_Apache Spark
byWill Du
 
PPTX
4Introduction+to+Spark.pptx sdfsdfsdfsdfsdf
byyafora8192
 
PDF
Apache Spark Introduction.pdf
byMaheshPandit16
 
PPTX
Unit II Real Time Data Processing tools.pptx
byRahul Borate
 
PDF
Spark Driven Big Data Analytics
byinoshg
 
PPTX
Apache Spark Fundamentals
byZahra Eskandari
 
PPTX
Apache Spark on HDinsight Training
bySynergetics Learning and Cloud Consulting
 
PPTX
Apache Spark Core
byGirish Khanzode
 
PPTX
APACHE SPARK.pptx
byDeepaThirumurugan
 
PPTX
Introduction to Spark - Phoenix Meetup 08-19-2014
bycdmaxime
 
PPTX
Big data vahidamiri-tabriz-13960226-datastack.ir
bydatastack
 
PPTX
Apache Spark Introduction @ University College London
byVitthal Gogate
 
PPTX
Spark.pptx to knowledge gaining in wdm days ago
byPreethamMCPreethamMC
 
PPT
Scala and spark
byFabio Fumarola
 
PPTX
An Introduction to Apache Spark
byDona Mary Philip
 
PDF
Secrets of Spark's success - Deenar Toraskar, Think Reactive
byhuguk
 
PDF
Introduction to Spark Training
bySpark Summit
 
PDF
Spark Summit East 2015 Advanced Devops Student Slides
byDatabricks
 
PPTX
Apache Spark for Beginners
byAnirudh
 
PPTX
Intro to Spark development
bySpark Summit
 
Ten tools for ten big data areas 03_Apache Spark
byWill Du
 
4Introduction+to+Spark.pptx sdfsdfsdfsdfsdf
byyafora8192
 
Apache Spark Introduction.pdf
byMaheshPandit16
 
Unit II Real Time Data Processing tools.pptx
byRahul Borate
 
Spark Driven Big Data Analytics
byinoshg
 
Apache Spark Fundamentals
byZahra Eskandari
 
Apache Spark on HDinsight Training
bySynergetics Learning and Cloud Consulting
 
Apache Spark Core
byGirish Khanzode
 
APACHE SPARK.pptx
byDeepaThirumurugan
 
Introduction to Spark - Phoenix Meetup 08-19-2014
bycdmaxime
 
Big data vahidamiri-tabriz-13960226-datastack.ir
bydatastack
 
Apache Spark Introduction @ University College London
byVitthal Gogate
 
Spark.pptx to knowledge gaining in wdm days ago
byPreethamMCPreethamMC
 
Scala and spark
byFabio Fumarola
 
An Introduction to Apache Spark
byDona Mary Philip
 
Secrets of Spark's success - Deenar Toraskar, Think Reactive
byhuguk
 
Introduction to Spark Training
bySpark Summit
 
Spark Summit East 2015 Advanced Devops Student Slides
byDatabricks
 
Apache Spark for Beginners
byAnirudh
 
Intro to Spark development
bySpark Summit
 

Recently uploaded

PDF
Data Migration Best Practices: A Visual Guide for Secure & Accurate Data Tran...
byalexmartincaneda
 
PDF
Let's build machine learning CNN in RUST + bonus_ ML in water meter with ESP...
byMarcin Bielak
 
PPTX
gas chromatograohy by Rasha Ibrahim.pptx
byri7061584
 
PDF
Machine Learning Lectures -- Outliers in Data
byZahra Sadeghi
 
PDF
Trends and Predictions 2026_Reuters Institute
bydominikamizerska1
 
PDF
Doctoral of Philosophy Thesis Defense Slides
bysuchanadatta3
 
PPTX
06 Proposal TIMES-TALKS A proposal for an IEA-ETSAP 12 part podcast series
byIEA-ETSAP
 
PPTX
Update on TIMES Features Maintainer Report
byIEA-ETSAP
 
PPTX
[DSC Europe 25] Josip Saban - Career building for data professionals.pptx
byDataScienceConferenc1
 
PPTX
EMTECH-LESSON-3.pptxggggggggggggggggggggggg
byERICADEMESA
 
PDF
Operations and Data Support Portfolio | Delbert Ross Wanjala Investor Relatio...
byDelbertRossWanjala
 
PDF
CodeMate_Autonomous_Engineering_Documents.pdf
byssuserc287c9
 
PDF
Glasgow Reading Group Invited Talk - Causality-driven Adhoc IR.pdf
bysuchanadatta3
 
PPTX
National Art School Diploma
by897eemzixp
 
PPTX
Recommended Practices for system studies with high shares of wind and solar
byIEA-ETSAP
 
PDF
MedGard - OBGYN for SMLE Exam presentation
bydrpreehaaslam
 
PDF
Solution Sheet / Objectives management (2026)
byOptymyze
 
PPTX
[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...
byDataScienceConferenc1
 
PPTX
Can AI see like a tourist? An exploration of Vision Language Models for autom...
byMODUL Technology GmbH
 
PDF
Powering_AI-Ready_MySQL_MyVector_ProxySQL_v3.pptx.pdf
byAlkin Tezuysal
 
Data Migration Best Practices: A Visual Guide for Secure & Accurate Data Tran...
byalexmartincaneda
 
Let's build machine learning CNN in RUST + bonus_ ML in water meter with ESP...
byMarcin Bielak
 
gas chromatograohy by Rasha Ibrahim.pptx
byri7061584
 
Machine Learning Lectures -- Outliers in Data
byZahra Sadeghi
 
Trends and Predictions 2026_Reuters Institute
bydominikamizerska1
 
Doctoral of Philosophy Thesis Defense Slides
bysuchanadatta3
 
06 Proposal TIMES-TALKS A proposal for an IEA-ETSAP 12 part podcast series
byIEA-ETSAP
 
Update on TIMES Features Maintainer Report
byIEA-ETSAP
 
[DSC Europe 25] Josip Saban - Career building for data professionals.pptx
byDataScienceConferenc1
 
EMTECH-LESSON-3.pptxggggggggggggggggggggggg
byERICADEMESA
 
Operations and Data Support Portfolio | Delbert Ross Wanjala Investor Relatio...
byDelbertRossWanjala
 
CodeMate_Autonomous_Engineering_Documents.pdf
byssuserc287c9
 
Glasgow Reading Group Invited Talk - Causality-driven Adhoc IR.pdf
bysuchanadatta3
 
National Art School Diploma
by897eemzixp
 
Recommended Practices for system studies with high shares of wind and solar
byIEA-ETSAP
 
MedGard - OBGYN for SMLE Exam presentation
bydrpreehaaslam
 
Solution Sheet / Objectives management (2026)
byOptymyze
 
[DSC Europe 25] Milos Belcevic - Product Professional's Journey to Full-Stack...
byDataScienceConferenc1
 
Can AI see like a tourist? An exploration of Vision Language Models for autom...
byMODUL Technology GmbH
 
Powering_AI-Ready_MySQL_MyVector_ProxySQL_v3.pptx.pdf
byAlkin Tezuysal
 

Apache Spark Architecture

Editor's Notes

  • #40 def printfunc (x): print 'Word "%s" occurs %d times' % (x[0], x[1]) infile = sc.textFile('hdfs://sparkdemo:8020/sparkdemo/textfiles/README.md', 4) rdd1 = infile.flatMap(lambda x: x.split()) rdd2 = rdd1.map(lambda x: (x, 1)).reduceByKey(lambda x, y: x+y) print rdd2.toDebugString() rdd2.foreach(printfunc)