SlideShare a Scribd company logo
1 of 114
1Pivotal Confidential–Internal Use Only 1Pivotal Confidential–Internal Use Only
Spark Architecture
A.Grishchenko
About me
Enterprise Architect @ Pivotal
 7 years in data processing
 5 years with MPP
 4 years with Hadoop
 Spark contributor
 http://0x0fff.com
Outline
 Spark Motivation
 Spark Pillars
 Spark Architecture
 Spark Shuffle
 Spark DataFrame
Outline
 Spark Motivation
 Spark Pillars
 Spark Architecture
 Spark Shuffle
 Spark DataFrame
Spark Motivation
 Difficultly of programming directly in Hadoop MapReduce
Spark Motivation
 Difficultly of programming directly in Hadoop MapReduce
 Performance bottlenecks, or batch not fitting use cases
Spark Motivation
 Difficultly of programming directly in Hadoop MapReduce
 Performance bottlenecks, or batch not fitting use cases
 Better support iterative jobs typical for machine learning
Difficulty of Programming in MR
Word Count implementations
 Hadoop MR – 61 lines in Java
 Spark – 1 line in interactive shell
sc.textFile('...').flatMap(lambda x: x.split())
.map(lambda x: (x, 1)).reduceByKey(lambda x, y: x+y)
.saveAsTextFile('...')
VS
Performance Bottlenecks
How many times the data is put to the HDD during a single
MapReduce Job?
 One
 Two
 Three
 More
Performance Bottlenecks
How many times the data is put to the HDD during a single
MapReduce Job?
 One
 Two
 Three
 More
Performance Bottlenecks
Consider Hive as main SQL tool
Performance Bottlenecks
Consider Hive as main SQL tool
 Typical Hive query is translated to 3-5 MR jobs
Performance Bottlenecks
Consider Hive as main SQL tool
 Typical Hive query is translated to 3-5 MR jobs
 Each MR would scan put data to HDD 3+ times
Performance Bottlenecks
Consider Hive as main SQL tool
 Typical Hive query is translated to 3-5 MR jobs
 Each MR would scan put data to HDD 3+ times
 Each put to HDD – write followed by read
Performance Bottlenecks
Consider Hive as main SQL tool
 Typical Hive query is translated to 3-5 MR jobs
 Each MR would scan put data to HDD 3+ times
 Each put to HDD – write followed by read
 Sums up to 18-30 scans of data during a single
Hive query
Performance Bottlenecks
Spark offers you
 Lazy Computations
– Optimize the job before executing
Performance Bottlenecks
Spark offers you
 Lazy Computations
– Optimize the job before executing
 In-memory data caching
– Scan HDD only once, then scan your RAM
Performance Bottlenecks
Spark offers you
 Lazy Computations
– Optimize the job before executing
 In-memory data caching
– Scan HDD only once, then scan your RAM
 Efficient pipelining
– Avoids the data hitting the HDD by all means
Outline
 Spark Motivation
 Spark Pillars
 Spark Architecture
 Spark Shuffle
 Spark DataFrame
Spark Pillars
Two main abstractions of Spark
Spark Pillars
Two main abstractions of Spark
 RDD – Resilient Distributed Dataset
Spark Pillars
Two main abstractions of Spark
 RDD – Resilient Distributed Dataset
 DAG – Direct Acyclic Graph
RDD
 Simple view
– RDD is collection of data items split into partitions and
stored in memory on worker nodes of the cluster
RDD
 Simple view
– RDD is collection of data items split into partitions and
stored in memory on worker nodes of the cluster
 Complex view
– RDD is an interface for data transformation
RDD
 Simple view
– RDD is collection of data items split into partitions and
stored in memory on worker nodes of the cluster
 Complex view
– RDD is an interface for data transformation
– RDD refers to the data stored either in persisted store
(HDFS, Cassandra, HBase, etc.) or in cache (memory,
memory+disks, disk only, etc.) or in another RDD
RDD
 Complex view (cont’d)
– Partitions are recomputed on failure or cache eviction
RDD
 Complex view (cont’d)
– Partitions are recomputed on failure or cache eviction
– Metadata stored for interface
▪ Partitions – set of data splits associated with this RDD
RDD
 Complex view (cont’d)
– Partitions are recomputed on failure or cache eviction
– Metadata stored for interface
▪ Partitions – set of data splits associated with this RDD
▪ Dependencies – list of parent RDDs involved in computation
RDD
 Complex view (cont’d)
– Partitions are recomputed on failure or cache eviction
– Metadata stored for interface
▪ Partitions – set of data splits associated with this RDD
▪ Dependencies – list of parent RDDs involved in computation
▪ Compute – function to compute partition of the RDD given the
parent partitions from the Dependencies
RDD
 Complex view (cont’d)
– Partitions are recomputed on failure or cache eviction
– Metadata stored for interface
▪ Partitions – set of data splits associated with this RDD
▪ Dependencies – list of parent RDDs involved in computation
▪ Compute – function to compute partition of the RDD given the
parent partitions from the Dependencies
▪ Preferred Locations – where is the best place to put
computations on this partition (data locality)
RDD
 Complex view (cont’d)
– Partitions are recomputed on failure or cache eviction
– Metadata stored for interface
▪ Partitions – set of data splits associated with this RDD
▪ Dependencies – list of parent RDDs involved in computation
▪ Compute – function to compute partition of the RDD given the
parent partitions from the Dependencies
▪ Preferred Locations – where is the best place to put
computations on this partition (data locality)
▪ Partitioner – how the data is split into partitions
RDD
 RDD is the main and only tool for data
manipulation in Spark
 Two classes of operations
– Transformations
– Actions
RDD
Lazy computations model
Transformation cause only metadata change
DAG
Direct Acyclic Graph – sequence of computations
performed on data
DAG
Direct Acyclic Graph – sequence of computations
performed on data
 Node – RDD partition
DAG
Direct Acyclic Graph – sequence of computations
performed on data
 Node – RDD partition
 Edge – transformation on top of data
DAG
Direct Acyclic Graph – sequence of computations
performed on data
 Node – RDD partition
 Edge – transformation on top of data
 Acyclic – graph cannot return to the older partition
DAG
Direct Acyclic Graph – sequence of computations
performed on data
 Node – RDD partition
 Edge – transformation on top of data
 Acyclic – graph cannot return to the older partition
 Direct – transformation is an action that transitions
data partition state (from A to B)
DAG
WordCount example
DAG
WordCount example
HDFSInputSplits
HDFS
RDDPartitions
RDD RDD RDD RDD RDD
sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Outline
 Spark Motivation
 Spark Pillars
 Spark Architecture
 Spark Shuffle
 Spark DataFrames
Spark Cluster
Driver Node
…
Worker Node Worker Node Worker Node
Spark Cluster
Driver Node
Driver
…
Worker Node
…
Executor Executor
Worker Node
…
Executor Executor
Worker Node
…
Executor Executor
Spark Cluster
Driver Node
Driver
Spark Context
…
Worker Node
…
Executor
Cache
Executor
Cache
Worker Node
…
Executor
Cache
Executor
Cache
Worker Node
…
Executor
Cache
Executor
Cache
Spark Cluster
Driver Node
Driver
Spark Context
…
Worker Node
…
Executor
Cache
Task
…
Task
Task
Executor
Cache
Task
…
Task
Task
Worker Node
…
Executor
Cache
Task
…
Task
Task
Executor
Cache
Task
…
Task
Task
Worker Node
…
Executor
Cache
Task
…
Task
Task
Executor
Cache
Task
…
Task
Task
Spark Cluster
 Driver
– Entry point of the Spark Shell (Scala, Python, R)
Spark Cluster
 Driver
– Entry point of the Spark Shell (Scala, Python, R)
– The place where SparkContext is created
Spark Cluster
 Driver
– Entry point of the Spark Shell (Scala, Python, R)
– The place where SparkContext is created
– Translates RDD into the execution graph
Spark Cluster
 Driver
– Entry point of the Spark Shell (Scala, Python, R)
– The place where SparkContext is created
– Translates RDD into the execution graph
– Splits graph into stages
Spark Cluster
 Driver
– Entry point of the Spark Shell (Scala, Python, R)
– The place where SparkContext is created
– Translates RDD into the execution graph
– Splits graph into stages
– Schedules tasks and controls their execution
Spark Cluster
 Driver
– Entry point of the Spark Shell (Scala, Python, R)
– The place where SparkContext is created
– Translates RDD into the execution graph
– Splits graph into stages
– Schedules tasks and controls their execution
– Stores metadata about all the RDDs and their partitions
Spark Cluster
 Driver
– Entry point of the Spark Shell (Scala, Python, R)
– The place where SparkContext is created
– Translates RDD into the execution graph
– Splits graph into stages
– Schedules tasks and controls their execution
– Stores metadata about all the RDDs and their partitions
– Brings up Spark WebUI with job information
Spark Cluster
 Executor
– Stores the data in cache in JVM heap or on HDDs
Spark Cluster
 Executor
– Stores the data in cache in JVM heap or on HDDs
– Reads data from external sources
Spark Cluster
 Executor
– Stores the data in cache in JVM heap or on HDDs
– Reads data from external sources
– Writes data to external sources
Spark Cluster
 Executor
– Stores the data in cache in JVM heap or on HDDs
– Reads data from external sources
– Writes data to external sources
– Performs all the data processing
Executor Memory
Spark Cluster – Detailed
Spark Cluster – PySpark
Application Decomposition
 Application
– Single instance of SparkContext that stores some data
processing logic and can schedule series of jobs,
sequentially or in parallel (SparkContext is thread-safe)
Application Decomposition
 Application
– Single instance of SparkContext that stores some data
processing logic and can schedule series of jobs,
sequentially or in parallel (SparkContext is thread-safe)
 Job
– Complete set of transformations on RDD that finishes
with action or data saving, triggered by the driver
application
Application Decomposition
 Stage
– Set of transformations that can be pipelined and
executed by a single independent worker. Usually it is
app the transformations between “read”, “shuffle”,
“action”, “save”
Application Decomposition
 Stage
– Set of transformations that can be pipelined and
executed by a single independent worker. Usually it is
app the transformations between “read”, “shuffle”,
“action”, “save”
 Task
– Execution of the stage on a single data partition. Basic
unit of scheduling
WordCount ExampleHDFSInputSplits
HDFS
RDDPartitions
RDD RDD RDD RDD RDD
sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Stage 1
WordCount ExampleHDFSInputSplits
HDFS
RDDPartitions
RDD RDD RDD RDD RDD
sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Stage 2Stage 1
WordCount ExampleHDFSInputSplits
HDFS
RDDPartitions
RDD RDD RDD RDD RDD
sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Stage 2Stage 1
WordCount ExampleHDFSInputSplits
HDFS
RDDPartitions
RDD RDD RDD RDD RDD
sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Task 1
Task 2
Task 3
Task 4
Stage 2Stage 1
WordCount ExampleHDFSInputSplits
HDFS
RDDPartitions
RDD RDD RDD RDD RDD
sc.textFile(‘hdfs://…’) flatMap map reduceByKey foreach
Task 1
Task 2
Task 3
Task 4
Task 5
Task 6
Task 7
Task 8
Stage 2Stage 1
WordCount ExampleHDFSInputSplits
HDFS
pipeline
Task 1
Task 2
Task 3
Task 4
Task 5
Task 6
Task 7
Task 8
partition shuffle pipeline
Persistence in Spark
Persistence Level Description
MEMORY_ONLY Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in
memory, some partitions will not be cached and will be recomputed on the fly each
time they're needed. This is the default level.
MEMORY_AND_DISK Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in
memory, store the partitions that don't fit on disk, and read them from there when
they're needed.
MEMORY_ONLY_SER Store RDD as serialized Java objects (one byte array per partition). This is
generally more space-efficient than deserialized objects, especially when using a
fast serializer, but more CPU-intensive to read.
MEMORY_AND_DISK_SER Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk
instead of recomputing them on the fly each time they're needed.
DISK_ONLY Store the RDD partitions only on disk.
MEMORY_ONLY_2,
DISK_ONLY_2, etc.
Same as the levels above, but replicate each partition on two cluster nodes.
Persistence in Spark
 Spark considers memory as a cache with LRU
eviction rules
 If “Disk” is involved, data is evicted to disks
rdd = sc.parallelize(xrange(1000))
rdd.cache().count()
rdd.persist(StorageLevel.MEMORY_AND_DISK_SER).count()
rdd.unpersist()
Outline
 Spark Motivation
 Spark Pillars
 Spark Architecture
 Spark Shuffle
 Spark DataFrame
Shuffles in Spark
 Hash Shuffle – default prior to 1.2.0
Shuffles in Spark
 Hash Shuffle – default prior to 1.2.0
 Sort Shuffle – default now
Shuffles in Spark
 Hash Shuffle – default prior to 1.2.0
 Sort Shuffle – default now
 Tungsten Sort – new optimized one!
Hash Shuffle
Executor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Hash Shuffle
Executor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
“map” task
spark.executor.cores/
spark.task.cpus
…
Hash Shuffle
Local DirectoryExecutor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
“map” task
spark.executor.cores/
spark.task.cpus
…
Output File
Output File
Output File
…
Numberof
“Reducers”
spark.local.dir
Hash Shuffle
Local DirectoryExecutor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
“map” task
spark.executor.cores/
spark.task.cpus
…
Output File
Output File
Output File
…
Numberof
“Reducers”
Output File
Output File
Output File
…
Numberof
“Reducers”
…
spark.local.dir
Hash Shuffle
Numberof“map”tasks
executedbythisexecutor
Hash Shuffle with Consolidation
Executor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Hash Shuffle With Consolidation
Local DirectoryExecutor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
…
Numberof
“Reducers”
Output File
Output File
Output File
spark.local.dir
Hash Shuffle With Consolidationspark.executor.cores/
spark.task.cpus
Local DirectoryExecutor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
“map” task
…
…
…
Numberof
“Reducers”
spark.local.dir
Hash Shuffle With Consolidationspark.executor.cores/
spark.task.cpus
spark.executor.cores/
spark.task.cpus
…
Numberof
“Reducers”
Output File
Output File
Output File
Output File
Output File
Output File
Local DirectoryExecutor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
…
…
…
Numberof
“Reducers”
spark.local.dir
Hash Shuffle With Consolidationspark.executor.cores/
spark.task.cpus
spark.executor.cores/
spark.task.cpus
…
Numberof
“Reducers”
Output File
Output File
Output File
Output File
Output File
Output File
Local DirectoryExecutor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
“map” task
…
…
…
Numberof
“Reducers”
spark.local.dir
Hash Shuffle With Consolidationspark.executor.cores/
spark.task.cpus
spark.executor.cores/
spark.task.cpus
Output File
Output File
Output File
…
Numberof
“Reducers”
Output File
Output File
Output File
Local DirectoryExecutor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
…
…
…
Numberof
“Reducers”
spark.local.dir
Hash Shuffle With Consolidationspark.executor.cores/
spark.task.cpus
spark.executor.cores/
spark.task.cpus
Output File
Output File
Output File
…
Numberof
“Reducers”
Output File
Output File
Output File
Local DirectoryExecutor JVM
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map” task
“map” task
…
…
…
Numberof
“Reducers”
spark.local.dir
Hash Shuffle With Consolidationspark.executor.cores/
spark.task.cpus
spark.executor.cores/
spark.task.cpus
…
Numberof
“Reducers”
Output File
Output File
Output File
Output File
Output File
Output File
Sort Shuffle
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
Partition
Partition
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
Partition
Partition
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
AppendOnlyMap
…
AppendOnlyMap
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
AppendOnlyMap
…
AppendOnlyMap
sort &
spill
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
AppendOnlyMap
…
AppendOnlyMap
sort &
spill
Local
Directory
Output File
index
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
AppendOnlyMap
…
AppendOnlyMap
sort &
spill
sort &
spill
Local
Directory
Output File
index
Output File
index
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
AppendOnlyMap
…
AppendOnlyMap
sort &
spill
sort &
spill
sort &
spill
…
Local
Directory
…
Output File
index
Output File
index
Output File
index
spark.executor.cores/spark.task.cpus
MinHeap
Merge
MinHeap
Merge
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
AppendOnlyMap
…
AppendOnlyMap
sort &
spill
sort &
spill
sort &
spill
…
Local
Directory
…
“reduce”task“reduce”task
…Output File
index
Output File
index
Output File
index
spark.executor.cores/spark.task.cpus
Tungsten Sort Shuffle
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
Partition
Partition
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
Partition
Partition
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
Serialized Data
LinkedList<MemoryBlock>
Array of data pointers and
Partition IDs, long[]
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
Serialized Data
LinkedList<MemoryBlock>
Array of data pointers and
Partition IDs, long[]
sort &
spill
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
Serialized Data
LinkedList<MemoryBlock>
Local Directory
Array of data pointers and
Partition IDs, long[]
sort &
spill
spark.local.dir
Output File
partition
partition
partition
partition
index
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
Serialized Data
LinkedList<MemoryBlock>
Local Directory
Array of data pointers and
Partition IDs, long[]
sort &
spill
sort &
spill
…
spark.local.dir
Output File
partition
partition
partition
partition
index
Output File
partition
partition
partition
partition
index
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
Serialized Data
LinkedList<MemoryBlock>
Local Directory
Array of data pointers and
Partition IDs, long[]
sort &
spill
sort &
spill
…
spark.local.dir
Output File
partition
partition
partition
partition
index
Output File
partition
partition
partition
partition
index
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
Serialized Data
LinkedList<MemoryBlock>
…
Local Directory
Array of data pointers and
Partition IDs, long[]
Serialized Data
LinkedList<MemoryBlock>
Array of data pointers and
Partition IDs, long[]
sort &
spill
sort &
spill
…
spark.local.dir
Output File
partition
partition
partition
partition
index
Output File
partition
partition
partition
partition
index
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
Serialized Data
LinkedList<MemoryBlock>
…
Local Directory
…
Array of data pointers and
Partition IDs, long[]
Serialized Data
LinkedList<MemoryBlock>
Array of data pointers and
Partition IDs, long[]
sort &
spill
sort &
spill
…
sort &
spill
spark.local.dir
Output File
partition
partition
partition
partition
index
Output File
partition
partition
partition
partition
index
Output File
partition
partition
partition
partition
index
spark.executor.cores/spark.task.cpus
Executor JVMPartition
Partition
Partition
Partition
Partition
Partition
Partition
Partition
“map”
task
“map”
task
…
Tungsten Sort Shuffle
spark.storage.[safetyFraction * memoryFraction]
spark.shuffle.
[safetyFraction * memoryFraction]
Partition
Partition
Serialized Data
LinkedList<MemoryBlock>
…
Local Directory
…
Array of data pointers and
Partition IDs, long[]
Serialized Data
LinkedList<MemoryBlock>
Array of data pointers and
Partition IDs, long[]
sort &
spill
sort &
spill
…
sort &
spill
spark.local.dir
Output File
partition
partition
partition
partition
index
Output File
partition
partition
partition
partition
index
Output File
partition
partition
partition
partition
index
Output File
partition
partition
partition
partition
index
merge
spark.executor.cores/spark.task.cpus
Outline
 Spark Motivation
 Spark Pillars
 Spark Architecture
 Spark Shuffle
 Spark DataFrame
DataFrame Idea
DataFrame Implementation
 Interface
– DataFrame is an RDD with schema – field names, field
data types and statistics
– Unified transformation interface in all languages, all the
transformations are passed to JVM
– Can be accessed as RDD, in this case transformed to
the RDD of Row objects
DataFrame Implementation
 Internals
– Internally it is the same RDD
– Data is stored in row-columnar format, row chunk size
is set by spark.sql.inMemoryColumnarStorage.batchSize
– Each column in each partition stores min-max values
for partition pruning
– Allows better compression ratio than standard RDD
– Delivers faster performance for small subsets of
columns
113Pivotal Confidential–Internal Use Only 113Pivotal Confidential–Internal Use Only
Questions?
BUILT FOR THE SPEED OF BUSINESS

More Related Content

What's hot

Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcachedJurriaan Persyn
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache SparkRahul Jain
 
Processing Large Data with Apache Spark -- HasGeek
Processing Large Data with Apache Spark -- HasGeekProcessing Large Data with Apache Spark -- HasGeek
Processing Large Data with Apache Spark -- HasGeekVenkata Naga Ravi
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to RedisDvir Volk
 
From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.Taras Matyashovsky
 
Scalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsJonas Bonér
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Databricks
 
Apache Spark overview
Apache Spark overviewApache Spark overview
Apache Spark overviewDataArt
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Sparkdatamantra
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in SparkSpark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in SparkBo Yang
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guideRyan Blue
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiDatabricks
 
Introduction to apache spark
Introduction to apache spark Introduction to apache spark
Introduction to apache spark Aakashdata
 
Apache spark - Architecture , Overview & libraries
Apache spark - Architecture , Overview & librariesApache spark - Architecture , Overview & libraries
Apache spark - Architecture , Overview & librariesWalaa Hamdy Assy
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsCloudera, Inc.
 
Top 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark ApplicationsTop 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark ApplicationsSpark Summit
 
Introducing DataFrames in Spark for Large Scale Data Science
Introducing DataFrames in Spark for Large Scale Data ScienceIntroducing DataFrames in Spark for Large Scale Data Science
Introducing DataFrames in Spark for Large Scale Data ScienceDatabricks
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Databricks
 
Introduction to Spark with Python
Introduction to Spark with PythonIntroduction to Spark with Python
Introduction to Spark with PythonGokhan Atil
 
Spark shuffle introduction
Spark shuffle introductionSpark shuffle introduction
Spark shuffle introductioncolorant
 

What's hot (20)

Introduction to memcached
Introduction to memcachedIntroduction to memcached
Introduction to memcached
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Spark
 
Processing Large Data with Apache Spark -- HasGeek
Processing Large Data with Apache Spark -- HasGeekProcessing Large Data with Apache Spark -- HasGeek
Processing Large Data with Apache Spark -- HasGeek
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.From cache to in-memory data grid. Introduction to Hazelcast.
From cache to in-memory data grid. Introduction to Hazelcast.
 
Scalability, Availability & Stability Patterns
Scalability, Availability & Stability PatternsScalability, Availability & Stability Patterns
Scalability, Availability & Stability Patterns
 
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
Improving SparkSQL Performance by 30%: How We Optimize Parquet Pushdown and P...
 
Apache Spark overview
Apache Spark overviewApache Spark overview
Apache Spark overview
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Spark
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in SparkSpark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
 
A Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and HudiA Thorough Comparison of Delta Lake, Iceberg and Hudi
A Thorough Comparison of Delta Lake, Iceberg and Hudi
 
Introduction to apache spark
Introduction to apache spark Introduction to apache spark
Introduction to apache spark
 
Apache spark - Architecture , Overview & libraries
Apache spark - Architecture , Overview & librariesApache spark - Architecture , Overview & libraries
Apache spark - Architecture , Overview & libraries
 
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark ApplicationsTop 5 Mistakes to Avoid When Writing Apache Spark Applications
Top 5 Mistakes to Avoid When Writing Apache Spark Applications
 
Top 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark ApplicationsTop 5 Mistakes When Writing Spark Applications
Top 5 Mistakes When Writing Spark Applications
 
Introducing DataFrames in Spark for Large Scale Data Science
Introducing DataFrames in Spark for Large Scale Data ScienceIntroducing DataFrames in Spark for Large Scale Data Science
Introducing DataFrames in Spark for Large Scale Data Science
 
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
Deep Dive into Spark SQL with Advanced Performance Tuning with Xiao Li & Wenc...
 
Introduction to Spark with Python
Introduction to Spark with PythonIntroduction to Spark with Python
Introduction to Spark with Python
 
Spark shuffle introduction
Spark shuffle introductionSpark shuffle introduction
Spark shuffle introduction
 

Viewers also liked

Introduction to Spark Internals
Introduction to Spark InternalsIntroduction to Spark Internals
Introduction to Spark InternalsPietro Michiardi
 
Apache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and SmarterApache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and SmarterDatabricks
 
Introduction to Apache Spark Developer Training
Introduction to Apache Spark Developer TrainingIntroduction to Apache Spark Developer Training
Introduction to Apache Spark Developer TrainingCloudera, Inc.
 
Deep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache SparkDeep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache SparkDatabricks
 
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)Spark Summit
 
SQL to Hive Cheat Sheet
SQL to Hive Cheat SheetSQL to Hive Cheat Sheet
SQL to Hive Cheat SheetHortonworks
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...DataWorks Summit/Hadoop Summit
 
MapR Tutorial Series
MapR Tutorial SeriesMapR Tutorial Series
MapR Tutorial Seriesselvaraaju
 
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)Amazon Web Services
 
MapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR Technologies
 
Simplifying Big Data Analytics with Apache Spark
Simplifying Big Data Analytics with Apache SparkSimplifying Big Data Analytics with Apache Spark
Simplifying Big Data Analytics with Apache SparkDatabricks
 
MapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase APIMapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase APImcsrivas
 
Hands on MapR -- Viadea
Hands on MapR -- ViadeaHands on MapR -- Viadea
Hands on MapR -- Viadeaviadea
 
Architectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop DistributionArchitectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop Distributionmcsrivas
 
Hadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsHadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsLynn Langit
 

Viewers also liked (20)

Modern Data Architecture
Modern Data ArchitectureModern Data Architecture
Modern Data Architecture
 
Introduction to Spark Internals
Introduction to Spark InternalsIntroduction to Spark Internals
Introduction to Spark Internals
 
Apache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and SmarterApache Spark 2.0: Faster, Easier, and Smarter
Apache Spark 2.0: Faster, Easier, and Smarter
 
Apache HAWQ Architecture
Apache HAWQ ArchitectureApache HAWQ Architecture
Apache HAWQ Architecture
 
MPP vs Hadoop
MPP vs HadoopMPP vs Hadoop
MPP vs Hadoop
 
Introduction to Apache Spark Developer Training
Introduction to Apache Spark Developer TrainingIntroduction to Apache Spark Developer Training
Introduction to Apache Spark Developer Training
 
Deep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache SparkDeep Dive: Memory Management in Apache Spark
Deep Dive: Memory Management in Apache Spark
 
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)
 
SQL to Hive Cheat Sheet
SQL to Hive Cheat SheetSQL to Hive Cheat Sheet
SQL to Hive Cheat Sheet
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
 
MapR Tutorial Series
MapR Tutorial SeriesMapR Tutorial Series
MapR Tutorial Series
 
Apache Spark & Hadoop
Apache Spark & HadoopApache Spark & Hadoop
Apache Spark & Hadoop
 
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
AWS re:Invent 2016: Fraud Detection with Amazon Machine Learning on AWS (FIN301)
 
MapR and Cisco Make IT Better
MapR and Cisco Make IT BetterMapR and Cisco Make IT Better
MapR and Cisco Make IT Better
 
Simplifying Big Data Analytics with Apache Spark
Simplifying Big Data Analytics with Apache SparkSimplifying Big Data Analytics with Apache Spark
Simplifying Big Data Analytics with Apache Spark
 
Deep Learning for Fraud Detection
Deep Learning for Fraud DetectionDeep Learning for Fraud Detection
Deep Learning for Fraud Detection
 
MapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase APIMapR M7: Providing an enterprise quality Apache HBase API
MapR M7: Providing an enterprise quality Apache HBase API
 
Hands on MapR -- Viadea
Hands on MapR -- ViadeaHands on MapR -- Viadea
Hands on MapR -- Viadea
 
Architectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop DistributionArchitectural Overview of MapR's Apache Hadoop Distribution
Architectural Overview of MapR's Apache Hadoop Distribution
 
Hadoop MapReduce Fundamentals
Hadoop MapReduce FundamentalsHadoop MapReduce Fundamentals
Hadoop MapReduce Fundamentals
 

Similar to Apache Spark Architecture

Bigdata processing with Spark - part II
Bigdata processing with Spark - part IIBigdata processing with Spark - part II
Bigdata processing with Spark - part IIArjen de Vries
 
Geek Night - Functional Data Processing using Spark and Scala
Geek Night - Functional Data Processing using Spark and ScalaGeek Night - Functional Data Processing using Spark and Scala
Geek Night - Functional Data Processing using Spark and ScalaAtif Akhtar
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...Databricks
 
Apache Spark and DataStax Enablement
Apache Spark and DataStax EnablementApache Spark and DataStax Enablement
Apache Spark and DataStax EnablementVincent Poncet
 
Apache Spark Introduction.pdf
Apache Spark Introduction.pdfApache Spark Introduction.pdf
Apache Spark Introduction.pdfMaheshPandit16
 
A Deep Dive Into Spark
A Deep Dive Into SparkA Deep Dive Into Spark
A Deep Dive Into SparkAshish kumar
 
Big Data Processing using Apache Spark and Clojure
Big Data Processing using Apache Spark and ClojureBig Data Processing using Apache Spark and Clojure
Big Data Processing using Apache Spark and ClojureDr. Christian Betz
 
Apache Spark Fundamentals Meetup Talk
Apache Spark Fundamentals Meetup TalkApache Spark Fundamentals Meetup Talk
Apache Spark Fundamentals Meetup TalkEren Avşaroğulları
 
Paris Data Geek - Spark Streaming
Paris Data Geek - Spark Streaming Paris Data Geek - Spark Streaming
Paris Data Geek - Spark Streaming Djamel Zouaoui
 
Spark Summit East 2015 Advanced Devops Student Slides
Spark Summit East 2015 Advanced Devops Student SlidesSpark Summit East 2015 Advanced Devops Student Slides
Spark Summit East 2015 Advanced Devops Student SlidesDatabricks
 
Big data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle DatabaseBig data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle DatabaseMartin Toshev
 
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...Chetan Khatri
 
Intro to Spark
Intro to SparkIntro to Spark
Intro to SparkKyle Burke
 
A really really fast introduction to PySpark - lightning fast cluster computi...
A really really fast introduction to PySpark - lightning fast cluster computi...A really really fast introduction to PySpark - lightning fast cluster computi...
A really really fast introduction to PySpark - lightning fast cluster computi...Holden Karau
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache SparkVincent Poncet
 

Similar to Apache Spark Architecture (20)

Spark
SparkSpark
Spark
 
Bigdata processing with Spark - part II
Bigdata processing with Spark - part IIBigdata processing with Spark - part II
Bigdata processing with Spark - part II
 
Spark learning
Spark learningSpark learning
Spark learning
 
Geek Night - Functional Data Processing using Spark and Scala
Geek Night - Functional Data Processing using Spark and ScalaGeek Night - Functional Data Processing using Spark and Scala
Geek Night - Functional Data Processing using Spark and Scala
 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
 
Apache Spark and DataStax Enablement
Apache Spark and DataStax EnablementApache Spark and DataStax Enablement
Apache Spark and DataStax Enablement
 
Apache Spark Introduction.pdf
Apache Spark Introduction.pdfApache Spark Introduction.pdf
Apache Spark Introduction.pdf
 
A Deep Dive Into Spark
A Deep Dive Into SparkA Deep Dive Into Spark
A Deep Dive Into Spark
 
Big Data Processing using Apache Spark and Clojure
Big Data Processing using Apache Spark and ClojureBig Data Processing using Apache Spark and Clojure
Big Data Processing using Apache Spark and Clojure
 
Apache Spark Fundamentals Meetup Talk
Apache Spark Fundamentals Meetup TalkApache Spark Fundamentals Meetup Talk
Apache Spark Fundamentals Meetup Talk
 
Paris Data Geek - Spark Streaming
Paris Data Geek - Spark Streaming Paris Data Geek - Spark Streaming
Paris Data Geek - Spark Streaming
 
Spark Summit East 2015 Advanced Devops Student Slides
Spark Summit East 2015 Advanced Devops Student SlidesSpark Summit East 2015 Advanced Devops Student Slides
Spark Summit East 2015 Advanced Devops Student Slides
 
Big data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle DatabaseBig data processing with Apache Spark and Oracle Database
Big data processing with Apache Spark and Oracle Database
 
Intro to Apache Spark
Intro to Apache SparkIntro to Apache Spark
Intro to Apache Spark
 
Intro to Apache Spark
Intro to Apache SparkIntro to Apache Spark
Intro to Apache Spark
 
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
HKOSCon18 - Chetan Khatri - Scaling TB's of Data with Apache Spark and Scala ...
 
Spark SQL
Spark SQLSpark SQL
Spark SQL
 
Intro to Spark
Intro to SparkIntro to Spark
Intro to Spark
 
A really really fast introduction to PySpark - lightning fast cluster computi...
A really really fast introduction to PySpark - lightning fast cluster computi...A really really fast introduction to PySpark - lightning fast cluster computi...
A really really fast introduction to PySpark - lightning fast cluster computi...
 
Introduction to Apache Spark
Introduction to Apache SparkIntroduction to Apache Spark
Introduction to Apache Spark
 

Recently uploaded

Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfNicoChristianSunaryo
 
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...Timothy Spann
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaManalVerma4
 
Algorithms Insight 2024 version 1.0 pdfg
Algorithms Insight 2024 version 1.0 pdfgAlgorithms Insight 2024 version 1.0 pdfg
Algorithms Insight 2024 version 1.0 pdfgChristopherLopez729314
 
Unlocking Anticipatory Text Generation- A Constrained Approach for Large Lan...
Unlocking Anticipatory Text Generation-  A Constrained Approach for Large Lan...Unlocking Anticipatory Text Generation-  A Constrained Approach for Large Lan...
Unlocking Anticipatory Text Generation- A Constrained Approach for Large Lan...Ingeol Baek
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...Jack Cole
 
library-project-report library-project-report
library-project-report library-project-reportlibrary-project-report library-project-report
library-project-report library-project-reportmediacontrol2000
 
testingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdftestingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdfDSP Mutual Fund
 
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbaAdobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbas73678sri
 
Statistical Analysis and Hypothesis Tesing
Statistical Analysis and Hypothesis TesingStatistical Analysis and Hypothesis Tesing
Statistical Analysis and Hypothesis TesingManishaPatil932723
 
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j
 
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfRabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j
 
Data Discovery With Power Query in excel
Data Discovery With Power Query in excelData Discovery With Power Query in excel
Data Discovery With Power Query in excelKapilSidhpuria3
 
concept of soil quality & soil health.pptx
concept of soil quality & soil health.pptxconcept of soil quality & soil health.pptx
concept of soil quality & soil health.pptxpranavmishrafzd
 
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvwAdobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvws73678sri
 
Micromeritics unit 4 physical pharmaceutics sem.4 b.pharm
Micromeritics unit 4 physical pharmaceutics  sem.4 b.pharmMicromeritics unit 4 physical pharmaceutics  sem.4 b.pharm
Micromeritics unit 4 physical pharmaceutics sem.4 b.pharmDarshanpatel609504
 
INTRODUCTION TO BUSINESS ANALYTICS BA4206 ANNA UNIVERSITY
INTRODUCTION TO BUSINESS ANALYTICS BA4206 ANNA UNIVERSITYINTRODUCTION TO BUSINESS ANALYTICS BA4206 ANNA UNIVERSITY
INTRODUCTION TO BUSINESS ANALYTICS BA4206 ANNA UNIVERSITYFreelance
 
Introductio to Data Science and types of data
Introductio to Data Science and types of dataIntroductio to Data Science and types of data
Introductio to Data Science and types of dataManishaPatil932723
 
prediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachprediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachAdekunleJoseph4
 

Recently uploaded (20)

Digital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdfDigital Indonesia Report 2024 by We Are Social .pdf
Digital Indonesia Report 2024 by We Are Social .pdf
 
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...2024 XTREMEJ_  Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
2024 XTREMEJ_ Building Real-time Pipelines with FLaNK_ A Case Study with Tra...
 
IBEF report on the Insurance market in India
IBEF report on the Insurance market in IndiaIBEF report on the Insurance market in India
IBEF report on the Insurance market in India
 
Algorithms Insight 2024 version 1.0 pdfg
Algorithms Insight 2024 version 1.0 pdfgAlgorithms Insight 2024 version 1.0 pdfg
Algorithms Insight 2024 version 1.0 pdfg
 
Unlocking Anticipatory Text Generation- A Constrained Approach for Large Lan...
Unlocking Anticipatory Text Generation-  A Constrained Approach for Large Lan...Unlocking Anticipatory Text Generation-  A Constrained Approach for Large Lan...
Unlocking Anticipatory Text Generation- A Constrained Approach for Large Lan...
 
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
why-transparency-and-traceability-are-essential-for-sustainable-supply-chains...
 
library-project-report library-project-report
library-project-report library-project-reportlibrary-project-report library-project-report
library-project-report library-project-report
 
testingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdftestingsdadadadaaddadadadadadadadaad.pdf
testingsdadadadaaddadadadadadadadaad.pdf
 
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsbaAdobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
Adobe Scan 06-Mar-2024 (1).pdfwvsbbsbsba
 
Statistical Analysis and Hypothesis Tesing
Statistical Analysis and Hypothesis TesingStatistical Analysis and Hypothesis Tesing
Statistical Analysis and Hypothesis Tesing
 
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdfNeo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
Neo4j_Exploring the Impact of Graph Technology on Financial Services.pdf
 
2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use2023 Survey Shows Dip in High School E-Cigarette Use
2023 Survey Shows Dip in High School E-Cigarette Use
 
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdfRabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
Rabobank_Exploring the Impact of Graph Technology on Financial Services.pdf
 
Data Discovery With Power Query in excel
Data Discovery With Power Query in excelData Discovery With Power Query in excel
Data Discovery With Power Query in excel
 
concept of soil quality & soil health.pptx
concept of soil quality & soil health.pptxconcept of soil quality & soil health.pptx
concept of soil quality & soil health.pptx
 
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvwAdobe Scan 06-Mar-2024 (1).pdf shavashwvw
Adobe Scan 06-Mar-2024 (1).pdf shavashwvw
 
Micromeritics unit 4 physical pharmaceutics sem.4 b.pharm
Micromeritics unit 4 physical pharmaceutics  sem.4 b.pharmMicromeritics unit 4 physical pharmaceutics  sem.4 b.pharm
Micromeritics unit 4 physical pharmaceutics sem.4 b.pharm
 
INTRODUCTION TO BUSINESS ANALYTICS BA4206 ANNA UNIVERSITY
INTRODUCTION TO BUSINESS ANALYTICS BA4206 ANNA UNIVERSITYINTRODUCTION TO BUSINESS ANALYTICS BA4206 ANNA UNIVERSITY
INTRODUCTION TO BUSINESS ANALYTICS BA4206 ANNA UNIVERSITY
 
Introductio to Data Science and types of data
Introductio to Data Science and types of dataIntroductio to Data Science and types of data
Introductio to Data Science and types of data
 
prediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approachprediction of default payment next month using a logistic approach
prediction of default payment next month using a logistic approach
 

Apache Spark Architecture

Editor's Notes

  1. def printfunc (x): print 'Word "%s" occurs %d times' % (x[0], x[1]) infile = sc.textFile('hdfs://sparkdemo:8020/sparkdemo/textfiles/README.md', 4) rdd1 = infile.flatMap(lambda x: x.split()) rdd2 = rdd1.map(lambda x: (x, 1)).reduceByKey(lambda x, y: x+y) print rdd2.toDebugString() rdd2.foreach(printfunc)