Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

© 2015 IBM Corporation
Apache Hadoop Day 2015
Intro to Apache Spark
LIGHTENING FAST CLUSTER COMPUTING

Mapreduce Limitations
• Lots of boilerplate , makes it
complex to program in MR.
• Disk based approach not good for
iterative usecases.
• Batch processing not fit for real
time.
In short no single solution, people build
specialized systems as workarounds.

Spark Goal
Batch
Interactiv
e
Streamin
g
Single
Framework!
Support batch, streaming, and interactive computations…
… in a unified framework
Easy to develop sophisticated algorithms (e.g., graph, ML algos)

Spark Core
Supports Scala,Java,Python,R
Spark Core
Supports Scala,Java,Python,R
Spark SQL
Interactive
Spark SQL
Interactive
Spark
Streaming
realtime
Spark
Streaming
realtime
Mlib/Spark.ml
Machine learning
Mlib/Spark.ml
Machine learning
GraphX
Graph processing
GraphX
Graph processing
Spark Stack
Unified engine across diverse workload and
environments

Data processing landscape
GraphLa
b
Girap
h
…
Graph
Grap
h
Graph
Pre
gel
Googl
e
Apache
Dato

Dreme
l
GraphLa
b
Girap
h
Drill
Impala
…
SQ
L
Graph
Grap
h
SQ
L
Graph
Pre
gel
Googl
e
Google
SQL
Apach
e
Apache
Cloudera
Dato

Dreme
l
GraphLa
b
Girap
h
Drill
Impala
…
SQ
L
Graph
Grap
h
SQ
L
Graph
Pre
gel
Googl
e
Google
SQL
Apach
e
Apache
DAG
Tez
Apache
Cloudera
Stream
Stor
m
Apache
Dato

Dr
e
aphLa
b
Girap
h
Graph
Grap
h
G
Pregel
Go Apache
DAG
ez
Apache
raph
ogle SQL
Drill
Apache
SQL
mel T
Google
SQL
Impala
Cloudera
Storm
Stream … Gr
Apache
Dato
Stop surarmement now !

Spark

Unifies batch, streaming, interactive comp.

Easy to build sophisticated applications

Support iterative, graph-parallel algorithms

Powerful APIs in Scala, Python, Java
Spark
Spark
Streaming Shark SQL
BlinkDB
GraphX MLlib
Streami
ng
Batch,
Interactiv
e
Batch,
Interactive
Interacti
ve
Data-parallel,
Iterative
Sophisticated
algos.

MapReduce Vs Spark

Mapreduce run each task in its own process, when
tasks completed the process dies
(MultithreadedMapper)

In Spark by default many tasks are concurrently run
in multi-threads on a single executor.

MR executor short lived and runs one large task

Spark executor is long live and runs many small
tasks

Process creation Vs Thread creation cost.

Problems in Spark

Applications cannot share data(mostly RDDs in
Spark Context) without writing to external
Storage.

Resource allocation inefficiency
[spark.dynamicAllocation.enabled].

Not Exactly designed for interactive applications.

Spark Internals -RDD
• RDD (Resilient Distributed Dataset)
• Lazy & Immutable
• Iterative operations before RDD
• Fault Tolerant
• Traditional way for achieving Fault Tolerance
• How does RDD achieve Fault Tolerance
• Partition

Spark Internals – RDDs
sc.textFile(“hdfs://<input>”)
.filter(_.startsWith(“ERROR”))
.map(_.split(“ “)(1))
.saveAsTextFile(“hdfs://<output>”)
Stage-1
HDFS HDFSHadoopRDD FilteredRDD MappedRDD

Spark Internals – RDDS
Narrow Vs Wide
Dependency
•Narrow dependency –
Each partition of parent is
Used by at max one
partition of child
•Wide dependency –
multiple child partition may
depend on one parent.

Narrow/Shuffle Dependency – class diagram

Task
Scheduler
Task Thread
Block
Manager
Spark Internal – Job Scheduling
RDD Object DAG Scheduler Task
Scheduler
Executor
Split DAG into Stages
and Tasks
Submit each Stage as
ready
Launches
individual
tasks
Execute tasks
Stores & serves
blocks
Rdd1.join(rdd2)
.groupBy(…)
.filter(…)

Resource Allocation
• Dynamic Resource Allocation.
• Resource Allocation Policy.
 Request Policy
 Remove Policy

Request/Remove Policy
Request
• Pending tasks to be scheduled.
• Spark request executors in rounds.
• spark.dynamicAllocation.schedulerBacklogTim
eout &
spark.dynamicAllocation.sustainedSchedulerB
acklogTimeout
Remove
• Removes when its idle for more than
spark.dynamicAllocation.executorIdleTimeo
ut seconds

Graceful Decommission of Executors
• State before Dynamic Allocation
• With Dynamic Allocation
• Complexity increases with Shuffle
• External Shuffle Service
• State of Cached data either in disk or memory

Fair Scheduler
• What is Fair Scheduling?
• How to enable Fair Scheduler
val conf = new
SparkConf().setMaster(...).setAppName(...)
conf.set("spark.scheduler.mode", "FAIR")
val sc = new SparkContext(conf)
• Fair Scheduler Pools

RDD Deep Dive
• RDD Basics
• How to create
• RDD Operations
• Lineage
• Partitions
• Shuffle
• Type of RDDs
• Extending RDD
• Caching in RDD

RDD Basics
• RDD (Resilient Distributed Dataset)
• Distributed collection of Object
• Resilient - Ability to re-compute missing partitions
(node failure)
• Distributed – Split across multiple partitions
• Dataset - Can contain any type, Python/Java/Scala
Object or User defined Object
• Fundamental unit of data in spark

RDD Basics – How to create
Two ways

Loading external datasets
− Spark supports wide range of sources
− Access HDFS data through InputFormat & OutputFormat
of Hadoop.
− Supports custom Input/Output format

Parallelizing collection in driver program
val lineRDD = sc.textFile(“hdfs:///path/to/Readme.md”)
textFile(“/my/directory/*”) or textFile(“/my/directory/*.gz”)
SparkContext.wholeTextFiles returns (filename,content) pair
val listRDD = sc.parallelize(List(“spark”,”meetup”,”deepdive”))

RDD Operations

Two type of Operations

Transformation

Action

Transformations are lazy, nothing actually happens until an action is
called.

Action triggers the computation

Action returns values to driver or writes data to external storage.

Lazy Evaluation
−
Transformation on RDD, don’t get performed immediately
−
Spark Internally records metadata to track the operation
−
Loading data into RDD also gets lazy evaluated
−
Lazy evaluation reduce number of passes on the data by
grouping operations
−
MapReduce – Burden on developer to merge the operation,
complex map.
−
Failure in Persisting the RDD will re-compute complete lineage
every time.

RDD In Action
sc.textFile(“hdfs://file.txt")
.flatMap(line=>line.split(" "))
.map(word => (word,1))
.reduceByKey(_+_)
.collect()
I scream you
scream lets all
scream for
icecream!
I wish I were
what I was when
I wished I were
what I am.
I
scream
you
scream
lets
all
scream
for
icecream
(I,1)
(scream,1)
(you,1)
(scream,1)
(lets,1)
(all,1)
(scream,1)
(icecream,1)
(icecream,1)
(scream,3)
(you,1)
(lets,1)
(I,1)
(all,1)

Lineage Demo

RDD Partition

Partition Definition

Fragments of RDD

Fragmentation allows Spark to execute in Parallel.

Partitions are distributed across cluster(Spark worker)

Partitioning

Impacts parallelism

Impacts performance

Importance of partition Tuning

Too few partitions

Less concurrency, unused cores.

More susceptible to data skew

Increased memory pressure for groupBy, reduceByKey,
sortByKey, etc.

Too many partitions

Framework overhead (more scheduling latency than the time
needed for actual task.)

Many CPU context-switching

Need “reasonable number” of partitions

Commonly between 100 and 10,000 partitions

Lower bound: At least ~2x number of cores in cluster

Upper bound: Ensure tasks take at least 100ms

How Spark Partitions data

Input data partition

Shuffle transformations

Custom Partitioner

Partition - Input Data

Spark uses same class as Hadoop to perform Input/Output

sc.textFile(“hdfs://…”) invokes Hadoop TextInputFormat

Below are Knobs which defines #Partitions

dfs.block.size – default 128MB(Hadoop 2.0)

numPartition – can be used to increase number of partition
default is 0 which means 1 partition

mapreduce.input.fileinputformat.split.minsize – default 1kb

Partition Size = Max(minsize,Min(goalSize,blockSize)

goalSize = totalInputSize/numPartitions

32MB, 0, 1KB, 640MB total size - Defaults
−Max(1kb,Min(640MB,32MB) ) = 20 partitions

32MB, 30, 1KB , 640MB total size - Want more partition
−Max(1kb,Min(32MB,32MB)) = 32 partition

32MB, 5, 1KB = Max(1kb,Min(120MB,32MB)) = 20 – Bigger size
partition

32MB,0, 64MB = Max(64MB,Min(640MB,32MB)) = 10 Bigger size
partition

Partition - Shuffle transformations

All shuffle transformation provides parameter
for desire number of partition

Default Behavior - Spark Uses HashPartitioner.
− If spark.default.parallelism is set , takes that as # of
partitions
− If spark.default.parallelism is not set
largest upstream RDD ‘s number of partition
− Reduces chances of out of memory
1. groupByKey
2. reduceByKey
3. aggregateByKey
4. sortByKey
5. join
6. cogroup
7. cartesian
8. coalesce
9. repartition
10.repartitionAndSort
WithinPartitions
Shuffle Transformation

Partition - Repartitioning

RDD provides two operators

repartition(numPartitions)
− Can Increase/decrease number of partitions
− Internally does shuffle
− expensive due to shuffle
− For decreasing partition use coalesce

Coalesce(numPartition,Shuffle:[true/false])
− Decreases partitions
− Goes for narrow dependencies
− Avoids shuffle
− In case of drastic reduction may trigger shuffle

Custom Partitioner

Partition the data according to use case & data structure

Provides control over no of partitions, distribution of data

Extends Partitioner class, need to implement getPartitions &
numPartitons

Partitioning Demo

Shuffle - GroupByKey Vs ReduceByKey
val wordCountsWithGroup = rdd
.groupByKey()
.map(t => (t._1, t._2.sum)) .collect()

Shuffle - GroupByKey Vs ReduceByKey
val wordPairsRDD = rdd.map(word => (word, 1))
val wordCountsWithReduce = wordPairsRDD
.reduceByKey(_ + _)
.collect()

The Shuffle

Redistribution of data among partition between stages.

Most of the Performance, Reliability Scalability Issues in Spark occurs
within Shuffle.

Like MapReduce Spark shuffle uses Pull model.

Consistently evolved and still an area of research in Spark

Shuffle Overview
• Spark run job stage by stage.
• Stages are build up by DAGScheduler according to RDD’s
ShuffleDependency
• e.g. ShuffleRDD / CoGroupedRDD will have a
ShuffleDependency
• Many operator will create ShuffleRDD / CoGroupedRDD under
the hood.
• Repartition/CombineByKey/GroupBy/ReduceByKey/cogrou
p
• Many other operator will further call into the above
operators
•
e.g. various join operator will call CoGroup.
•

You have seen this
join
union
groupBy
Stage 3
Stage 1
Stage 2
A: B:
C: D:
map
E:
F:
G:

Shuffle is Expensive
• When doing shuffle, data no longer stay in memory only, gets
written to disk.
• For spark, shuffle process might involve
• Data partition: which might involve very expensive data
sorting works etc.
• Data ser/deser: to enable data been transfer through
network or across processes.
• Data compression: to reduce IO bandwidth etc.
• Disk IO: probably multiple times on one single data block
• E.g. Shuffle Spill, Merge combine

Shuffle History

Shuffle module in Spark has evolved over time.

Spark(0.6-0.7) – Same code path as RDD’s persist method.
MEMORY_ONLY , DISK_ONLY options available.

Spark (0.8-0.9)
-
Separate code for shuffle, ShuffleBlockManager &
BlockObjectWriter for shuffle only.
-
Shuffle optimization - Consolidate Shuffle Write.

Spark 1.0 – Introduced pluggable shuffle framework

Spark 1.1 – Sort based Shuffle Implementation

Spark 1.2 - Netty transfer Implementation. Sort based shuffle is
default now.

Spark 1.2+ - External shuffle service etc.

Understanding Shuffle

Input Aggregation

Types of Shuffle

Hash based
− Basic Hash Shuffle
− Consolidate Hash Shuffle

Sort Based Shuffle

Input Aggregation

Like MapReduce, Spark involves aggregate(Combiner) on map side.

Aggregation is done in ShuffleMapTask using

AppendOnlyMap (In Memory Hash Table combiner)
− Key’s are never removed , values gets updated

ExternalAppendOnlyMap (In Memory and disk Hash Table combiner)
− A Hash Map which can spill to disk
− Append Only Map that spill data to disk if insufficient memory

Shuffle file In-Memory Buffer – Shuffle writes to In-memory buffer before
writing to a shuffle file.

Shuffle Types – Basic Hash Shuffle

Hash Based shuffle (spark.shuffle.manager). Hash Partitions the data
for reducers

Each map task writes each bucket to a file.

#Map Tasks = M

#Reduce Tasks = R

#Shuffle File = M*R , #In-Memory Buffer = M*R

Shuffle Types – Basic Hash Shuffle

Problem

Lets use 100KB as buffer size

We have 10000 reducers

10 Mapper tasks Per Executor

In-Memory Buffer size will = 100KB*10000*10

Buffer need will be 10GB/Executor

This huge amount of Buffer is not acceptable and this
Implementation cant support 10000 reducer.

Shuffle Types – Consolidate Hash Shuffle

Solution to decrease the IN-Memory Buffer size , No of File.

Within Executor, Map Tasks writes each Bucket to a Segment of the file.

#Shuffle file/Executor = #Reducers,

# In-Memory Buffer/ Executor=#R( Reducers)

Shuffle Types – Sort Based Shuffle

Consolidate Hash Shuffle needs one file for each reducer.
- Total C*R intermediate file , C = # of executor running map
tasks

Still too many files(e.g ~10k reducers),

Need significant memory for compression & serialization
buffer.

Too many open files issue.

Sort Based Shuflle is similar to map-side shuffle from
MapReduce

Introduced in Spark 1.1 , now its default shuffle

Shuffle Types – Sort Based Shuffle

Map output records from each task are kept in memory till they can fit.

Once full , data gets sorted by partition and spilled to single file.

Each Map task generate 1 data file and one index file

Utilize external sorter to do the sort work

If map side combiner is required data will be sorted by key and partition
otherwise only by partition

#reducer <=200, no sorting uses hash approach, generate file per reducer
and merge them into a single file

Shuffle Reader

On Reader side both Sort & Hash Shuffle uses Hash Shuffle Reader

On reducer side a set of thread fetch remote output map blocks

Once block comes its records are de-serialized and passed into a
result queue.

Records are passed to ExternalAppendOnlyMap , for ordering
operation like sortByKey records are passed to externalSorter.
20
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Bucket
Reduce Task
Aggregator Aggregator Aggregator Aggregator
Reduce Task Reduce Task Reduce Task

Type of RDDS - RDD Interface
Base for all RDDs (RDD.scala), consists of

A Set of partitions (“splits” in Hadoop)

A List of dependencies on parent RDDs

A Function to compute the partition from its
parents

Optional preferred locations for each partition

A Partitioner defines strategy for partitionig
hash/range

Basic operations like map, filter, persist etc
Partitions
Dependencies
Compute
PreferredLocations
Partitioner
map,filter,persist
s
Lineage
Optimized execution
Operations

Example: HadoopRDD

partitions = one per HDFS block

dependencies = none

compute(partition) = read corresponding block

preferredLocations(part) = HDFS block location

partitioner = none

Example: MapPartitionRDD

partitions = Parent Partition

dependencies = “one-to-one “parent RDD

compute(partition) = apply map on parent

preferredLocations(part) = none (ask parent)

partitioner = none

Example: CoGroupRDD

partitions = one per reduce task

dependencies = could be narrow or wide dependency

compute(partition) = read and join shuffled data

preferredLocations(part) = none

partitioner = HashPartitioner(numTasks)

Extending RDDs
Extend RDDs to

To add transformation/actions

Allow developer to express domain specific calculation in
cleaner way

Improves code readability

Easy to maintain

Custom RDD for Input Source, Domain

Way to add new Input data source

Better way to express domain specific data

Better control on partitioning and distribution

How to Extend

Add custom operators to RDD

Use scala Impilicits

Feels and works like built in operator

You can add operator to Specific RDD or to all

Custom RDD

Extend RDD API to create our own RDD

Implement compute & getPartitions abstract method

Implicit Class

Creates an extension method to existing type

Introduced in Scala 2.10

Implicits are compile time checked. Implicit class gets resolved
into a class definition with implict conversion

We will use Implicit to add new method in RDD

Adding new Operator to RDD

We will use Scala Implicit feature to add a new operator to an
existingRDD

This operator will show up only in our RDD

Implicit conversions are handled by Scala

Custom RDD Implementation

Extending RDD allow you to create your own custom RDD
structure

Custom RDD allow control on computation, change partition &
locality information

Caching in RDD

Spark allows caching/Persisting entire dataset in memory

Persisting RDD in cache

First time when it is computed it will be kept in memory

Reuse the the cache partition in next set of operation

Fault-tolerant, recomputed in case of failure

Caching is key tool for interactive and iterative algorithm

Persist support different storage level

Storage level - In memory , Disk or both , Techyon

Serialized Vs Deserialized

Caching In RDD

Spark Context tracks persistent RDDs

Block Manager puts partition in memory when first evaluated

Cache is lazy evaluation , no caching without an action.

Shuffle also keeps its data in Cache after shuffle operations.

We still need to cache shuffle RDDs

Caching Demo

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Similar to Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive (20)

Recently uploaded

Recently uploaded (20)

Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive

Editor's Notes