This document provides an overview of Spark Pair RDDs and persistence. It defines Pair RDDs as RDDs of key-value pairs that enable aggregations. It describes how to create Pair RDDs in Scala, Python, and Java and covers common Pair RDD transformations like groupByKey, reduceByKey, aggregateByKey, combineByKey, joins, sorting by key, and actions. It also discusses the differences between groupByKey and reduceByKey and demonstrates various Pair RDD functions and persistence levels.

1. Spark Pair RDD and Persistence
Spark and Scala

2. Spark Pair RDD and Persistence
 Spark Pair RDDs
 Creating Pair RDDs
 Pair RDD Reduce Transformations
 groupByKey
 reduceByKey
 aggregateByKey
 combineByKey
 Pair RDD Joins Transformations
 Pair RDD Sort by Key
 Pair RDD Actions
 countByKey
 collectAsMap
 lookUp
 RDD Cache and RDD Persistence
 RDD Storage Levels
 Q & A
Spark and Scala

3. Spark Pair RDD
 Pair RDDs are rdds of key/value pair
 Key/value RDDs are used to perform aggregations
 Key/value RDDs expose new operations
 Counting reviews for each product, grouping together data with the
same key, and grouping together two different RDDs
Spark and Scala

4. Creating Pair RDD (Scala)
 In Scala, for the functions on key data to be available, we also need to
return tuples
 Creating a pair RDD using the first word as the key word in Scala
 val pairs = lines.map(x => (x.split(” “)(0), x))
Spark and Scala

5. Creating Pair RDD
 Pair RDDs can be created by running a map() function that returns
key or value pairs
 The procedure to build the key-value RDDs differs by language.
 In Python language, for the functions on key data to work we need to
return an RDD composed of tuples
 Creating a pair RDD using the first word as the key in Python
programming language.
 pairs = lines.map(lambda x: (x.split(” “)[0], x))
Spark and Scala

6. Creating Pair RDD (Java)
 Java doesn’t have a built-in function of tuple
 so Spark’s Java API users create tuples using the scala.Tuple2 class.
 Java users can construct a new tuple by writing new Tuple2(elem1,
elem2)
 Access its relevant elements with the _1() and _2() methods
 Java users also need to call special versions of Spark’s functions when
you are creating pair of RDDs.
 For instance, the mapToPair () function should be used in place of the
basic map() function.
Spark and Scala

7. Creating Pair RDD (Java)
 Creating a pair RDD using the first word as the key word in Java
program.
 PairFunction<String, String, String> keyData =
new PairFunction<String, String, String>() {
public Tuple2<String, String> call(String x) {
return new Tuple2(x.split(” “)[0], x);
}
};
 JavaPairRDD<String, String> pairs = lines.mapToPair(keyData);
Spark and Scala

8. PairRDD Transformations
 Most Spark operations work on RDDs containing any type of objects,
a few special operations are only available on RDDs of key-value pairs
 The most common ones are distributed “shuffle” operations, such as
grouping or aggregating the elements by a key
 In Scala, these operations are automatically available on RDDs
containing Tuple2 objects, writing (a, b)
 The key-value pair operations are available in the PairRDDFunctions
class, which automatically wraps around an RDD of tuples
Spark and Scala

9. PairRDD Transformations (Aggregation)
 When datasets are described in terms of key/value pairs, it is
common feature that is required to aggregate statistics across all
elements with the same key value
 Spark has a set of operations that combines values that own the
same key
 These operations return RDDs and thus are transformations rather
than actions i.e. reduceByKey(), foldByKey(), combineByKey()
Spark and Scala

10. PairRDD Transformations (Grouping)
 With key data is a common type of use case in grouping our data sets
is used with respect to predefined key value
 For example, viewing all of a customer’s orders together in one file.
 If our data is already keyed in the way we want to implement,
groupByKey() will group our data using the key value using our RDD.
 On an RDD consisting of keys of type K and values of type V, we get
back an RDD operation of type [K, Iterable[V]]
Spark and Scala

11. PairRDD Transformations (Joins)
 The most useful and effective operations we get with keyed data
values comes from using it together with other keyed data.
 Joining datasets together is probably one of the most common type
of operations you can find out on a pair RDD
 Supports following type of joins
 Inner Join
 left OuterJoin
 rightOuterJoin
Spark and Scala

12. PairRDD Transformations (Sort)
 We can sort an RDD with key or value pairs if there is an ordering
defined on the key set.
 Once we have sorted our data elements, any subsequent call on the
sorted data to collect() or save() will result in ordered dataset.
Spark and Scala

13. PairRDD Transformations
Transformations Description
groupByKey([numTasks]
)
When called on a dataset of (K, V) pairs, returns a dataset of (K,
Iterable<V>) pairs. Note: If you are grouping in order to perform an
aggregation (such as a sum or average) over each key, using
reduceByKey or aggregateByKey will yield much better performance.
Note: By default, the level of parallelism in the output depends on the
number of partitions of the parent RDD.
You can pass an optional numTasks argument to set a different
number of tasks.
reduceByKey(func,
[numTasks])
When called on a dataset of (K, V) pairs, returns a dataset of (K, V)
pairs where the values for each key are aggregated using the given
reduce function func, which must be of type (V,V) => V. Like in
groupByKey, the number of reduce tasks is configurable through an
optional second argument
sortByKey([ascending],
[numTasks])
When called on a dataset of (K, V) pairs where K implements Ordered,
returns a dataset of (K, V) pairs sorted by keys in ascending or
descending order, as specified in the Boolean ascending argument.
Spark and Scala

14. PairRDD Transformations
Transformations Description
combineByKey[C](
createCombiner: V =>
C,
mergeValue: (C, V) =>
C,
mergeCombiners: (C,
C) => C): RDD[(K, C)]
1st Argument : createCombiner is called when a key(in the RDD
element) is found for the first time in a given Partition. This method
creates an initial value for the accumulator for that key
2nd Argument : mergeValue is called when the key already has an
accumulator
3rd Argument : mergeCombiners is called when more that one
partition has accumulator for the same key
mapvalues When called on a dataset of (K, V) pairs, returns a dataset of (K, V)
pairs where the values for each key are aggregated using the given
reduce function func, which must be of type (V,V) => V. Like in
groupByKey, the number of reduce tasks is configurable through an
optional second argument
Keys/values When called on a dataset of (K, V) pairs where K implements Ordered,
returns a dataset of (K, V) pairs sorted by keys in ascending or
descending order, as specified in the boolean ascending argument.
Spark and Scala

15. PairRDD Transformations
Transformations Description
aggregateByKey(zeroVal
ue)(seqOp, combOp,
[numTasks])
When called on a dataset of (K, V) pairs, returns a dataset of (K, U)
pairs where the values for each key are aggregated using the given
combine functions and a neutral "zero" value. Allows an aggregated
value type that is different than the input value type, while avoiding
unnecessary allocations. Like in groupByKey, the number of reduce
tasks is configurable through an optional second argument.
join(otherDataset,
[numTasks])
When called on datasets of type (K, V) and (K, W), returns a dataset of
(K, (V, W)) pairs with all pairs of elements for each key. Outer joins are
supported through leftOuterJoin, rightOuterJoin, and fullOuterJoin
cogroup(otherDataset,
[numTasks])
When called on datasets of type (K, V) and (K, W), returns a dataset of
(K, (Iterable<V>, Iterable<W>)) tuples. This operation is also called
groupWith
Spark and Scala

16. PairRDD Actions
 countByKey() : Count the number of elements for each key pair.
 collectAsMap() : Collect the result outputs as a map to provide easy
lookup.
 lookup(key) : Return all values associated with the provided key pair.
Spark and Scala

17. Demo of Pair RDD Transformations
• val rddnums= sc.parallelize(List("Hadoop Spark Scala
Python","DataScience Python C# Java","Hadoop Scala Python","Spark
Scala"))
• val inputmarkrdd = sc.parallelize(Seq(("maths", 50), ("maths",
60),("english", 65), ("physics", 66), ("physics", 61), ("physics", 87))
Spark and Scala

18. Spark Create Pair RDD
 Create Pair RDD from String separated with space
 Use flat Map to create RDD of key/value pair
Spark and Scala

19. Spark Group By Key
 Use Group By Key to output count of words
Spark and Scala

20. Spark Reduce By Key
 Use reduce By Key to output count of words
Spark and Scala

21. Diff between Group/Reduce By Key
Spark and Scala

22. Diff between Group/Reduce By Key
 reduce By Key combine output with a common key on each partition
before shuffling the data.
 Reduce function is called again to reduce all the values from each
partition to produce one result.
 On the other hand, when calling groupByKey - all the key-value pairs
are shuffled around. This is a lot of unnecessary data to being
transferred over the network.
 To determine which machine to shuffle a pair to, Spark calls a
partitioning function on the key of the pair. Spark spills data to disk
when there is more data shuffled onto a single executor machine
than can fit in memory.
Spark and Scala

23. Diff between Group/Reduce By Key
 However, it flushes out the data to disk one key at a time - so if a
single key has more key-value pairs than can fit in memory, an out of
memory exception occurs.
 This will be more gracefully handled in a later release of Spark so the
job can still proceed, but should still be avoided - when Spark needs
to spill to disk, performance is severely impacted.
Spark and Scala

24. Aggregate By Key
Spark and Scala

25. Combine By Key
Spark and Scala

26. Combine By Key
Spark and Scala

27. Spark Sort By Key
Spark and Scala

28. MapValues/Keys/Values
Spark and Scala

29. CountByKey (Action)
Spark and Scala

30. CollectAsMap (Action)
Spark and Scala

31. Spark Persistence
Spark and Scala
 One of the most important capabilities in Spark is persisting (or
caching) a dataset in memory across operations.
 When you persist an RDD, each node stores any partitions of it that it
computes in memory and reuses them in other actions on that
dataset (or datasets derived from it).
 This allows future actions to be much faster (often by more than 10x)
 Caching is a key tool for iterative algorithms and fast interactive use

32. Spark Persistence
Spark and Scala
 RDD.cache is also a lazy operation.
 If you run textFile.count the first time, the file will be loaded, cached,
and counted.
 If you call textFile.count a second time, the operation will use the
cache.

33. Spark Persistence
Spark and Scala
 You can mark an RDD to be persisted using the persist() or cache()
methods on it.
 The first time it is computed in an action, it will be kept in memory on
the nodes.
 Spark’s cache is fault-tolerant – if any partition of an RDD is lost, it will
automatically be recomputed using the transformations that
originally created it.

34. Memory vs Memory + Disk
Spark and Scala
 Cache RDD in Memory
 Cache RDD in memory +Disk

35. Spark Persistence
Spark and Scala
 Each persisted RDD can be stored using a different storage level,
allowing you, for example, to persist the dataset on disk, persist it in
memory but as serialized Java objects (to save space), replicate it
across nodes.
 These levels are set by passing a StorageLevel object (Scala, Java,
Python) to persist().
 The cache() method is a shorthand for using the default storage level,
which is StorageLevel.MEMORY_ONLY (store deserialized objects in
memory).

36. RDD Cache
Spark and Scala

37. Spark Persistence
Spark and Scala
Storage Level Explanation
MEMORY_ONLY Store RDD as deserialized Java objects in the JVM. If the RDD does not fit
in memory, some partitions will not be cached and will be recomputed
on the fly each time they're needed. This is the default level.
MEMORY_AND_DISK Store RDD as deserialized Java objects in the JVM. If the RDD does not fit
in memory, store the partitions that don't fit on disk, and read them
from there when they're needed.
MEMORY_ONLY_SER
(Java and Scala)
Store RDD as serialized Java objects (one byte array per partition). This is
generally more space-efficient than deserialized objects, especially
when using a fast serializer, but more CPU-intensive to read.

38. Spark Persistence
Spark and Scala
Storage Level Explanation
MEMORY_AND_DISK_SER
(Java and Scala)
Similar to MEMORY_ONLY_SER, but spill partitions that
don't fit in memory to disk instead of recomputing them on
the fly each time they're needed.
DISK_ONLY Store the RDD partitions only on disk
MEMORY_ONLY_2,
MEMORY_AND_DISK_2, etc
Same as the levels above, but replicate each partition on
two cluster nodes

39. Spark Persistence
Spark and Scala
 In Python, stored objects will always be serialized with the Pickle
library, so it does not matter whether you choose a serialized level.
 The available storage levels in Python include MEMORY_ONLY,
MEMORY_ONLY_2, MEMORY_AND_DISK, MEMORY_AND_DISK_2,
DISK_ONLY, and DISK_ONLY_2

40. Spark Persistence Space and CPU
Spark and Scala

41. Spark UnPersist
Spark and Scala
 RDD.unpersist()

42. Spark Cache
Spark and Scala

43. Spark Cache
Spark and Scala

44. Spark Cache
Spark and Scala

45. Spark Persist
Spark and Scala