Get started with Apache Spark parallel computing framework. This is a keynote from the series "by Developer for Developers"

1. Big Data processing
with Apache Spark
Lucian Neghina
Big Data Architect
by Developer for Developers

2. Outline
● Spark Overview
● Structured APIs
● Low-Level APIs
● Streaming
● Writing an Application

3. Who uses Spark, and for what
Data Science
● Analyze and model the data
● Transforming the data into a
usable format
● Ad-hoc analysis, statistics,
machine learning
Data Processing
● Parallelize across clusters
● Hides the complexity of
distributed systems
○ Programming
○ Networking communication
○ Fault tolerance

4. Ecosystem

5. Functionality

6. Basic Architecture

7. Master-Workers

8. Directed Acyclic Graph (DAG)

9. DataFrame & Partitions

10. Core data structures
● Immutable
● Lives in memory
● Strongly typed
● Operations
○ Transformations (lazy)
○ Actions

11. Transformations and Actions
● Transformations return new RDDs as results
They are lazy, their result is not immediately computed
● Actions compute a result based on an RDD, and either
returned or saved to a storage
They are eager, their result is immediately computed

12. Transformations

13. Transformations

14. Actions
lines
#
#
#
#
count() cause Spark to:
● read data
● sum within partitions
● combine sums in driver

15. Actions
lines
#
#
#
#
Spark recomputes lines:
● read data (again)
● sum within partitions
● combine sums in driver
comments
#
#
#
#

16. What happens when an action is executed?
Driver
Worker Worker Worker

17. What happens when an action is executed?
The data is partitioned into different blocks
Driver
Worker Worker Worker
Block 1 Block 2 Block 3

18. What happens when an action is executed?
Driver sends the code to be executed on
each block
Driver
Worker Worker Worker
Block 1 Block 2 Block 3

19. What happens when an action is executed?
Read HDFS block
Driver
Worker Worker Worker
Block 1 Block 2 Block 3

20. What happens when an action is executed?
Read HDFS block and cache the data
Process and send the result to the driver
Driver
Worker Worker Worker
Block 1 Block 2 Block 3
Cache Cache Cache

21. What happens when an action is executed?
Driver combine the results / sum
Driver
Worker Worker Worker
Block 1 Block 2 Block 3
Cache Cache Cache

22. What happens when an action is executed?
Process from cache
Driver
Worker Worker Worker
Block 1 Block 2 Block 3
Cache Cache Cache

23. What happens when an action is executed?
Send the data back to the driver
Driver
Worker Worker Worker
Block 1 Block 2 Block 3
Cache Cache Cache

24. Distributed processing

25. Distributed processing
The result is calculated as follows,
Partition 1 : Sum(all Elements) + 3 (Zero value)
Partition 2 : Sum(all Elements) + 3 (Zero value)
Partition 3 : Sum(all Elements) + 3 (Zero value)
Result = Partition1 + Partition2 + Partition3 + 3(Zero value)
So we get 21 + 22 + 31 + (4 * 3) = 86

26. Structured APIs
DataFrames, Datasets and SQL

27. Structured API Overview
Structured APIs apply to both batch and streaming
computation.
Core type of distributed collections:
● Datasets (typed) - checks schema at runtime
● DataFrames (untyped) - checks schema at compile time
● SQL tables and views

28. Structured API Execution

29. Logical Planning

30. Physical Planning

31. Execution Pipeline

32. Basic Operations
● Schemas (schema-on-read)
Defines the column names and types of a DataFrames
● Columns and Expressions
Columns in Spark are similar to columns in a spreadsheet.
Expressions are the operations like select, manipulate, remove
columns
● Records and Rows
Each row is a single record represented as an object of type Row.

33. Data Sources
Read API structure
DataFrameReader.format(...).option(“key”, “value”).schema(...).load()
● CSV
● JSON
● Parquet
● ORC
● JDBC/ODBC connections
● Plain-text files
● and many, many others from community

34. Aggregations
● Aggregation Functions
count, countDistinct, first and last, min and max, sum, avg
● Grouping
with Expressions, with Maps
● Grouping Sets
Rollups, Cube, Pivot

35. ● Join Types
● Computation strategy
○ node-to-node (shuffle join)
○ per-node (broadcast join)
Joins

36. Big table to big table - shuffle join

37. Big table to small table - broadcast join

38. Low-Level APIs
Resilient Distributed Datasets (RDDs)

39. What is an RDD

40. RDD as a Distributed Dataset

41. RDD is Fault-Tolerance

42. RDD is Immutable

43. Working with RDDs

44. RDD Transformations

45. RDD Transformations

46. RDD Actions
Actions that return results to the driver program

47. RDD Actions
Action with side effects

48. Key-Value Pairs

49. What is a Key-Value Pair RDD?
● Any RDD whose elements are key-value pairs
○ Key-Value pair is a tuple with two components (key, value)
○ Different pair may have the same keys
○ Both keys and values can be of primitive or complex data
type

50. Transformations on Pair RDDs

51. Transformations on Pair RDDs

52. Actions on Pair RDDs

53. Aggregation on Pair RDDs

54. Aggregation on Pair RDDs

55. Grouping and Sorting on Pair RDDs
● Grouping values with the same key
○ Reorganizing data by a new key
○ Post-processing per-key groups
● Sorting values using keys
○ Generating special-purpose datasets
○ Generating reports that require ordering

56. Grouping and Sorting on Pair RDDs

57. Grouping and Sorting on Pair RDDs

58. Grouping and Sorting on Pair RDDs

59. Distributed Variables

60. Broadcast Variables
Main use cases:
● Application tasks across multiple stage need the same, relatively large and
immutable dataset
● Application tasks need the same, relatively large and immutable dataset cached
in deserialized form

61. Accumulators
Main use cases:
● Counting and summation
● Application needs to compute multiple aggregates on the same dataset
● Application needs custom aggregation not supported by existing Spark operations

62. Persistence

63. Persistence

64. Persistence

65. Persistence

66. Persistence

67. Persistence
cache
cache
cache

68. Tuning Partitioning

69. Distributed Collection of Partitions
● Spark automatically partitions RDDs
● Spark automatically distributes partitions among nodes

70. RDD Partitioning Properties
Number of partitions
Property Description
partitions Returns an array with all partition references for the sources RDD
partitions.size Returns a number of partitions in the source RDD
Partitioner
Property Description
partitioner Returns an Option[Partitioner] for the source RDD
Partitioner can refer to HashPartitioner, RangePartitioner or custom

71. Partitioning and Computation
● Partition is the smallest unit of data
● Task is the smallest unit of computation
● Number or partitions = Number of tasks

72. ● Number of partitions
○ Affects a number of tasks and the level of parallelism
○ Goal: balancing task execution and scheduling times
● Partitioner
○ Affects key-based operations
○ Goal: Avoiding shuffling the same dataset multiple times
Controlling Partitioning

73. Partitioning Rules
● Parallelizing a Scala Collection
○ partitions size = defaultParallelism
○ partitioner = None
● Reading data from HDFS
○ partitions size = max(number of file blocks or defaultParallelism)
○ partitioner = None
● Retrieving data from Cassandra
○ partitions size = max(data-size / 64 MBs or defaultParallelism)
○ partitioner = None

74. Setting and Reusing a Partitioner

75. Mechanisms for Controlling Partitioning

76. Data Shuffling