An Introduction to Apache Spark with Amazon EMR. Dr. Peter Smith's presentation slides from the Vancouver Amazon Web Services User Group Meetup on November 20, 2018 at ACL hosted and presented by Onica.

1. An Introduction to Apache Spark
with Amazon EMR
Peter Smith, Principal Software Engineer, ACL

2. Overview
• What is Spark?
• A Brief Timeline of Spark
• Storing data in RDDs and DataFrames
• Distributed Processing of Distributed Data
• Loading data: CSV, Parquet, JDBC
• Queries: SQL, Scala, PySpark
• Setting up Spark with EMR (Demo)

3. What is Spark?
Apache Spark is a unified analytics engine
for large-scale data processing.
• Can process Terabytes of data (billions of rows)
• Click streams from a web application.
• IoT data.
• Financial trades.
• Computation performed over multiple (potentially 1000s) of compute nodes.
• Has held the world record for sorting 100TB of data in 23 minutes (2014)

4. Usage Scenarios
• Batch Processing
– Large amounts of data, read from disk, then processed.
• Streaming Data
– Data is processed in real-time
• Machine Learning
– Predicting outcomes based on past experience
• Graph Processing
– Arbitrary data relationship, not just rows and columns

5. Spark versus Database
SQL
Interpreter /
Execution
Engine
SQL Database
SQL Client
SQL Select, Insert, Update
• Disk files in proprietary
format (e.g. B-Trees,
WALs)
• Users never look directly
at data files.
• Execution engine has
100% control over file
storage.
• Often database server is
a single machine (lots of
CPUs)

6. Spark versus Database
Spark
Worker
Spark
Worker
Spark
Worker
Spark
Worker
Spark Driver
SQL, Java, Scala,
Python, R
S3 Bucket
• Disk formats and
locations are 100%
controlled by user.
• No Transactional
Inserts or Updates!
• Compute is spread
over multiple servers
to improve scale.
Separate EC2
Servers
Amazon EFS

7. A Brief Timeline of Spark
2003 - Google File System
2004 - Map Reduce
2006 2010
2013
2011
Hadoop 2.8.4
Spark 2.3.2
(as of today)

8. How is Data Stored?
Spark allows data to be read or written from disk in a range of formats:
• CSV – Possibly the simplest and most common: Fred,Jones,1998,10,Stan
• JSON – Often generated by web applications. { “first_name”: “Fred”, … }
• JDBC – If the source data is in a database.
• Parquet and ORC – Optimized storage for column-oriented queries.
• Others – You’re free to write your own connectors.
Data is read or written as complete files – doesn’t support inserts or updates.
(unlike a transactional database that completely controls the structure of the data).

9. RDDs - How Spark Stores Data
• RDD – Resilient Distributed Dataset
• Data is stored in RAM, partitioned across multiple servers
• Each partition operates in parallel.
Instead of using database replication
(for resilience), Spark will re-perform
the work on a different worker.

10. Example: Sort people by age.
1. Divide into partitions of 8 people
2. Within group, sort by age.
3. Shuffle people based on decade
of birth.
4. Sort within each group.

11. Data Frames – Rows/Columns (> Spark v1.5)
• RDD – Rows of Java Objects
• DataFrame – Rows of Typed Fields (like a Database table)
Id
(Integer)
First_name
(String)
Last_name
(String)
BirthYear
(Integer)
Shoe size
(Float)
Dog’s name
(String)
1 Fran Brown 1982 10.5 Stan
2 Mary Jones 1976 9.0 Fido
3 Brad Pitt 1963 11.0 Barker
4 Jane Simpson 1988 8.0 Rex
… … … … … …
• DataFrames allow better type-safety and performance optimization.

12. Example: Loading data from a CSV File
data.csv:
1,Fran,Brown,1982,10.5,Stan
2,Mary,Jones,1976,9.0,Fido
3,Brad,Pitt,1963,11.0,Barker
4,Jane,Simpson,1988,8.0,Rex
5,James,Thompson,1980,9.5,Bif
6,Paul,Wilson,1967,8.5,Ariel
7,Alice,Carlton,1984,11.5,Hank
8,Mike,Taylor,1981,9.5,Quincy
9,Shona,Smith,1975,9.0,Juneau
10,Phil,Arnold,1978,10.0,Koda

13. Example: Loading data from a CSV File
$ spark-shell
scala> val df = spark.read.csv("data.csv")
Notes:
• Similar methods exist for JSON, JDBC, Parquet, etc.
• You can write your own!
• Scala is a general purpose programming language (not like SQL)

14. Example: Examing a Data Frame
scala> df.show(5)
+---+-----+--------+----+----+------+
|_c0| _c1| _c2| _c3| _c4| _c5|
+---+-----+--------+----+----+------+
| 1| Fran| Brown|1982|10.5| Stan|
| 2| Mary| Jones|1976| 9.0| Fido|
| 3| Brad| Pitt|1963|11.0|Barker|
| 4| Jane| Simpson|1988| 8.0| Rex|
| 5|James|Thompson|1980| 9.5| Bif|
+---+-----+--------+----+----+------+

15. Example: Defining a Schema
scala> df.printSchema
root
|-- _c0: string (nullable = true)
|-- _c1: string (nullable = true)
|-- _c2: string (nullable = true)
|-- _c3: string (nullable = true)
|-- _c4: string (nullable = true)
|-- _c5: string (nullable = true)

16. Example: Defining a Schema
scala> df.printSchema
root
|-- _c0: string (nullable = true)
|-- _c1: string (nullable = true)
|-- _c2: string (nullable = true)
|-- _c3: string (nullable = true)
|-- _c4: string (nullable = true)
|-- _c5: string (nullable = true)

17. Example: Defining a Schema
scala> val mySchema = StructType(
Array(
StructField("id", LongType),
StructField("first_name", StringType),
StructField("last_name", StringType),
StructField("birth_year", IntegerType),
StructField("shoe_size", FloatType),
StructField("dog_name", StringType)
)
)
scala> val df = spark.read.schema(mySchema).csv("data.csv")

18. Example: Defining a Schema
scala> df.show(5)
+---+----------+---------+----------+---------+--------+
| id|first_name|last_name|birth_year|shoe_size|dog_name|
+---+----------+---------+----------+---------+--------+
| 1| Fran| Brown| 1982| 10.5| Stan|
| 2| Mary| Jones| 1976| 9.0| Fido|
| 3| Brad| Pitt| 1963| 11.0| Barker|
| 4| Jane| Simpson| 1988| 8.0| Rex|
| 5| James| Thompson| 1980| 9.5| Bif|
+---+----------+---------+----------+---------+--------+
scala> df.printSchema
root
|-- id: long (nullable = true)
|-- first_name: string (nullable = true)
|-- last_name: string (nullable = true)
|-- birth_year: integer (nullable = true)
|-- shoe_size: float (nullable = true)
|-- dog_name: string (nullable = true)

19. Example: Counting Records
scala> df.count()
res21: Long = 10
Imagine 10 Billion rows over 1000 servers?

20. Example: Selecting Columns
scala> val df_dog = df.select(
col("first_name"),
col("dog_name”))
scala> df_dog.show(5)
+----------+--------+
|first_name|dog_name|
+----------+--------+
| Fran| Stan|
| Mary| Fido|
| Brad| Barker|
| Jane| Rex|
| James| Bif|
+----------+--------+

21. Example: Aggregations
scala> df.agg(
min(col("birth_year")),
avg(col("birth_year"))
).show
+---------------+---------------+
|min(birth_year)|avg(birth_year)|
+---------------+---------------+
| 1963| 1977.4|
+---------------+---------------+

22. Example: Filtering
scala> df.where("birth_year > 1980").show
+---+----------+---------+----------+---------+--------+
| id|first_name|last_name|birth_year|shoe_size|dog_name|
+---+----------+---------+----------+---------+--------+
| 1| Fran| Brown| 1982| 10.5| Stan|
| 4| Jane| Simpson| 1988| 8.0| Rex|
| 7| Alice| Carlton| 1984| 11.5| Hank|
| 8| Mike| Taylor| 1981| 9.5| Quincy|
+---+----------+---------+----------+---------+--------+

23. Example: Grouping
scala> df.groupBy(
(floor(col("birth_year") / 10) * 10) as "Decade"
).count.show
+------+-----+
|Decade|count|
+------+-----+
| 1960| 2|
| 1970| 3|
| 1980| 5|
+------+-----+

24. Example: More Advanced
scala> df.select(
col("first_name"),
col("dog_name"),
levenshtein(
col("first_name"),
col("dog_name")
) as "Diff"
).show(5)
+----------+--------+----+
|first_name|dog_name|Diff|
+----------+--------+----+
| Fran| Stan| 2|
| Mary| Fido| 4|
| Brad| Barker| 4|
| Jane| Rex| 4|
| James| Bif| 5|
+----------+--------+----+
Shorter version:
df.select(
'first_name,
'dog_name,
levenshtein(
'first_name,
'dog_name) as "Diff”
).show(5)

25. Queries: User Defined Functions
def taxRateFunc(year: Int) = {
if (year >= 1984) 0.20 else 0.05
}
val taxRate = udf(taxRateFunc _)
df.select('birth_year, taxRate('birth_year)).show(5)
+----------+---------------+
|birth_year|UDF(birth_year)|
+----------+---------------+
| 1982| 0.05|
| 1976| 0.05|
| 1963| 0.05|
| 1988| 0.20|
| 1980| 0.05|
+----------+---------------+
UDAFs - Check out
http://build.acl.com
Computing Average
Dates in Spark!

26. Why is Spark better than a Database?
It looks a lot like SQL, but:
• Can read/write data in arbitrary formats.
• Can be extended with general purpose program code.
• Can be split across 1000s of compute nodes.
• Can do ML, Streaming, Graph queries.
• Can use cheap storage (such as S3)
But yeah, if you’re happy with your database, that’s OK too.

27. Queries: PySpark
Very similar API, but written in Python:
$ pyspark
>>>> spark.read.csv(”data.csv").show(5)
+---+-----+--------+----+----+------+
|_c0| _c1| _c2| _c3| _c4| _c5|
+---+-----+--------+----+----+------+
| 1| Fran| Brown|1982|10.5| Stan|
| 2| Mary| Jones|1976| 9.0| Fido|
| 3| Brad| Pitt|1963|11.0|Barker|
| 4| Jane| Simpson|1988| 8.0| Rex|
| 5|James|Thompson|1980| 9.5| Bif|
+---+-----+--------+----+----+------+

28. Demo Time…

29. Using EMR
Driver
m4.2xlarge
Worker
m4.2xlarge
Worker
m4.2xlarge
Worker
m4.2xlarge
Worker
m4.2xlarge
Query
Data
Zeppelin
Spark History