An overview of Apache Spark and AWS Glue. Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, Python and R, and an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. AWS Glue is a fully managed extract, transform, and load (ETL) service that makes it easy for customers to prepare and load their data for analytics. You can create and run an ETL job with a few clicks in the AWS Management Console. You simply point AWS Glue to your data stored on AWS, and AWS Glue discovers your data and stores the associated metadata (e.g. table definition and schema) in the AWS Glue Data Catalog. Once cataloged, your data is immediately searchable, queryable, and available for ETL.

1. Apache Spark e AWS Glue
An overview about Apache Spark and AWS Glue
Laercio Serra
Data Engineer at ContaAzul

2. Agenda
1. Apache Spark
● How does it work
● Use cases
2. AWS Glue
● How does it work
● Use cases

3. Spark Features
● Spark is written in Scala Programming Language and runs in JVM.
● API: Scala, Java, Python, R.
● Interactive shell: Scala and Python.
● Data sources: SQL, NoSQL, S3, HDFS, Local File System, etc.
● Good fit for iterative tasks like Machine Learning (ML) algorithms.

4. Spark Components

5. Spark RDD (Resilient Distributed Dataset)
● Fault-tolerant collection of elements that can be operated on in parallel
● Immutable
● Two types of RDD operations
○ Transformations
○ Actions
● Dataset from any storage supported by Haddop
● Types of files supported: text files, sequence files and Hadoop InputFormat

6. RDD Transformations
● https://spark.apache.org/docs/latest/rdd-programming-guide.html#transformations
● Transformations are lazy evaluations
● Returns a pointer to new RDD
● Some functions:

7. RDD Actions
● https://spark.apache.org/docs/latest/rdd-programming-guide.html#actions
● Actions returns values
● Some functions:

8. RDD Persistence
● Each node stores any partitions of the cache that it computes in memory
● Reuses them in other actions on that dataset (or datasets derived from it)
● Two methods for RDD persistence:
○ persist()
○ cache()
● Different storage level of caching:
○ MEMORY_ONLY
○ MEMORY_AND_DISK
○ MEMORY_ONLY_SER
○ MEMORY_AND_DISK_SER
○ DISK_ONLY
○ MEMORY_ONLY_2, MEMORY_AND_DISK_2, etc
○ OFF_HEAP

9. Programming in Spark
● API available in Java, Scala and Python
● Create a RDD
● Process the data (transformations and actions)
● Persist the RDD to improve performance
● Use broadcast variable or accumulators

10. Understanding actions

11. Understanding actions
driver
worker1 worker2 worker3
block 1 block 2 block 3
The data is partitioned into different blocks

12. driver
worker1 worker2 worker3
block 1 block 2 block 3
Driver sends the code to be executed on each block
Understanding actions

13. driver
worker1 worker2 worker3
block 1 block 2 block 3
Reads HDFS block
Understanding actions

14. driver
worker1 worker2 worker3
block 1 block 2 block 3
Processing and caching data
cache cache cache
Understanding actions

15. driver
worker1 worker2 worker3
block 1 block 2 block 3
Send the data back to the driver
cache cache cache
Understanding actions

16. driver
worker1 worker2 worker3
block 1 block 2 block 3
Processing from cache and sending the data back to the
driver
cache cache cache
Understanding actions

17. Spark Streaming
● Easy, reliable and fast processing
of live data streams.
● Developers can reuse the same
code for stream and batch
processing and can also
integrate the streaming data
with historical data.
● Has exactly-once message
guarantees and helps recover
lost work without having to
write any extra code or adding
additional configurations.
● Supports inclusion of Spark
MLlib for machine learning
pipelines into data pathways.

18. Spark MLib

19. ETL using Spark Map Reduce (RDD)
Use case

20. Data Analysis using Spark SQL (Sample 1)
Use case

21. Data Analysis using Spark SQL (Sample 2)
Use case
API Reference PySpark Dataframe

22. Data Analysis using Spark SQL (Sample 3)
Use case

23. Data Analysis using Spark SQL (Sample 4)
Use case
● Jupyter PySpark SQL Dataframes

24. Real-Time using Spark Streaming
Use case

25. Data Classification using Spark MLib (Sample 1)
Use case

26. Data Classification using Spark MLib (Sample 2)
● Jupyter PySpark Machine Learning
Use case

27. Spark UI

28. AWS Glue
Connections Crawlers
Source
Target
Tables Jobs Cloudwatch

29. Crawler

30. Tables

31. Jobs

32. Jobs

33. AWS Glue Transformations
● Built-in transformations
● AWS Glue PySpark Reference
● Spark Transformations

34. AWS Cloudwatch

35. Demo

36. THANK YOU!