This document provides an introduction to Apache Spark and Scala. It discusses that Apache Spark is a general purpose cluster computing system that is faster than Hadoop MapReduce, runs locally and in the cloud. It has high-level APIs for Scala, Python, Java and R. The document outlines Spark's core components including SQL, MLlib, GraphX and streaming. It describes Spark's main data collections of RDDs for unstructured data and DataFrames/Datasets for structured data. Finally, it provides an overview of demonstrations that will be covered including the Spark shell, notebook, streaming and deploying a mini project to Google Cloud Dataproc.

1. Getting Started with Apache
Spark and Scala
Dinko Srkoč
Instantor Technology Services

2. About Apache Spark
(inevitable but hopefully quick intro)
● Started at UC Berkeley in 2009
● General purpose cluster computing system
● Fast: 10x on disk, 100x in memory vs Hadoop MapReduce
● Runs locally, in the cloud, on Hadoop, Mesos
● High level APIs in:
○ Scala
○ Python
○ Java
○ R

3. About Apache Spark
The Stack:
● SQL - SQL and semi/structured data processing
● MLLib - machine learning algorithms
● GraphX - graph processing
● Streaming - stream processing of
live data streams

4. Data collections in Spark
Collections: immutable, distributed, partitioned across nodes, operated in parallel
● Resilient Distributed Dataset (RDD)
○ Basic abstraction
○ Low-level API
○ Suitable for unstructured data (media, streams of text)
● Dataset/DataFrame
○ Dataset[T] - typed API, DataFrame (a.k.a. DataSet[Row]) - untyped API
○ High-level expressions: filters/maps, aggregations, averages, SQL queries, columnar access
○ optimizations

5. Demo
The Menu:
● Starter - spark shell
○ Loading from different sources
○ The inevitable word count example
● Intermediate - spark notebook
○ Documentation, data visualization
● Main course - back to shell
○ streaming
○ Spark UI
● Dessert - mini project:
○ SBT
○ Deploying to Google Cloud Dataproc

6. Thank you!
Questions?