Spark Introduction

1. Apache Spark
Syed
Solutions Engineer - Big Data
mail.syed786@gmail.com
info.syedacademy@gmail.com
+91-9030477368

2. History
Developed in 2009 at UC Berkeley AMPLab.
● Open sourced in 2010.
● Spark becomes one of the largest big-data
projects with more 400 contributors in 50+ organizations such
as:
– Databricks, Yahoo!, Intel, Cloudera, IBM, …

3. • Fast and general cluster computing system
interoperable with Hadoop datasets.
What is Spark?

4. Where Does Big Data Come From?
It’s all happening online – could record every:
» Click
» Ad impression
» Billing event
» Fast Forward, pause,…
» Server request
» Transaction
» Network message
» Fault
» Facebook
» Instagram
» TripAdvisor
» Twitter
» YouTube
»…

5. Graph Data
Lots of interesting data has a graph structure:
• Social networks
• Telecommunication Networks
• Computer Networks
• Road networks
• Collaborations/Relationships
• …
Some of these graphs can get quite large
(e.g., Facebook user graph)
Log Files – Apache Web Server Log

6. Why Apache Spark?
General purpose cluster computing system
• Originally developed at UC Berkeley, now one of the
largest Apache projects
• Typically faster than Hadoop due to main-memory
processing
• High-level APIs in Java, Scala, Python and R
Functionality for:
• Map/Reduce
• SQL processing
• Real-time stream processing
• Machine learning
• Graph processing

7. Apache Spark EcoSystem
• Apache Spark
• RDDs
• Spark SQL
• Once known as Shark
before completely
integrated into Spark
• For SQL, structured and
semi-structured data
processing
• Spark Streaming
• Processing of live data
streams
• MLlib/ML
• Machine Learning
Algorithms
• GraphX
• Graph Processing

8. MapReduce vs Spark
PIG HIVE MAHOUT
(machine
learning)
MapReduce

9. Hadoop MapReduce

14. Programming Models
• MapReduce – 50 lines of code
• Spark – 1 line of code

18. MapReduce Bottlenecks and Improvements
• Bottlenecks
• MapReduce is a very I/O heavy operation
• Map phase needs to read from disk then write back out
• Reduce phase needs to read from disk and then write
back out
• How can we improve it?
• RAM is becoming very cheap and abundant
• Use RAM for in-data sharing

19. MapReduce vs. Spark (Performance)
MapReduce Record Spark Record Spark Record 1PB
Data Size 102.5 TB 100 TB 1000 TB
# Nodes 2100 206 190
# Cores 50400 physical 6592 virtualized 6080 virtualized
Elapsed Time 72 mins 23 mins 234 mins
Sort rate 1.42 TB/min 4.27 TB/min 4.27 TB/min
Sort rate/node 0.67 GB/min 20.7 GB/min 22.5 GB/min

21. Spark Architecture

22. RDDs
• Primary abstraction object used by Apache Spark
• Resilient Distributed Dataset
• Fault-tolerant
• Collection of elements that can be operated on in parallel
• Distributed collection of data from any source
• Contained in an RDD:
• Set of dependencies on parent RDDs
• Lineage (Directed Acyclic Graph – DAG)
• Set of partitions
• Atomic pieces of a dataset
• A function for computing the RDD based on its parents
• Metadata about its partitioning scheme and data
placement
• RDDs are Immutable
• Allows for more effective fault tolerance
• Intended to support abstract datasets while also maintain
MapReduce properties like automatic fault tolerance,
locality-aware scheduling and scalability.

23. SPARK SQL
• DataFrames
• DataSets

24. Spark Streaming
• Spark Streaming is an extension of the core Spark API that
enables scalable, high-throughput, fault-tolerant stream
processing of live data streams

26. Thank you!
www.syedacademy.com
mail.syed786@gmail.com
info.syedacademy@gmail.com
+91-9030477368