This document provides an overview of Spark, including how it differs from MapReduce by loading more data into memory and implementing caching mechanisms. It discusses Spark's ecosystem, features like RDDs, and how it can run on Hadoop clusters. It also summarizes Spark Streaming, describing how it processes data in micro-batches, provides windowing and transformation operations on DStreams, and unifies batch, streaming, and interactive analytics.

1. TechLabs by
A la découverte de Machine Learning, de Redis et de Spark

2. TechLabs by
2
Maturin BADO
@mccstanmbg
github.com/mccstan
SPARK

3. Spark : Introduction

4. Outline
❏ Data processing today
❏ Spark, hadoop, MapReduce
❏ Spark ecosystem
❏ Spark basics

5. Data processing today
Data intensive application
Definition :
“We call an application data-intensive if data is its primary challenge—the
quantity of data, the complexity of data, or the speed at which it is changing—as
opposed to compute-intensive, where CPU cycles are the bottleneck.”
Martin Klepmann

6. Data processing today
Today apps needs :
❏ Store data (databases)
❏ Caches
❏ search data (search index)
❏ Asynchronously message handling (stream processing)
❏ batch processing

7. Spark, hadoop, MapReduce

8. Spark, hadoop, MapReduce
Spark : main differences with Map Reduce
❏ Spark load most of the dataset in memory
❏ Implement cache mechanisms which reduce read from disk
❏ Is much faster than MapReduce : Job scheduling
❏ Does not implement any data distribution technology but
can run on top of hadoop clusters (HDFS )

9. Spark ecosystem : open source

10. Spark ecosystem : features

11. Spark ecosystem : deployment

12. Spark basics : RDD
RDD : Resilient Distributed data
❏ Primary spark abstraction
❏ Fault tolerant collection of elements
❏ Partitioned and Immutable
❏ Two types operations
❏ Lazy Transformation

13. Spark basics : An execution flow

14. Spark Streaming

15. Outline
❏ Why In-stream processing ?
❏ Runtime and Programming Model
❏ Spark Streaming : Overview
❏ Benefits of Discretized Stream Processing
❏ Processing flow
❏ Transform operations
❏ Window operations

16. Why In-stream processing ?

17. Why In-stream processing ?

18. Runtime and Programming Model
Native Streaming

19. Runtime and Programming Model
Micro-batch Streaming

20. Spark Streaming : Overview

21. Benefits of Discretized Stream Processing
Dynamic load balancing

22. Benefits of Discretized Stream Processing
Fast failure and straggler recovery

23. Benefits of Discretized Stream Processing
❏ Unification of batch, streaming and interactive analytics
❏ Advanced analytics like machine learning and interactive SQL
❏ Streaming + SQL and DataFrames
❏ Streaming + MLlib

24. Spark Streaming : Processing flow

25. Spark Streaming : DStreams
Discretized Streams (DStreams) :
❏ The basic spark streaming abstraction
❏ A continuous series of RDDs

26. Spark Streaming : Transformations
Transform Operations : Any operation applied on a DStream translates
to operations on the underlying RDDs

27. Spark Streaming : Transformations
Window Operations :

28. Spark Streaming : Time abstractions
Batch interval
Sliding interval
Window size

29. Spark Streaming : Time abstractions
Batch interval
Window size
Sliding interval

30. Spark Streaming : Some examples
❏ Wordcount
❏ stateless operation, counting words for every batch
❏ Basic Error count
❏ stateless operation, using a filter : contains(“ERROR”)
❏ Cumulative Error count
❏ Stateful operation, errors from the beginning of the processing
❏ Windowed Errors counts
❏ Stateful operation, errors from the sliding window of time

31. The git repo
https://github.com/SoatGroup/spark-streaming-java-examples
https://github.com/SoatGroup/spark-streaming-python