Pour ce mois de mars, nous vous proposons une thématique Big Data autour de Spark et du Machine Learning ! Nous attaquerons par une présentation d'Apache Spark 1.5 : son architecture distribuée et ses possibilités n'auront plus de secret pour vous. Nous enchaînerons ensuite avec les fondamentaux du Machine Learning : vocabulaire (pour enfin comprendre ce que raconte les data scientists / dataminer ! ), usages et explication des algorithmes les plus populaires ... Promis la présentation ne comporte pas de formules de maths barbares ;) Puis nous mettrons en pratique ces deux présentations en développant ensemble votre première application prédictive avec Apache Spark et Apache Zeppelin !

1. Spark &
Zeppelin
Introduction
#NightClazz Spark & ML
10/03/16
Fabrice Sznajderman

2. Agenda
● Apache Spark
● Apache Zeppelin
Introduction

3. Who I am?
● Fabrice Sznajderman
○ Java/Scala/Web developer
■ Java/Scala trainer
● BrownBagLunch.fr

4. Spark
Introduction

5. Big picture
Spark introduction

6. What is it about?
● A cluster computing framework
● Open source
● Written in Scala

7. History
2009 : Project start at MIT research lab
2010 : Project open-sourced
2013 : Become a Apache project and creation of the
Databricks company
2014 : Become a top level Apache project and the most active
project in the Apache fundation (500+ contributors)
2014 : Release of Spark 1.0, 1.1 and 1.2
2015 : Release of Spark 1.3, 1.4, 1.5 and 1.6
2015 : IBM, SAP… investment in Spark
2015 : 2000 registration in Spark Summit SF, 1000 in Spark
Summit Amsterdam
2016 : new Spark Summit in San Francisco in June 2016

8. Multi-languages
● Scala
● Java
● Python
● R

9. Spark Shell
● REPL
● Learn API
● Interactive Analysis

10. RDD
Core concept

11. Definition
● Resilient
● Distributed
● Datasets

12. Properties
● Immutable
● Serializable
● Can be persist in RAM and / or
disk
● Simple or complexe type

13. Use as a collection
● DSL
● Monadic type
● Several operators
○ map, filter, count, distinct, flatmap, ...
○ join, groupBy, union, ...

14. Created from
● A collection (List, Set)
● Various formats of file
○ json, text, Hadoop SequenceFile, ...
● Various database
○ JDBC, Cassandra, ...
● Others RDD

15. Sample
val conf = new SparkConf()
.setAppName("sample")
.setMaster("local")
val sc = new SparkContext(conf)
val rdd = sc.textFile("data.csv")
val nb = rdd.map(s => s.length).filter(i => i> 10).count()

16. Lazy-evaluation
● Intermediate operators
○ map, filter, distinct, flatmap, …
● final operators
○ count, mean, fold, first, ...
val nb = rdd.map(s => s.length).filter(i => i> 10).count()

17. Caching
● Reused an intermediate result
● Cache operator
● Avoid re-computing
val r = rdd.map(s => s.length).cache()
val nb = r.filter(i => i> 10).count()
val sum = r.filter(i => i> 10).sum()

18. Distributed
Architecture
Core concept

19. Run locally
val master = "local"
val master = "local[*]"
val master = "local[4]"
val conf = new SparkConf().setAppName("sample")
.setMaster(master)
val sc = new SparkContext(conf)

20. Run on cluster
val master = "spark://..."
val conf = new SparkConf().setAppName("sample")
.setMaster(master)
val sc = new SparkContext(conf)

21. Standalone Cluster
Spark
Master
Spark
Slave
Spark
Slave
Spark
Slave
E
E E
E
E
E
Spark
client
Spark
client
Spark
client

22. Modules
Core concept

23. Composed by
Spark Core
Spark
Streaming
MLlib GraphX
Spark
SQL
ML PipelineDataFrames
Several data sources

24. Several data sources
http://prog3.com/article/2015-06-18/2824958

25. Spark SQL
● Structured data processing
● SQL Language
● DataFrame

26. DataFrame 1/3
● A distributed collection of rows
organized into named columns
● An abstraction for selecting,
filtering, aggregating and
plotting structured data
● Provide a schema
● Not a RDD replacement
What?

27. DataFrame 1/3
● RDD more efficient than before
(Hadoop)
● But RDD is still too complicated
for common tasks
● DataFrame is more simple and
faster
Why?

28. DataFrame 2/3
Optimized

29. DataFrame 3/3
● From Spark 1.3
● DataFrame API is just an
interface
○ Implementation is done one time in
Spark engine
○ All languages take benefits of
optimization with out rewriting
anything
How ?

30. Spark Streaming
● Framework over RDD and
Dataframe API
● Real-time data processing
● RDD is DStream here
● Same as before but dataset is
not static

31. Spark Streaming
Internal flow
http://spark.apache.org/docs/latest/img/streaming-flow.png

32. Spark Streaming
Inputs / Ouputs
http://spark.apache.org/docs/latest/img/streaming-arch.png

33. Spark MLlib
● Make pratical machine learning
scalable and easy
● Provide commons learning
algorithms & utilities

34. Spark MLlib
● Divides into 2 packages
○ spark.mllib
○ spark.ml

35. Spark MLlib
● Original API based on RDD
● Each model has its own
interface
spark.mllib

36. Spark MLlib
spark.mllib
val sc = //init sparkContext
val (trainingData, checkData) = sc.textFile
("train.csv")
/*transform*/
.randomSplit(Array(0.98, 0.02))
val model = RandomForest.trainClassifier(
trainingData,
10,
Map[Int, Int](),
30,
"auto",
"gini",
7,
100,
0)
val prediction = model.predict(...)
//init sparkContext
val (trainingData, checkData) = sc.textFile("train.
csv")
/*transform*/
.randomSplit(Array(0.98, 0.02))
val model = new
LogisticRegressionWithLBFGS()
.setNumClasses(10)
.run(train)
val prediction = model.predict(...)
Each model exposes its own
interface

37. Spark MLlib
● Provides uniform set of high-
level APIs
● Based on top of Dataframe
● Pipeline concepts
○ Transformer
○ Estimator
○ Pipeline
spark.ml

38. Spark MLlib
spark.ml
● Transformer : transform(DF)
○ map a dataFrame by adding new
column
○ predict the label and adding result in
new column
● Estimator : fit(DF)
○ learning algorithm
○ produces a model from dataFrame

39. Spark MLlib
spark.ml
● Pipeline
○ sequence of stages (transformer or
estimator)
○ specific order

40. Spark MLlib
spark.ml
val training:DataFrame = ???
val test:DataFrame = ???
val lr = new LogisticRegression()
lr.setMaxIter(10).setRegParam(0.01)
//training model
val model1 = lr.fit(training)
//prediction on data test
model1.transform(test)

41. Spark MLlib
spark.ml
val training:DataFrame = ???
val test:DataFrame = ???
val tokenizer = new Tokenizer()
.setInputCol("text")
.setOutputCol("words")
val hashingTF = new HashingTF()
.setNumFeatures(1000)
.setInputCol(tokenizer.getOutputCol)
.setOutputCol("features")
val lr = new RandomForestClassifier()()
/*.add parameter*/
val pipeline = new Pipeline()
.setStages(Array(tokenizer, hashingTF, lr))
// Fit the pipeline to training documents.
val model = pipeline.fit(training)
model.transform(test)
val training:DataFrame = ???
val test:DataFrame = ???
val tokenizer = new Tokenizer()
.setInputCol("text")
.setOutputCol("words")
val hashingTF = new HashingTF()
.setNumFeatures(1000)
.setInputCol(tokenizer.getOutputCol)
.setOutputCol("features")
val lr = new LogisticRegression()
/*.add parameter*/
val pipeline = new Pipeline()
.setStages(Array(tokenizer, hashingTF, lr))
// Fit the pipeline to training documents.
val model = pipeline.fit(training)
model.transform(test)
Differents models
Same manner to create
the pipeline

42. Zeppelin
Introduction

43. Big picture
Zeppelin introduction

44. What it is about?
● “A web-based notebook that
enables interactive data analytics”
● 100% opensource
● Undergoing Incubation

45. Multi-purpose
● Data Ingestion
● Data Discovery
● Data Analytics
● Data Visualization &
Collaboration

46. Multiple Language
backend
● Scala
● shell
● python
● markdown
● your language by creation your
own interpreter

47. Data visualization
Easy way to build graph
from data

48. Demo

49. Merci