An example-driven presentation on implementing Machine Learning with Apache Spark.

1. Service Symphony Ltd
Apache Spark
Machine Learning by Example
Meeraj Kunnumpurath
25th of February 2017
1

2. Introduction
❖ Working as technologist and software architect for couple of decades, at number of leading
financial institutions in the UK
❖ Authored a number books on Enterprise Java, Web Services and SOA
❖ Spoken at a number of technology conferences
❖ Founded Service Symphony Ltd in 2009 serving leading financial services customers
building mission critical middleware
❖ Engineer with a keen interest in ML, AI and Data Science
❖ Blog: http://www.servicesymphony.com/blog
❖ Email: meeraj@servicesymphony.com
❖ Presentation: https://www.slideshare.net/MeerajKunnumpurath/machine-learning-by-
example-apache-spark
❖ GitHub: https://github.com/kunnum/sandbox/tree/master/notebooks
2

3. Agenda
❖ Introduction to using ML with Apache Spark
❖ Hands-on example driven approach
❖ Not a deep dive into Apache Spark Architecture
❖ Neither a deep dive into ML algorithms
❖ Examples built using Apache Zeppelin
❖ Some of the examples are from Spark ASF
documentation
3

4. Apache Spark - Overview
❖ Open source large scale distributed data processing fabric
❖ Offers multiple components addressing different facets of data science for big and
fast data processing, ML, analytics and data ingestion
❖ Ability to process large amount of data in memory spanning multiple process
spaces
❖ Initially started as a research project in UC Berkeley
❖ Originally released under BSD, top level ASF project licensed under ASL 2.0 since
2014
❖ One of the most active open source project, arguably the most active ASF project
❖ Adopted, extended and commercialised by multiple vendors playing in the data
science realm
4

5. Apache Spark - Architecture
5

6. Apache Spark - Architecture
6

7. Apache Spark - Architecture
7

8. Scala - Spark Natural Transition
❖ Interest in Spark stemmed from deep interest in Scala
and functional programming
❖ Data processing echo system built around Scala, with a
strong synergy in Scala’s design motivations
❖ Extends Scala’s idiomatic functional programming
model to transcend beyond process boundaries
❖ Spark RDDs - Scala collections on steroids
8

9. Spark - Scala Notebook
9

10. Spark - Scala Notebook
10

11. ML Components
11

12. ML Components
❖ Data Structures
❖ Vectors and Matrices
❖ Data Frames
❖ Feature Extractors and Transformers
❖ Estimators
❖ Models
❖ Pipelines
❖ Evaluators
❖ Tuning Aids
12

13. ML Components - Notebook
13

14. ML Components - Notebook
14

15. ML Components - Notebook
15

16. ML Components - Notebook
16

17. Spark ML - Pipeline Architecture
❖ Dataframe
❖ Estimator
❖ Transformer
❖ Pipeline
❖ Parameter
17

18. Spark ML - Pipeline Architecture
18
Training time flow
Pipeline in estimator mode
Pipeline.fit()
Creates a pipeline model

19. Spark ML - Pipeline Architecture
19
Test time flow
Pipeline in transformer mode
PipelineModel.transform()
Creates dataframe with augmented prediction columns

20. ML Pipeline Notebook
20

21. ML Pipeline Notebook
21

22. ML Pipeline Notebook
22

23. ML Pipeline Notebook
23

24. ML Pipeline Notebook
24

25. Regression
❖ Supervised Learning Algorithm for predicting continuous labels
❖ Multiple Algorithms
❖ Linear Regression
❖ Generalised Linear Regression
❖ Decision Tree Regression
❖ Random Forest Regression
❖ Gradient Boosted Tree Regression
❖ Survival Regression
❖ Isotonic Regression
❖ Works with input feature vectors and labelled points
25

26. Regression
26

27. Linear Regression - Notebook
27

28. Linear Regression - Notebook
28

29. Linear Regression - Notebook
29

30. Linear Regression - Notebook
30

31. Linear Regression - Notebook
31

32. Classification
❖ Supervised learning for predicting discrete labels
❖ Multiple algorithms
❖ Binomial and polynomial logistic regression
❖ Decision tree classifier
❖ Random forest classifier
❖ Gradient boosted tree classifier
❖ Multi-layer neural network classifier
❖ Naive Bayes Classifier
32

33. Classification
33

34. Classification - Notebook
34

35. Classification - Notebook
35

36. Classification - Notebook
36

37. Classification - Notebook
37

38. Classification - Notebook
38

39. Classification - Notebook
39

40. Classification - Notebook
40

41. Classification - Notebook
41

42. Clustering
❖ Unsupervised learning algorithm based on similarity
vectors
❖ Multiple algorithms
❖ K-Means Clustering
❖ LDA - Latent Dirichlet Allocation
❖ Bisecting K-Means
❖ Gaussian Mixture Model
42

43. Clustering
43

44. Clustering - Notebook
44

45. Clustering - Notebook
45

46. Clustering - Notebook
46

47. Clustering - Notebook
47

48. Clustering - Notebook
48

49. Clustering - Notebook
49

50. Clustering - Notebook
50

51. Clustering - Notebook
51

52. Collaborative Filtering
❖ Commonly used for recommender systems
❖ Uses ALS (Alternating Least Squares) to learn latent
factors in user to item association
❖ Default assumption is based on explicit feedback for
matrix factorization
❖ You an explicitly enable implicit preferences
52

53. Collaborative Filtering
53

54. Collaborative Filtering - Notebook
54

55. Collaborative Filtering - Notebook
55

56. Collaborative Filtering - Notebook
56

57. Collaborative Filtering - Notebook
57

58. Collaborative Filtering - Notebook
58

59. Collaborative Filtering - Notebook
59

60. Model Tuning
❖ API to tune an individual estimator or the entire
pipeline using a normalised parameter model
❖ API to support k-fold cross validation
❖ API to evaluate performance on linear regression, as
well as binomial and polynomial classification
❖ API for performing training validation split
60

61. Model Tuning - Notebook
61

62. Model Tuning - Notebook
62

63. Model Tuning - Notebook
63

64. Model Tuning - Notebook
64

65. Model Tuning - Notebook
65

66. Model Tuning - Notebook
66

67. Questions
67