Presented by David Taieb, Architect, IBM Cloud Data Services Along with Spark Streaming, Spark SQL and GraphX, MLLib is one of the four key architectural components of Spark. It provides easy-to-use (even for beginners), powerful Machine Learning APIs that are designed to work in parallel using Spark RDDs. In this session, we’ll introduce the different algorithms available in MLLib, e.g. supervised learning with classification (binary and multi class) and regression but also unsupervised learning with clustering (K-means) and recommendation systems. We’ll conclude the presentation with a deep dive on a sample machine learning application built with Spark MLLib that predicts whether a scheduled flight will be delayed or not. This application trains a model using data from real flight information. The labeled flight data is combined with weather data from the “Insight for Weather” service available on IBM Bluemix Cloud Platform to form the training, test and blind data. Even if you are not a black belt in machine learning, you will learn in this session how to leverage powerful Machine Learning algorithms available in Spark to build interesting predictive and prescriptive applications. About the Speaker: For the last 4 years, David has been the lead architect for the Watson Core UI & Tooling team based in Littleton, Massachusetts. During that time, he led the design and development of a Unified Tooling Platform to support all the Watson Tools including accuracy analysis, test experiments, corpus ingestion, and training data generation. Before that, he was the lead architect for the Domino Server OSGi team responsible for integrating the eXpeditor J2EE Web Container in Domino and building first class APIs for the developer community. He started with IBM in 1996, working on various globalization technologies and products including Domino Global Workbench (used to develop multilingual Notes/Domino NSF applications) and a multilingual Content Management system for the Websphere Application Server. David enjoys sharing his experience by speaking at conferences. You’ll find him at various events like the Unicode conference, Eclipsecon, and Lotusphere. He’s also passionate about building tools that help improve developer productivity and overall experience.

1. Machine Learning
With Apache Spark
CodeMash, Sandusky, Ohio, Jan 5-8, 2016
David Taieb
STSM-IBM Cloud Data Services

2. ©2015 IBM Corporation
Introduction
David Taieb
david_taieb@us.ibm.com
Developer Advocate
IBM Cloud Data Services
Our mission:
We are here to help developers realize their most ambitious projects.
https://developer.ibm.com/clouddataservices/connect/

3. ©2015 IBM Corporation
Big data, cloud and the rise of business Analytics
‣ Data being collected by enterprises
grows exponentially : ERP,
embedded systems (IOT)
‣ Cloud, with high availability and huge
capacity, make more data available
for analytics
‣ Big data and cloud create new
opportunities:
- Organizations: more effective decision-
making process, richer client interactions
- Business users: discover new insights,
better decision-making process
- Developers: access to diverse data sources
and new tools that increase productivity

4. ©2015 IBM Corporation
Why Business Analytics with big data
“In God we trust.
All others bring data”
W. Edwards Deming
‣ Every day, companies make bet-the-business
decisions about their customers, competitors and
new products
‣ Time available for decision-making is shrinking
(sometimes real-time)
‣ As more and more companies go digital, data
becomes the world’s newest resource for
competitive advantage
‣ Decision making has moved from the elite few to
the empowered many
‣ Few organizations can keep pace with the
appetite for data

5. Business Analytics Types
Descriptive Analytics Predictive Analytics Prescriptive Analytics
Look at the reason for
past success or failure
What is probably going
to happen in the future?
What’s my best actions?
• Use interactive querying and
visualization to explore and
communicate data
• Discover insight and trends
• correlation between 2
seemingly unrelated
variables
• Data mining
• Generate hypothesis and
models
• Predict occurrence of future
events using probability
(confidence)
• Product recommendations
• Classification
• Help make the right decision
based on the data
• Find optimal solution to a
given problem

6. Taking Analytics a step further with Cognitive Systems
‣ Use natural language processing and machine learning algorithms to unlock knowledge
from massive amount of structured and unstructured data
Decide
• Ingest and analyze domain sources, info models
• Generate evidence based decisions with confidence
• Learn with new outcomes and actions
• e.g. - Next generation Apps  Probabilistic Apps
Ask
• Leverage vast amounts of data
• Ask questions for greater insights
• Natural language inquiries
• e.g. - Next generation Chat
Discover
• Find the rationale for given answers
• Prompt for inputs to yield improved responses
• Inspire considerations of new ideas
• e.g. - Next generation Search  Discovery
IBM Watson

7. IBM Cloud Data Services
Resources for developers to get, build, and analyze on the IBM Cloud

8. ©2015 IBM Corporation
What is spark
Spark is an open source
in-memory
computing framework for
distributed data processing
and
iterative analysis
on massive data volumes

9. ©2015 IBM Corporation
Spark Core Libraries
Spark Core
general compute engine, handles
distributed task dispatching, scheduling
and basic I/O functions
Spark
SQL
Spark
Streaming
Mllib
(machine
learning)
GraphX
(graph)
executes
SQL
statements
performs
streaming
analytics using
micro-batches
common
machine
learning and
statistical
algorithms
distributed
graph
processing
framework

10. ©2015 IBM Corporation
Key reasons for interest in Spark
Open Source
Fast
distributed data
processing
Productive
Web Scale
•In-memory storage greatly reduces disk I/O
•Up to 100x faster in memory, 10x faster on disk
•Largest project and one of the most active on Apache
•Vibrant growing community of developers continuously improve code
base and extend capabilities
•Fast adoption in the enterprise (IBM, Databricks, etc…)
•Fault tolerant, seamlessly recompute lost data from hardware failure
•Scalable: easily increase number of worker nodes
•Flexible job execution: Batch, Streaming, Interactive
•Easily handle Petabytes of data without special code handling
•Compatible with existing Hadoop ecosystem
•Unified programming model across a range of use cases
•Rich and expressive apis hide complexities of parallel computing and worker node
management
•Support for Java, Scala, Python and R: less code written
•Include a set of core libraries that enable various analytic methods: Saprk SQL, Mllib, GraphX

11. ©2015 IBM Corporation
IBM is all-in on its commitment to Spark
11
Foster
Community
Educate 1M+ data scientists
and engineers via online
courses
Sponsor AMPLab, creators and
evangelists of Spark
Infuse the
Portfolio
Integrate Spark throughout
portfolio
3,500 employees working
on Spark-related topics
Spark however customers
want it – standalone,
platform or products
Source: https://www-03.ibm.com/press/us/en/pressrelease/47107.wss
Launch Spark Technology Cluster
(STC), 300 engineers
Open source SystemML
Partner with databricks
Contribute to the
Core

12. ©2015 IBM Corporation
Spark MLLib
‣ Extension to the Spark Core API that provide a library of easy to use Machine
learning algorithms.
‣ Highly scalable: Leverages Spark ability to work with massive amount of data
‣ Fast: Designed for parallel computing
‣ Cover common Machine Learning algorithms:
- Regression
- Classification
- Clustering
- Recommender Systems
- Text Analytics

13. ©2015 IBM Corporation
What is Machine Learning and where is it used
‣Subfield of computer science that focuses on getting computers to
learn from data:
- Recognize patterns
- Make predictions
‣Example use:
- Spam filters
- Netflix recommendations
- Self-driving cars
- Watson
- …

14. ©2015 IBM Corporation
Typical Machine Learning Flow diagram
Data
Acquisition
Data
Preparation
Data Annotation
(Ground Truth)
Model
Training
• Cleansing
• Shaping
• Enrichment
Model
Testing
Training
Set
Test
Set
Blind
Set
Iterative
Cross-Validation
Evaluate Performance and optimize model
Train Model

15. ©2015 IBM Corporation
MLLib Algorithm Overview
• Predictive analytics
• Recommendations
• Collaborative Filtering
• Matrix Factorization
• Feature extraction and Transformation
• TF-IDF
• HashingTF
• Word2Vec
• StandardScaler
• Normalizer
• Model Evaluation/Metrics
• Binary Classification Metrics
• Multi Class Metrics
• Regression Metrics

16. ©2015 IBM Corporation
Predictive analytics
Continuous Output Discrete Output
Supervised
Learning
(require Ground-Truth)
• Regression
- Linear
- Ridge
- Lasso
- Isotonic
• Decision Tree
• RandomForest
• GradientBoostedTree
• Classification
- Logistic Regression
- SVM
- NaiveBayes
• Decision Tree
• RandomForest
• GradientBoostedTree
• K-NN (available as add-on spark
package)
Unsupervised
Learning
(no Ground-Truth data required)
• Clustering
- KMeans
- Gaussian Mixture
• Dimensionality Reduction
- PCA
- SVD
• FP-Growth

17. ©2015 IBM Corporation
Featured demo: Flight Delay Predictor
‣ Use training data collected from flight stats and enriched with weather observations from
“Insight for Weather” service on Bluemix
‣ Train multi-class classifier that, given and flight departure weather observations, can predict the
flight delay class:
- 0 = Canceled
- 1 = On Time
- 2 = Delay less than 2 hours
- 3 = Delay between 2 and 4 hours
- 4 = Delay more than 4 hours
‣ Provide metrics measurement for each algorithms
- Accuracy
- Precision
- Recall

18. ©2015 IBM Corporation
Architecture
Weather
Simple Data
Pipes
Airports
Flight Schedules
Flight Status
Metadata
Training
Set
Test
Set
Blind
Set
Custom
Connector run
every 24 hours
Notebook

19. ©2015 IBM Corporation
Get
‣ Identify data sources:
- flightstats.com: https://developer.flightstats.com
- Airport metadata: FS Code, geolocation,…
- Flight Schedules
- Flight Status
- Weather Observations
- Insight for Weather on Bluemix
‣ Storage:
- Cloudant
‣ Tool used:
- Simple Data Pipes custom connector to build Training, Test and Blind data set
‣ Constraints:
- Weather service provide past observations as far as 24 hours back only
- Flightstats API key is a 30 day trial version, limited to 20,000 calls only

20. ©2015 IBM Corporation
Custom Pipes Connector to build training data set
https://developer.ibm.com/clouddataservices/simple-data-pipe/

21. ©2015 IBM Corporation
Run every 24 hours
Because Weather service doesn’t return observations older than 24 hours, the data
set must be ran every 24 hours

22. ©2015 IBM Corporation
Build: Explore the data with Notebook

23. ©2015 IBM Corporation
Loading training data set

24. ©2015 IBM Corporation
Build: Visualize and explore data set
Scatter plot of flights delays based on temperature in Departing and Arrival airports

25. ©2015 IBM Corporation
Build: Visualize and explore data set
Scatter plot of flights delays based on wind speed in Departing and Arrival airports

26. ©2015 IBM Corporation
Constraints
‣ Past weather observations provided by the “Insight for Weather” service have more details than
forecast data:
- Limit the number of features used to train the models to the intersections of the 2.
‣ Restrict the training data to weather forecast at departure and arrival airport
- Would adding weather data from various point in the route increase the model performance?
‣ Difficult to get enough representative data because I was using a trial account on flightstats
- Ideally, I would use more airports with better representative weather
‣ Didn’t use any categorical features
‣ For simplicity: Use IPython Notebook as the user interface
- Make the experience less compelling for Business users
- To avoid writing too much code in the Notebook, encapsulate some of the business logic in a Python library
- Doesn’t cover as much of the Spark API as Scala

27. ©2015 IBM Corporation
Load labeled data RDD

28. ©2015 IBM Corporation
Load labeled data RDD

29. ©2015 IBM Corporation
Build: NaiveBayes Classification

30. ©2015 IBM Corporation
Build: Decision Tree classification

31. ©2015 IBM Corporation
Build: Random Forest classification

32. ©2015 IBM Corporation
Build: Performance measurements
Load blind data

33. ©2015 IBM Corporation
Build: Compare metrics between different
models

34. ©2015 IBM Corporation
Naïve Bayes vs Decision Tree
‣ Probabilistic: compute the probability of a
data instance to be in a specific class
‣ Assume that each feature (variable) is
independent from the others
‣ Performance depends on the predictive
nature of the features (non predictive
features will affect the accuracy)
‣ Works well with low amount of training data.
Doesn’t need all the possibilities
‣ Doesn’t work with categorical features.
‣ Non-Probabilistic: partition the data into
subsets that best describe the variable
‣ The deeper the tree, the better the model fits
the data
‣ Watch out for overfiting: need to prune the tree
‣ Can handle categorical or continuous features
‣ No need for input to be scaled or standardized:
Set you features and go!
‣ Requires a lot of data covering all possibilities

35. ©2015 IBM Corporation
Analyze: Run model

36. ©2015 IBM Corporation
Code: Run Model

37. ©2015 IBM Corporation
If you want to know more
‣https://developer.ibm.com/clouddataservices/
‣https://github.com/ibm-cds-labs/pipes-connector-flightstats
‣http://spark.apache.org/docs/latest/mllib-guide.html
‣https://console.ng.bluemix.net/data/analytics/

38. ©2015 IBM Corporation