Machine Learning with Apache Spark

Machine Learning
With Apache Spark
CodeMash, Sandusky, Ohio, Jan 5-8, 2016
David Taieb
STSM-IBM Cloud Data Services

©2015 IBM Corporation
Introduction
David Taieb
david_taieb@us.ibm.com
Developer Advocate
IBM Cloud Data Services
Our mission:
We are here to help developers realize their most ambitious projects.
https://developer.ibm.com/clouddataservices/connect/

Big data, cloud and the rise of business Analytics
‣ Data being collected by enterprises
grows exponentially : ERP,
embedded systems (IOT)
‣ Cloud, with high availability and huge
capacity, make more data available
for analytics
‣ Big data and cloud create new
opportunities:
- Organizations: more effective decision-
making process, richer client interactions
- Business users: discover new insights,
better decision-making process
- Developers: access to diverse data sources
and new tools that increase productivity

Why Business Analytics with big data
“In God we trust.
All others bring data”
W. Edwards Deming
‣ Every day, companies make bet-the-business
decisions about their customers, competitors and
new products
‣ Time available for decision-making is shrinking
(sometimes real-time)
‣ As more and more companies go digital, data
becomes the world’s newest resource for
competitive advantage
‣ Decision making has moved from the elite few to
the empowered many
‣ Few organizations can keep pace with the
appetite for data

Business Analytics Types
Descriptive Analytics Predictive Analytics Prescriptive Analytics
Look at the reason for
past success or failure
What is probably going
to happen in the future?
What’s my best actions?
• Use interactive querying and
visualization to explore and
communicate data
• Discover insight and trends
• correlation between 2
seemingly unrelated
variables
• Data mining
• Generate hypothesis and
models
• Predict occurrence of future
events using probability
(confidence)
• Product recommendations
• Classification
• Help make the right decision
based on the data
• Find optimal solution to a
given problem

Taking Analytics a step further with Cognitive Systems
‣ Use natural language processing and machine learning algorithms to unlock knowledge
from massive amount of structured and unstructured data
Decide
• Ingest and analyze domain sources, info models
• Generate evidence based decisions with confidence
• Learn with new outcomes and actions
• e.g. - Next generation Apps  Probabilistic Apps
Ask
• Leverage vast amounts of data
• Ask questions for greater insights
• Natural language inquiries
• e.g. - Next generation Chat
Discover
• Find the rationale for given answers
• Prompt for inputs to yield improved responses
• Inspire considerations of new ideas
• e.g. - Next generation Search  Discovery
IBM Watson

IBM Cloud Data Services
Resources for developers to get, build, and analyze on the IBM Cloud

What is spark
Spark is an open source
in-memory
computing framework for
distributed data processing
and
iterative analysis
on massive data volumes

Spark Core Libraries
Spark Core
general compute engine, handles
distributed task dispatching, scheduling
and basic I/O functions
Spark
SQL
Spark
Streaming
Mllib
(machine
learning)
GraphX
(graph)
executes
SQL
statements
performs
streaming
analytics using
micro-batches
common
machine
learning and
statistical
algorithms
distributed
graph
processing
framework

Key reasons for interest in Spark
Open Source
Fast
distributed data
processing
Productive
Web Scale
•In-memory storage greatly reduces disk I/O
•Up to 100x faster in memory, 10x faster on disk
•Largest project and one of the most active on Apache
•Vibrant growing community of developers continuously improve code
base and extend capabilities
•Fast adoption in the enterprise (IBM, Databricks, etc…)
•Fault tolerant, seamlessly recompute lost data from hardware failure
•Scalable: easily increase number of worker nodes
•Flexible job execution: Batch, Streaming, Interactive
•Easily handle Petabytes of data without special code handling
•Compatible with existing Hadoop ecosystem
•Unified programming model across a range of use cases
•Rich and expressive apis hide complexities of parallel computing and worker node
management
•Support for Java, Scala, Python and R: less code written
•Include a set of core libraries that enable various analytic methods: Saprk SQL, Mllib, GraphX

IBM is all-in on its commitment to Spark
11
Foster
Community
Educate 1M+ data scientists
and engineers via online
courses
Sponsor AMPLab, creators and
evangelists of Spark
Infuse the
Portfolio
Integrate Spark throughout
portfolio
3,500 employees working
on Spark-related topics
Spark however customers
want it – standalone,
platform or products
Source: https://www-03.ibm.com/press/us/en/pressrelease/47107.wss
Launch Spark Technology Cluster
(STC), 300 engineers
Open source SystemML
Partner with databricks
Contribute to the
Core

Spark MLLib
‣ Extension to the Spark Core API that provide a library of easy to use Machine
learning algorithms.
‣ Highly scalable: Leverages Spark ability to work with massive amount of data
‣ Fast: Designed for parallel computing
‣ Cover common Machine Learning algorithms:
- Regression
- Classification
- Clustering
- Recommender Systems
- Text Analytics

What is Machine Learning and where is it used
‣Subfield of computer science that focuses on getting computers to
learn from data:
- Recognize patterns
- Make predictions
‣Example use:
- Spam filters
- Netflix recommendations
- Self-driving cars
- Watson
- …

Typical Machine Learning Flow diagram
Data
Acquisition
Data
Preparation
Data Annotation
(Ground Truth)
Model
Training
• Cleansing
• Shaping
• Enrichment
Model
Testing
Training
Set
Test
Set
Blind
Set
Iterative
Cross-Validation
Evaluate Performance and optimize model
Train Model

MLLib Algorithm Overview
• Predictive analytics
• Recommendations
• Collaborative Filtering
• Matrix Factorization
• Feature extraction and Transformation
• TF-IDF
• HashingTF
• Word2Vec
• StandardScaler
• Normalizer
• Model Evaluation/Metrics
• Binary Classification Metrics
• Multi Class Metrics
• Regression Metrics

Predictive analytics
Continuous Output Discrete Output
Supervised
Learning
(require Ground-Truth)
• Regression
- Linear
- Ridge
- Lasso
- Isotonic
• Decision Tree
• RandomForest
• GradientBoostedTree
• Classification
- Logistic Regression
- SVM
- NaiveBayes
• Decision Tree
• RandomForest
• GradientBoostedTree
• K-NN (available as add-on spark
package)
Unsupervised
Learning
(no Ground-Truth data required)
• Clustering
- KMeans
- Gaussian Mixture
• Dimensionality Reduction
- PCA
- SVD
• FP-Growth

Featured demo: Flight Delay Predictor
‣ Use training data collected from flight stats and enriched with weather observations from
“Insight for Weather” service on Bluemix
‣ Train multi-class classifier that, given and flight departure weather observations, can predict the
flight delay class:
- 0 = Canceled
- 1 = On Time
- 2 = Delay less than 2 hours
- 3 = Delay between 2 and 4 hours
- 4 = Delay more than 4 hours
‣ Provide metrics measurement for each algorithms
- Accuracy
- Precision
- Recall

Architecture
Weather
Simple Data
Pipes
Airports
Flight Schedules
Flight Status
Metadata
Training
Set
Test
Set
Blind
Set
Custom
Connector run
every 24 hours
Notebook

Get
‣ Identify data sources:
- flightstats.com: https://developer.flightstats.com
- Airport metadata: FS Code, geolocation,…
- Flight Schedules
- Flight Status
- Weather Observations
- Insight for Weather on Bluemix
‣ Storage:
- Cloudant
‣ Tool used:
- Simple Data Pipes custom connector to build Training, Test and Blind data set
‣ Constraints:
- Weather service provide past observations as far as 24 hours back only
- Flightstats API key is a 30 day trial version, limited to 20,000 calls only

Custom Pipes Connector to build training data set
https://developer.ibm.com/clouddataservices/simple-data-pipe/

Run every 24 hours
Because Weather service doesn’t return observations older than 24 hours, the data
set must be ran every 24 hours

Build: Explore the data with Notebook

Loading training data set

Build: Visualize and explore data set
Scatter plot of flights delays based on temperature in Departing and Arrival airports

Build: Visualize and explore data set
Scatter plot of flights delays based on wind speed in Departing and Arrival airports

Constraints
‣ Past weather observations provided by the “Insight for Weather” service have more details than
forecast data:
- Limit the number of features used to train the models to the intersections of the 2.
‣ Restrict the training data to weather forecast at departure and arrival airport
- Would adding weather data from various point in the route increase the model performance?
‣ Difficult to get enough representative data because I was using a trial account on flightstats
- Ideally, I would use more airports with better representative weather
‣ Didn’t use any categorical features
‣ For simplicity: Use IPython Notebook as the user interface
- Make the experience less compelling for Business users
- To avoid writing too much code in the Notebook, encapsulate some of the business logic in a Python library
- Doesn’t cover as much of the Spark API as Scala

Load labeled data RDD

Build: NaiveBayes Classification

Build: Decision Tree classification

Build: Random Forest classification

Build: Performance measurements
Load blind data

Build: Compare metrics between different
models

Naïve Bayes vs Decision Tree
‣ Probabilistic: compute the probability of a
data instance to be in a specific class
‣ Assume that each feature (variable) is
independent from the others
‣ Performance depends on the predictive
nature of the features (non predictive
features will affect the accuracy)
‣ Works well with low amount of training data.
Doesn’t need all the possibilities
‣ Doesn’t work with categorical features.
‣ Non-Probabilistic: partition the data into
subsets that best describe the variable
‣ The deeper the tree, the better the model fits
the data
‣ Watch out for overfiting: need to prune the tree
‣ Can handle categorical or continuous features
‣ No need for input to be scaled or standardized:
Set you features and go!
‣ Requires a lot of data covering all possibilities

Analyze: Run model

Code: Run Model

If you want to know more
‣https://developer.ibm.com/clouddataservices/
‣https://github.com/ibm-cds-labs/pipes-connector-flightstats
‣http://spark.apache.org/docs/latest/mllib-guide.html
‣https://console.ng.bluemix.net/data/analytics/

Machine Learning with Apache Spark

More Related Content

What's hot

Similar to Machine Learning with Apache Spark

More from IBM Cloud Data Services

Recently uploaded

Machine Learning with Apache Spark