SlideShare a Scribd company logo
Multinomial Logistic Regression
with Apache Spark
DB Tsai
Machine Learning Engineer
May 1st
, 2014
Machine Learning with Big Data
● Hadoop MapReduce solutions
● MapReduce is scaling well for batch processing
● However, lots of machine learning algorithms
are iterative by nature.
● There are lots of tricks people do, like training
with subsamples of data, and then average the
models. Why big data with approximation?
+ =
Lightning-fast cluster computing
● Empower users to iterate through the data by
utilizing the in-memory cache.
● Logistic regression runs
up to 100x faster than
Hadoop M/R in memory.
● We're able to train
exact model without doing any approximation.
Binary Logistic Regression
d =
ax1+bx2+cx0
√a2
+b2
where x0=1
P( y=1∣⃗x , ⃗w)=
exp(d )
1+exp(d )
=
exp(⃗x ⃗w)
1+exp(⃗x ⃗w)
P ( y=0∣⃗x , ⃗w)=
1
1+exp(⃗x ⃗w)
log
P ( y=1∣⃗x , ⃗w)
P( y=0∣⃗x , ⃗w)
=⃗x ⃗w
w0=
c
√a2
+b2
w1=
a
√a2
+b2
w2=
b
√a2
+b2
wherew0 iscalled as intercept
Training Binary Logistic Regression
● Maximum Likelihood estimation
From a training data
and labels
● We want to find that maximizes the
likelihood of data defined by
● We can take log of the equation, and minimize
it instead. The Log-Likelihood becomes the
loss function.
X =( ⃗x1 , ⃗x2 , ⃗x3 ,...)
Y =( y1, y2, y3, ...)
⃗w
L( ⃗w , ⃗x1, ... , ⃗xN )=P ( y1∣⃗x1 , ⃗w)P ( y2∣⃗x2 , ⃗w)... P( yN∣ ⃗xN , ⃗w)
l( ⃗w ,⃗x)=log P(y1∣⃗x1 , ⃗w)+log P( y2∣⃗x2 , ⃗w)...+log P( yN∣ ⃗xN , ⃗w)
Optimization
● First Order Minimizer
Require loss, gradient of loss function
– Gradient Decent is step size
– Limited-memory BFGS (L-BFGS)
– Orthant-Wise Limited-memory Quasi-Newton
(OWLQN)
– Coordinate Descent (CD)
– Trust Region Newton Method (TRON)
● Second Order Minimizer
Require loss, gradient and hessian of loss function
– Newton-Raphson, quadratic convergence. Fast!
● Ref: Journal of Machine Learning Research 11 (2010) 3183-
3234, Chih-Jen Lin et al.
⃗wn+1=⃗wn−γ ⃗G , γ
⃗wn+1=⃗wn−H−1 ⃗G
Problem of Second Order Minimizer
● Scale horizontally (the numbers of training
data) by leveraging Spark to parallelize this
iterative optimization process.
● Don't scale vertically (the numbers of training
features). Dimension of Hessian is
● Recent applications from document
classification and computational linguistics are
of this type.
dim(H )=[(k−1)(n+1)]
2
where k is numof class ,nis num of features
L-BFGS
● It's a quasi-Newton method.
● Hessian matrix of second derivatives doesn't
need to be evaluated directly.
● Hessian matrix is approximated using gradient
evaluations.
● It converges a way faster than the default
optimizer in Spark, Gradient Decent.
● We love open source! Alpine Data Labs
contributed our L-BFGS to Spark, and it's
already merged in Spark-1157.
Training Binary Logistic Regression
l( ⃗w ,⃗x) = ∑
k=1
N
log P( yk∣⃗xk , ⃗w)
= ∑k=1
N
yk log P ( yk=1∣⃗xk , ⃗w)+(1−yk )log P ( yk =0∣⃗xk , ⃗w)
= ∑k=1
N
yk log
exp( ⃗xk ⃗w)
1+exp( ⃗xk ⃗w)
+(1−yk )log
1
1+exp( ⃗xk ⃗w)
= ∑k=1
N
yk ⃗xk ⃗w−log(1+exp( ⃗xk ⃗w))
Gradient : Gi (⃗w ,⃗x)=
∂l( ⃗w ,⃗x)
∂ wi
=∑k=1
N
yk xki−
exp( ⃗xk ⃗w)
1+exp( ⃗xk ⃗w)
xki
Hessian: H ij (⃗w ,⃗x)=
∂∂l( ⃗w ,⃗x)
∂wi ∂w j
=−∑k=1
N
exp( ⃗xk ⃗w)
(1+exp( ⃗xk ⃗w))2
xki xkj
Overfitting
P ( y=1∣⃗x , ⃗w)=
exp(zd )
1+exp(zd )
=
exp(⃗x ⃗w)
1+exp(⃗x ⃗w)
Regularization
● The loss function becomes
● The loss function of regularizer doesn't
depend on data. Common regularizers are
– L2 Regularization:
– L1 Regularization:
● L1 norm is not differentiable at zero!
ltotal (⃗w ,⃗x)=lmodel (⃗w ,⃗x)+lreg( ⃗w)
lreg (⃗w)=λ∑i=1
N
wi
2
lreg (⃗w)=λ∑i=1
N
∣wi∣
⃗G (⃗w ,⃗x)total=⃗G(⃗w ,⃗x)model+⃗G( ⃗w)reg
̄H ( ⃗w ,⃗x)total= ̄H (⃗w ,⃗x)model+ ̄H (⃗w)reg
Mini School of Spark APIs
● map(func) : Return a new distributed dataset
formed by passing each element of the source
through a function func.
● reduce(func) : Aggregate the elements of the
dataset using a function func (which takes two
arguments and returns one). The function
should be commutative and associative so
that it can be computed correctly in parallel.
No “key” thing here compared with Hadoop
M/R.
Example – No More “Word Count”!
Let's compute the mean of numbers.
3.0
2.0
1.0
5.0
Executor 1
Executor 2
map
map
(1.0, 1)
(5.0, 1)
(3.0, 1)
(2.0, 1)
reduce
(5.0, 2)
reduce (11.0, 4)
Shuffle
● The previous example using map(func) will
create new tuple object, which may cause
Garbage Collection issue.
● Like the combiner in Hadoop Map Reduce,
for each tuple emitted from map(func), there is
no guarantee that they will be combined
locally in the same executor by reduce(func).
It may increase the traffic in shuffle phase.
● In Hadoop, we address this by in-mapper
combiner or aggregating the result in global
variables which have scope to entire partition.
The same approach can be used in Spark.
Mini School of Spark APIs
● mapPartitions(func) : Similar to map, but runs
separately on each partition (block) of the
RDD, so func must be of type Iterator[T] =>
Iterator[U] when running on an RDD of type T.
● This API allows us to have global variables on
entire partition to aggregate the result locally
and efficiently.
Better Mean
of Numbers
Implementation
3.0
2.0
Executor 1
reduce (11.0, 4)
Shuffle
var counts = 0
var sum = 0.0
loop
var counts = 2
var sum = 5.0
3.0
2.0
(5.0, 2)
1.0
5.0
Executor 2
var counts = 0
var sum = 0.0
loop
var counts = 2
var sum = 6.0
1.0
5.0
(6.0, 2)
mapPartitions
More Idiomatic Scala Implementation
● aggregate(zeroValue: U)(seqOp: (U, T) => U,
combOp: (U, U) => U): U zeroValue is a neutral "zero value"
with type U for initialization. This is
analogous to
in previous example.
var counts = 0
var sum = 0.0
seqOp is function taking
(U, T) => U, where U is aggregator
initialized as zeroValue. T is each
line of values in RDD. This is
analogous to mapPartition in
previous example.
combOp is function taking
(U, U) => U. This is essentially
combing the results between
different executors. The
functionality is the same as
reduce(func) in previous example.
Approach of Parallelization in Spark
Spark Driver
JVM
Time
Spark Executor 1
JVM
Spark Executor 2
JVM
1) Find available resources in
cluster, and launch executor JVMs.
Also initialize the weights.
2) Ask executors to load the data
into executors' JVMs
2) Trying to load the data into memory. If the data is bigger
than memory, it will be partial cached.
(The data locality from source will be taken care by Spark)
3) Ask executors to compute loss, and
gradient of each training sample
(each row) given the current weights.
Get the aggregated results after
the Reduce Phase in executors.
5) If the regularization is enabled,
compute the loss and gradient of
regularizer in driver since it doesn't
depend on training data but only
depends on weights. Add them into
the results from executors.
3) Map Phase:
Compute the loss and gradient of each row of training data
locally given the weights obtained from the driver. Can either
emit each result or sum them up in local aggregators.
4) Reduce Phase:
Sum up the losses and gradients emitted from the Map Phase
Taking a rest!
6) Plug the loss and gradient from
model and regularizer into optimizer
to get the new weights. If the
differences of weights and losses
are larger than criteria,
GO BACK TO 3)
Taking a rest!
7) Finish the model training! Taking a rest!
Step 3) and 4)
● This is the implementation of step 3) and 4) in
MLlib before Spark 1.0
● gradient can have implementations of
Logistic Regression, Linear Regression, SVM,
or any customized cost function.
● Each training data will create new “grad”
object after gradient.compute.
Step 3) and 4) with mapPartitions
Step 3) and 4) with aggregate
● This is the implementation of step 3) and 4) in
MLlib in coming Spark 1.0
● No unnecessary object creation! It's helpful
when we're dealing with large features training
data. GC will not kick in the executor JVMs.
Extension to Multinomial Logistic Regression
● In Binary Logistic Regression
● For K classes multinomial problem where labels
ranged from [0, K-1], we can generalize it via
● The model, weights becomes
(K-1)(N+1) matrix, where N is number of features.
log
P( y=1∣⃗x , ̄w)
P ( y=0∣⃗x , ̄w)
=⃗x ⃗w1
log
P ( y=2∣⃗x , ̄w)
P ( y=0∣⃗x , ̄w)
=⃗x ⃗w2
...
log
P ( y=K −1∣⃗x , ̄w)
P ( y=0∣⃗x , ̄w)
=⃗x ⃗wK −1
log
P( y=1∣⃗x , ⃗w)
P ( y=0∣⃗x , ⃗w)
=⃗x ⃗w
̄w=(⃗w1, ⃗w2, ... , ⃗wK −1)T
P( y=0∣⃗x , ̄w)=
1
1+∑
i=1
K −1
exp(⃗x ⃗wi )
P( y=1∣⃗x , ̄w)=
exp(⃗x ⃗w2)
1+∑
i=1
K −1
exp(⃗x ⃗wi)
...
P( y=K −1∣⃗x , ̄w)=
exp(⃗x ⃗wK −1)
1+∑i=1
K −1
exp(⃗x ⃗wi )
Training Multinomial Logistic Regression
l( ̄w ,⃗x) = ∑
k=1
N
log P( yk∣⃗xk , ̄w)
= ∑k=1
N
α( yk)log P( y=0∣⃗xk , ̄w)+(1−α( yk ))log P ( yk∣⃗xk , ̄w)
= ∑k=1
N
α( yk)log
1
1+∑
i=1
K −1
exp(⃗x ⃗wi)
+(1−α( yk ))log
exp(⃗x ⃗w yk
)
1+∑
i=1
K−1
exp(⃗x ⃗wi)
= ∑
k=1
N
(1−α( yk ))⃗x ⃗wyk
−log(1+∑
i=1
K−1
exp(⃗x ⃗wi))
Gradient : Gij ( ̄w ,⃗x)=
∂l( ̄w ,⃗x)
∂wij
=∑k=1
N
(1−α( yk )) xkj δi , yk
−
exp( ⃗xk ⃗w)
1+exp( ⃗xk ⃗w)
xkj
α( yk )=1 if yk=0
α( yk )=0 if yk≠0
Define:
Note that the first index “i” is for classes, and the second index “j” is for features.
Hessian: H ij ,lm( ̄w ,⃗x)=
∂∂l ( ̄w ,⃗x)
∂wij ∂wlm
0 5 10 15 20 25 30 35
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
Logistic Regression with a9a Dataset (11M rows, 123 features, 11% non-zero elements)
16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster
L-BFGS Dense Features
L-BFGS Sparse Features
GD Sparse Features
GD Dense Features
Seconds
Log-Likelihood/NumberofSamples
a9a Dataset Benchmark
a9a Dataset Benchmark
-1 1 3 5 7 9 11 13 15
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
Logistic Regression with a9a Dataset (11M rows, 123 features, 11% non-zero elements)
16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster
L-BFGS
GD
Iterations
Log-Likelihood/NumberofSamples
0 5 10 15 20 25 30
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Logistic Regression with rcv1 Dataset (6.8M rows, 677,399 features, 0.15% non-zero elements)
16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster
LBFGS Sparse Vector
GD Sparse Vector
Second
Log-Likelihood/NumberofSamples
rcv1 Dataset Benchmark
news20 Dataset Benchmark
0 10 20 30 40 50 60 70 80
0
0.2
0.4
0.6
0.8
1
1.2
Logistic Regression with news20 Dataset (0.14M rows, 1,355,191 features, 0.034% non-zero elements)
16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster
LBFGS Sparse Vector
GD Sparse Vector
Second
Log-Likelihood/NumberofSamples
Alpine Demo
Alpine Demo
Alpine Demo
Conclusion
● Alpine supports state of the art Cloudera CDH5
● Spark runs programs 100x faster than Hadoop
● Spark turns iterative big data machine learning problems
into single machine problems
● MLlib provides lots of state of art machine learning
implementations, e.g. K-means, linear regression,
logistic regression, ALS, collaborative filtering, and
Naive Bayes, etc.
● Spark provides ease of use APIs in Java, Scala, or
Python, and interactive Scala and Python shell
● Spark is Apache project, and it's the most active big data
open source project nowadays.

More Related Content

What's hot

Iceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data AnalyticsIceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data Analytics
Alluxio, Inc.
 
A Deep Dive into Query Execution Engine of Spark SQL
A Deep Dive into Query Execution Engine of Spark SQLA Deep Dive into Query Execution Engine of Spark SQL
A Deep Dive into Query Execution Engine of Spark SQL
Databricks
 
Apache Spark - Dataframes & Spark SQL - Part 1 | Big Data Hadoop Spark Tutori...
Apache Spark - Dataframes & Spark SQL - Part 1 | Big Data Hadoop Spark Tutori...Apache Spark - Dataframes & Spark SQL - Part 1 | Big Data Hadoop Spark Tutori...
Apache Spark - Dataframes & Spark SQL - Part 1 | Big Data Hadoop Spark Tutori...
CloudxLab
 
Google Bigtable Paper Presentation
Google Bigtable Paper PresentationGoogle Bigtable Paper Presentation
Google Bigtable Paper Presentation
vanjakom
 
How Pulsar Stores Your Data - Pulsar Summit NA 2021
How Pulsar Stores Your Data - Pulsar Summit NA 2021How Pulsar Stores Your Data - Pulsar Summit NA 2021
How Pulsar Stores Your Data - Pulsar Summit NA 2021
StreamNative
 
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveHive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
DataWorks Summit
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
Ryan Blue
 
Linear algebra
Linear algebraLinear algebra
Linear algebra
Sungbin Lim
 
HDFS Analysis for Small Files
HDFS Analysis for Small FilesHDFS Analysis for Small Files
HDFS Analysis for Small Files
DataWorks Summit/Hadoop Summit
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
DataWorks Summit/Hadoop Summit
 
Cardinality Estimation through Histogram in Apache Spark 2.3 with Ron Hu and ...
Cardinality Estimation through Histogram in Apache Spark 2.3 with Ron Hu and ...Cardinality Estimation through Histogram in Apache Spark 2.3 with Ron Hu and ...
Cardinality Estimation through Histogram in Apache Spark 2.3 with Ron Hu and ...
Databricks
 
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
StreamNative
 
[232] TensorRT를 활용한 딥러닝 Inference 최적화
[232] TensorRT를 활용한 딥러닝 Inference 최적화[232] TensorRT를 활용한 딥러닝 Inference 최적화
[232] TensorRT를 활용한 딥러닝 Inference 최적화
NAVER D2
 
Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1
Sadayuki Furuhashi
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in SparkSpark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Bo Yang
 
RDD
RDDRDD
Hadoop
Hadoop Hadoop
Hadoop
ABHIJEET RAJ
 
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/AvroThe Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
Databricks
 
Enabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkEnabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache Spark
Kazuaki Ishizaki
 
Locality sensitive hashing
Locality sensitive hashingLocality sensitive hashing
Locality sensitive hashing
Sameera Horawalavithana
 

What's hot (20)

Iceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data AnalyticsIceberg + Alluxio for Fast Data Analytics
Iceberg + Alluxio for Fast Data Analytics
 
A Deep Dive into Query Execution Engine of Spark SQL
A Deep Dive into Query Execution Engine of Spark SQLA Deep Dive into Query Execution Engine of Spark SQL
A Deep Dive into Query Execution Engine of Spark SQL
 
Apache Spark - Dataframes & Spark SQL - Part 1 | Big Data Hadoop Spark Tutori...
Apache Spark - Dataframes & Spark SQL - Part 1 | Big Data Hadoop Spark Tutori...Apache Spark - Dataframes & Spark SQL - Part 1 | Big Data Hadoop Spark Tutori...
Apache Spark - Dataframes & Spark SQL - Part 1 | Big Data Hadoop Spark Tutori...
 
Google Bigtable Paper Presentation
Google Bigtable Paper PresentationGoogle Bigtable Paper Presentation
Google Bigtable Paper Presentation
 
How Pulsar Stores Your Data - Pulsar Summit NA 2021
How Pulsar Stores Your Data - Pulsar Summit NA 2021How Pulsar Stores Your Data - Pulsar Summit NA 2021
How Pulsar Stores Your Data - Pulsar Summit NA 2021
 
Hive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep DiveHive + Tez: A Performance Deep Dive
Hive + Tez: A Performance Deep Dive
 
Parquet performance tuning: the missing guide
Parquet performance tuning: the missing guideParquet performance tuning: the missing guide
Parquet performance tuning: the missing guide
 
Linear algebra
Linear algebraLinear algebra
Linear algebra
 
HDFS Analysis for Small Files
HDFS Analysis for Small FilesHDFS Analysis for Small Files
HDFS Analysis for Small Files
 
How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...How to understand and analyze Apache Hive query execution plan for performanc...
How to understand and analyze Apache Hive query execution plan for performanc...
 
Cardinality Estimation through Histogram in Apache Spark 2.3 with Ron Hu and ...
Cardinality Estimation through Histogram in Apache Spark 2.3 with Ron Hu and ...Cardinality Estimation through Histogram in Apache Spark 2.3 with Ron Hu and ...
Cardinality Estimation through Histogram in Apache Spark 2.3 with Ron Hu and ...
 
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
Security and Multi-Tenancy with Apache Pulsar in Yahoo! (Verizon Media) - Pul...
 
[232] TensorRT를 활용한 딥러닝 Inference 최적화
[232] TensorRT를 활용한 딥러닝 Inference 최적화[232] TensorRT를 활용한 딥러닝 Inference 최적화
[232] TensorRT를 활용한 딥러닝 Inference 최적화
 
Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1Understanding Presto - Presto meetup @ Tokyo #1
Understanding Presto - Presto meetup @ Tokyo #1
 
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in SparkSpark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
Spark Shuffle Deep Dive (Explained In Depth) - How Shuffle Works in Spark
 
RDD
RDDRDD
RDD
 
Hadoop
Hadoop Hadoop
Hadoop
 
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/AvroThe Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
The Rise of ZStandard: Apache Spark/Parquet/ORC/Avro
 
Enabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache SparkEnabling Vectorized Engine in Apache Spark
Enabling Vectorized Engine in Apache Spark
 
Locality sensitive hashing
Locality sensitive hashingLocality sensitive hashing
Locality sensitive hashing
 

Viewers also liked

Linear Regression Using SPSS
Linear Regression Using SPSSLinear Regression Using SPSS
Linear Regression Using SPSS
Dr Athar Khan
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
DrZahid Khan
 
Binary Logistic Regression
Binary Logistic RegressionBinary Logistic Regression
Binary Logistic Regression
Seth Anandaram Jaipuria College
 
Logistic Regression: Predicting The Chances Of Coronary Heart Disease
Logistic Regression: Predicting The Chances Of Coronary Heart DiseaseLogistic Regression: Predicting The Chances Of Coronary Heart Disease
Logistic Regression: Predicting The Chances Of Coronary Heart Disease
Michael Lieberman
 
Multinomial logisticregression basicrelationships
Multinomial logisticregression basicrelationshipsMultinomial logisticregression basicrelationships
Multinomial logisticregression basicrelationships
Anirudha si
 
Logistic regression with SPSS examples
Logistic regression with SPSS examplesLogistic regression with SPSS examples
Logistic regression with SPSS examples
Gaurav Kamboj
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
DrZahid Khan
 
Logistic Regression Analysis
Logistic Regression AnalysisLogistic Regression Analysis
Logistic Regression Analysis
COSTARCH Analytical Consulting (P) Ltd.
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
Venkata Reddy Konasani
 
Logistics management
Logistics managementLogistics management
Logistics management
Navin Raj Saroj
 
Third party logistics
Third party logisticsThird party logistics
Third party logistics
Kuldeep Uttam
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
saba khan
 
Reverse Logistics
Reverse  LogisticsReverse  Logistics
Reverse Logistics
sanket_123
 
Logistic management
Logistic managementLogistic management
Logistic management
jimcyvarghese
 

Viewers also liked (14)

Linear Regression Using SPSS
Linear Regression Using SPSSLinear Regression Using SPSS
Linear Regression Using SPSS
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
 
Binary Logistic Regression
Binary Logistic RegressionBinary Logistic Regression
Binary Logistic Regression
 
Logistic Regression: Predicting The Chances Of Coronary Heart Disease
Logistic Regression: Predicting The Chances Of Coronary Heart DiseaseLogistic Regression: Predicting The Chances Of Coronary Heart Disease
Logistic Regression: Predicting The Chances Of Coronary Heart Disease
 
Multinomial logisticregression basicrelationships
Multinomial logisticregression basicrelationshipsMultinomial logisticregression basicrelationships
Multinomial logisticregression basicrelationships
 
Logistic regression with SPSS examples
Logistic regression with SPSS examplesLogistic regression with SPSS examples
Logistic regression with SPSS examples
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
 
Logistic Regression Analysis
Logistic Regression AnalysisLogistic Regression Analysis
Logistic Regression Analysis
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
 
Logistics management
Logistics managementLogistics management
Logistics management
 
Third party logistics
Third party logisticsThird party logistics
Third party logistics
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
 
Reverse Logistics
Reverse  LogisticsReverse  Logistics
Reverse Logistics
 
Logistic management
Logistic managementLogistic management
Logistic management
 

Similar to Multinomial Logistic Regression with Apache Spark

Introduction to PyTorch
Introduction to PyTorchIntroduction to PyTorch
Introduction to PyTorch
Jun Young Park
 
CSCI 2033 Elementary Computational Linear Algebra(Spring 20.docx
CSCI 2033 Elementary Computational Linear Algebra(Spring 20.docxCSCI 2033 Elementary Computational Linear Algebra(Spring 20.docx
CSCI 2033 Elementary Computational Linear Algebra(Spring 20.docx
mydrynan
 
Functional go
Functional goFunctional go
Functional go
Geison Goes
 
Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural Networks
Stratio
 
20181212 - PGconfASIA - LT - English
20181212 - PGconfASIA - LT - English20181212 - PGconfASIA - LT - English
20181212 - PGconfASIA - LT - English
Kohei KaiGai
 
Algorithm analysis basics - Seven Functions/Big-Oh/Omega/Theta
Algorithm analysis basics - Seven Functions/Big-Oh/Omega/ThetaAlgorithm analysis basics - Seven Functions/Big-Oh/Omega/Theta
Algorithm analysis basics - Seven Functions/Big-Oh/Omega/Theta
Priyanka Rana
 
Java 8
Java 8Java 8
Java 8
vilniusjug
 
Basic concept of MATLAB.ppt
Basic concept of MATLAB.pptBasic concept of MATLAB.ppt
Basic concept of MATLAB.ppt
aliraza2732
 
SPU Optimizations - Part 2
SPU Optimizations - Part 2SPU Optimizations - Part 2
SPU Optimizations - Part 2
Naughty Dog
 
parallel
parallelparallel
MUMS: Transition & SPUQ Workshop - Practical Bayesian Optimization for Urban ...
MUMS: Transition & SPUQ Workshop - Practical Bayesian Optimization for Urban ...MUMS: Transition & SPUQ Workshop - Practical Bayesian Optimization for Urban ...
MUMS: Transition & SPUQ Workshop - Practical Bayesian Optimization for Urban ...
The Statistical and Applied Mathematical Sciences Institute
 
Backpropagation - Elisa Sayrol - UPC Barcelona 2018
Backpropagation - Elisa Sayrol - UPC Barcelona 2018Backpropagation - Elisa Sayrol - UPC Barcelona 2018
Backpropagation - Elisa Sayrol - UPC Barcelona 2018
Universitat Politècnica de Catalunya
 
Matlab1
Matlab1Matlab1
Matlab1
guest8ba004
 
Dafunctor
DafunctorDafunctor
Dafunctor
Buganini Chiu
 
MapReduce
MapReduceMapReduce
MapReduce
ahmedelmorsy89
 
Pytorch meetup
Pytorch meetupPytorch meetup
Pytorch meetup
Dmitri Azarnyh
 
Unit3 MapReduce
Unit3 MapReduceUnit3 MapReduce
Dsp manual completed2
Dsp manual completed2Dsp manual completed2
Dsp manual completed2
bilawalali74
 
On Implementation of Neuron Network(Back-propagation)
On Implementation of Neuron Network(Back-propagation)On Implementation of Neuron Network(Back-propagation)
On Implementation of Neuron Network(Back-propagation)
Yu Liu
 
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUsEarly Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Jeff Larkin
 

Similar to Multinomial Logistic Regression with Apache Spark (20)

Introduction to PyTorch
Introduction to PyTorchIntroduction to PyTorch
Introduction to PyTorch
 
CSCI 2033 Elementary Computational Linear Algebra(Spring 20.docx
CSCI 2033 Elementary Computational Linear Algebra(Spring 20.docxCSCI 2033 Elementary Computational Linear Algebra(Spring 20.docx
CSCI 2033 Elementary Computational Linear Algebra(Spring 20.docx
 
Functional go
Functional goFunctional go
Functional go
 
Introduction to Artificial Neural Networks
Introduction to Artificial Neural NetworksIntroduction to Artificial Neural Networks
Introduction to Artificial Neural Networks
 
20181212 - PGconfASIA - LT - English
20181212 - PGconfASIA - LT - English20181212 - PGconfASIA - LT - English
20181212 - PGconfASIA - LT - English
 
Algorithm analysis basics - Seven Functions/Big-Oh/Omega/Theta
Algorithm analysis basics - Seven Functions/Big-Oh/Omega/ThetaAlgorithm analysis basics - Seven Functions/Big-Oh/Omega/Theta
Algorithm analysis basics - Seven Functions/Big-Oh/Omega/Theta
 
Java 8
Java 8Java 8
Java 8
 
Basic concept of MATLAB.ppt
Basic concept of MATLAB.pptBasic concept of MATLAB.ppt
Basic concept of MATLAB.ppt
 
SPU Optimizations - Part 2
SPU Optimizations - Part 2SPU Optimizations - Part 2
SPU Optimizations - Part 2
 
parallel
parallelparallel
parallel
 
MUMS: Transition & SPUQ Workshop - Practical Bayesian Optimization for Urban ...
MUMS: Transition & SPUQ Workshop - Practical Bayesian Optimization for Urban ...MUMS: Transition & SPUQ Workshop - Practical Bayesian Optimization for Urban ...
MUMS: Transition & SPUQ Workshop - Practical Bayesian Optimization for Urban ...
 
Backpropagation - Elisa Sayrol - UPC Barcelona 2018
Backpropagation - Elisa Sayrol - UPC Barcelona 2018Backpropagation - Elisa Sayrol - UPC Barcelona 2018
Backpropagation - Elisa Sayrol - UPC Barcelona 2018
 
Matlab1
Matlab1Matlab1
Matlab1
 
Dafunctor
DafunctorDafunctor
Dafunctor
 
MapReduce
MapReduceMapReduce
MapReduce
 
Pytorch meetup
Pytorch meetupPytorch meetup
Pytorch meetup
 
Unit3 MapReduce
Unit3 MapReduceUnit3 MapReduce
Unit3 MapReduce
 
Dsp manual completed2
Dsp manual completed2Dsp manual completed2
Dsp manual completed2
 
On Implementation of Neuron Network(Back-propagation)
On Implementation of Neuron Network(Back-propagation)On Implementation of Neuron Network(Back-propagation)
On Implementation of Neuron Network(Back-propagation)
 
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUsEarly Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
 

More from DB Tsai

2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
DB Tsai
 
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
DB Tsai
 
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
DB Tsai
 
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
DB Tsai
 
2014-08-14 Alpine Innovation to Spark
2014-08-14 Alpine Innovation to Spark2014-08-14 Alpine Innovation to Spark
2014-08-14 Alpine Innovation to Spark
DB Tsai
 
Large-Scale Machine Learning with Apache Spark
Large-Scale Machine Learning with Apache SparkLarge-Scale Machine Learning with Apache Spark
Large-Scale Machine Learning with Apache Spark
DB Tsai
 
Unsupervised Learning with Apache Spark
Unsupervised Learning with Apache SparkUnsupervised Learning with Apache Spark
Unsupervised Learning with Apache Spark
DB Tsai
 

More from DB Tsai (7)

2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
 
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
 
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
 
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
 
2014-08-14 Alpine Innovation to Spark
2014-08-14 Alpine Innovation to Spark2014-08-14 Alpine Innovation to Spark
2014-08-14 Alpine Innovation to Spark
 
Large-Scale Machine Learning with Apache Spark
Large-Scale Machine Learning with Apache SparkLarge-Scale Machine Learning with Apache Spark
Large-Scale Machine Learning with Apache Spark
 
Unsupervised Learning with Apache Spark
Unsupervised Learning with Apache SparkUnsupervised Learning with Apache Spark
Unsupervised Learning with Apache Spark
 

Recently uploaded

Digital Twins Computer Networking Paper Presentation.pptx
Digital Twins Computer Networking Paper Presentation.pptxDigital Twins Computer Networking Paper Presentation.pptx
Digital Twins Computer Networking Paper Presentation.pptx
aryanpankaj78
 
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
upoux
 
一比一原版(osu毕业证书)美国俄勒冈州立大学毕业证如何办理
一比一原版(osu毕业证书)美国俄勒冈州立大学毕业证如何办理一比一原版(osu毕业证书)美国俄勒冈州立大学毕业证如何办理
一比一原版(osu毕业证书)美国俄勒冈州立大学毕业证如何办理
upoux
 
SCALING OF MOS CIRCUITS m .pptx
SCALING OF MOS CIRCUITS m                 .pptxSCALING OF MOS CIRCUITS m                 .pptx
SCALING OF MOS CIRCUITS m .pptx
harshapolam10
 
Mechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdfMechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdf
21UME003TUSHARDEB
 
Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...
Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...
Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...
Transcat
 
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODELDEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
ijaia
 
Assistant Engineer (Chemical) Interview Questions.pdf
Assistant Engineer (Chemical) Interview Questions.pdfAssistant Engineer (Chemical) Interview Questions.pdf
Assistant Engineer (Chemical) Interview Questions.pdf
Seetal Daas
 
FULL STACK PROGRAMMING - Both Front End and Back End
FULL STACK PROGRAMMING - Both Front End and Back EndFULL STACK PROGRAMMING - Both Front End and Back End
FULL STACK PROGRAMMING - Both Front End and Back End
PreethaV16
 
AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...
AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...
AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...
Paris Salesforce Developer Group
 
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICSUNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
vmspraneeth
 
Supermarket Management System Project Report.pdf
Supermarket Management System Project Report.pdfSupermarket Management System Project Report.pdf
Supermarket Management System Project Report.pdf
Kamal Acharya
 
原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样
原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样
原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样
ydzowc
 
openshift technical overview - Flow of openshift containerisatoin
openshift technical overview - Flow of openshift containerisatoinopenshift technical overview - Flow of openshift containerisatoin
openshift technical overview - Flow of openshift containerisatoin
snaprevwdev
 
Accident detection system project report.pdf
Accident detection system project report.pdfAccident detection system project report.pdf
Accident detection system project report.pdf
Kamal Acharya
 
Generative AI Use cases applications solutions and implementation.pdf
Generative AI Use cases applications solutions and implementation.pdfGenerative AI Use cases applications solutions and implementation.pdf
Generative AI Use cases applications solutions and implementation.pdf
mahaffeycheryld
 
P5 Working Drawings.pdf floor plan, civil
P5 Working Drawings.pdf floor plan, civilP5 Working Drawings.pdf floor plan, civil
P5 Working Drawings.pdf floor plan, civil
AnasAhmadNoor
 
Null Bangalore | Pentesters Approach to AWS IAM
Null Bangalore | Pentesters Approach to AWS IAMNull Bangalore | Pentesters Approach to AWS IAM
Null Bangalore | Pentesters Approach to AWS IAM
Divyanshu
 
Object Oriented Analysis and Design - OOAD
Object Oriented Analysis and Design - OOADObject Oriented Analysis and Design - OOAD
Object Oriented Analysis and Design - OOAD
PreethaV16
 
Blood finder application project report (1).pdf
Blood finder application project report (1).pdfBlood finder application project report (1).pdf
Blood finder application project report (1).pdf
Kamal Acharya
 

Recently uploaded (20)

Digital Twins Computer Networking Paper Presentation.pptx
Digital Twins Computer Networking Paper Presentation.pptxDigital Twins Computer Networking Paper Presentation.pptx
Digital Twins Computer Networking Paper Presentation.pptx
 
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
一比一原版(uofo毕业证书)美国俄勒冈大学毕业证如何办理
 
一比一原版(osu毕业证书)美国俄勒冈州立大学毕业证如何办理
一比一原版(osu毕业证书)美国俄勒冈州立大学毕业证如何办理一比一原版(osu毕业证书)美国俄勒冈州立大学毕业证如何办理
一比一原版(osu毕业证书)美国俄勒冈州立大学毕业证如何办理
 
SCALING OF MOS CIRCUITS m .pptx
SCALING OF MOS CIRCUITS m                 .pptxSCALING OF MOS CIRCUITS m                 .pptx
SCALING OF MOS CIRCUITS m .pptx
 
Mechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdfMechanical Engineering on AAI Summer Training Report-003.pdf
Mechanical Engineering on AAI Summer Training Report-003.pdf
 
Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...
Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...
Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...
 
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODELDEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
DEEP LEARNING FOR SMART GRID INTRUSION DETECTION: A HYBRID CNN-LSTM-BASED MODEL
 
Assistant Engineer (Chemical) Interview Questions.pdf
Assistant Engineer (Chemical) Interview Questions.pdfAssistant Engineer (Chemical) Interview Questions.pdf
Assistant Engineer (Chemical) Interview Questions.pdf
 
FULL STACK PROGRAMMING - Both Front End and Back End
FULL STACK PROGRAMMING - Both Front End and Back EndFULL STACK PROGRAMMING - Both Front End and Back End
FULL STACK PROGRAMMING - Both Front End and Back End
 
AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...
AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...
AI + Data Community Tour - Build the Next Generation of Apps with the Einstei...
 
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICSUNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
UNIT 4 LINEAR INTEGRATED CIRCUITS-DIGITAL ICS
 
Supermarket Management System Project Report.pdf
Supermarket Management System Project Report.pdfSupermarket Management System Project Report.pdf
Supermarket Management System Project Report.pdf
 
原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样
原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样
原版制作(Humboldt毕业证书)柏林大学毕业证学位证一模一样
 
openshift technical overview - Flow of openshift containerisatoin
openshift technical overview - Flow of openshift containerisatoinopenshift technical overview - Flow of openshift containerisatoin
openshift technical overview - Flow of openshift containerisatoin
 
Accident detection system project report.pdf
Accident detection system project report.pdfAccident detection system project report.pdf
Accident detection system project report.pdf
 
Generative AI Use cases applications solutions and implementation.pdf
Generative AI Use cases applications solutions and implementation.pdfGenerative AI Use cases applications solutions and implementation.pdf
Generative AI Use cases applications solutions and implementation.pdf
 
P5 Working Drawings.pdf floor plan, civil
P5 Working Drawings.pdf floor plan, civilP5 Working Drawings.pdf floor plan, civil
P5 Working Drawings.pdf floor plan, civil
 
Null Bangalore | Pentesters Approach to AWS IAM
Null Bangalore | Pentesters Approach to AWS IAMNull Bangalore | Pentesters Approach to AWS IAM
Null Bangalore | Pentesters Approach to AWS IAM
 
Object Oriented Analysis and Design - OOAD
Object Oriented Analysis and Design - OOADObject Oriented Analysis and Design - OOAD
Object Oriented Analysis and Design - OOAD
 
Blood finder application project report (1).pdf
Blood finder application project report (1).pdfBlood finder application project report (1).pdf
Blood finder application project report (1).pdf
 

Multinomial Logistic Regression with Apache Spark

  • 1. Multinomial Logistic Regression with Apache Spark DB Tsai Machine Learning Engineer May 1st , 2014
  • 2. Machine Learning with Big Data ● Hadoop MapReduce solutions ● MapReduce is scaling well for batch processing ● However, lots of machine learning algorithms are iterative by nature. ● There are lots of tricks people do, like training with subsamples of data, and then average the models. Why big data with approximation? + =
  • 3. Lightning-fast cluster computing ● Empower users to iterate through the data by utilizing the in-memory cache. ● Logistic regression runs up to 100x faster than Hadoop M/R in memory. ● We're able to train exact model without doing any approximation.
  • 4. Binary Logistic Regression d = ax1+bx2+cx0 √a2 +b2 where x0=1 P( y=1∣⃗x , ⃗w)= exp(d ) 1+exp(d ) = exp(⃗x ⃗w) 1+exp(⃗x ⃗w) P ( y=0∣⃗x , ⃗w)= 1 1+exp(⃗x ⃗w) log P ( y=1∣⃗x , ⃗w) P( y=0∣⃗x , ⃗w) =⃗x ⃗w w0= c √a2 +b2 w1= a √a2 +b2 w2= b √a2 +b2 wherew0 iscalled as intercept
  • 5. Training Binary Logistic Regression ● Maximum Likelihood estimation From a training data and labels ● We want to find that maximizes the likelihood of data defined by ● We can take log of the equation, and minimize it instead. The Log-Likelihood becomes the loss function. X =( ⃗x1 , ⃗x2 , ⃗x3 ,...) Y =( y1, y2, y3, ...) ⃗w L( ⃗w , ⃗x1, ... , ⃗xN )=P ( y1∣⃗x1 , ⃗w)P ( y2∣⃗x2 , ⃗w)... P( yN∣ ⃗xN , ⃗w) l( ⃗w ,⃗x)=log P(y1∣⃗x1 , ⃗w)+log P( y2∣⃗x2 , ⃗w)...+log P( yN∣ ⃗xN , ⃗w)
  • 6. Optimization ● First Order Minimizer Require loss, gradient of loss function – Gradient Decent is step size – Limited-memory BFGS (L-BFGS) – Orthant-Wise Limited-memory Quasi-Newton (OWLQN) – Coordinate Descent (CD) – Trust Region Newton Method (TRON) ● Second Order Minimizer Require loss, gradient and hessian of loss function – Newton-Raphson, quadratic convergence. Fast! ● Ref: Journal of Machine Learning Research 11 (2010) 3183- 3234, Chih-Jen Lin et al. ⃗wn+1=⃗wn−γ ⃗G , γ ⃗wn+1=⃗wn−H−1 ⃗G
  • 7. Problem of Second Order Minimizer ● Scale horizontally (the numbers of training data) by leveraging Spark to parallelize this iterative optimization process. ● Don't scale vertically (the numbers of training features). Dimension of Hessian is ● Recent applications from document classification and computational linguistics are of this type. dim(H )=[(k−1)(n+1)] 2 where k is numof class ,nis num of features
  • 8. L-BFGS ● It's a quasi-Newton method. ● Hessian matrix of second derivatives doesn't need to be evaluated directly. ● Hessian matrix is approximated using gradient evaluations. ● It converges a way faster than the default optimizer in Spark, Gradient Decent. ● We love open source! Alpine Data Labs contributed our L-BFGS to Spark, and it's already merged in Spark-1157.
  • 9. Training Binary Logistic Regression l( ⃗w ,⃗x) = ∑ k=1 N log P( yk∣⃗xk , ⃗w) = ∑k=1 N yk log P ( yk=1∣⃗xk , ⃗w)+(1−yk )log P ( yk =0∣⃗xk , ⃗w) = ∑k=1 N yk log exp( ⃗xk ⃗w) 1+exp( ⃗xk ⃗w) +(1−yk )log 1 1+exp( ⃗xk ⃗w) = ∑k=1 N yk ⃗xk ⃗w−log(1+exp( ⃗xk ⃗w)) Gradient : Gi (⃗w ,⃗x)= ∂l( ⃗w ,⃗x) ∂ wi =∑k=1 N yk xki− exp( ⃗xk ⃗w) 1+exp( ⃗xk ⃗w) xki Hessian: H ij (⃗w ,⃗x)= ∂∂l( ⃗w ,⃗x) ∂wi ∂w j =−∑k=1 N exp( ⃗xk ⃗w) (1+exp( ⃗xk ⃗w))2 xki xkj
  • 10. Overfitting P ( y=1∣⃗x , ⃗w)= exp(zd ) 1+exp(zd ) = exp(⃗x ⃗w) 1+exp(⃗x ⃗w)
  • 11. Regularization ● The loss function becomes ● The loss function of regularizer doesn't depend on data. Common regularizers are – L2 Regularization: – L1 Regularization: ● L1 norm is not differentiable at zero! ltotal (⃗w ,⃗x)=lmodel (⃗w ,⃗x)+lreg( ⃗w) lreg (⃗w)=λ∑i=1 N wi 2 lreg (⃗w)=λ∑i=1 N ∣wi∣ ⃗G (⃗w ,⃗x)total=⃗G(⃗w ,⃗x)model+⃗G( ⃗w)reg ̄H ( ⃗w ,⃗x)total= ̄H (⃗w ,⃗x)model+ ̄H (⃗w)reg
  • 12. Mini School of Spark APIs ● map(func) : Return a new distributed dataset formed by passing each element of the source through a function func. ● reduce(func) : Aggregate the elements of the dataset using a function func (which takes two arguments and returns one). The function should be commutative and associative so that it can be computed correctly in parallel. No “key” thing here compared with Hadoop M/R.
  • 13. Example – No More “Word Count”! Let's compute the mean of numbers. 3.0 2.0 1.0 5.0 Executor 1 Executor 2 map map (1.0, 1) (5.0, 1) (3.0, 1) (2.0, 1) reduce (5.0, 2) reduce (11.0, 4) Shuffle
  • 14. ● The previous example using map(func) will create new tuple object, which may cause Garbage Collection issue. ● Like the combiner in Hadoop Map Reduce, for each tuple emitted from map(func), there is no guarantee that they will be combined locally in the same executor by reduce(func). It may increase the traffic in shuffle phase. ● In Hadoop, we address this by in-mapper combiner or aggregating the result in global variables which have scope to entire partition. The same approach can be used in Spark.
  • 15. Mini School of Spark APIs ● mapPartitions(func) : Similar to map, but runs separately on each partition (block) of the RDD, so func must be of type Iterator[T] => Iterator[U] when running on an RDD of type T. ● This API allows us to have global variables on entire partition to aggregate the result locally and efficiently.
  • 16. Better Mean of Numbers Implementation 3.0 2.0 Executor 1 reduce (11.0, 4) Shuffle var counts = 0 var sum = 0.0 loop var counts = 2 var sum = 5.0 3.0 2.0 (5.0, 2) 1.0 5.0 Executor 2 var counts = 0 var sum = 0.0 loop var counts = 2 var sum = 6.0 1.0 5.0 (6.0, 2) mapPartitions
  • 17. More Idiomatic Scala Implementation ● aggregate(zeroValue: U)(seqOp: (U, T) => U, combOp: (U, U) => U): U zeroValue is a neutral "zero value" with type U for initialization. This is analogous to in previous example. var counts = 0 var sum = 0.0 seqOp is function taking (U, T) => U, where U is aggregator initialized as zeroValue. T is each line of values in RDD. This is analogous to mapPartition in previous example. combOp is function taking (U, U) => U. This is essentially combing the results between different executors. The functionality is the same as reduce(func) in previous example.
  • 18. Approach of Parallelization in Spark Spark Driver JVM Time Spark Executor 1 JVM Spark Executor 2 JVM 1) Find available resources in cluster, and launch executor JVMs. Also initialize the weights. 2) Ask executors to load the data into executors' JVMs 2) Trying to load the data into memory. If the data is bigger than memory, it will be partial cached. (The data locality from source will be taken care by Spark) 3) Ask executors to compute loss, and gradient of each training sample (each row) given the current weights. Get the aggregated results after the Reduce Phase in executors. 5) If the regularization is enabled, compute the loss and gradient of regularizer in driver since it doesn't depend on training data but only depends on weights. Add them into the results from executors. 3) Map Phase: Compute the loss and gradient of each row of training data locally given the weights obtained from the driver. Can either emit each result or sum them up in local aggregators. 4) Reduce Phase: Sum up the losses and gradients emitted from the Map Phase Taking a rest! 6) Plug the loss and gradient from model and regularizer into optimizer to get the new weights. If the differences of weights and losses are larger than criteria, GO BACK TO 3) Taking a rest! 7) Finish the model training! Taking a rest!
  • 19. Step 3) and 4) ● This is the implementation of step 3) and 4) in MLlib before Spark 1.0 ● gradient can have implementations of Logistic Regression, Linear Regression, SVM, or any customized cost function. ● Each training data will create new “grad” object after gradient.compute.
  • 20. Step 3) and 4) with mapPartitions
  • 21. Step 3) and 4) with aggregate ● This is the implementation of step 3) and 4) in MLlib in coming Spark 1.0 ● No unnecessary object creation! It's helpful when we're dealing with large features training data. GC will not kick in the executor JVMs.
  • 22. Extension to Multinomial Logistic Regression ● In Binary Logistic Regression ● For K classes multinomial problem where labels ranged from [0, K-1], we can generalize it via ● The model, weights becomes (K-1)(N+1) matrix, where N is number of features. log P( y=1∣⃗x , ̄w) P ( y=0∣⃗x , ̄w) =⃗x ⃗w1 log P ( y=2∣⃗x , ̄w) P ( y=0∣⃗x , ̄w) =⃗x ⃗w2 ... log P ( y=K −1∣⃗x , ̄w) P ( y=0∣⃗x , ̄w) =⃗x ⃗wK −1 log P( y=1∣⃗x , ⃗w) P ( y=0∣⃗x , ⃗w) =⃗x ⃗w ̄w=(⃗w1, ⃗w2, ... , ⃗wK −1)T P( y=0∣⃗x , ̄w)= 1 1+∑ i=1 K −1 exp(⃗x ⃗wi ) P( y=1∣⃗x , ̄w)= exp(⃗x ⃗w2) 1+∑ i=1 K −1 exp(⃗x ⃗wi) ... P( y=K −1∣⃗x , ̄w)= exp(⃗x ⃗wK −1) 1+∑i=1 K −1 exp(⃗x ⃗wi )
  • 23. Training Multinomial Logistic Regression l( ̄w ,⃗x) = ∑ k=1 N log P( yk∣⃗xk , ̄w) = ∑k=1 N α( yk)log P( y=0∣⃗xk , ̄w)+(1−α( yk ))log P ( yk∣⃗xk , ̄w) = ∑k=1 N α( yk)log 1 1+∑ i=1 K −1 exp(⃗x ⃗wi) +(1−α( yk ))log exp(⃗x ⃗w yk ) 1+∑ i=1 K−1 exp(⃗x ⃗wi) = ∑ k=1 N (1−α( yk ))⃗x ⃗wyk −log(1+∑ i=1 K−1 exp(⃗x ⃗wi)) Gradient : Gij ( ̄w ,⃗x)= ∂l( ̄w ,⃗x) ∂wij =∑k=1 N (1−α( yk )) xkj δi , yk − exp( ⃗xk ⃗w) 1+exp( ⃗xk ⃗w) xkj α( yk )=1 if yk=0 α( yk )=0 if yk≠0 Define: Note that the first index “i” is for classes, and the second index “j” is for features. Hessian: H ij ,lm( ̄w ,⃗x)= ∂∂l ( ̄w ,⃗x) ∂wij ∂wlm
  • 24. 0 5 10 15 20 25 30 35 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 Logistic Regression with a9a Dataset (11M rows, 123 features, 11% non-zero elements) 16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster L-BFGS Dense Features L-BFGS Sparse Features GD Sparse Features GD Dense Features Seconds Log-Likelihood/NumberofSamples a9a Dataset Benchmark
  • 25. a9a Dataset Benchmark -1 1 3 5 7 9 11 13 15 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 Logistic Regression with a9a Dataset (11M rows, 123 features, 11% non-zero elements) 16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster L-BFGS GD Iterations Log-Likelihood/NumberofSamples
  • 26. 0 5 10 15 20 25 30 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Logistic Regression with rcv1 Dataset (6.8M rows, 677,399 features, 0.15% non-zero elements) 16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster LBFGS Sparse Vector GD Sparse Vector Second Log-Likelihood/NumberofSamples rcv1 Dataset Benchmark
  • 27. news20 Dataset Benchmark 0 10 20 30 40 50 60 70 80 0 0.2 0.4 0.6 0.8 1 1.2 Logistic Regression with news20 Dataset (0.14M rows, 1,355,191 features, 0.034% non-zero elements) 16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster LBFGS Sparse Vector GD Sparse Vector Second Log-Likelihood/NumberofSamples
  • 31. Conclusion ● Alpine supports state of the art Cloudera CDH5 ● Spark runs programs 100x faster than Hadoop ● Spark turns iterative big data machine learning problems into single machine problems ● MLlib provides lots of state of art machine learning implementations, e.g. K-means, linear regression, logistic regression, ALS, collaborative filtering, and Naive Bayes, etc. ● Spark provides ease of use APIs in Java, Scala, or Python, and interactive Scala and Python shell ● Spark is Apache project, and it's the most active big data open source project nowadays.