SlideShare a Scribd company logo
Multinomial Logistic Regression
with Apache Spark
DB Tsai
dbtsai@alpinenow.com
Machine Learning Engineer
June 20th
, 2014
Silicon Valley Machine Learning Meetup
What is Alpine Data Labs doing?
● Collaborative, Code-Free, Advanced Analytics
Solution for Big Data
● Technology we're using: Scala/Java, Akka, Spray,
Hadoop, Spark/SparkSQL, Pig, Sqoop
● Our platform runs against different flavors of Hadoop
distributions such as Cloudera, Pivotal Hadoop, MapR,
HortonWorks and Apache.
● Actively involved in the open source community: almost
of all our newly developed algorithms in Spark will be
contributed back to MLLib.
● Already committed a L-BFGS optimizer to Spark, and
helped fix couple bugs. Working on multinational logistic
regression, GLM, Decision Tree and a Random Forest
● In addition we are the maintainer of several open source
projects including Chorus, SBT plugin for JUnit test
Listener and several other projects.
We're open source friendly!
We're hiring!
● Machine Learning Engineer
● Data Scientist
● UI/UX Engineer
● Front-end Engineer
● Infrastructure Engineer
● Back-end Engineer
● Automation Test Engineer
Shoot me an email at
dbtsai@alpinenow.com
Machine Learning with Big Data
● Hadoop MapReduce solutions
● MapReduce is scaling well for batch processing
● Lots of machine learning algorithms are iterative
by nature.
● There are lots of tricks people do, like training
with subsamples of data, and then average the
models. Why big data with approximation?
+ =
Lightning-fast cluster computing
● Empower users to iterate through the data by
utilizing the in-memory cache.
● Logistic regression runs
up to 100x faster than
Hadoop M/R in memory.
● We're able to train
exact model without doing any approximation.
Binary Logistic Regression
d =
ax1+bx2+cx0
√a2
+b2
where x0=1
P( y=1∣⃗x , ⃗w)=
exp(d )
1+exp(d )
=
exp(⃗x ⃗w)
1+exp(⃗x ⃗w)
P ( y=0∣⃗x , ⃗w)=
1
1+exp(⃗x ⃗w)
log
P ( y=1∣⃗x , ⃗w)
P( y=0∣⃗x , ⃗w)
=⃗x ⃗w
w0=
c
√a2
+b2
w1=
a
√a2
+b2
w2=
b
√a2
+b2
where w0 iscalled as intercept
Training Binary Logistic Regression
● Maximum Likelihood estimation
From a training data
and labels
● We want to find that maximizes the
likelihood of data defined by
● We can take log of the equation, and minimize
it instead. The Log-Likelihood becomes the
loss function.
X =( ⃗x1 , ⃗x2 , ⃗x3 ,...)
Y =( y1, y2, y3, ...)
⃗w
L( ⃗w , ⃗x1, ... , ⃗xN )=P ( y1∣⃗x1 , ⃗w)P ( y2∣⃗x2 , ⃗w)... P( yN∣ ⃗xN , ⃗w)
l( ⃗w ,⃗x)=log P(y1∣⃗x1 , ⃗w)+log P( y2∣⃗x2 , ⃗w)...+log P( yN∣ ⃗xN , ⃗w)
Optimization
● First Order Minimizer
Require loss, gradient of loss function
– Gradient Decent is step size
– Limited-memory BFGS (L-BFGS)
– Orthant-Wise Limited-memory Quasi-Newton
(OWLQN)
– Coordinate Descent (CD)
– Trust Region Newton Method (TRON)
● Second Order Minimizer
Require loss, gradient and hessian of loss function
– Newton-Raphson, quadratic convergence. Fast!
● Ref: Journal of Machine Learning Research 11 (2010) 3183-
3234, Chih-Jen Lin et al.
⃗wn+1=⃗wn−γ ⃗G , γ
⃗wn+1=⃗wn−H−1 ⃗G
Problem of Second Order Minimizer
● Scale horizontally (the numbers of training
data) by leveraging Spark to parallelize this
iterative optimization process.
● Don't scale vertically (the numbers of training
features). Dimension of Hessian is
● Recent applications from document
classification and computational linguistics are
of this type.
dim(H )=[(k−1)(n+1)]
2
where k is numof class ,nis num of features
L-BFGS
● It's a quasi-Newton method.
● Hessian matrix of second derivatives doesn't
need to be evaluated directly.
● Hessian matrix is approximated using gradient
evaluations.
● It converges a way faster than the default
optimizer in Spark, Gradient Decent.
● We love open source! Alpine Data Labs
contributed our L-BFGS to Spark, and it's
already merged in Spark-1157.
Training Binary Logistic Regression
l( ⃗w ,⃗x) = ∑
k=1
N
log P( yk∣⃗xk , ⃗w)
= ∑k=1
N
yk log P ( yk=1∣⃗xk , ⃗w)+(1−yk )log P ( yk =0∣⃗xk , ⃗w)
= ∑k=1
N
yk log
exp( ⃗xk ⃗w)
1+exp( ⃗xk ⃗w)
+(1−yk )log
1
1+exp( ⃗xk ⃗w)
= ∑k=1
N
yk ⃗xk ⃗w−log(1+exp( ⃗xk ⃗w))
Gradient : Gi (⃗w ,⃗x)=
∂l( ⃗w ,⃗x)
∂ wi
=∑k=1
N
yk xki−
exp( ⃗xk ⃗w)
1+exp( ⃗xk ⃗w)
xki
Hessian: H ij (⃗w ,⃗x)=
∂∂l( ⃗w ,⃗x)
∂wi ∂w j
=−∑k=1
N
exp( ⃗xk ⃗w)
(1+exp( ⃗xk ⃗w))2
xki xkj
Overfitting
P ( y=1∣⃗x , ⃗w)=
exp(zd )
1+exp(zd )
=
exp(⃗x ⃗w)
1+exp(⃗x ⃗w)
Regularization
● The loss function becomes
● The loss function of regularizer doesn't
depend on data. Common regularizers are
– L2 Regularization:
– L1 Regularization:
● L1 norm is not differentiable at zero!
ltotal (⃗w ,⃗x)=lmodel (⃗w ,⃗x)+lreg ( ⃗w)
lreg (⃗w)=λ∑i=1
N
wi
2
lreg (⃗w)=λ∑i=1
N
∣wi∣
⃗G (⃗w ,⃗x)total=⃗G(⃗w ,⃗x)model+⃗G( ⃗w)reg
̄H ( ⃗w ,⃗x)total= ̄H (⃗w ,⃗x)model+ ̄H (⃗w)reg
Mini School of Spark APIs
● map(func) : Return a new distributed dataset
formed by passing each element of the source
through a function func.
● reduce(func) : Aggregate the elements of the
dataset using a function func (which takes two
arguments and returns one). The function
should be commutative and associative so
that it can be computed correctly in parallel.
This is O(n) operation where n is number of
partition. In SPARK-2174, treeReduce will be
merged for O(log(n)) operation.
Example – compute the mean of numbers.
3.0
2.0
1.0
5.0
Executor 1
Executor 2
map
(3.0, 1)
(2.0, 1)
reduce
(5.0, 2)
reduce (11.0, 4)
Shuffle
reduce
(6.0, 2)
(1.0, 1)
(5.0, 1)
map
Executor 3
Mini School of Spark APIs
● mapPartitions(func) : Similar to map, but runs
separately on each partition (block) of the
RDD, so func must be of type Iterator[T] =>
Iterator[U] when running on an RDD of type T.
● This API allows us to have global variables on
entire partition to aggregate the result locally
and efficiently.
Better Mean
of Numbers
Implementation
3.0
2.0
Executor 1
reduce (11.0, 4)
Shuffle
var counts = 0
var sum = 0.0
loop
var counts = 2
var sum = 5.0
3.0
2.0
(5.0, 2)
1.0
5.0
Executor 2
var counts = 0
var sum = 0.0
loop
var counts = 2
var sum = 6.0
1.0
5.0
(6.0, 2)
mapPartitions
More Idiomatic Scala Implementation
● aggregate(zeroValue: U)(seqOp: (U, T) => U,
combOp: (U, U) => U): U zeroValue is a neutral "zero value"
with type U for initialization. This is
analogous to
in previous example.
var counts = 0
var sum = 0.0
seqOp is function taking
(U, T) => U, where U is aggregator
initialized as zeroValue. T is each
line of values in RDD. This is
analogous to mapPartition in
previous example.
combOp is function taking
(U, U) => U. This is essentially
combing the results between
different executors. The
functionality is the same as
reduce(func) in previous example.
Approach of Parallelization in Spark
Spark Driver
JVM
Time
Spark Executor 1
JVM
Spark Executor 2
JVM
1) Find available resources in
cluster, and launch executor JVMs.
Also initialize the weights.
2) Ask executors to load the data
into executors' JVMs
2) Trying to load the data into memory. If the data is bigger
than memory, it will be partial cached.
(The data locality from source will be taken care by Spark)
3) Ask executors to compute loss, and
gradient of each training sample
(each row) given the current weights.
Get the aggregated results after
the Reduce Phase in executors.
5) If the regularization is enabled,
compute the loss and gradient of
regularizer in driver since it doesn't
depend on training data but only
depends on weights. Add them into
the results from executors.
3) Map Phase:
Compute the loss and gradient of each row of training data
locally given the weights obtained from the driver. Can either
emit each result or sum them up in local aggregators.
4) Reduce Phase:
Sum up the losses and gradients emitted from the Map Phase
Taking a rest!
6) Plug the loss and gradient from
model and regularizer into optimizer
to get the new weights. If the
differences of weights and losses
are larger than criteria,
GO BACK TO 3)
Taking a rest!
7) Finish the model training! Taking a rest!
Step 3) and 4)
● This is the implementation of step 3) and 4) in
MLlib before Spark 1.0
● gradient can have implementations of
Logistic Regression, Linear Regression, SVM,
or any customized cost function.
● Each training data will create new “grad”
object after gradient.compute.
Step 3) and 4) with mapPartitions
Step 3) and 4) with aggregate
● This is the implementation of step 3) and 4) in
MLlib in coming Spark 1.0
● No unnecessary object creation! It's helpful
when we're dealing with large features training
data. GC will not kick in the executor JVMs.
Extension to Multinomial Logistic Regression
● In Binary Logistic Regression
● For K classes multinomial problem where labels
ranged from [0, K-1], we can generalize it via
● The model, weights becomes
(K-1)(N+1) matrix, where N is number of features.
log
P( y=1∣⃗x , ̄w)
P ( y=0∣⃗x , ̄w)
=⃗x ⃗w1
log
P ( y=2∣⃗x , ̄w)
P ( y=0∣⃗x , ̄w)
=⃗x ⃗w2
...
log
P ( y=K −1∣⃗x , ̄w)
P ( y=0∣⃗x , ̄w)
=⃗x ⃗wK −1
log
P( y=1∣⃗x , ⃗w)
P ( y=0∣⃗x , ⃗w)
=⃗x ⃗w
̄w=(⃗w1, ⃗w2, ... , ⃗wK −1)T
P( y=0∣⃗x , ̄w)=
1
1+∑
i=1
K −1
exp(⃗x ⃗wi )
P( y=1∣⃗x , ̄w)=
exp(⃗x ⃗w2)
1+∑
i=1
K −1
exp(⃗x ⃗wi)
...
P( y=K −1∣⃗x , ̄w)=
exp(⃗x ⃗wK −1)
1+∑i=1
K −1
exp(⃗x ⃗wi )
Training Multinomial Logistic Regression
l( ̄w ,⃗x) = ∑
k=1
N
log P( yk∣⃗xk , ̄w)
= ∑k=1
N
α( yk)log P( y=0∣⃗xk , ̄w)+(1−α( yk ))log P ( yk∣⃗xk , ̄w)
= ∑k=1
N
α( yk)log
1
1+∑
i=1
K −1
exp(⃗x ⃗wi)
+(1−α( yk ))log
exp(⃗x ⃗w yk
)
1+∑
i=1
K−1
exp(⃗x ⃗wi)
= ∑
k=1
N
(1−α( yk ))⃗x ⃗wyk
−log(1+∑
i=1
K−1
exp(⃗x ⃗wi))
Gradient : Gij ( ̄w ,⃗x)=
∂l( ̄w ,⃗x)
∂wij
=∑k=1
N
(1−α( yk )) xkj δi , yk
−
exp( ⃗xk ⃗w)
1+exp( ⃗xk ⃗w)
xkj
α( yk )=1 if yk=0
α( yk )=0 if yk≠0
Define:
Note that the first index “i” is for classes, and the second index “j” is for features.
Hessian: H ij ,lm( ̄w ,⃗x)=
∂∂l ( ̄w ,⃗x)
∂wij ∂wlm
0 5 10 15 20 25 30 35
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
Logistic Regression with a9a Dataset (11M rows, 123 features, 11% non-zero elements)
16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster
L-BFGS Dense Features
L-BFGS Sparse Features
GD Sparse Features
GD Dense Features
Seconds
Log-Likelihood/NumberofSamples
a9a Dataset Benchmark
a9a Dataset Benchmark
-1 1 3 5 7 9 11 13 15
0.3
0.35
0.4
0.45
0.5
0.55
0.6
0.65
0.7
Logistic Regression with a9a Dataset (11M rows, 123 features, 11% non-zero elements)
16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster
L-BFGS
GD
Iterations
Log-Likelihood/NumberofSamples
0 5 10 15 20 25 30
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Logistic Regression with rcv1 Dataset (6.8M rows, 677,399 features, 0.15% non-zero elements)
16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster
LBFGS Sparse Vector
GD Sparse Vector
Second
Log-Likelihood/NumberofSamples
rcv1 Dataset Benchmark
news20 Dataset Benchmark
0 10 20 30 40 50 60 70 80
0
0.2
0.4
0.6
0.8
1
1.2
Logistic Regression with news20 Dataset (0.14M rows, 1,355,191 features, 0.034% non-zero elements)
16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster
LBFGS Sparse Vector
GD Sparse Vector
Second
Log-Likelihood/NumberofSamples
Alpine Demo
Alpine Demo
Alpine Demo
Conclusion
● We're hiring!
● Spark runs programs 100x faster than Hadoop
● Spark turns iterative big data machine learning problems
into single machine problems
● MLlib provides lots of state of art machine learning
implementations, e.g. K-means, linear regression,
logistic regression, ALS, collaborative filtering, and Naive
Bayes, etc.
● Spark provides ease of use APIs in Java, Scala, or
Python, and interactive Scala and Python shell
● Spark is Apache project, and it's the most active big data
open source project nowadays.

More Related Content

What's hot

Fractional Knapsack Problem
Fractional Knapsack ProblemFractional Knapsack Problem
Fractional Knapsack Problem
harsh kothari
 
What is the Expectation Maximization (EM) Algorithm?
What is the Expectation Maximization (EM) Algorithm?What is the Expectation Maximization (EM) Algorithm?
What is the Expectation Maximization (EM) Algorithm?
Kazuki Yoshida
 
Apache flink
Apache flinkApache flink
Apache flink
Ahmed Nader
 
Knapsack problem using greedy approach
Knapsack problem using greedy approachKnapsack problem using greedy approach
Knapsack problem using greedy approach
padmeshagrekar
 
Machine Learning With Logistic Regression
Machine Learning  With Logistic RegressionMachine Learning  With Logistic Regression
Machine Learning With Logistic Regression
Knoldus Inc.
 
Apache Spark PDF
Apache Spark PDFApache Spark PDF
Apache Spark PDF
Naresh Rupareliya
 
Architecture for building scalable and highly available Postgres Cluster
Architecture for building scalable and highly available Postgres ClusterArchitecture for building scalable and highly available Postgres Cluster
Architecture for building scalable and highly available Postgres Cluster
Ashnikbiz
 
Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...
Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...
Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...
Flink Forward
 
Apache PIG
Apache PIGApache PIG
Apache PIG
Anuja Gunale
 
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxData
 
Spark SQL Bucketing at Facebook
 Spark SQL Bucketing at Facebook Spark SQL Bucketing at Facebook
Spark SQL Bucketing at Facebook
Databricks
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent method
Prof. Neeta Awasthy
 
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
NAVER D2
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used for
Aljoscha Krettek
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent method
Sanghyuk Chun
 
Achieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQLAchieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQL
Mydbops
 
Fuzzy Clustering(C-means, K-means)
Fuzzy Clustering(C-means, K-means)Fuzzy Clustering(C-means, K-means)
Fuzzy Clustering(C-means, K-means)
Fellowship at Vodafone FutureLab
 
strassen matrix multiplication algorithm
strassen matrix multiplication algorithmstrassen matrix multiplication algorithm
strassen matrix multiplication algorithm
evil eye
 
AI_Planning.pdf
AI_Planning.pdfAI_Planning.pdf
AI_Planning.pdf
SUSHMARATHI3
 
Лекция 13. YARN
Лекция 13. YARNЛекция 13. YARN
Лекция 13. YARN
Technopark
 

What's hot (20)

Fractional Knapsack Problem
Fractional Knapsack ProblemFractional Knapsack Problem
Fractional Knapsack Problem
 
What is the Expectation Maximization (EM) Algorithm?
What is the Expectation Maximization (EM) Algorithm?What is the Expectation Maximization (EM) Algorithm?
What is the Expectation Maximization (EM) Algorithm?
 
Apache flink
Apache flinkApache flink
Apache flink
 
Knapsack problem using greedy approach
Knapsack problem using greedy approachKnapsack problem using greedy approach
Knapsack problem using greedy approach
 
Machine Learning With Logistic Regression
Machine Learning  With Logistic RegressionMachine Learning  With Logistic Regression
Machine Learning With Logistic Regression
 
Apache Spark PDF
Apache Spark PDFApache Spark PDF
Apache Spark PDF
 
Architecture for building scalable and highly available Postgres Cluster
Architecture for building scalable and highly available Postgres ClusterArchitecture for building scalable and highly available Postgres Cluster
Architecture for building scalable and highly available Postgres Cluster
 
Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...
Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...
Time to-live: How to Perform Automatic State Cleanup in Apache Flink - Andrey...
 
Apache PIG
Apache PIGApache PIG
Apache PIG
 
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
InfluxDB IOx Tech Talks: Query Engine Design and the Rust-Based DataFusion in...
 
Spark SQL Bucketing at Facebook
 Spark SQL Bucketing at Facebook Spark SQL Bucketing at Facebook
Spark SQL Bucketing at Facebook
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent method
 
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
[135] 오픈소스 데이터베이스, 은행 서비스에 첫발을 내밀다.
 
Apache Flink and what it is used for
Apache Flink and what it is used forApache Flink and what it is used for
Apache Flink and what it is used for
 
Gradient descent method
Gradient descent methodGradient descent method
Gradient descent method
 
Achieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQLAchieving High Availability in PostgreSQL
Achieving High Availability in PostgreSQL
 
Fuzzy Clustering(C-means, K-means)
Fuzzy Clustering(C-means, K-means)Fuzzy Clustering(C-means, K-means)
Fuzzy Clustering(C-means, K-means)
 
strassen matrix multiplication algorithm
strassen matrix multiplication algorithmstrassen matrix multiplication algorithm
strassen matrix multiplication algorithm
 
AI_Planning.pdf
AI_Planning.pdfAI_Planning.pdf
AI_Planning.pdf
 
Лекция 13. YARN
Лекция 13. YARNЛекция 13. YARN
Лекция 13. YARN
 

Viewers also liked

2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
DB Tsai
 
Large scale logistic regression and linear support vector machines using spark
Large scale logistic regression and linear support vector machines using sparkLarge scale logistic regression and linear support vector machines using spark
Large scale logistic regression and linear support vector machines using spark
Mila, Université de Montréal
 
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Mich Talebzadeh (Ph.D.)
 
Generalized Linear Models in Spark MLlib and SparkR by Xiangrui Meng
Generalized Linear Models in Spark MLlib and SparkR by Xiangrui MengGeneralized Linear Models in Spark MLlib and SparkR by Xiangrui Meng
Generalized Linear Models in Spark MLlib and SparkR by Xiangrui Meng
Spark Summit
 
Forecasting stock market movement direction with support vector machine
Forecasting stock market movement direction with support vector machineForecasting stock market movement direction with support vector machine
Forecasting stock market movement direction with support vector machine
Mohamed DHAOUI
 
Generalized Linear Models in Spark MLlib and SparkR
Generalized Linear Models in Spark MLlib and SparkRGeneralized Linear Models in Spark MLlib and SparkR
Generalized Linear Models in Spark MLlib and SparkR
Databricks
 
Scaling out logistic regression with Spark
Scaling out logistic regression with SparkScaling out logistic regression with Spark
Scaling out logistic regression with Spark
Barak Gitsis
 
Introduction to Machine Learning & Classification
Introduction to Machine Learning & ClassificationIntroduction to Machine Learning & Classification
Introduction to Machine Learning & Classification
Christopher Sharkey
 
Machine Learning with Big Data using Apache Spark
Machine Learning with Big Data using Apache SparkMachine Learning with Big Data using Apache Spark
Machine Learning with Big Data using Apache Spark
InSemble
 
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...
DataWorks Summit/Hadoop Summit
 
Building Random Forest at Scale
Building Random Forest at ScaleBuilding Random Forest at Scale
Building Random Forest at Scale
Sri Ambati
 
Elasticity & forecasting
Elasticity & forecastingElasticity & forecasting
Elasticity & forecasting
ishwarijoshi
 
Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...
Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...
Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...
Jen Aman
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
Venkata Reddy Konasani
 
Elasticity+and+its+application
Elasticity+and+its+applicationElasticity+and+its+application
Elasticity+and+its+application
Khalid Aziz
 
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Spark Summit
 
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
台灣資料科學年會
 
Introduction to Big Data/Machine Learning
Introduction to Big Data/Machine LearningIntroduction to Big Data/Machine Learning
Introduction to Big Data/Machine Learning
Lars Marius Garshol
 
What Makes Great Infographics
What Makes Great InfographicsWhat Makes Great Infographics
What Makes Great Infographics
SlideShare
 
Masters of SlideShare
Masters of SlideShareMasters of SlideShare
Masters of SlideShare
Kapost
 

Viewers also liked (20)

2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
2015-06-15 Large-Scale Elastic-Net Regularized Generalized Linear Models at S...
 
Large scale logistic regression and linear support vector machines using spark
Large scale logistic regression and linear support vector machines using sparkLarge scale logistic regression and linear support vector machines using spark
Large scale logistic regression and linear support vector machines using spark
 
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
Query Engines for Hive: MR, Spark, Tez with LLAP – Considerations!
 
Generalized Linear Models in Spark MLlib and SparkR by Xiangrui Meng
Generalized Linear Models in Spark MLlib and SparkR by Xiangrui MengGeneralized Linear Models in Spark MLlib and SparkR by Xiangrui Meng
Generalized Linear Models in Spark MLlib and SparkR by Xiangrui Meng
 
Forecasting stock market movement direction with support vector machine
Forecasting stock market movement direction with support vector machineForecasting stock market movement direction with support vector machine
Forecasting stock market movement direction with support vector machine
 
Generalized Linear Models in Spark MLlib and SparkR
Generalized Linear Models in Spark MLlib and SparkRGeneralized Linear Models in Spark MLlib and SparkR
Generalized Linear Models in Spark MLlib and SparkR
 
Scaling out logistic regression with Spark
Scaling out logistic regression with SparkScaling out logistic regression with Spark
Scaling out logistic regression with Spark
 
Introduction to Machine Learning & Classification
Introduction to Machine Learning & ClassificationIntroduction to Machine Learning & Classification
Introduction to Machine Learning & Classification
 
Machine Learning with Big Data using Apache Spark
Machine Learning with Big Data using Apache SparkMachine Learning with Big Data using Apache Spark
Machine Learning with Big Data using Apache Spark
 
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...
Apache HBase + Spark: Leveraging your Non-Relational Datastore in Batch and S...
 
Building Random Forest at Scale
Building Random Forest at ScaleBuilding Random Forest at Scale
Building Random Forest at Scale
 
Elasticity & forecasting
Elasticity & forecastingElasticity & forecasting
Elasticity & forecasting
 
Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...
Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...
Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...
 
Logistic regression
Logistic regressionLogistic regression
Logistic regression
 
Elasticity+and+its+application
Elasticity+and+its+applicationElasticity+and+its+application
Elasticity+and+its+application
 
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
 
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
[DSC 2016] 系列活動:李宏毅 / 一天搞懂深度學習
 
Introduction to Big Data/Machine Learning
Introduction to Big Data/Machine LearningIntroduction to Big Data/Machine Learning
Introduction to Big Data/Machine Learning
 
What Makes Great Infographics
What Makes Great InfographicsWhat Makes Great Infographics
What Makes Great Infographics
 
Masters of SlideShare
Masters of SlideShareMasters of SlideShare
Masters of SlideShare
 

Similar to 2014-06-20 Multinomial Logistic Regression with Apache Spark

Alpine Spark Implementation - Technical
Alpine Spark Implementation - TechnicalAlpine Spark Implementation - Technical
Alpine Spark Implementation - Technical
alpinedatalabs
 
SPU Optimizations - Part 2
SPU Optimizations - Part 2SPU Optimizations - Part 2
SPU Optimizations - Part 2
Naughty Dog
 
Pytorch meetup
Pytorch meetupPytorch meetup
Pytorch meetup
Dmitri Azarnyh
 
Introduction to PyTorch
Introduction to PyTorchIntroduction to PyTorch
Introduction to PyTorch
Jun Young Park
 
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUsEarly Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Jeff Larkin
 
04 Multi-layer Feedforward Networks
04 Multi-layer Feedforward Networks04 Multi-layer Feedforward Networks
04 Multi-layer Feedforward Networks
Tamer Ahmed Farrag, PhD
 
Efficient Hill Climber for Multi-Objective Pseudo-Boolean Optimization
Efficient Hill Climber for Multi-Objective Pseudo-Boolean OptimizationEfficient Hill Climber for Multi-Objective Pseudo-Boolean Optimization
Efficient Hill Climber for Multi-Objective Pseudo-Boolean Optimization
jfrchicanog
 
Learning Deep Learning
Learning Deep LearningLearning Deep Learning
Learning Deep Learning
simaokasonse
 
Dafunctor
DafunctorDafunctor
Dafunctor
Buganini Chiu
 
Time complexity.ppt
Time complexity.pptTime complexity.ppt
Time complexity.ppt
YekoyeTigabuYeko
 
how to calclute time complexity of algortihm
how to calclute time complexity of algortihmhow to calclute time complexity of algortihm
how to calclute time complexity of algortihm
Sajid Marwat
 
DFA minimization algorithms in map reduce
DFA minimization algorithms in map reduceDFA minimization algorithms in map reduce
DFA minimization algorithms in map reduce
Iraj Hedayati
 
lecture01_lecture01_lecture0001_ceva.pdf
lecture01_lecture01_lecture0001_ceva.pdflecture01_lecture01_lecture0001_ceva.pdf
lecture01_lecture01_lecture0001_ceva.pdf
AnaNeacsu5
 
Using CNTK's Python Interface for Deep LearningDave DeBarr -
Using CNTK's Python Interface for Deep LearningDave DeBarr - Using CNTK's Python Interface for Deep LearningDave DeBarr -
Using CNTK's Python Interface for Deep LearningDave DeBarr -
PyData
 
Algorithm Design and Complexity - Course 11
Algorithm Design and Complexity - Course 11Algorithm Design and Complexity - Course 11
Algorithm Design and Complexity - Course 11
Traian Rebedea
 
20181212 - PGconfASIA - LT - English
20181212 - PGconfASIA - LT - English20181212 - PGconfASIA - LT - English
20181212 - PGconfASIA - LT - English
Kohei KaiGai
 
parallel
parallelparallel
Scala as a Declarative Language
Scala as a Declarative LanguageScala as a Declarative Language
Scala as a Declarative Language
vsssuresh
 
Design and Implementation of Parallel and Randomized Approximation Algorithms
Design and Implementation of Parallel and Randomized Approximation AlgorithmsDesign and Implementation of Parallel and Randomized Approximation Algorithms
Design and Implementation of Parallel and Randomized Approximation Algorithms
Ajay Bidyarthy
 
Spark 4th Meetup Londond - Building a Product with Spark
Spark 4th Meetup Londond - Building a Product with SparkSpark 4th Meetup Londond - Building a Product with Spark
Spark 4th Meetup Londond - Building a Product with Spark
samthemonad
 

Similar to 2014-06-20 Multinomial Logistic Regression with Apache Spark (20)

Alpine Spark Implementation - Technical
Alpine Spark Implementation - TechnicalAlpine Spark Implementation - Technical
Alpine Spark Implementation - Technical
 
SPU Optimizations - Part 2
SPU Optimizations - Part 2SPU Optimizations - Part 2
SPU Optimizations - Part 2
 
Pytorch meetup
Pytorch meetupPytorch meetup
Pytorch meetup
 
Introduction to PyTorch
Introduction to PyTorchIntroduction to PyTorch
Introduction to PyTorch
 
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUsEarly Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
Early Results of OpenMP 4.5 Portability on NVIDIA GPUs & CPUs
 
04 Multi-layer Feedforward Networks
04 Multi-layer Feedforward Networks04 Multi-layer Feedforward Networks
04 Multi-layer Feedforward Networks
 
Efficient Hill Climber for Multi-Objective Pseudo-Boolean Optimization
Efficient Hill Climber for Multi-Objective Pseudo-Boolean OptimizationEfficient Hill Climber for Multi-Objective Pseudo-Boolean Optimization
Efficient Hill Climber for Multi-Objective Pseudo-Boolean Optimization
 
Learning Deep Learning
Learning Deep LearningLearning Deep Learning
Learning Deep Learning
 
Dafunctor
DafunctorDafunctor
Dafunctor
 
Time complexity.ppt
Time complexity.pptTime complexity.ppt
Time complexity.ppt
 
how to calclute time complexity of algortihm
how to calclute time complexity of algortihmhow to calclute time complexity of algortihm
how to calclute time complexity of algortihm
 
DFA minimization algorithms in map reduce
DFA minimization algorithms in map reduceDFA minimization algorithms in map reduce
DFA minimization algorithms in map reduce
 
lecture01_lecture01_lecture0001_ceva.pdf
lecture01_lecture01_lecture0001_ceva.pdflecture01_lecture01_lecture0001_ceva.pdf
lecture01_lecture01_lecture0001_ceva.pdf
 
Using CNTK's Python Interface for Deep LearningDave DeBarr -
Using CNTK's Python Interface for Deep LearningDave DeBarr - Using CNTK's Python Interface for Deep LearningDave DeBarr -
Using CNTK's Python Interface for Deep LearningDave DeBarr -
 
Algorithm Design and Complexity - Course 11
Algorithm Design and Complexity - Course 11Algorithm Design and Complexity - Course 11
Algorithm Design and Complexity - Course 11
 
20181212 - PGconfASIA - LT - English
20181212 - PGconfASIA - LT - English20181212 - PGconfASIA - LT - English
20181212 - PGconfASIA - LT - English
 
parallel
parallelparallel
parallel
 
Scala as a Declarative Language
Scala as a Declarative LanguageScala as a Declarative Language
Scala as a Declarative Language
 
Design and Implementation of Parallel and Randomized Approximation Algorithms
Design and Implementation of Parallel and Randomized Approximation AlgorithmsDesign and Implementation of Parallel and Randomized Approximation Algorithms
Design and Implementation of Parallel and Randomized Approximation Algorithms
 
Spark 4th Meetup Londond - Building a Product with Spark
Spark 4th Meetup Londond - Building a Product with SparkSpark 4th Meetup Londond - Building a Product with Spark
Spark 4th Meetup Londond - Building a Product with Spark
 

More from DB Tsai

2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
DB Tsai
 
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
DB Tsai
 
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
DB Tsai
 
2014-08-14 Alpine Innovation to Spark
2014-08-14 Alpine Innovation to Spark2014-08-14 Alpine Innovation to Spark
2014-08-14 Alpine Innovation to Spark
DB Tsai
 
Large-Scale Machine Learning with Apache Spark
Large-Scale Machine Learning with Apache SparkLarge-Scale Machine Learning with Apache Spark
Large-Scale Machine Learning with Apache Spark
DB Tsai
 
Unsupervised Learning with Apache Spark
Unsupervised Learning with Apache SparkUnsupervised Learning with Apache Spark
Unsupervised Learning with Apache Spark
DB Tsai
 

More from DB Tsai (6)

2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
2017 Netflix's Recommendation ML Pipeline Using Apache Spark: Spark Summit Ea...
 
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
 
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
 
2014-08-14 Alpine Innovation to Spark
2014-08-14 Alpine Innovation to Spark2014-08-14 Alpine Innovation to Spark
2014-08-14 Alpine Innovation to Spark
 
Large-Scale Machine Learning with Apache Spark
Large-Scale Machine Learning with Apache SparkLarge-Scale Machine Learning with Apache Spark
Large-Scale Machine Learning with Apache Spark
 
Unsupervised Learning with Apache Spark
Unsupervised Learning with Apache SparkUnsupervised Learning with Apache Spark
Unsupervised Learning with Apache Spark
 

Recently uploaded

Protect YugabyteDB with Hashicorp Vault.pdf
Protect YugabyteDB with Hashicorp Vault.pdfProtect YugabyteDB with Hashicorp Vault.pdf
Protect YugabyteDB with Hashicorp Vault.pdf
Gwenn Etourneau
 
Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...
Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...
Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...
John Gallagher
 
Digital Image Processing - Module 4 Chapter 2
Digital Image Processing - Module 4 Chapter 2Digital Image Processing - Module 4 Chapter 2
Digital Image Processing - Module 4 Chapter 2
821priyankaj
 
How to Formulate A Good Research Question
How to Formulate A  Good Research QuestionHow to Formulate A  Good Research Question
How to Formulate A Good Research Question
rkpv2002
 
Developing a Genetic Algorithm Based Daily Calorie Recommendation System for ...
Developing a Genetic Algorithm Based Daily Calorie Recommendation System for ...Developing a Genetic Algorithm Based Daily Calorie Recommendation System for ...
Developing a Genetic Algorithm Based Daily Calorie Recommendation System for ...
AIRCC Publishing Corporation
 
UNIT 1 - INTRODUCTION ON DISASTER MANAGEMENT.ppt
UNIT 1 - INTRODUCTION ON DISASTER MANAGEMENT.pptUNIT 1 - INTRODUCTION ON DISASTER MANAGEMENT.ppt
UNIT 1 - INTRODUCTION ON DISASTER MANAGEMENT.ppt
shanmugamram247
 
ANATOMY OF SOA - Thomas Erl - Service Oriented Architecture
ANATOMY OF SOA - Thomas Erl - Service Oriented ArchitectureANATOMY OF SOA - Thomas Erl - Service Oriented Architecture
ANATOMY OF SOA - Thomas Erl - Service Oriented Architecture
Divya Rajasekar
 
AFCAT STATIC Genral knowledge important CAPSULE.pdf
AFCAT STATIC Genral knowledge important CAPSULE.pdfAFCAT STATIC Genral knowledge important CAPSULE.pdf
AFCAT STATIC Genral knowledge important CAPSULE.pdf
vibhapatil140
 
DPWH - DEPARTMENT OF PUBLIC WORKS AND HIGHWAYS
DPWH - DEPARTMENT OF PUBLIC WORKS AND HIGHWAYSDPWH - DEPARTMENT OF PUBLIC WORKS AND HIGHWAYS
DPWH - DEPARTMENT OF PUBLIC WORKS AND HIGHWAYS
RyanMacayan
 
Ebara corporation introduction. looking ahead, going beyond expectation
Ebara corporation introduction. looking ahead, going beyond expectationEbara corporation introduction. looking ahead, going beyond expectation
Ebara corporation introduction. looking ahead, going beyond expectation
nanjay2
 
李易峰祝绪丹做爱视频流出【网芷:ht28.co】可爱学生妹>>>[网趾:ht28.co】]<<<
李易峰祝绪丹做爱视频流出【网芷:ht28.co】可爱学生妹>>>[网趾:ht28.co】]<<<李易峰祝绪丹做爱视频流出【网芷:ht28.co】可爱学生妹>>>[网趾:ht28.co】]<<<
李易峰祝绪丹做爱视频流出【网芷:ht28.co】可爱学生妹>>>[网趾:ht28.co】]<<<
amzhoxvzidbke
 
Concepts of Automatic Block Signalling.ppt
Concepts of Automatic Block Signalling.pptConcepts of Automatic Block Signalling.ppt
Concepts of Automatic Block Signalling.ppt
princeshah76
 
The Control of Relative Humidity & Moisture Content in The Air
The Control of Relative Humidity & Moisture Content in The AirThe Control of Relative Humidity & Moisture Content in The Air
The Control of Relative Humidity & Moisture Content in The Air
Ashraf Ismail
 
Indian Railway Signalling concepts and basics.pdf
Indian Railway Signalling concepts and basics.pdfIndian Railway Signalling concepts and basics.pdf
Indian Railway Signalling concepts and basics.pdf
princeshah76
 
"Operational and Technical Overview of Electric Locomotives at the Kanpur Ele...
"Operational and Technical Overview of Electric Locomotives at the Kanpur Ele..."Operational and Technical Overview of Electric Locomotives at the Kanpur Ele...
"Operational and Technical Overview of Electric Locomotives at the Kanpur Ele...
nanduchaihan9
 
Updated Limitations of Simplified Methods for Evaluating the Potential for Li...
Updated Limitations of Simplified Methods for Evaluating the Potential for Li...Updated Limitations of Simplified Methods for Evaluating the Potential for Li...
Updated Limitations of Simplified Methods for Evaluating the Potential for Li...
Robert Pyke
 
Safety Operating Procedure for Testing Lifting Tackles
Safety Operating Procedure for Testing Lifting TacklesSafety Operating Procedure for Testing Lifting Tackles
Safety Operating Procedure for Testing Lifting Tackles
ssuserfcf701
 
Generative AI and Large Language Models (LLMs)
Generative AI and Large Language Models (LLMs)Generative AI and Large Language Models (LLMs)
Generative AI and Large Language Models (LLMs)
rkpv2002
 
AI chapter1 introduction to artificial intelligence
AI chapter1 introduction to artificial intelligenceAI chapter1 introduction to artificial intelligence
AI chapter1 introduction to artificial intelligence
GeethaAL
 
TestRigor - Element Location Rules and UI Grid system
TestRigor - Element Location Rules and UI Grid systemTestRigor - Element Location Rules and UI Grid system
TestRigor - Element Location Rules and UI Grid system
artembondar5
 

Recently uploaded (20)

Protect YugabyteDB with Hashicorp Vault.pdf
Protect YugabyteDB with Hashicorp Vault.pdfProtect YugabyteDB with Hashicorp Vault.pdf
Protect YugabyteDB with Hashicorp Vault.pdf
 
Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...
Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...
Fix Production Bugs Quickly - The Power of Structured Logging in Ruby on Rail...
 
Digital Image Processing - Module 4 Chapter 2
Digital Image Processing - Module 4 Chapter 2Digital Image Processing - Module 4 Chapter 2
Digital Image Processing - Module 4 Chapter 2
 
How to Formulate A Good Research Question
How to Formulate A  Good Research QuestionHow to Formulate A  Good Research Question
How to Formulate A Good Research Question
 
Developing a Genetic Algorithm Based Daily Calorie Recommendation System for ...
Developing a Genetic Algorithm Based Daily Calorie Recommendation System for ...Developing a Genetic Algorithm Based Daily Calorie Recommendation System for ...
Developing a Genetic Algorithm Based Daily Calorie Recommendation System for ...
 
UNIT 1 - INTRODUCTION ON DISASTER MANAGEMENT.ppt
UNIT 1 - INTRODUCTION ON DISASTER MANAGEMENT.pptUNIT 1 - INTRODUCTION ON DISASTER MANAGEMENT.ppt
UNIT 1 - INTRODUCTION ON DISASTER MANAGEMENT.ppt
 
ANATOMY OF SOA - Thomas Erl - Service Oriented Architecture
ANATOMY OF SOA - Thomas Erl - Service Oriented ArchitectureANATOMY OF SOA - Thomas Erl - Service Oriented Architecture
ANATOMY OF SOA - Thomas Erl - Service Oriented Architecture
 
AFCAT STATIC Genral knowledge important CAPSULE.pdf
AFCAT STATIC Genral knowledge important CAPSULE.pdfAFCAT STATIC Genral knowledge important CAPSULE.pdf
AFCAT STATIC Genral knowledge important CAPSULE.pdf
 
DPWH - DEPARTMENT OF PUBLIC WORKS AND HIGHWAYS
DPWH - DEPARTMENT OF PUBLIC WORKS AND HIGHWAYSDPWH - DEPARTMENT OF PUBLIC WORKS AND HIGHWAYS
DPWH - DEPARTMENT OF PUBLIC WORKS AND HIGHWAYS
 
Ebara corporation introduction. looking ahead, going beyond expectation
Ebara corporation introduction. looking ahead, going beyond expectationEbara corporation introduction. looking ahead, going beyond expectation
Ebara corporation introduction. looking ahead, going beyond expectation
 
李易峰祝绪丹做爱视频流出【网芷:ht28.co】可爱学生妹>>>[网趾:ht28.co】]<<<
李易峰祝绪丹做爱视频流出【网芷:ht28.co】可爱学生妹>>>[网趾:ht28.co】]<<<李易峰祝绪丹做爱视频流出【网芷:ht28.co】可爱学生妹>>>[网趾:ht28.co】]<<<
李易峰祝绪丹做爱视频流出【网芷:ht28.co】可爱学生妹>>>[网趾:ht28.co】]<<<
 
Concepts of Automatic Block Signalling.ppt
Concepts of Automatic Block Signalling.pptConcepts of Automatic Block Signalling.ppt
Concepts of Automatic Block Signalling.ppt
 
The Control of Relative Humidity & Moisture Content in The Air
The Control of Relative Humidity & Moisture Content in The AirThe Control of Relative Humidity & Moisture Content in The Air
The Control of Relative Humidity & Moisture Content in The Air
 
Indian Railway Signalling concepts and basics.pdf
Indian Railway Signalling concepts and basics.pdfIndian Railway Signalling concepts and basics.pdf
Indian Railway Signalling concepts and basics.pdf
 
"Operational and Technical Overview of Electric Locomotives at the Kanpur Ele...
"Operational and Technical Overview of Electric Locomotives at the Kanpur Ele..."Operational and Technical Overview of Electric Locomotives at the Kanpur Ele...
"Operational and Technical Overview of Electric Locomotives at the Kanpur Ele...
 
Updated Limitations of Simplified Methods for Evaluating the Potential for Li...
Updated Limitations of Simplified Methods for Evaluating the Potential for Li...Updated Limitations of Simplified Methods for Evaluating the Potential for Li...
Updated Limitations of Simplified Methods for Evaluating the Potential for Li...
 
Safety Operating Procedure for Testing Lifting Tackles
Safety Operating Procedure for Testing Lifting TacklesSafety Operating Procedure for Testing Lifting Tackles
Safety Operating Procedure for Testing Lifting Tackles
 
Generative AI and Large Language Models (LLMs)
Generative AI and Large Language Models (LLMs)Generative AI and Large Language Models (LLMs)
Generative AI and Large Language Models (LLMs)
 
AI chapter1 introduction to artificial intelligence
AI chapter1 introduction to artificial intelligenceAI chapter1 introduction to artificial intelligence
AI chapter1 introduction to artificial intelligence
 
TestRigor - Element Location Rules and UI Grid system
TestRigor - Element Location Rules and UI Grid systemTestRigor - Element Location Rules and UI Grid system
TestRigor - Element Location Rules and UI Grid system
 

2014-06-20 Multinomial Logistic Regression with Apache Spark

  • 1. Multinomial Logistic Regression with Apache Spark DB Tsai dbtsai@alpinenow.com Machine Learning Engineer June 20th , 2014 Silicon Valley Machine Learning Meetup
  • 2. What is Alpine Data Labs doing? ● Collaborative, Code-Free, Advanced Analytics Solution for Big Data
  • 3. ● Technology we're using: Scala/Java, Akka, Spray, Hadoop, Spark/SparkSQL, Pig, Sqoop ● Our platform runs against different flavors of Hadoop distributions such as Cloudera, Pivotal Hadoop, MapR, HortonWorks and Apache. ● Actively involved in the open source community: almost of all our newly developed algorithms in Spark will be contributed back to MLLib. ● Already committed a L-BFGS optimizer to Spark, and helped fix couple bugs. Working on multinational logistic regression, GLM, Decision Tree and a Random Forest ● In addition we are the maintainer of several open source projects including Chorus, SBT plugin for JUnit test Listener and several other projects. We're open source friendly!
  • 4. We're hiring! ● Machine Learning Engineer ● Data Scientist ● UI/UX Engineer ● Front-end Engineer ● Infrastructure Engineer ● Back-end Engineer ● Automation Test Engineer Shoot me an email at dbtsai@alpinenow.com
  • 5. Machine Learning with Big Data ● Hadoop MapReduce solutions ● MapReduce is scaling well for batch processing ● Lots of machine learning algorithms are iterative by nature. ● There are lots of tricks people do, like training with subsamples of data, and then average the models. Why big data with approximation? + =
  • 6. Lightning-fast cluster computing ● Empower users to iterate through the data by utilizing the in-memory cache. ● Logistic regression runs up to 100x faster than Hadoop M/R in memory. ● We're able to train exact model without doing any approximation.
  • 7. Binary Logistic Regression d = ax1+bx2+cx0 √a2 +b2 where x0=1 P( y=1∣⃗x , ⃗w)= exp(d ) 1+exp(d ) = exp(⃗x ⃗w) 1+exp(⃗x ⃗w) P ( y=0∣⃗x , ⃗w)= 1 1+exp(⃗x ⃗w) log P ( y=1∣⃗x , ⃗w) P( y=0∣⃗x , ⃗w) =⃗x ⃗w w0= c √a2 +b2 w1= a √a2 +b2 w2= b √a2 +b2 where w0 iscalled as intercept
  • 8. Training Binary Logistic Regression ● Maximum Likelihood estimation From a training data and labels ● We want to find that maximizes the likelihood of data defined by ● We can take log of the equation, and minimize it instead. The Log-Likelihood becomes the loss function. X =( ⃗x1 , ⃗x2 , ⃗x3 ,...) Y =( y1, y2, y3, ...) ⃗w L( ⃗w , ⃗x1, ... , ⃗xN )=P ( y1∣⃗x1 , ⃗w)P ( y2∣⃗x2 , ⃗w)... P( yN∣ ⃗xN , ⃗w) l( ⃗w ,⃗x)=log P(y1∣⃗x1 , ⃗w)+log P( y2∣⃗x2 , ⃗w)...+log P( yN∣ ⃗xN , ⃗w)
  • 9. Optimization ● First Order Minimizer Require loss, gradient of loss function – Gradient Decent is step size – Limited-memory BFGS (L-BFGS) – Orthant-Wise Limited-memory Quasi-Newton (OWLQN) – Coordinate Descent (CD) – Trust Region Newton Method (TRON) ● Second Order Minimizer Require loss, gradient and hessian of loss function – Newton-Raphson, quadratic convergence. Fast! ● Ref: Journal of Machine Learning Research 11 (2010) 3183- 3234, Chih-Jen Lin et al. ⃗wn+1=⃗wn−γ ⃗G , γ ⃗wn+1=⃗wn−H−1 ⃗G
  • 10. Problem of Second Order Minimizer ● Scale horizontally (the numbers of training data) by leveraging Spark to parallelize this iterative optimization process. ● Don't scale vertically (the numbers of training features). Dimension of Hessian is ● Recent applications from document classification and computational linguistics are of this type. dim(H )=[(k−1)(n+1)] 2 where k is numof class ,nis num of features
  • 11. L-BFGS ● It's a quasi-Newton method. ● Hessian matrix of second derivatives doesn't need to be evaluated directly. ● Hessian matrix is approximated using gradient evaluations. ● It converges a way faster than the default optimizer in Spark, Gradient Decent. ● We love open source! Alpine Data Labs contributed our L-BFGS to Spark, and it's already merged in Spark-1157.
  • 12. Training Binary Logistic Regression l( ⃗w ,⃗x) = ∑ k=1 N log P( yk∣⃗xk , ⃗w) = ∑k=1 N yk log P ( yk=1∣⃗xk , ⃗w)+(1−yk )log P ( yk =0∣⃗xk , ⃗w) = ∑k=1 N yk log exp( ⃗xk ⃗w) 1+exp( ⃗xk ⃗w) +(1−yk )log 1 1+exp( ⃗xk ⃗w) = ∑k=1 N yk ⃗xk ⃗w−log(1+exp( ⃗xk ⃗w)) Gradient : Gi (⃗w ,⃗x)= ∂l( ⃗w ,⃗x) ∂ wi =∑k=1 N yk xki− exp( ⃗xk ⃗w) 1+exp( ⃗xk ⃗w) xki Hessian: H ij (⃗w ,⃗x)= ∂∂l( ⃗w ,⃗x) ∂wi ∂w j =−∑k=1 N exp( ⃗xk ⃗w) (1+exp( ⃗xk ⃗w))2 xki xkj
  • 13. Overfitting P ( y=1∣⃗x , ⃗w)= exp(zd ) 1+exp(zd ) = exp(⃗x ⃗w) 1+exp(⃗x ⃗w)
  • 14. Regularization ● The loss function becomes ● The loss function of regularizer doesn't depend on data. Common regularizers are – L2 Regularization: – L1 Regularization: ● L1 norm is not differentiable at zero! ltotal (⃗w ,⃗x)=lmodel (⃗w ,⃗x)+lreg ( ⃗w) lreg (⃗w)=λ∑i=1 N wi 2 lreg (⃗w)=λ∑i=1 N ∣wi∣ ⃗G (⃗w ,⃗x)total=⃗G(⃗w ,⃗x)model+⃗G( ⃗w)reg ̄H ( ⃗w ,⃗x)total= ̄H (⃗w ,⃗x)model+ ̄H (⃗w)reg
  • 15. Mini School of Spark APIs ● map(func) : Return a new distributed dataset formed by passing each element of the source through a function func. ● reduce(func) : Aggregate the elements of the dataset using a function func (which takes two arguments and returns one). The function should be commutative and associative so that it can be computed correctly in parallel. This is O(n) operation where n is number of partition. In SPARK-2174, treeReduce will be merged for O(log(n)) operation.
  • 16. Example – compute the mean of numbers. 3.0 2.0 1.0 5.0 Executor 1 Executor 2 map (3.0, 1) (2.0, 1) reduce (5.0, 2) reduce (11.0, 4) Shuffle reduce (6.0, 2) (1.0, 1) (5.0, 1) map Executor 3
  • 17. Mini School of Spark APIs ● mapPartitions(func) : Similar to map, but runs separately on each partition (block) of the RDD, so func must be of type Iterator[T] => Iterator[U] when running on an RDD of type T. ● This API allows us to have global variables on entire partition to aggregate the result locally and efficiently.
  • 18. Better Mean of Numbers Implementation 3.0 2.0 Executor 1 reduce (11.0, 4) Shuffle var counts = 0 var sum = 0.0 loop var counts = 2 var sum = 5.0 3.0 2.0 (5.0, 2) 1.0 5.0 Executor 2 var counts = 0 var sum = 0.0 loop var counts = 2 var sum = 6.0 1.0 5.0 (6.0, 2) mapPartitions
  • 19. More Idiomatic Scala Implementation ● aggregate(zeroValue: U)(seqOp: (U, T) => U, combOp: (U, U) => U): U zeroValue is a neutral "zero value" with type U for initialization. This is analogous to in previous example. var counts = 0 var sum = 0.0 seqOp is function taking (U, T) => U, where U is aggregator initialized as zeroValue. T is each line of values in RDD. This is analogous to mapPartition in previous example. combOp is function taking (U, U) => U. This is essentially combing the results between different executors. The functionality is the same as reduce(func) in previous example.
  • 20. Approach of Parallelization in Spark Spark Driver JVM Time Spark Executor 1 JVM Spark Executor 2 JVM 1) Find available resources in cluster, and launch executor JVMs. Also initialize the weights. 2) Ask executors to load the data into executors' JVMs 2) Trying to load the data into memory. If the data is bigger than memory, it will be partial cached. (The data locality from source will be taken care by Spark) 3) Ask executors to compute loss, and gradient of each training sample (each row) given the current weights. Get the aggregated results after the Reduce Phase in executors. 5) If the regularization is enabled, compute the loss and gradient of regularizer in driver since it doesn't depend on training data but only depends on weights. Add them into the results from executors. 3) Map Phase: Compute the loss and gradient of each row of training data locally given the weights obtained from the driver. Can either emit each result or sum them up in local aggregators. 4) Reduce Phase: Sum up the losses and gradients emitted from the Map Phase Taking a rest! 6) Plug the loss and gradient from model and regularizer into optimizer to get the new weights. If the differences of weights and losses are larger than criteria, GO BACK TO 3) Taking a rest! 7) Finish the model training! Taking a rest!
  • 21. Step 3) and 4) ● This is the implementation of step 3) and 4) in MLlib before Spark 1.0 ● gradient can have implementations of Logistic Regression, Linear Regression, SVM, or any customized cost function. ● Each training data will create new “grad” object after gradient.compute.
  • 22. Step 3) and 4) with mapPartitions
  • 23. Step 3) and 4) with aggregate ● This is the implementation of step 3) and 4) in MLlib in coming Spark 1.0 ● No unnecessary object creation! It's helpful when we're dealing with large features training data. GC will not kick in the executor JVMs.
  • 24. Extension to Multinomial Logistic Regression ● In Binary Logistic Regression ● For K classes multinomial problem where labels ranged from [0, K-1], we can generalize it via ● The model, weights becomes (K-1)(N+1) matrix, where N is number of features. log P( y=1∣⃗x , ̄w) P ( y=0∣⃗x , ̄w) =⃗x ⃗w1 log P ( y=2∣⃗x , ̄w) P ( y=0∣⃗x , ̄w) =⃗x ⃗w2 ... log P ( y=K −1∣⃗x , ̄w) P ( y=0∣⃗x , ̄w) =⃗x ⃗wK −1 log P( y=1∣⃗x , ⃗w) P ( y=0∣⃗x , ⃗w) =⃗x ⃗w ̄w=(⃗w1, ⃗w2, ... , ⃗wK −1)T P( y=0∣⃗x , ̄w)= 1 1+∑ i=1 K −1 exp(⃗x ⃗wi ) P( y=1∣⃗x , ̄w)= exp(⃗x ⃗w2) 1+∑ i=1 K −1 exp(⃗x ⃗wi) ... P( y=K −1∣⃗x , ̄w)= exp(⃗x ⃗wK −1) 1+∑i=1 K −1 exp(⃗x ⃗wi )
  • 25. Training Multinomial Logistic Regression l( ̄w ,⃗x) = ∑ k=1 N log P( yk∣⃗xk , ̄w) = ∑k=1 N α( yk)log P( y=0∣⃗xk , ̄w)+(1−α( yk ))log P ( yk∣⃗xk , ̄w) = ∑k=1 N α( yk)log 1 1+∑ i=1 K −1 exp(⃗x ⃗wi) +(1−α( yk ))log exp(⃗x ⃗w yk ) 1+∑ i=1 K−1 exp(⃗x ⃗wi) = ∑ k=1 N (1−α( yk ))⃗x ⃗wyk −log(1+∑ i=1 K−1 exp(⃗x ⃗wi)) Gradient : Gij ( ̄w ,⃗x)= ∂l( ̄w ,⃗x) ∂wij =∑k=1 N (1−α( yk )) xkj δi , yk − exp( ⃗xk ⃗w) 1+exp( ⃗xk ⃗w) xkj α( yk )=1 if yk=0 α( yk )=0 if yk≠0 Define: Note that the first index “i” is for classes, and the second index “j” is for features. Hessian: H ij ,lm( ̄w ,⃗x)= ∂∂l ( ̄w ,⃗x) ∂wij ∂wlm
  • 26. 0 5 10 15 20 25 30 35 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 Logistic Regression with a9a Dataset (11M rows, 123 features, 11% non-zero elements) 16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster L-BFGS Dense Features L-BFGS Sparse Features GD Sparse Features GD Dense Features Seconds Log-Likelihood/NumberofSamples a9a Dataset Benchmark
  • 27. a9a Dataset Benchmark -1 1 3 5 7 9 11 13 15 0.3 0.35 0.4 0.45 0.5 0.55 0.6 0.65 0.7 Logistic Regression with a9a Dataset (11M rows, 123 features, 11% non-zero elements) 16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster L-BFGS GD Iterations Log-Likelihood/NumberofSamples
  • 28. 0 5 10 15 20 25 30 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Logistic Regression with rcv1 Dataset (6.8M rows, 677,399 features, 0.15% non-zero elements) 16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster LBFGS Sparse Vector GD Sparse Vector Second Log-Likelihood/NumberofSamples rcv1 Dataset Benchmark
  • 29. news20 Dataset Benchmark 0 10 20 30 40 50 60 70 80 0 0.2 0.4 0.6 0.8 1 1.2 Logistic Regression with news20 Dataset (0.14M rows, 1,355,191 features, 0.034% non-zero elements) 16 executors in INTEL Xeon E3-1230v3 32GB Memory * 5 nodes Hadoop 2.0.5 alpha cluster LBFGS Sparse Vector GD Sparse Vector Second Log-Likelihood/NumberofSamples
  • 33. Conclusion ● We're hiring! ● Spark runs programs 100x faster than Hadoop ● Spark turns iterative big data machine learning problems into single machine problems ● MLlib provides lots of state of art machine learning implementations, e.g. K-means, linear regression, logistic regression, ALS, collaborative filtering, and Naive Bayes, etc. ● Spark provides ease of use APIs in Java, Scala, or Python, and interactive Scala and Python shell ● Spark is Apache project, and it's the most active big data open source project nowadays.