Machine Learning on Big Data

Machine Learning on Big DataLessons Learned from Google ProjectsMax LinSoftware Engineer | Google ResearchMassively Parallel Computing | Harvard CS 264Guest Lecture | March 29th, 2011

Outline• Machine Learning intro• Scaling machine learning algorithms up• Design choices of large scale ML systems

“Machine Learning is a studyof computer algorithms that improve automatically through experience.”

The quick brown foxjumped over the lazy dog.

The quick brown fox Englishjumped over the lazy dog.

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up youneed a computer.

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.No hay mal que por bienno venga.

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.No hay mal que por bien Spanishno venga.

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.No hay mal que por bien Spanishno venga.La tercera es la vencida.

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.No hay mal que por bien Spanishno venga.La tercera es la vencida. Spanish

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.No hay mal que por bien Spanishno venga.La tercera es la vencida. SpanishTo be or not to be -- thatis the question

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.No hay mal que por bien Spanishno venga.La tercera es la vencida. SpanishTo be or not to be -- that ?is the question

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.No hay mal que por bien Spanishno venga.La tercera es la vencida. SpanishTo be or not to be -- that ?is the questionLa fe mueve montañas.

The quick brown fox Englishjumped over the lazy dog.To err is human, but toreally foul things up you Englishneed a computer.No hay mal que por bien Spanishno venga.La tercera es la vencida. SpanishTo be or not to be -- that ?is the questionLa fe mueve montañas. ?

The quick brown fox English jumped over the lazy dog. To err is human, but to really foul things up you EnglishTraining need a computer. No hay mal que por bien Spanish no venga. La tercera es la vencida. Spanish To be or not to be -- that ? is the question La fe mueve montañas. ?

The quick brown fox English jumped over the lazy dog. To err is human, but to really foul things up you EnglishTraining Input X need a computer. No hay mal que por bien Spanish no venga. La tercera es la vencida. Spanish To be or not to be -- that ? is the question La fe mueve montañas. ?

The quick brown fox English jumped over the lazy dog. To err is human, but to really foul things up you EnglishTraining Input X need a computer. Output Y No hay mal que por bien Spanish no venga. La tercera es la vencida. Spanish To be or not to be -- that ? is the question La fe mueve montañas. ?

The quick brown fox English jumped over the lazy dog. To err is human, but to really foul things up you EnglishTraining Input X need a computer. Output Y No hay mal que por bien Spanish no venga. Model f(x) La tercera es la vencida. Spanish To be or not to be -- that ? is the question La fe mueve montañas. ?

The quick brown fox English jumped over the lazy dog. To err is human, but to really foul things up you EnglishTraining Input X need a computer. Output Y No hay mal que por bien Spanish no venga. Model f(x) La tercera es la vencida. Spanish To be or not to be -- that ?Testing is the question La fe mueve montañas. ?

The quick brown fox English jumped over the lazy dog. To err is human, but to really foul things up you EnglishTraining Input X need a computer. Output Y No hay mal que por bien Spanish no venga. Model f(x) La tercera es la vencida. Spanish To be or not to be -- that ?Testing f(x’) is the question La fe mueve montañas. ?

The quick brown fox English jumped over the lazy dog. To err is human, but to really foul things up you EnglishTraining Input X need a computer. Output Y No hay mal que por bien Spanish no venga. Model f(x) La tercera es la vencida. Spanish To be or not to be -- that ?Testing f(x’) is the question = y’ La fe mueve montañas. ?

Linear Classiﬁer

Linear ClassiﬁerThe quick brown fox jumped over the lazy dog.

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ...0,

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ... 0,

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ... 0, ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ... 0, ... 1,

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ... 0, ... 1, ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ... 0, ... 1, ... 1,

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ... 0, ... 1, ... 1, ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ... 0, ... 1, ... 1, ... 0,

Linear Classiﬁer The quick brown fox jumped over the lazy dog.‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ... 0, ... 0, ... 1, ... 1, ... 0, ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog. ‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ...[ 0, ... 0, ... 1, ... 1, ... 0, ...

Linear Classiﬁer The quick brown fox jumped over the lazy dog. ‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ...[ 0, ... 0, ... 1, ... 1, ... 0, ... ]

Linear Classiﬁer The quick brown fox jumped over the lazy dog. ‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ...x [ 0, ... 0, ... 1, ... 1, ... 0, ... ]

Linear Classiﬁer The quick brown fox jumped over the lazy dog. ‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ...x [ 0, ... 0, ... 1, ... 1, ... 0, ... ] [ 0.1, ... 132, ... 150, ... 200, ... -153, ... ]

Linear Classiﬁer The quick brown fox jumped over the lazy dog. ‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ...x [ 0, ... 0, ... 1, ... 1, ... 0, ... ]w [ 0.1, ... 132, ... 150, ... 200, ... -153, ... ]

Linear Classiﬁer The quick brown fox jumped over the lazy dog. ‘a’ ... ‘aardvark’ ... ‘dog’ ... ‘the’ ... ‘montañas’ ...x [ 0, ... 0, ... 1, ... 1, ... 0, ... ]w [ 0.1, ... 132, ... 150, ... 200, ... -153, ... ] P f (x) = w · x = w p ∗ xp p=1

Training Data Input X Ouput Y P ... ... ...N ... ... ... ... ... ... ...

Typical machine learningdata at GoogleN: 100 billions / 1 billionP: 1 billion / 10 million(mean / median) http://www.ﬂickr.com/photos/mr_t_in_dc/5469563053

Classiﬁer Training• Training: Given {(x, y)} and f, minimize the following objective function N arg min L(yi , f (xi ; w)) + R(w) w n=1

Use Newton’s method? t +1 t t −1 tw ← w − H(w ) J(w ) http://www.ﬂickr.com/photos/visitﬁnland/5424369765/

Scaling Up

Scaling Up• Why big data?

Scaling Up• Why big data?• Parallelize machine learning algorithms

Scaling Up• Why big data?• Parallelize machine learning algorithms • Embarrassingly parallel

Scaling Up• Why big data?• Parallelize machine learning algorithms • Embarrassingly parallel • Parallelize sub-routines

Scaling Up• Why big data?• Parallelize machine learning algorithms • Embarrassingly parallel • Parallelize sub-routines • Distributed learning

Subsampling

Subsampling Big Data

Subsampling Big DataShard 1 Shard 2 Shard 3 Shard M ...

Subsampling Big DataReduce N Shard 1

Subsampling Big DataReduce N Shard 1 Machine

Subsampling Big DataReduce N Machine Shard 1

Subsampling Big DataReduce N Machine Shard 1 Model

Why not Small Data? [Banko and Brill, 2001]

Parallelize Estimates• Naive Bayes Classiﬁer N P i arg min − P (xp |yi ; w)P (yi ; w) w i=1 p=1• Maximum Likelihood Estimates N i i=1 1EN,the (x ) wthe|EN = N i=1 1EN (xi )

Word Counting

Word CountingMap

Word Counting X: “The quick brown fox ...”Map Y: EN

Word Counting (‘the|EN’, 1) X: “The quick brown fox ...”Map Y: EN

Word Counting (‘the|EN’, 1) X: “The quick brown fox ...”Map (‘quick|EN’, 1) Y: EN

Word Counting (‘the|EN’, 1) X: “The quick brown fox ...”Map (‘quick|EN’, 1) Y: EN (‘brown|EN’, 1)

Word Counting (‘the|EN’, 1) X: “The quick brown fox ...” Map (‘quick|EN’, 1) Y: EN (‘brown|EN’, 1)Reduce

Word Counting

Word Counting Big Data

Word Counting Big DataShard 1 Shard 2 Shard 3 ... Shard M

Word Counting Big Data Mapper 1 Mapper 2 Mapper 3 Mapper MMap Shard 1 Shard 2 Shard 3 ... Shard M (‘the’ | EN, 1)

Word Counting Big Data Mapper 1 Mapper 2 Mapper 3 Mapper M Map Shard 1 Shard 2 Shard 3 ... Shard M (‘the’ | EN, 1) (‘fox’ | EN, 1) ... (‘montañas’ | ES, 1) ReducerReduce Tally counts and update w

Word Counting Big Data Mapper 1 Mapper 2 Mapper 3 Mapper M Map Shard 1 Shard 2 Shard 3 ... Shard M (‘the’ | EN, 1) (‘fox’ | EN, 1) ... (‘montañas’ | ES, 1) ReducerReduce Tally counts and update w Model

Parallelize Optimization N P i yi exp( p=1 wp ∗ xp ) arg min P w i=1 1 + exp( p=1 wp ∗ xi ) p

Parallelize Optimization• Maximum Entropy Classiﬁers P N i yi exp( p=1 wp ∗ xp ) arg min P w i=1 1 + exp( p=1 wp ∗ xi ) p

Parallelize Optimization• Maximum Entropy Classiﬁers P N i yi exp( p=1 wp ∗ xp ) arg min P w i=1 1 + exp( p=1 wp ∗ xi ) p• Good: J(w) is concave

Parallelize Optimization• Maximum Entropy Classiﬁers P N i yi exp( p=1 wp ∗ xp ) arg min P w i=1 1 + exp( p=1 wp ∗ xi ) p• Good: J(w) is concave• Bad: no closed-form solution like NB

Parallelize Optimization• Maximum Entropy Classiﬁers P N i yi exp( p=1 wp ∗ xp ) arg min P w i=1 1 + exp( p=1 wp ∗ xi ) p• Good: J(w) is concave• Bad: no closed-form solution like NB• Ugly: Large N

Gradient Descent http://www.cs.cmu.edu/~epxing/Class/10701/Lecture/lecture7.pdf

Gradient Descent

Gradient Descent• w is initialized as zero• for t in 1 to T • Calculate gradients •

Gradient Descent• w is initialized as zero• for t in 1 to T • Calculate gradients J(w) •

Gradient Descent• w is initialized as zero• for t in 1 to T • Calculate gradients J(w) • w ← w − η J(w) t+1 t

Gradient Descent• w is initialized as zero• for t in 1 to T • Calculate gradients J(w) • w ← w − η J(w) t+1 t N J(w) = P (w, xi , yi ) i=1

Distribute Gradient• w is initialized as zero• for t in 1 to T • Calculate gradients in parallel• Training CPU: O(TPN) to O(TPN / M)

Distribute Gradient• w is initialized as zero• for t in 1 to T • Calculate gradients in parallel wt+1 ← wt − η J(w)• Training CPU: O(TPN) to O(TPN / M)

Distribute Gradient

Distribute Gradient Big Data

Distribute Gradient Big Data Shard 1 Shard 2 Shard 3 ... Shard M

Distribute Gradient Big Data Machine 1 Machine 2 Machine 3 Machine MMap Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, partial gradient sum)

Distribute Gradient Big Data Machine 1 Machine 2 Machine 3 Machine M Map Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, partial gradient sum)Reduce Sum and Update w

Distribute Gradient Big Data Machine 1 Machine 2 Machine 3 Machine M Map Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, partial gradient sum)Reduce Sum and Update w Repeat M/R until converge Model

Parallelize Subroutines• Support Vector Machines 1 n 2 arg min ||w||2 +C ζi w,b,ζ 2 i=1 s.t. 1 − yi (w · φ(xi ) + b) ≤ ζi , ζi ≥ 0• Solve the dual problem 1 T arg min α Qα − αT 1 α 2 s.t. 0 ≤ α ≤ C, yT α = 0

The computationalcost for the Primal-Dual Interior PointMethod is O(n^3) intime and O(n^2) in memoryhttp://www.ﬂickr.com/photos/sea-turtle/198445204/

Parallel SVM [Chang et al, 2007] √ N

Parallel SVM [Chang et al, 2007]• Parallel, row-wise incomplete Cholesky Factorization for Q √ N

Parallel SVM [Chang et al, 2007]• Parallel, row-wise incomplete Cholesky Factorization for Q• Parallel interior point method • Time O(n^3) becomes O(n^2 / M) √ • Memory O(n^2) becomes O(n N / M)

Parallel SVM [Chang et al, 2007]• Parallel, row-wise incomplete Cholesky Factorization for Q• Parallel interior point method • Time O(n^3) becomes O(n^2 / M) √ • Memory O(n^2) becomes O(n N / M)• Parallel Support Vector Machines (psvm) http:// code.google.com/p/psvm/ • Implement in MPI

Parallel ICF• Distribute Q by row into M machines Machine 1 Machine 2 Machine 3 row 1 row 3 row 5 ... row 2 row 4 row 6• For each dimension n < N √ • Send local pivots to master • Master selects largest local pivots and broadcast the global pivot to workers

Majority Vote

Majority Vote Big Data

Majority Vote Big DataShard 1 Shard 2 Shard 3 ... Shard M

Majority Vote Big Data Machine 1 Machine 2 Machine 3 Machine MMap Shard 1 Shard 2 Shard 3 ... Shard M

Majority Vote Big Data Machine 1 Machine 2 Machine 3 Machine MMap Shard 1 Shard 2 Shard 3 ... Shard M Model 1 Model 2 Model 3 Model 4

Majority Vote• Train individual classiﬁers independently• Predict by taking majority votes• Training CPU: O(TPN) to O(TPN / M)

Parameter Mixture [Mann et al, 2009]

Parameter Mixture [Mann et al, 2009] Big Data

Parameter Mixture [Mann et al, 2009] Big Data Shard 1 Shard 2 Shard 3 ... Shard M

Parameter Mixture [Mann et al, 2009] Big Data Machine 1 Machine 2 Machine 3 Machine MMap Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, w1) (dummy key, w2) ...

Parameter Mixture [Mann et al, 2009] Big Data Machine 1 Machine 2 Machine 3 Machine M Map Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, w1) (dummy key, w2) ...Reduce Average w

Parameter Mixture [Mann et al, 2009] Big Data Machine 1 Machine 2 Machine 3 Machine M Map Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, w1) (dummy key, w2) ...Reduce Average w Model

Much Less network usage than distributed gradient descent O(MN) vs. O(MNT)ttp://www.ﬂickr.com/photos/annamatic3000/127945652/

Iterative Param Mixture [McDonald et al., 2010]

Iterative Param Mixture[McDonald et al., 2010] Big Data

Iterative Param Mixture [McDonald et al., 2010] Big Data Shard 1 Shard 2 Shard 3 ... Shard M

Iterative Param Mixture [McDonald et al., 2010] Big Data Machine 1 Machine 2 Machine 3 Machine MMap Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, w1) (dummy key, w2) ...

Iterative Param Mixture [McDonald et al., 2010] Big Data Machine 1 Machine 2 Machine 3 Machine M Map Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, w1) (dummy key, w2) ... Reduceafter each Average w epoch

Iterative Param Mixture [McDonald et al., 2010] Big Data Machine 1 Machine 2 Machine 3 Machine M Map Shard 1 Shard 2 Shard 3 ... Shard M (dummy key, w1) (dummy key, w2) ... Reduceafter each Average w epoch Model

Scalable http://www.ﬂickr.com/photos/mr_t_in_dc/5469563053

Parallelhttp://www.ﬂickr.com/photos/aloshbennett/3209564747/

Accuracyhttp://www.ﬂickr.com/photos/wanderlinse/4367261825/

http://www.ﬂickr.com/photos/imagelink/4006753760/

Binary Classiﬁcationhttp://www.ﬂickr.com/photos/brenderous/4532934181/

Automatic FeatureDiscovery http://www.ﬂickr.com/photos/mararie/2340572508/

Fast Responsehttp://www.ﬂickr.com/photos/prunejuice/3687192643/

Memory is new hard disk.http://www.ﬂickr.com/photos/jepoirrier/840415676/

Algorithm + Infrastructurehttp://www.ﬂickr.com/photos/neubie/854242030/

Design forMulticores http://www.ﬂickr.com/photos/geektechnique/2344029370/

Combiner

Multi-shard Combiner[Chandra et al., 2010]

MachineLearning on Big Data

Parallelize ML Algorithms

Parallelize ML Algorithms• Embarrassingly parallel

Parallelize ML Algorithms• Embarrassingly parallel• Parallelize sub-routines

Parallelize ML Algorithms• Embarrassingly parallel• Parallelize sub-routines• Distributed learning

Parallel

Parallel Accuracy

Parallel Accuracy FastResponse

Google APIs

Google APIs• Prediction API • machine learning service on the cloud • http://code.google.com/apis/predict

Google APIs• Prediction API • machine learning service on the cloud • http://code.google.com/apis/predict• BigQuery • interactive analysis of massive data on the cloud • http://code.google.com/apis/bigquery

http://blog.acronym.co.kr	644
http://datascience101.wordpress.com	366
http://overdelivering.blogspot.com	359
http://overdelivery.blogspot.com	80
http://techbytes365.blogspot.in	59
http://m4xl1n.com	53
http://m4xl1n.blogspot.com	37
http://tweetedtimes.com	23
https://twitter.com	23
http://xmlmanager.tistory.com	21
http://irr.posterous.com	14
http://overdelivering.blogspot.in	14
http://twitter.com	14
http://www.linkedin.com	13
http://selins.posterous.com	10
http://techobytes.blogspot.in	9
http://overdelivering.blogspot.co.uk	7
http://globalmaximum.wordpress.com	6
http://trunk.ly	6
http://overdelivering.blogspot.tw	5
http://www.hanrss.com	5
http://overdelivering.blogspot.com.ar	4
http://testddiction.blogspot.com	4
http://posterous.com	4
http://cti97.ro	4
http://www.mefeedia.com	3
http://testddiction.blogspot.in	3
http://overdelivering.blogspot.fr	3
http://overdelivering.blogspot.se	3
http://a0.twimg.com	3
http://cti97.tumblr.com	2
http://feeds.feedburner.com	2
http://overdelivering.blogspot.mx	2
http://overdelivering.blogspot.kr	2
http://overdelivering.blogspot.sg	2
http://overdelivering.blogspot.ch	1
http://m4xl1n.blogspot.sg	1
http://overdelivering.blogspot.it	1
http://www.blogger.com	1
http://technikov.blogspot.in	1
http://overdelivering.blogspot.com.br	1
http://overdelivering.blogspot.be	1
http://techbytes365.blogspot.com	1
http://overdelivering.blogspot.ca	1
http://overdelivering.blogspot.de	1
http://overdelivering.blogspot.pt	1
http://overdelivering.blogspot.com.au	1
http://webcache.googleusercontent.com	1

Machine Learning on Big Data

by Max Lin

Accessibility

Categories

Upload Details

Usage Rights

48 Embeds 1,822

Statistics