Your SlideShare is downloading. ×
H2O Open Source Deep Learning, Arno Candel 03-20-14
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

H2O Open Source Deep Learning, Arno Candel 03-20-14


Published on

More information in our Deep Learning webinar: …

More information in our Deep Learning webinar:

Latest slide deck:

Published in: Technology

  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Deep Learning with H2O !
 Scalable In-Memory Machine Learning ! H20 Meetup, Mountain View, 3/20/14 Arno Candel
  • 2. Who am I? PhD in Computational Physics, 2005
 from ETH Zurich Switzerland ! 6 years at SLAC - Accelerator Physics Modeling 2 years at Skytree, Inc - Machine Learning 3 months at 0xdata/H2O - Machine Learning ! 10+ years in HPC, C++, MPI, Supercomputing Arno Candel
  • 3. Outline Intro Theory Implementation Results MNIST handwritten digits classification Live Demo Prostate cancer classification and age regression text classification
  • 4. Distributed in-memory math platform 
 ➔ GLM, GBM, RF, K-Means, PCA, Deep Learning
 Easy to use SDK / API
 ➔ Java, R, Scala, Python, JSON, Browser-based GUI ! Businesses can use ALL of their data (w or w/o Hadoop)
 ➔ Modeling without Sampling
 Big Data + Better Algorithms 
 ➔ Better Predictions H2O Open Source in-memory
 Prediction Engine for Big Data
  • 5. About H20 (aka 0xdata) Pure Java, Apache v2 Open Source Join the!
  • 6. H2O w or w/o Hadoop H2O H2O H2O HDFS HDFS HDFS YARN Hadoop MR R Java Scala JSON Python Standalone Over YARN On MRv1
  • 7. H2O Architecture in-memory K-V store MapReduce compression Machine Learning Algorithms R Engine Nano fast Scoring Engine Prediction Engine memory manager e.g. Deep Learning
  • 8. Wikipedia:
 Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using architectures composed of multiple non-linear transformations. ! ! ! ! ! Facebook DeepFace (LeCun): “Almost as good as humans at recognising faces” ! Google Brain (Andrew Ng, Jeff Dean & Geoffrey Hinton) ! FBI FACE: $1 billion face recognition project What is Deep Learning? Example: Input data
 (facial image) Prediction (person’s ID)
  • 9. Deep Learning is trending 20132012 Google trends 2011
  • 10. 1970s multi-layer feed-forward Neural Network (supervised learning with back-propagation) ! + distributed processing for big data (H2O in-memory MapReduce paradigm on distributed data) ! + multi-threaded speedup (H2O Fork/Join worker threads update the model asynchronously) ! + smart algorithms for accuracy (weight initialization, adaptive learning, momentum, dropout, regularization) ! = Top-notch prediction engine! Deep Learning in H2O
  • 11. “fully connected” directed graph of neurons age income employment married not married Input layer Hidden layer 1 Hidden layer 2 Output layer 3x4 4x3 3x2#connections information flow input/output neuron hidden neuron 4 3 2#neurons 3 Example Neural Network
  • 12. age income employment yj = tanh(sumi(xi*uij)+bj) uij xi yj per-class probabilities
 sum(pl) = 1 zk = tanh(sumj(yj*vjk)+ck) vjk zk pl pl = softmax(sumk(zk*wkl)+dl) wkl softmax(xk) = exp(xk) / sumk(exp(xk)) “neurons activate each other via weighted sums” Prediction: Forward Propagation married not married activation function: tanh alternative:
 x -> max(0,x) “rectifier” pl is a non-linear function of xi: can approximate ANY function with enough layers! bj, ck, dl: bias values
 (indep. of inputs)
  • 13. age income employment xi standardize input xi: mean = 0, stddev = 1 ! horizontalize categorical variables, e.g. {full-time, part-time, none, self-employed} 
 {0,1,0} = part-time, {0,0,0} = self-employed Poor man’s initialization: random weights ! Better: Uniform distribution in
 +/- sqrt(6/(#units + #units_previous_layer)) Data preparation & Initialization Neural Networks are sensitive to numerical noise,
 operate best in the linear regime (not saturated) married not married
  • 14. Mean Square Error = (0.2^2 + 0.2^2)/2 “penalize differences per-class” ! Cross-entropy = -log(0.8) “strongly penalize non-1-ness” Stochastic Gradient Descent SGD: improve weights and biases for EACH training row married not married For each training row, we make a prediction and compare with the actual label (supervised training): 1 0 0.8 0.2 predicted actual Objective: minimize prediction error (MSE or cross-entropy) w <— w - rate * ∂E/∂w 1
  • 15. Backward Propagation 
 ! ∂E/∂wi = ∂E/∂y * ∂y/∂net * ∂net/∂wi = ∂(error(y))/∂y * ∂(activation(net))/∂net * xi Backprop: Compute ∂E/∂wi via chain rule going backwards wi net = sumi(wi*xi) + b xi E = error(y) y = activation(net) How to compute ∂E/∂wi for wi <— wi - rate * ∂E/∂wi ? Naive: For every i, evaluate E twice at (w1,…,wi±∆,…,wN)… Slow!
  • 16. H2O Deep Learning Architecture K-V K-V HTTPD HTTPD nodes/JVMs: sync threads: async communication w w w w w w w w1 w3 w2 w4 w2+w4 w1+w3 w* = (w1+w2+w3+w4)/4 map:
 each node trains a copy of the weights and biases with (some* or all of) its local data with asynchronous F/J threads initial weights and biases w updated weights and biases w* H2O atomic in-memory
 K-V store reduce:
 average weights and biases from all nodes Keep iterating over the data (“epochs”), score from time to time Query & display the model via JSON, WWW 2 2 431 1 1 1 4 3 2 1 2 1 i *mini-batch: number of total rows per iteration, can be less than 1 epoch
  • 17. “Secret” Sauce to Higher Accuracy Momentum training:
 keep changing weights and biases (even if there’s no error) 
 “find other local minima, and go faster along valleys” Adaptive learning rate - ADADELTA (Google):
 automatically set learning rate for each neuron based on its training history, combines annealing and momentum features Learning rate annealing:
 rate r = r0 / (1+ß*N), N = training samples “dig deeper into local minimum” Grid Search and Checkpointing:
 Run a grid search over multiple hyper-parameters, then continue training the best model L1/L2/Dropout/MaxSumWeights regularization:
 L1: penalizes non-zero weights, L2: penalizes large weights
 Dropout: randomly ignore certain inputs “train exp. many models at once” MaxSumWeights: Reduce all incoming weights if the sum > max value “regularization avoids overtraining and improves generalization error”
  • 18. MNIST: digits classification Train: 60,000 rows 784 integer columns 10 classes Test: 10,000 rows 784 integer columns 10 classes MNIST: Digitized handwritten digits database (Yann LeCun) Data: 28x28=784 pixels with values in 0…255 (gray-scale) One of the most popular multi-class classification problems Without distortions or convolutions (which help), the best-ever published error rate on test set: 0.83% (Microsoft)
  • 19. most frequent mistakes:
 confuse 4 with 6 and 9, and 7 with 2 test set error: 1.5% after 40 epochs 1.02% after 400 epochs
 0.95% after 4000 epochs H2O Deep Learning on MNIST: 0.95% test set error (so far) 1 node
  • 20. Prostate Cancer Dataset
  • 21. Live Demo: Cancer Prediction Interactive ROC curve with real- time updates
  • 22. Live Demo: Cancer Prediction 0% training error with only 322 model parameters in seconds!
  • 23. Live Demo: Grid Search Regression Doing a grid search to find good hyper-parameters to predict AGE from other 7 features Then continue training the best model 5 hidden 50 tanh layers, rho=0.99, epsilon = 1e-10 MSE < 1 for test set ages in 44…79 Regression: 1 linear output neuron
  • 24. Live Demo: ebay Text Classification Users enter a description when selling an item Task: Predict the type of item Data prep: Binary word vector 0,0,1,0,0,0,0,0,1,0,0,0,1,…,0 H2O parses SVMLight sparse format: label 3:1 9:1 13:1 … ! “Small” sample dataset on jewelry and watches: Train: 578,361 rows 8,647 cols 467 classes Test: 64,263 rows 8,647 cols 143 classes ! H2O compressed columnar in-memory store: Only needs 60MB to store 5 billion entries (never inflated)
  • 25. Live Demo: ebay Text Classification Work in progress, shown results are for illustration only! Default parameters, no tuning, 4 nodes (16-cores each) Train: 578,361 rows 8,647 cols 467 classes Test: 64,263 rows 8,647 cols 143 classes
  • 26. Tips for H2O Deep Learning ! General: More layers: more complex functions (non-linearity) More neurons per layer: detect finer structure in data More regularization: less overfitting (better validation error) ! Do a grid search to get a feel for convergence, then continue training. Try Tanh first. For Rectifier, try max_w2 = 50 and/or L1=1e-5. Try TanhDropout or RectifierDropout with test/validation set after finding good parameters for convergence on training set. Distributed: Smaller mini-batch: more comm., slower, but higher accuracy. With ADADELTA: Try epsilon = 1e-4,1e-6,1e-8,1e-10, rho = 0.9,0.95,0.99 Without ADADELTA: Try rate = 1e-4…1e-2, rate_annealing = 1e-5…1e-8 Try momentum_start = 0.5, momentum_stable = 0.99,
 momentum_ramp = 1/rate_annealing Try balance_classes = true for imbalanced classes. Try force_load_balance for small datasets.
  • 27. Summary H2O is a distributed in-memory math platform that allows fast prototyping in Java, R, Scala and Python. ! H2o enables the development of enterprise-quality blazing fast machine learning applications. ! H2O Deep Learning is distributed, easy to use, and early results compete with the world’s best. ! Deep Learning makes better predictions! ! Try it yourself and join our next meetup!
 git clone