Sri Ambati, profile picture
Uploaded bySri Ambati
4,331 views

H2ODeepLearningThroughExamples021215

- The document describes a presentation on deep learning given by Arno Candel of H2O.ai. - The presentation covered deep learning methods and implementations, results from case studies in Higgs boson classification, handwritten digit recognition, and text classification. - It also demonstrated H2O's scalability and the ability of its deep learning algorithm to achieve state-of-the-art results on benchmark datasets.

Embed presentation

Downloaded 160 times
Deep Learning 
 Through Examples Silicon Valley Big Data Science Meetup Mountain View, 2/12/2015 Arno Candel, H2O.ai
Who am I? PhD in Computational Physics, 2005
 from ETH Zurich Switzerland 6 years at SLAC - Accelerator Physics Modeling 2 years at Skytree - Machine Learning 13 months at H2O.ai - Machine Learning 15 years in Supercomputing & Modeling Named “2014 Big Data All-Star” by Fortune Magazine @ArnoCandel
H2O Deep Learning, @ArnoCandel Outline Introduction (10 mins) Methods & Implementation (20 mins) Results and Live Demos (20 mins) Higgs boson classification MNIST handwritten digits Ebay text classification h2o-dev Outlook: Flow, Python Part 2: Hands-On Session (40 mins) Web GUI: Higgs dataset R Studio: Adult, Higgs, MNIST datasets 4
H2O Deep Learning, @ArnoCandel Teamwork at H2O.ai Java, Apache v2 Open-Source #1 Java Machine Learning in Github Join the community! 5
H2O Deep Learning, @ArnoCandel H2O: Open-Source (Apache v2) Predictive Analytics Platform 6
H2O Deep Learning, @ArnoCandel 7 H2O Architecture - Designed for speed, scale, accuracy & ease of use Key technical points: • distributed JVMs + REST API • no Java GC issues 
 (data in byte[], Double) • loss-less number compression • Hadoop integration (v1,YARN) • R package (CRAN) Pre-built fully featured algos:
 K-Means, NB, PCA, CoxPH,
 GLM, RF, GBM, DeepLearning
H2O Deep Learning, @ArnoCandel Wikipedia:
 Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using architectures composed of multiple 
 non-linear transformations. What is Deep Learning? Input:
 Image Output:
 User ID 8 Example: Facebook DeepFace
H2O Deep Learning, @ArnoCandel What is NOT Deep Linear models are not deep (by definition) Neural nets with 1 hidden layer are not deep (only 1 layer - no feature hierarchy) SVMs and Kernel methods are not deep (2 layers: kernel + linear) Classification trees are not deep (operate on original input space, no new features generated) 9
H2O Deep Learning, @ArnoCandel 1970s multi-layer feed-forward Neural Network (stochastic gradient descent with back-propagation) + distributed processing for big data (fine-grain in-memory MapReduce on distributed data) + multi-threaded speedup (async fork/join worker threads operate at FORTRAN speeds) + smart algorithms for fast & accurate results (automatic standardization, one-hot encoding of categoricals, missing value imputation, weight & bias initialization, adaptive learning rate, momentum, dropout/l1/L2 regularization, grid search, 
 N-fold cross-validation, checkpointing, load balancing, auto-tuning, model averaging, etc.) = powerful tool for (un)supervised machine learning on real-world data H2O Deep Learning 10 all 320 cores maxed out
H2O Deep Learning, @ArnoCandel “fully connected” directed graph of neurons age income employment married single Input layer Hidden layer 1 Hidden layer 2 Output layer 3x4 4x3 3x2#connections information flow input/output neuron hidden neuron 4 3 2#neurons 3 Example Neural Network 11
H2O Deep Learning, @ArnoCandel age income employment yj = tanh(sumi(xi*uij)+bj) uij xi yj per-class probabilities
 sum(pl) = 1 zk = tanh(sumj(yj*vjk)+ck) vjk zk pl pl = softmax(sumk(zk*wkl)+dl) wkl softmax(xk) = exp(xk) / sumk(exp(xk)) “neurons activate each other via weighted sums” Prediction: Forward Propagation activation function: tanh alternative:
 x -> max(0,x) “rectifier” pl is a non-linear function of xi: can approximate ANY function with enough layers! bj, ck, dl: bias values
 (indep. of inputs) 12 married single
H2O Deep Learning, @ArnoCandel age income employment xi Automatic standardization of data
 xi: mean = 0, stddev = 1 horizontalize categorical variables, e.g. {full-time, part-time, none, self-employed} 
 ->
 {0,1,0} = part-time, {0,0,0} = self-employed Automatic initialization of weights Poor man’s initialization: random weights wkl Default (better): Uniform distribution in
 +/- sqrt(6/(#units + #units_previous_layer)) Data preparation & Initialization Neural Networks are sensitive to numerical noise,
 operate best in the linear regime (not saturated) 13 married single wkl
H2O Deep Learning, @ArnoCandel Mean Square Error = (0.22 + 0.22)/2 “penalize differences per-class” Cross-entropy = -log(0.8) “strongly penalize non-1-ness” Training: Update Weights & Biases Stochastic Gradient Descent: Update weights and biases via gradient of the error (via back-propagation): For each training row, we make a prediction and compare with the actual label (supervised learning): married10.8 predicted actual Objective: minimize prediction error (MSE or cross-entropy) w <— w - rate * ∂E/∂w 1 14 single00.2 E w rate
H2O Deep Learning, @ArnoCandel Backward Propagation 
 ∂E/∂wi = ∂E/∂y * ∂y/∂net * ∂net/∂wi = ∂(error(y))/∂y * ∂(activation(net))/∂net * xi Backprop: Compute ∂E/∂wi via chain rule going backwards wi net = sumi(wi*xi) + b xi E = error(y) y = activation(net) How to compute ∂E/∂wi for wi <— wi - rate * ∂E/∂wi ? Naive: For every i, evaluate E twice at (w1,…,wi±∆,…,wN)… Slow! 15
H2O Deep Learning, @ArnoCandel H2O Deep Learning Architecture K-V K-V HTTPD HTTPD nodes/JVMs: sync threads: async communication w w w w w w w w1 w3 w2 w4 w2+w4 w1+w3 w* = (w1+w2+w3+w4)/4 map:
 each node trains a copy of the weights and biases with (some* or all of) its local data with asynchronous F/J threads initial model: weights and biases w updated model: w* H2O atomic in-memory
 K-V store reduce:
 model averaging: average weights and biases from all nodes, speedup is at least #nodes/log(#rows) arxiv:1209.4129v3 Keep iterating over the data (“epochs”), score from time to time Query & display the model via JSON, WWW 2 2 431 1 1 1 4 3 2 1 2 1 i *auto-tuned (default) or user-specified number of points per MapReduce iteration 16
H2O Deep Learning, @ArnoCandel Adaptive learning rate - ADADELTA (Google)
 Automatically set learning rate for each neuron based on its training history Grid Search and Checkpointing
 Run a grid search to scan many hyper- parameters, then continue training the most promising model(s) Regularization
 L1: penalizes non-zero weights
 L2: penalizes large weights
 Dropout: randomly ignore certain inputs Hogwild!: intentional race conditions Distributed mode: weight averaging 17 “Secret” Sauce to Higher Accuracy
H2O Deep Learning, @ArnoCandel Detail: Adaptive Learning Rate Compute moving average of ∆wi 2 at time t for window length rho: E[∆wi 2]t = rho * E[∆wi 2]t-1 + (1-rho) * ∆wi 2 Compute RMS of ∆wi at time t with smoothing epsilon: RMS[∆wi]t = sqrt( E[∆wi 2]t + epsilon ) Adaptive annealing / progress: Gradient-dependent learning rate, moving window prevents “freezing” (unlike ADAGRAD: no window) Adaptive acceleration / momentum: accumulate previous weight updates, but over a window of time RMS[∆wi]t-1 RMS[∂E/∂wi]t rate(wi, t) = Do the same for ∂E/∂wi, then obtain per-weight learning rate: cf. ADADELTA paper 18
H2O Deep Learning, @ArnoCandel Detail: Dropout Regularization 19 Training: For each hidden neuron, for each training sample, for each iteration, ignore (zero out) a different random fraction p of input activations. age income employment married single X X X Testing: Use all activations, but reduce them by a factor p (to “simulate” the missing activations during training). cf. Geoff Hinton's paper
H2O Deep Learning, @ArnoCandel 20 Application: Higgs Boson Classification Higgs
 vs
 Background Large Hadron Collider: Largest experiment of mankind! $13+ billion, 16.8 miles long, 120 MegaWatts, -456F, 1PB/day, etc. Higgs boson discovery (July ’12) led to 2013 Nobel prize! http://arxiv.org/pdf/1402.4735v2.pdf Images courtesy CERN / LHC HIGGS UCI Dataset: 21 low-level features AND 7 high-level derived features (physics formulae) Train: 10M rows, Valid: 500k, Test: 500k rows
H2O Deep Learning, @ArnoCandel 21 Live Demo: Let’s see what Deep Learning can do with low-level features alone! ? ? ? Former baseline for AUC: 0.733 and 0.816 H2O Algorithm low-level H2O AUC all features H2O AUC Generalized Linear Model 0.596 0.684 Random Forest 0.764 0.840 Gradient Boosted Trees 0.753 0.839 Neural Net 1 hidden layer 0.760 0.830 H2O Deep Learning ? add
 derived
 features Higgs: Derived features are important!
H2O Deep Learning, @ArnoCandel MNIST: digits classification Standing world record:
 Without distortions or convolutions, the best-ever published error rate on test set: 0.83% (Microsoft) 22 Train: 60,000 rows 784 integer columns 10 classes Test: 10,000 rows 784 integer columns 10 classes MNIST = Digitized handwritten digits database (Yann LeCun) Data: 28x28=784 pixels with (gray-scale) values in 0…255 Yann LeCun: “Yet another advice: don't get fooled by people who claim to have a solution to Artificial General Intelligence. Ask them what error rate they get on MNIST or ImageNet.”
H2O Deep Learning, @ArnoCandel 23 H2O Deep Learning beats MNIST Standard 60k/10k data No distortions No convolutions No unsupervised training No ensemble 10 hours on 10 16-core nodes World-record! 0.83% test set error http://learn.h2o.ai/content/hands-on_training/deep_learning.html
H2O Deep Learning, @ArnoCandel POJO Model Export for Production Scoring 24 Plain old Java code is auto-generated to take your H2O Deep Learning models into production!
H2O Deep Learning, @ArnoCandel Parallel Scalability (for 64 epochs on MNIST, with “0.83%” parameters) 25 Speedup 0.00 10.00 20.00 30.00 40.00 1 2 4 8 16 32 63 H2O Nodes (4 cores per node, 1 epoch per node per MapReduce) 2.7 mins Training Time 0 25 50 75 100 1 2 4 8 16 32 63 H2O Nodes in minutes
H2O Deep Learning, @ArnoCandel Goal: Predict the item from seller’s text description 26 Train: 578,361 rows 8,647 cols 467 classes Test: 64,263 rows 8,647 cols 143 classes “Vintage 18KT gold Rolex 2 Tone in great condition” Data: Bag of words vector 0,0,1,0,0,0,0,0,1,0,0,0,1,…,0 vintagegold condition Text Classification
H2O Deep Learning, @ArnoCandel Out-Of-The-Box: 11.6% test set error after 10 epochs! Predicts the correct class (out of 143) 88.4% of the time! 27 Note 2: No tuning was done
 (results are for illustration only) Train: 578,361 rows 8,647 cols 467 classes Test: 64,263 rows 8,647 cols 143 classes Note 1: H2O columnar-compressed in-memory store only needs 60 MB to store 5 billion values (dense CSV needs 18 GB) Text Classification
H2O Deep Learning, @ArnoCandel MNIST: Unsupervised Anomaly Detection with Deep Learning (Autoencoder) 28 The good The bad The ugly Download the script and run it yourself!
H2O Deep Learning, @ArnoCandel 29 How well did Deep Learning do? Let’s see how H2O did in the past 10 minutes! Higgs: Live Demo (Continued) <your guess?> reference paper results Any guesses for AUC on low-level features? AUC=0.76 was the best for RF/GBM/NN (H2O)
H2O Deep Learning, @ArnoCandel H2O Steam: Scoring Platform 30 Higgs Dataset Demo on 10-node cluster Let’s score all our H2O models and compare them! http://server:port/steam/index.html Live Demo
H2O Deep Learning, @ArnoCandel 31 Live Demo on 10-node cluster: <10 minutes runtime for all H2O algos! Better than LHC baseline of AUC=0.73! Scoring Higgs Models in H2O Steam
H2O Deep Learning, @ArnoCandel 32 Algorithm Paper’s* l-l AUC low-level H2O AUC all features
 H2O AUC Parameters (not heavily tuned), 
 H2O running on 10 nodes Generalized Linear Model - 0.596 0.684 default, binomial Random Forest - 0.764 0.840 50 trees, max depth 50 Gradient Boosted Trees 0.73 0.753 0.839 50 trees, max depth 15 Neural Net 1 layer 0.733 0.760 0.830 1x300 Rectifier, 100 epochs Deep Learning 3 hidden layers 0.836 0.850 - 3x1000 Rectifier, L2=1e-5, 40 epochs Deep Learning 4 hidden layers 0.868 0.869 - 4x500 Rectifier, L1=L2=1e-5, 300 epochs Deep Learning 5 hidden layers 0.880 0.871 - 5x500 Rectifier, L1=L2=1e-5 Deep Learning on low-level features alone beats everything else! Prelim. H2O results compare well with paper’s results* (TMVA & Theano) Higgs Particle Detection with H2O *Nature paper: http://arxiv.org/pdf/1402.4735v2.pdf HIGGS UCI Dataset: 21 low-level features AND 7 high-level derived features Train: 10M rows, Test: 500k rows
H2O Deep Learning, @ArnoCandel H2O Kaggle Starter R Scripts 33 Final ranking:
 #26 out of 1604
H2O Deep Learning, @ArnoCandel Currently Ongoing Challenge 34
H2O Deep Learning, @ArnoCandel Re-Live H2O World! 35 http://h2o.ai/h2o-world/ http://learn.h2o.ai Watch the Videos Day 2 • Speakers from Academia & Industry • Trevor Hastie (ML) • John Chambers (S, R) • Josh Bloch (Java API) • Many use cases from customers • 3 Top Kaggle Contestants (Top 10) • 3 Panel discussions Day 1 • Hands-On Training • Supervised • Unsupervised • Advanced Topics • Markting Usecase • Product Demos • Hacker-Fest with 
 Cliff Click (CTO, Hotspot)
H2O Deep Learning, @ArnoCandel 36 H2O GitBooks Also available: GBM & GLM GitBooks at http://h2o.gitbooks.io H2O World learn.h2o.ai R, EC2, Hadoop Deep Learning
H2O Deep Learning, @ArnoCandel h2o-dev: Flow, Python, JS, Spark 37
H2O Deep Learning, @ArnoCandel h2o-dev Python Example 38
H2O Deep Learning, @ArnoCandel You can participate! 39 - Images: Convolutional & Pooling Layers PUB-644 - Sequences: Recurrent Neural Networks PUB-1052 - Faster Training: GPGPU support PUB-1013 - Pre-Training: Stacked Auto-Encoders PUB-1014 - Ensembles PUB-1072 - Use H2O at Kaggle Challenges!
H2O Deep Learning, @ArnoCandel Key Take-Aways H2O is an open source predictive analytics platform for data scientists and business analysts who need scalable and fast machine learning. H2O Deep Learning is ready to take your advanced analytics to the next level - Try it on your data! Join our Community and Meetups! https://github.com/h2oai h2ostream community forum www.h2o.ai @h2oai 40 Thank you!

More Related Content

PDF
MLconf - Distributed Deep Learning for Classification and Regression Problems...
bySri Ambati
 
PDF
H2O Deep Learning at Next.ML
bySri Ambati
 
PDF
How to win data science competitions with Deep Learning
bySri Ambati
 
PDF
H2O Distributed Deep Learning by Arno Candel 071614
bySri Ambati
 
PDF
San Francisco Hadoop User Group Meetup Deep Learning
bySri Ambati
 
PDF
Alex Tellez, Deep Learning Applications
bySri Ambati
 
PDF
H2O Open Source Deep Learning, Arno Candel 03-20-14
bySri Ambati
 
PDF
Deep Learning in the Wild with Arno Candel
bySri Ambati
 
MLconf - Distributed Deep Learning for Classification and Regression Problems...
bySri Ambati
 
H2O Deep Learning at Next.ML
bySri Ambati
 
How to win data science competitions with Deep Learning
bySri Ambati
 
H2O Distributed Deep Learning by Arno Candel 071614
bySri Ambati
 
San Francisco Hadoop User Group Meetup Deep Learning
bySri Ambati
 
Alex Tellez, Deep Learning Applications
bySri Ambati
 
H2O Open Source Deep Learning, Arno Candel 03-20-14
bySri Ambati
 
Deep Learning in the Wild with Arno Candel
bySri Ambati
 

What's hot

PDF
Deep Learning through Examples
bySri Ambati
 
PDF
H2O.ai's Distributed Deep Learning by Arno Candel 04/03/14
bySri Ambati
 
PDF
Deep Learning And Business Models (VNITC 2015-09-13)
byHa Phuong
 
PDF
Webinar: Deep Learning with H2O
bySri Ambati
 
PPTX
Introduction to Deep Learning
byMustafa Aldemir
 
PDF
Jan vitek distributedrandomforest_5-2-2013
bySri Ambati
 
PDF
Deep Learning Cases: Text and Image Processing
byGrigory Sapunov
 
PDF
Using Deep Learning to do Real-Time Scoring in Practical Applications - 2015-...
byGreg Makowski
 
PDF
Introduction to Deep Learning with Python
byindico data
 
PDF
Building Random Forest at Scale
bySri Ambati
 
PDF
Machine Learning and Deep Learning with R
byPoo Kuan Hoong
 
PPTX
What Deep Learning Means for Artificial Intelligence
byJonathan Mugan
 
PDF
Deep Learning and Reinforcement Learning
byRenārs Liepiņš
 
PDF
Ted Willke, Senior Principal Engineer & GM, Datacenter Group, Intel at MLconf SF
byMLconf
 
PDF
Distance oracle - Truy vấn nhanh khoảng cách giữa hai điểm bất kỳ trên đồ thị
byHong Ong
 
PPTX
Deep Learning with Python (PyData Seattle 2015)
byAlexander Korbonits
 
PPTX
Introduction to Deep Learning
byAdam Rogers
 
PPTX
Machine Learning for Smarter Apps - Jacksonville Meetup
bySri Ambati
 
PPTX
An introduction to Deep Learning
byDavid Rostcheck
 
PPTX
Promises of Deep Learning
byDavid Khosid
 
Deep Learning through Examples
bySri Ambati
 
H2O.ai's Distributed Deep Learning by Arno Candel 04/03/14
bySri Ambati
 
Deep Learning And Business Models (VNITC 2015-09-13)
byHa Phuong
 
Webinar: Deep Learning with H2O
bySri Ambati
 
Introduction to Deep Learning
byMustafa Aldemir
 
Jan vitek distributedrandomforest_5-2-2013
bySri Ambati
 
Deep Learning Cases: Text and Image Processing
byGrigory Sapunov
 
Using Deep Learning to do Real-Time Scoring in Practical Applications - 2015-...
byGreg Makowski
 
Introduction to Deep Learning with Python
byindico data
 
Building Random Forest at Scale
bySri Ambati
 
Machine Learning and Deep Learning with R
byPoo Kuan Hoong
 
What Deep Learning Means for Artificial Intelligence
byJonathan Mugan
 
Deep Learning and Reinforcement Learning
byRenārs Liepiņš
 
Ted Willke, Senior Principal Engineer & GM, Datacenter Group, Intel at MLconf SF
byMLconf
 
Distance oracle - Truy vấn nhanh khoảng cách giữa hai điểm bất kỳ trên đồ thị
byHong Ong
 
Deep Learning with Python (PyData Seattle 2015)
byAlexander Korbonits
 
Introduction to Deep Learning
byAdam Rogers
 
Machine Learning for Smarter Apps - Jacksonville Meetup
bySri Ambati
 
An introduction to Deep Learning
byDavid Rostcheck
 
Promises of Deep Learning
byDavid Khosid
 

Similar to H2ODeepLearningThroughExamples021215

PDF
Handwritten Recognition using Deep Learning with R
byPoo Kuan Hoong
 
PPTX
Project "Deep Water"
byJo-fai Chow
 
PPTX
2016 Deep Learning with R and h2o
byJan Eite Bullema
 
PDF
Arno candel scalabledatascienceanddeeplearningwithh2o_reworkboston2015
bySri Ambati
 
PDF
ArnoCandelScalabledatascienceanddeeplearningwithh2o_gotochg
bySri Ambati
 
PDF
Arno candel scalabledatascienceanddeeplearningwithh2o_odsc_boston2015
bySri Ambati
 
PDF
Scalable Data Science and Deep Learning with H2O
byodsc
 
PDF
H2O Deep Water - Making Deep Learning Accessible to Everyone
bySri Ambati
 
PPTX
Deep Learning with Microsoft R Open
byPoo Kuan Hoong
 
PDF
Introduction to Machine Learning with H2O and Python
byJo-fai Chow
 
PDF
H2O with Erin LeDell at Portland R User Group
bySri Ambati
 
PDF
Introduction to Machine Learning with H2O - Jo-Fai (Joe) Chow, H2O
byData Science Milan
 
PDF
H2O Deep Water - Making Deep Learning Accessible to Everyone
byJo-fai Chow
 
PDF
H2O World - H2O Deep Learning with Arno Candel
bySri Ambati
 
PDF
H2O at Poznan R Meetup
byJo-fai Chow
 
PDF
Intro to Machine Learning and H2O
byRaymond Peck
 
PDF
H2O Deep Water - Making Deep Learning Accessible to Everyone
byJo-fai Chow
 
PDF
H2 o deep water making deep learning accessible to everyone -jo-fai chow
byEvention
 
PDF
Berlin R Meetup
bySri Ambati
 
PDF
H2O at Berlin R Meetup
byJo-fai Chow
 
Handwritten Recognition using Deep Learning with R
byPoo Kuan Hoong
 
Project "Deep Water"
byJo-fai Chow
 
2016 Deep Learning with R and h2o
byJan Eite Bullema
 
Arno candel scalabledatascienceanddeeplearningwithh2o_reworkboston2015
bySri Ambati
 
ArnoCandelScalabledatascienceanddeeplearningwithh2o_gotochg
bySri Ambati
 
Arno candel scalabledatascienceanddeeplearningwithh2o_odsc_boston2015
bySri Ambati
 
Scalable Data Science and Deep Learning with H2O
byodsc
 
H2O Deep Water - Making Deep Learning Accessible to Everyone
bySri Ambati
 
Deep Learning with Microsoft R Open
byPoo Kuan Hoong
 
Introduction to Machine Learning with H2O and Python
byJo-fai Chow
 
H2O with Erin LeDell at Portland R User Group
bySri Ambati
 
Introduction to Machine Learning with H2O - Jo-Fai (Joe) Chow, H2O
byData Science Milan
 
H2O Deep Water - Making Deep Learning Accessible to Everyone
byJo-fai Chow
 
H2O World - H2O Deep Learning with Arno Candel
bySri Ambati
 
H2O at Poznan R Meetup
byJo-fai Chow
 
Intro to Machine Learning and H2O
byRaymond Peck
 
H2O Deep Water - Making Deep Learning Accessible to Everyone
byJo-fai Chow
 
H2 o deep water making deep learning accessible to everyone -jo-fai chow
byEvention
 
Berlin R Meetup
bySri Ambati
 
H2O at Berlin R Meetup
byJo-fai Chow
 

More from Sri Ambati

PDF
H2O Gen AI Ecosystem Overview - Level 1 - Slide Deck
bySri Ambati
 
PDF
LLM Learning Path Level 2 - Presentation Slides
bySri Ambati
 
PDF
Evaluating Generative AI.Models with Eval Studio - Support Presentation
bySri Ambati
 
PPTX
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
bySri Ambati
 
PDF
Large Language Models (LLMs) - Level 3 Slides
bySri Ambati
 
PDF
Practical MLOps with H2O.ai - Support Slide Deck.pdf
bySri Ambati
 
PDF
Presentation Resources - H2O Gen AI Ecosystem Overview - Level 2
bySri Ambati
 
PDF
LLM Learning Path Level 1 - Presentation Slides
bySri Ambati
 
PDF
Enterprise h2o GPTe Learning Path Slide Deck
bySri Ambati
 
PDF
H2O.ai Agents : From Theory to Practice - Support Presentation
bySri Ambati
 
PDF
H2O Generative AI Starter Track - Support Presentation Slides.pdf
bySri Ambati
 
PDF
H2O Label Genie Starter Track - Support Presentation
bySri Ambati
 
PDF
H2O Driverless AI Starter Course - Slides and Assignments
bySri Ambati
 
PDF
H2O Wave Course Starter - Presentation Slides
bySri Ambati
 
PDF
Data Science and Machine Learning Platforms (2024) Slides
bySri Ambati
 
PDF
H2O Cloud AI Developer Services - Slides (2024)
bySri Ambati
 
PDF
Data Prep for H2O Driverless AI - Slides
bySri Ambati
 
PDF
An In-depth Exploration of Enterprise h2oGPTe Slide Deck
bySri Ambati
 
PDF
Hydrogen Torch - Starter Course - Presentation Slides
bySri Ambati
 
PDF
Intro to Enterprise h2oGPTe Presentation Slides
bySri Ambati
 
H2O Gen AI Ecosystem Overview - Level 1 - Slide Deck
bySri Ambati
 
LLM Learning Path Level 2 - Presentation Slides
bySri Ambati
 
Evaluating Generative AI.Models with Eval Studio - Support Presentation
bySri Ambati
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
bySri Ambati
 
Large Language Models (LLMs) - Level 3 Slides
bySri Ambati
 
Practical MLOps with H2O.ai - Support Slide Deck.pdf
bySri Ambati
 
Presentation Resources - H2O Gen AI Ecosystem Overview - Level 2
bySri Ambati
 
LLM Learning Path Level 1 - Presentation Slides
bySri Ambati
 
Enterprise h2o GPTe Learning Path Slide Deck
bySri Ambati
 
H2O.ai Agents : From Theory to Practice - Support Presentation
bySri Ambati
 
H2O Generative AI Starter Track - Support Presentation Slides.pdf
bySri Ambati
 
H2O Label Genie Starter Track - Support Presentation
bySri Ambati
 
H2O Driverless AI Starter Course - Slides and Assignments
bySri Ambati
 
H2O Wave Course Starter - Presentation Slides
bySri Ambati
 
Data Science and Machine Learning Platforms (2024) Slides
bySri Ambati
 
H2O Cloud AI Developer Services - Slides (2024)
bySri Ambati
 
Data Prep for H2O Driverless AI - Slides
bySri Ambati
 
An In-depth Exploration of Enterprise h2oGPTe Slide Deck
bySri Ambati
 
Hydrogen Torch - Starter Course - Presentation Slides
bySri Ambati
 
Intro to Enterprise h2oGPTe Presentation Slides
bySri Ambati
 

Recently uploaded

PDF
Solved First Semester B.C.A. C Programming Question Paper – Jan/Feb 2025
byKuvempu University
 
PPTX
Privacy Preserving Detection of Sensitive Data Exposure (1).pptx
byNikitaMandhan4
 
PPTX
application security presentation 2 by harman
byharmanideapad
 
PDF
Using Agentic RAG for Research - Demonstration of NotebookLM
byRathish Chandra Gatti,Ph.D
 
PDF
nsfconvertersoftwaretoconvertNSFtoPST.pdf
byNSF Converter Software
 
PDF
One Agent to Rule Them All - The Fellowship of AI Agents
byIvo Andreev
 
PDF
TNG_Architecting Green Software - An Introduction_Nils-Oliver Linden&Alexande...
byQAware GmbH
 
PPTX
SNG460-CNG489_Week8_17-21Nov25_Chp8-OdtuClass.pptx
byberayseray382
 
PPTX
Computer_Virus_Presentation_With_Slide7.pptx
byHEVISLIVE
 
PPTX
List on Python Programming Language.pptx
byRICHARDALONSAGAY1
 
PPTX
SNG460-CNG489_Week6_3-7Nov25_Chp6-Part1.pptx
byberayseray382
 
PDF
From Experiment to Enterprise: Scaling AI Coding Assistants Across Engineerin...
byMaxim Salnikov
 
PDF
Cybersecurity Alert- What Organisations Must Watch Out For This Christmas Fes...
byCybernetic Global Intelligence
 
PDF
Cloud-Based Underwriting Software for Insurance
byInsurance Tech Services
 
PDF
Jeremy Millul - An NYU Computer Science Graduate
byJeremy Millul
 
PDF
Coding Assessment Tools .pdf
byCodexpro
 
PDF
DSD-INT 2025 Hydrological response of the Puget Sound area to a changing clim...
byDeltares
 
PDF
Kiến trúc hướng dịch vụ UET!!!!!!!!!!!!!!!!!!!!!!!
bydoquangminh962004
 
PPTX
Membershipanywhere - Digital Membership Card - USA , Canada, Australia.pptx
bymembershipanywhere
 
PPTX
OTT Content Analytics Enhanced by ALTBalaji Data Scraping.pptx
byyashpatric
 
Solved First Semester B.C.A. C Programming Question Paper – Jan/Feb 2025
byKuvempu University
 
Privacy Preserving Detection of Sensitive Data Exposure (1).pptx
byNikitaMandhan4
 
application security presentation 2 by harman
byharmanideapad
 
Using Agentic RAG for Research - Demonstration of NotebookLM
byRathish Chandra Gatti,Ph.D
 
nsfconvertersoftwaretoconvertNSFtoPST.pdf
byNSF Converter Software
 
One Agent to Rule Them All - The Fellowship of AI Agents
byIvo Andreev
 
TNG_Architecting Green Software - An Introduction_Nils-Oliver Linden&Alexande...
byQAware GmbH
 
SNG460-CNG489_Week8_17-21Nov25_Chp8-OdtuClass.pptx
byberayseray382
 
Computer_Virus_Presentation_With_Slide7.pptx
byHEVISLIVE
 
List on Python Programming Language.pptx
byRICHARDALONSAGAY1
 
SNG460-CNG489_Week6_3-7Nov25_Chp6-Part1.pptx
byberayseray382
 
From Experiment to Enterprise: Scaling AI Coding Assistants Across Engineerin...
byMaxim Salnikov
 
Cybersecurity Alert- What Organisations Must Watch Out For This Christmas Fes...
byCybernetic Global Intelligence
 
Cloud-Based Underwriting Software for Insurance
byInsurance Tech Services
 
Jeremy Millul - An NYU Computer Science Graduate
byJeremy Millul
 
Coding Assessment Tools .pdf
byCodexpro
 
DSD-INT 2025 Hydrological response of the Puget Sound area to a changing clim...
byDeltares
 
Kiến trúc hướng dịch vụ UET!!!!!!!!!!!!!!!!!!!!!!!
bydoquangminh962004
 
Membershipanywhere - Digital Membership Card - USA , Canada, Australia.pptx
bymembershipanywhere
 
OTT Content Analytics Enhanced by ALTBalaji Data Scraping.pptx
byyashpatric
 

H2ODeepLearningThroughExamples021215

  • 1.
    Deep Learning 
 ThroughExamples Silicon Valley Big Data Science Meetup Mountain View, 2/12/2015 Arno Candel, H2O.ai
  • 2.
    Who am I? PhDin Computational Physics, 2005
 from ETH Zurich Switzerland 6 years at SLAC - Accelerator Physics Modeling 2 years at Skytree - Machine Learning 13 months at H2O.ai - Machine Learning 15 years in Supercomputing & Modeling Named “2014 Big Data All-Star” by Fortune Magazine @ArnoCandel
  • 4.
    H2O Deep Learning,@ArnoCandel Outline Introduction (10 mins) Methods & Implementation (20 mins) Results and Live Demos (20 mins) Higgs boson classification MNIST handwritten digits Ebay text classification h2o-dev Outlook: Flow, Python Part 2: Hands-On Session (40 mins) Web GUI: Higgs dataset R Studio: Adult, Higgs, MNIST datasets 4
  • 5.
    H2O Deep Learning,@ArnoCandel Teamwork at H2O.ai Java, Apache v2 Open-Source #1 Java Machine Learning in Github Join the community! 5
  • 6.
    H2O Deep Learning,@ArnoCandel H2O: Open-Source (Apache v2) Predictive Analytics Platform 6
  • 7.
    H2O Deep Learning,@ArnoCandel 7 H2O Architecture - Designed for speed, scale, accuracy & ease of use Key technical points: • distributed JVMs + REST API • no Java GC issues 
 (data in byte[], Double) • loss-less number compression • Hadoop integration (v1,YARN) • R package (CRAN) Pre-built fully featured algos:
 K-Means, NB, PCA, CoxPH,
 GLM, RF, GBM, DeepLearning
  • 8.
    H2O Deep Learning,@ArnoCandel Wikipedia:
 Deep learning is a set of algorithms in machine learning that attempt to model high-level abstractions in data by using architectures composed of multiple 
 non-linear transformations. What is Deep Learning? Input:
 Image Output:
 User ID 8 Example: Facebook DeepFace
  • 9.
    H2O Deep Learning,@ArnoCandel What is NOT Deep Linear models are not deep (by definition) Neural nets with 1 hidden layer are not deep (only 1 layer - no feature hierarchy) SVMs and Kernel methods are not deep (2 layers: kernel + linear) Classification trees are not deep (operate on original input space, no new features generated) 9
  • 10.
    H2O Deep Learning,@ArnoCandel 1970s multi-layer feed-forward Neural Network (stochastic gradient descent with back-propagation) + distributed processing for big data (fine-grain in-memory MapReduce on distributed data) + multi-threaded speedup (async fork/join worker threads operate at FORTRAN speeds) + smart algorithms for fast & accurate results (automatic standardization, one-hot encoding of categoricals, missing value imputation, weight & bias initialization, adaptive learning rate, momentum, dropout/l1/L2 regularization, grid search, 
 N-fold cross-validation, checkpointing, load balancing, auto-tuning, model averaging, etc.) = powerful tool for (un)supervised machine learning on real-world data H2O Deep Learning 10 all 320 cores maxed out
  • 11.
    H2O Deep Learning,@ArnoCandel “fully connected” directed graph of neurons age income employment married single Input layer Hidden layer 1 Hidden layer 2 Output layer 3x4 4x3 3x2#connections information flow input/output neuron hidden neuron 4 3 2#neurons 3 Example Neural Network 11
  • 12.
    H2O Deep Learning,@ArnoCandel age income employment yj = tanh(sumi(xi*uij)+bj) uij xi yj per-class probabilities
 sum(pl) = 1 zk = tanh(sumj(yj*vjk)+ck) vjk zk pl pl = softmax(sumk(zk*wkl)+dl) wkl softmax(xk) = exp(xk) / sumk(exp(xk)) “neurons activate each other via weighted sums” Prediction: Forward Propagation activation function: tanh alternative:
 x -> max(0,x) “rectifier” pl is a non-linear function of xi: can approximate ANY function with enough layers! bj, ck, dl: bias values
 (indep. of inputs) 12 married single
  • 13.
    H2O Deep Learning,@ArnoCandel age income employment xi Automatic standardization of data
 xi: mean = 0, stddev = 1 horizontalize categorical variables, e.g. {full-time, part-time, none, self-employed} 
 ->
 {0,1,0} = part-time, {0,0,0} = self-employed Automatic initialization of weights Poor man’s initialization: random weights wkl Default (better): Uniform distribution in
 +/- sqrt(6/(#units + #units_previous_layer)) Data preparation & Initialization Neural Networks are sensitive to numerical noise,
 operate best in the linear regime (not saturated) 13 married single wkl
  • 14.
    H2O Deep Learning,@ArnoCandel Mean Square Error = (0.22 + 0.22)/2 “penalize differences per-class” Cross-entropy = -log(0.8) “strongly penalize non-1-ness” Training: Update Weights & Biases Stochastic Gradient Descent: Update weights and biases via gradient of the error (via back-propagation): For each training row, we make a prediction and compare with the actual label (supervised learning): married10.8 predicted actual Objective: minimize prediction error (MSE or cross-entropy) w <— w - rate * ∂E/∂w 1 14 single00.2 E w rate
  • 15.
    H2O Deep Learning,@ArnoCandel Backward Propagation 
 ∂E/∂wi = ∂E/∂y * ∂y/∂net * ∂net/∂wi = ∂(error(y))/∂y * ∂(activation(net))/∂net * xi Backprop: Compute ∂E/∂wi via chain rule going backwards wi net = sumi(wi*xi) + b xi E = error(y) y = activation(net) How to compute ∂E/∂wi for wi <— wi - rate * ∂E/∂wi ? Naive: For every i, evaluate E twice at (w1,…,wi±∆,…,wN)… Slow! 15
  • 16.
    H2O Deep Learning,@ArnoCandel H2O Deep Learning Architecture K-V K-V HTTPD HTTPD nodes/JVMs: sync threads: async communication w w w w w w w w1 w3 w2 w4 w2+w4 w1+w3 w* = (w1+w2+w3+w4)/4 map:
 each node trains a copy of the weights and biases with (some* or all of) its local data with asynchronous F/J threads initial model: weights and biases w updated model: w* H2O atomic in-memory
 K-V store reduce:
 model averaging: average weights and biases from all nodes, speedup is at least #nodes/log(#rows) arxiv:1209.4129v3 Keep iterating over the data (“epochs”), score from time to time Query & display the model via JSON, WWW 2 2 431 1 1 1 4 3 2 1 2 1 i *auto-tuned (default) or user-specified number of points per MapReduce iteration 16
  • 17.
    H2O Deep Learning,@ArnoCandel Adaptive learning rate - ADADELTA (Google)
 Automatically set learning rate for each neuron based on its training history Grid Search and Checkpointing
 Run a grid search to scan many hyper- parameters, then continue training the most promising model(s) Regularization
 L1: penalizes non-zero weights
 L2: penalizes large weights
 Dropout: randomly ignore certain inputs Hogwild!: intentional race conditions Distributed mode: weight averaging 17 “Secret” Sauce to Higher Accuracy
  • 18.
    H2O Deep Learning,@ArnoCandel Detail: Adaptive Learning Rate Compute moving average of ∆wi 2 at time t for window length rho: E[∆wi 2]t = rho * E[∆wi 2]t-1 + (1-rho) * ∆wi 2 Compute RMS of ∆wi at time t with smoothing epsilon: RMS[∆wi]t = sqrt( E[∆wi 2]t + epsilon ) Adaptive annealing / progress: Gradient-dependent learning rate, moving window prevents “freezing” (unlike ADAGRAD: no window) Adaptive acceleration / momentum: accumulate previous weight updates, but over a window of time RMS[∆wi]t-1 RMS[∂E/∂wi]t rate(wi, t) = Do the same for ∂E/∂wi, then obtain per-weight learning rate: cf. ADADELTA paper 18
  • 19.
    H2O Deep Learning,@ArnoCandel Detail: Dropout Regularization 19 Training: For each hidden neuron, for each training sample, for each iteration, ignore (zero out) a different random fraction p of input activations. age income employment married single X X X Testing: Use all activations, but reduce them by a factor p (to “simulate” the missing activations during training). cf. Geoff Hinton's paper
  • 20.
    H2O Deep Learning,@ArnoCandel 20 Application: Higgs Boson Classification Higgs
 vs
 Background Large Hadron Collider: Largest experiment of mankind! $13+ billion, 16.8 miles long, 120 MegaWatts, -456F, 1PB/day, etc. Higgs boson discovery (July ’12) led to 2013 Nobel prize! http://arxiv.org/pdf/1402.4735v2.pdf Images courtesy CERN / LHC HIGGS UCI Dataset: 21 low-level features AND 7 high-level derived features (physics formulae) Train: 10M rows, Valid: 500k, Test: 500k rows
  • 21.
    H2O Deep Learning,@ArnoCandel 21 Live Demo: Let’s see what Deep Learning can do with low-level features alone! ? ? ? Former baseline for AUC: 0.733 and 0.816 H2O Algorithm low-level H2O AUC all features H2O AUC Generalized Linear Model 0.596 0.684 Random Forest 0.764 0.840 Gradient Boosted Trees 0.753 0.839 Neural Net 1 hidden layer 0.760 0.830 H2O Deep Learning ? add
 derived
 features Higgs: Derived features are important!
  • 22.
    H2O Deep Learning,@ArnoCandel MNIST: digits classification Standing world record:
 Without distortions or convolutions, the best-ever published error rate on test set: 0.83% (Microsoft) 22 Train: 60,000 rows 784 integer columns 10 classes Test: 10,000 rows 784 integer columns 10 classes MNIST = Digitized handwritten digits database (Yann LeCun) Data: 28x28=784 pixels with (gray-scale) values in 0…255 Yann LeCun: “Yet another advice: don't get fooled by people who claim to have a solution to Artificial General Intelligence. Ask them what error rate they get on MNIST or ImageNet.”
  • 23.
    H2O Deep Learning,@ArnoCandel 23 H2O Deep Learning beats MNIST Standard 60k/10k data No distortions No convolutions No unsupervised training No ensemble 10 hours on 10 16-core nodes World-record! 0.83% test set error http://learn.h2o.ai/content/hands-on_training/deep_learning.html
  • 24.
    H2O Deep Learning,@ArnoCandel POJO Model Export for Production Scoring 24 Plain old Java code is auto-generated to take your H2O Deep Learning models into production!
  • 25.
    H2O Deep Learning,@ArnoCandel Parallel Scalability (for 64 epochs on MNIST, with “0.83%” parameters) 25 Speedup 0.00 10.00 20.00 30.00 40.00 1 2 4 8 16 32 63 H2O Nodes (4 cores per node, 1 epoch per node per MapReduce) 2.7 mins Training Time 0 25 50 75 100 1 2 4 8 16 32 63 H2O Nodes in minutes
  • 26.
    H2O Deep Learning,@ArnoCandel Goal: Predict the item from seller’s text description 26 Train: 578,361 rows 8,647 cols 467 classes Test: 64,263 rows 8,647 cols 143 classes “Vintage 18KT gold Rolex 2 Tone in great condition” Data: Bag of words vector 0,0,1,0,0,0,0,0,1,0,0,0,1,…,0 vintagegold condition Text Classification
  • 27.
    H2O Deep Learning,@ArnoCandel Out-Of-The-Box: 11.6% test set error after 10 epochs! Predicts the correct class (out of 143) 88.4% of the time! 27 Note 2: No tuning was done
 (results are for illustration only) Train: 578,361 rows 8,647 cols 467 classes Test: 64,263 rows 8,647 cols 143 classes Note 1: H2O columnar-compressed in-memory store only needs 60 MB to store 5 billion values (dense CSV needs 18 GB) Text Classification
  • 28.
    H2O Deep Learning,@ArnoCandel MNIST: Unsupervised Anomaly Detection with Deep Learning (Autoencoder) 28 The good The bad The ugly Download the script and run it yourself!
  • 29.
    H2O Deep Learning,@ArnoCandel 29 How well did Deep Learning do? Let’s see how H2O did in the past 10 minutes! Higgs: Live Demo (Continued) <your guess?> reference paper results Any guesses for AUC on low-level features? AUC=0.76 was the best for RF/GBM/NN (H2O)
  • 30.
    H2O Deep Learning,@ArnoCandel H2O Steam: Scoring Platform 30 Higgs Dataset Demo on 10-node cluster Let’s score all our H2O models and compare them! http://server:port/steam/index.html Live Demo
  • 31.
    H2O Deep Learning,@ArnoCandel 31 Live Demo on 10-node cluster: <10 minutes runtime for all H2O algos! Better than LHC baseline of AUC=0.73! Scoring Higgs Models in H2O Steam
  • 32.
    H2O Deep Learning,@ArnoCandel 32 Algorithm Paper’s* l-l AUC low-level H2O AUC all features
 H2O AUC Parameters (not heavily tuned), 
 H2O running on 10 nodes Generalized Linear Model - 0.596 0.684 default, binomial Random Forest - 0.764 0.840 50 trees, max depth 50 Gradient Boosted Trees 0.73 0.753 0.839 50 trees, max depth 15 Neural Net 1 layer 0.733 0.760 0.830 1x300 Rectifier, 100 epochs Deep Learning 3 hidden layers 0.836 0.850 - 3x1000 Rectifier, L2=1e-5, 40 epochs Deep Learning 4 hidden layers 0.868 0.869 - 4x500 Rectifier, L1=L2=1e-5, 300 epochs Deep Learning 5 hidden layers 0.880 0.871 - 5x500 Rectifier, L1=L2=1e-5 Deep Learning on low-level features alone beats everything else! Prelim. H2O results compare well with paper’s results* (TMVA & Theano) Higgs Particle Detection with H2O *Nature paper: http://arxiv.org/pdf/1402.4735v2.pdf HIGGS UCI Dataset: 21 low-level features AND 7 high-level derived features Train: 10M rows, Test: 500k rows
  • 33.
    H2O Deep Learning,@ArnoCandel H2O Kaggle Starter R Scripts 33 Final ranking:
 #26 out of 1604
  • 34.
    H2O Deep Learning,@ArnoCandel Currently Ongoing Challenge 34
  • 35.
    H2O Deep Learning,@ArnoCandel Re-Live H2O World! 35 http://h2o.ai/h2o-world/ http://learn.h2o.ai Watch the Videos Day 2 • Speakers from Academia & Industry • Trevor Hastie (ML) • John Chambers (S, R) • Josh Bloch (Java API) • Many use cases from customers • 3 Top Kaggle Contestants (Top 10) • 3 Panel discussions Day 1 • Hands-On Training • Supervised • Unsupervised • Advanced Topics • Markting Usecase • Product Demos • Hacker-Fest with 
 Cliff Click (CTO, Hotspot)
  • 36.
    H2O Deep Learning,@ArnoCandel 36 H2O GitBooks Also available: GBM & GLM GitBooks at http://h2o.gitbooks.io H2O World learn.h2o.ai R, EC2, Hadoop Deep Learning
  • 37.
    H2O Deep Learning,@ArnoCandel h2o-dev: Flow, Python, JS, Spark 37
  • 38.
    H2O Deep Learning,@ArnoCandel h2o-dev Python Example 38
  • 39.
    H2O Deep Learning,@ArnoCandel You can participate! 39 - Images: Convolutional & Pooling Layers PUB-644 - Sequences: Recurrent Neural Networks PUB-1052 - Faster Training: GPGPU support PUB-1013 - Pre-Training: Stacked Auto-Encoders PUB-1014 - Ensembles PUB-1072 - Use H2O at Kaggle Challenges!
  • 40.
    H2O Deep Learning,@ArnoCandel Key Take-Aways H2O is an open source predictive analytics platform for data scientists and business analysts who need scalable and fast machine learning. H2O Deep Learning is ready to take your advanced analytics to the next level - Try it on your data! Join our Community and Meetups! https://github.com/h2oai h2ostream community forum www.h2o.ai @h2oai 40 Thank you!