3. MSD IRELAND
+ 50 years
Approx. 2,000 employees
$2.5 billion investment to date
Approx 50% MSD’s top 20 products manufactured here
Export to + 60 countries
€6.1 billion turnover in 2017
2017 + 300 jobs & €280m investment
MSD Biotech, Dublin, coming in 2021
https://www.msd-ireland.com/
5. CORE TOPICS
• What is it?Deep Learning
• 2 of the most popular frameworks for DLKeras and Tensorflow
• Why is it so difficult?
Why Distributed Deep
Learning on Spark?
• Why and How?
DL in Python on the
JVM
6. DEEP LEARNING
It is a subset of Machine
Learning which is based on
Multilayer Neural Networks
9. TENSORFLOW
It is an end-to-end open source
platform for ML. It has a
comprehensive, flexible
ecosystem of tools, libraries and
community resources for
researchers and developers.
https://www.tensorflow.org/
10. KERAS
Keras is a high-level neural
networks API, written in Python
and capable of running on top of
TensorFlow, CNTK, or Theano.
It allows for easy prototyping
and runs seamlessly on CPUs
and GPUs.
https://keras.io/
12. Speed
It achieves high performance for
both batch and streaming data,
using a state-of-the-art DAG
scheduler, a query optimizer, and a
physical execution engine.
Ease of Use
It offers over 80 high-level
operators that make it easy to build
parallel apps. And you can use it
interactively from the Scala,
Python, R, and SQL shells.
Generality
Combine SQL, streaming, and
complex analytics.
Runs Everywhere
It runs on Hadoop, Apache
Mesos, Kubernetes,
standalone, or in the cloud. It
can access diverse data
sources.
13. WHEN WOULDYOU NEED TO TRAIN
MNNS IN SPARK
• Availability of a cluster of machines for training
• Scarcity of GPUs
• Networks very large
• Huge data sets
By the way, DL4J isn’t for Spark only: you can use it on a single machine
with multiple GPUs or multiple physical processors.
14. CHALLENGES OF TRAINING MNNS
IN SPARK
• Different execution models between Spark and the DL frameworks
• GPU configuration and management
• Performance
• Accuracy
16. DEEPLEARNING4J
It is an Open Source,
distributed, Deep Learning
framework written for JVM
languages.
It is integrated with
Hadoop and Apache
Spark.
It can be used on
distributed GPUs and
CPUs.
18. DL4J MODULES
• DataVec
• Arbiter
• NN
• Datasets
• RL4J
• DL4J-Spark
• Model Import
• ND4J
It is an Open Source linear algebra
and matrix manipulation library which
supports n-dimensional arrays and it
is integrated with Apache Hadoop
and Spark.
19. DL4J + APACHE SPARK
• DL4J provides high level API to design, configure train and evaluate
MNNs.
• Spark performances are excellent in particular for ETL/streaming, but
in terms of computation, in a MNN training context, some data
transformation/aggregation needs to be done using a low-level
language.
• DL4J uses ND4J, which is a C++ library that provides high level Scala
API to developers.
20. MODEL IMPORT IN DL4J
Keras TensorFlow
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
Train the Model
Save it as .pb
Load Model and
Weights
Load New Data
Predict
KerasModelImport
TFGraphMapper
Transfer Learning
21. MODEL IMPORT IN DL4J
Keras TensorFlow
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
Train the Model
Save it as .pb
Load Model and
Weights
Load New Data
Predict
22. KERAS MODEL IMPORT: SUPPORTED
FEATURES
• Layers
• Losses
• Activations
• Initializers
• Regularizers
• Constraints
• Metrics
• Optimizers
23. MODEL IMPORT IN DL4J: EXAMPLE
Keras
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
Import the VGG16
Model.
Test it.
24. MODEL IMPORT IN DL4J: EXAMPLE
Keras
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
25. MODEL IMPORT IN DL4J: EXAMPLE
Keras
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
26. MODEL IMPORT IN DL4J: EXAMPLE
Keras
Train the Model
Save it as .h5
Load Model and
Weights
Load New Data
Predict
29. HOW TRAINING HAPPENS IN SPARK
WITH DL4J
Parameter Averaging
(DL4J 1.0.0-alpha)
Asynchronous SDG
(DL4J 1.0.0-beta+)
30. HOW TRAINING HAPPENS IN SPARK
WITH DL4J
The key classes users should be familiar with to get started with distributed
training in DL4J are:
• TrainingMaster: It specifies how distributed training will be conducted in
practice. Implementations include Gradient Sharing or Parameter Averaging .
• SparkDl4jMultiLayer and SparkComputationGraph: They are wrappers
around the MultiLayerNetwork and ComputationGraph classes in DL4J that
enable the functionality related to distributed training.
• RDD<DataSet> and RDD<MultiDataSet>: Spark RDDs with DL4J’s
DataSet or MultiDataSet classes that define the source of the training or
evaluation data.
31. RE-TRAIN AN IMPORTED MODEL
Define the Spark Context
Choose the TrainingMaster implementation
Create the Spark network
Start the training
Get the model configuration
