With most machine learning (ML) and deep learning (DL) frameworks, it can take hours to move data for ETL, and hours to train models. It's also hard to scale, with data sets increasingly being larger than the capacity of any single server. The amount of the data also makes it hard to incrementally test and retrain models in near real-time. Learn how Apache Ignite and GridGain help to address limitations like ETL costs, scaling issues and Time-To-Market for the new models and help achieve near-real-time, continuous learning. Yuriy Babak, the head of ML/DL framework development at GridGain and Apache Ignite committer, will explain how ML/DL work with Apache Ignite, and how to get started. Topics include: — Overview of distributed ML/DL including architecture, implementation, usage patterns, pros and cons — Overview of Apache Ignite ML/DL, including built-in ML/DL algorithms, and how to implement your own — Model inference with Apache Ignite, including how to train models with other libraries, like Apache Spark, and deploy them in Ignite — How Apache Ignite and TensorFlow can be used together to build distributed DL model training and inference

1. 2019 © GridGain Systems
Distributed Machine
and Deep Learning at Scale
with Apache Ignite
Yuriy Babak

2. 2019 © GridGain Systems
Yuriy Babak
Head of ML/DL framework
development at GridGain
Apache Ignite committer

3. 2019 © GridGain Systems2019 © GridGain Systems
Agenda
● Overview of distributed ML/DL
● Model training in distributed environment
● Data preprocessing and building pipelines
● Model updates and online learning
● Some extra features

4. 2019 © GridGain Systems
Distributed machine
learning with Apache Ignite

5. 2019 © GridGain Systems2019 © GridGain Systems
Why do we need distributed ML?
● Data preprocessing
○ Big amount of data on different machines
○ High costs of ETL
● Model training
○ We cannot store all data on single server
○ Data irregularity
● Inference
○ Intensive data flow
○ Data co-located inference
○ Network latency

6. 2019 © GridGain Systems2019 © GridGain Systems
Why do we need distributed ML?
● Data preprocessing
○ Big amount of data on different machines
○ High costs of ETL
● Model training
○ We cannot store all data on single server
○ Data irregularity
● Inference
○ Intensive data flow
○ Data co-located inference
○ Network latency

7. 2019 © GridGain Systems2019 © GridGain Systems
Why do we need distributed ML?
● Data preprocessing
○ Big amount of data on different machines
○ High costs of ETL
● Model training
○ We cannot store all data on single server
○ Data irregularity
● Inference
○ Intensive data flow
○ Data co-located inference
○ Network latency

8. 2019 © GridGain Systems2019 © GridGain Systems
Common approach
Local calculations Result aggregationModel
Model update

9. 2019 © GridGain Systems2019 © GridGain Systems
Ignite partition-based distributed dataset
Partition Data Dataset Context Dataset Data
Upstream Cache Context Cache On-Heap
Learning Env
On-Heap
Durable Stateless Durable Recoverable
double[][] x = …
double[] y = ...
double[][] x = …
double[] y = ...
Partition Based Dataset StructuresSource Data

10. 2019 © GridGain Systems
Model training

11. 2019 © GridGain Systems
Algorithms: Classification
● Logistic Regression
● SVM
● KNN
● ANN
● Decision trees
● Random Forest
● Naive Bayes
#UnifiedAnalytics #SparkAISummit

12. 2019 © GridGain Systems
Algorithms: Regression
• KNN Regression
• Linear Regression
• Decision tree regression
• Random forest
regression
• Gradient-boosted tree
regression
#UnifiedAnalytics #SparkAISummit

13. 2019 © GridGain Systems
Algorithms: Clusterization
• K-means
• GMM
#UnifiedAnalytics #SparkAISummit

14. 2019 © GridGain Systems
Multilayer Perceptron Neural Network
#UnifiedAnalytics #SparkAISummit

15. 2019 © GridGain Systems2019 © GridGain Systems
Ignite ML API
Vectorizer<Integer, BinaryObject, String, Double> vectorizer =
new BinaryObjectVectorizer<Integer>(
"feature1",
“feature2”
).labeled("label");
KNNClassificationTrainer trainer = new KNNClassificationTrainer();
KNNClassificationModel mdl = trainer.fit(ignite, dataCache,
vectorizer).withK(3)
.withDistanceMeasure(new EuclideanDistance())
.withStrategy(NNStrategy.WEIGHTED);

16. 2019 © GridGain Systems
ML Pipelines
with Apache Ignite

17. 2019 © GridGain Systems2019 © GridGain Systems
Preprocessing and Pipelines

18. 2019 © GridGain Systems2019 © GridGain Systems
Preprocessing
● Imputing
● Normalization
● MinMax Scaler
● MaxAbs Scaler
● Standard Scaler
● Binarization
● One-Hot Encoder
● String Encoder

19. 2019 © GridGain Systems2019 © GridGain Systems
Preprocessing and Pipelines
IgniteCache<Integer, Vector> dataCache = TitanicUtils.readPassengers(ignite);
// Extracts "pclass", "sibsp", "parch", "sex", "embarked", "age", "fare".
Vectorizer<Integer, Vector, Integer, Double> vectorizer
= new DummyVectorizer<Integer>(0, 3, 4, 5, 6, 8, 10).labeled(1);
PipelineMdl<Integer, Vector> mdl =
new Pipeline<Integer, Vector, Integer, Double>()
.addVectorizer(vectorizer)
.addPreprocessingTrainer(new EncoderTrainer<Integer, Vector>()
.withEncoderType(EncoderType.STRING_ENCODER)
.withEncodedFeature(1)
.withEncodedFeature(6))
.addPreprocessingTrainer(new ImputerTrainer<Integer, Vector>())
.addPreprocessingTrainer(new MinMaxScalerTrainer<Integer, Vector>())
.addPreprocessingTrainer(new NormalizationTrainer<Integer, Vector>()
.withP(1))
.addTrainer(new DecisionTreeClassificationTrainer(5, 0))
.fit(ignite, dataCache);

20. 2019 © GridGain Systems2019 © GridGain Systems
Bagging, Boosting and Stacking
DatasetTrainer<LogisticRegressionModel, Double> trainer =
new LogisticRegressionSGDTrainer()
.withUpdatesStgy(new UpdatesStrategy<>(new SimpleGDUpdateCalculator(0.2),
SimpleGDParameterUpdate.SUM_LOCAL, SimpleGDParameterUpdate.AVG))
.withMaxIterations(30000)
.withLocIterations(100)
.withBatchSize(10)
.withSeed(123L);
BaggedTrainer<Double> baggedTrainer = TrainerTransformers.makeBagged(trainer,
// ensemble size, subsample ration, feature vector size, features subspace dim
7, 0.7, 2, 2,
new onMajorityPredictionsAggregator());

21. 2019 © GridGain Systems
Extra features

22. 2019 © GridGain Systems2019 © GridGain Systems
Online learning
SVMLinearClassificationTrainer trainer = new SVMLinearClassificationTrainer();
SVMLinearClassificationModel mdl1 = trainer.fit(ignite, dataCache1, vectorizer);
SVMLinearClassificationModel mdl2 = trainer.update(mdl1, ignite, dataCache2,
vectorizer);

23. 2019 © GridGain Systems2019 © GridGain Systems
Model import

24. 2019 © GridGain Systems
TensorFlow on Apache Ignite
• Ignite Dataset
• IGFS Plugin
• Distributed Training
• More info here
#UnifiedAnalytics #SparkAISummit
>>> import tensorflow as tf
>>> from tensorflow.contrib.ignite import IgniteDataset
>>>
>>> dataset = IgniteDataset(cache_name="SQL_PUBLIC_KITTEN_CACHE")
>>> iterator = dataset.make_one_shot_iterator()
>>> next_obj = iterator.get_next()
>>>
>>> with tf.Session() as sess:
>>> for _ in range(3):
>>> print(sess.run(next_obj))
{'key': 1, 'val': {'NAME': b'WARM KITTY'}}
{'key': 2, 'val': {'NAME': b'SOFT KITTY'}}
{'key': 3, 'val': {'NAME': b'LITTLE BALL OF FUR'}}

25. 2019 © GridGain Systems2019 © GridGain Systems
Inference in Ignite ML
2019 © GridGain Systems
Request queue
Response queue
Inference
Service
Inference
Service
Inference
Gateway

26. 2019 © GridGain Systems2019 © GridGain Systems
IMDB with built-in ML
IgniteModelStorageUtil.saveModel(ignite, model, “titanik_model_tree”);
QueryCursor<List<?>> cursor = cache.query(new SqlFieldsQuery("select " +
"survived as truth, " +
"predict('titanik_model_tree', pclass, age, sibsp, parch, fare, case
sex when 'male' then 1 else 0 end) as prediction " +
"from titanik_train"))
2019 © GridGain Systems

27. 2019 © GridGain Systems2019 © GridGain Systems
Apache Ignite with GridGain ML Python API
GridGain ML client library provides user applications the ability to work
with GridGain ML functionality using Py4J as an integration mechanism.
If you want to use ggml in your project, you may install it from PyPI:
$ pip install ggml

28. 2019 © GridGain Systems2019 © GridGain Systems
Apache Ignite with GridGain ML Python API
GridGain ML client library provides user applications the ability to work
with GridGain ML functionality using Py4J as an integration mechanism.
If you want to use ggml in your project, you may install it from PyPI:
$ pip install ggml
NB: available only for Apache Ignite master and for GG 8.7.6 (17 Jul)

29. 2019 © GridGain Systems2019 © GridGain Systems
Apache Ignite with GridGain ML Python API
from ggml.regression import LinearRegressionTrainer
trainer = LinearRegressionTrainer()
model = trainer.fit(x_train, y_train)
r2_score(y_test, model.predict(x_test))

30. 2019 © GridGain Systems2019 © GridGain Systems
Apache Ignite ML Tutorial
https://github.com/apache/ignite/
org.apache.ignite.examples.ml.tutorial
2019 © GridGain Systems

31. 2019 © GridGain Systems2019 © GridGain Systems
Distributed Machine and Deep Learning at
Scale with Apache Ignite
Links:
• http://ignite.apache.org/
• https://medium.com/tensorflow/tensorflow-on-apache-ignite-99f1fc60efeb
• https://github.com/gridgain/ml-python-api
• http://bit.ly/LondonJune5
Email:
● user@ignite.apache.org
● dev@ignite.apache.org
● ybabak@gridgain.com