This document discusses pipelines for model deployment using Kubernetes (k8s) and Jenkins CI/CD. It describes deploying both static and dynamic models. Static models involve calculating a model offline and deploying pre-trained models. Dynamic models allow for distributed continuous training and serving multiple live models. The architecture uses k8s, a REST API, processing engines, and a queue system to extract features, train models, and serve predictions. Models and metadata are stored in a distributed data platform called Guillotina. Tensorflow Serving pods are used to serve models via gRPC. Dynamic flows allocate workers, train models, and write results to the shared storage.

1. Pipelines for model deployment
Ramon Navarro Bosch, CTO Onna

2. What is
• Startup Barcelona / San
Francisco
• SaaS / On-Premise
• Connect sources of
information, gather, analyze
and offer a UI to search for
information
• Knowledge Management and
apps for eDiscovery and
contract management
• API centric solution

3. “41 components architecture distributed with
kubernetes (k8s) and deployed with jenkins
CI/CD”

4. Static models
• Calculate a model offline
• SVM / CNN / RF / RNN /
LSTM / Summary / …
• Mostly tensorflow, sklearn

5. Static models
Dataset
Algorithm
Test
Pack
Deploy
dev
stage
prod
tf…
Train

6. Static models
• Detect interesting content type
• Get dataset - apply generic feature extraction
• Train model … for example: SVM (binary) - versions of the
models as contained files
• Embed in processing container jenkinsfile / dockerfile
• Build container and push version to registry
• Adding to production stack
• Continuous build on each commit

7. Arquitecture - k8s
Canonical REST API
Guillotina
Processing engine
…
Account namespace Shared namespace

8. Queue system
New
document
Pipeline RabbitMQ
Feature
Extraction
Beat Beat
Write
Each component redirect messages processed to next
exchange
Director Exit

9. • Testing accuracy on production processing new messages.
Plug to queue and don’t ack messages to get reprocessed
• Scale the # of pods/beats for each one based on the size of the
queue
• Dead letter queues for not well processed
• Delay queues for incremental processing
• Experimental beats
• REST API to our Beats engine to extract information about a
resource
Static models

10. “We allow users to create their own models”

11. Guillotina
• Open Source scalable resource management framework (data
plaform)
• Provides an extensible Transactional Traversal REST API with
distributed DB support
• Event triggering
• Security model
• All async
• PyData talk 21/5

12. Dynamic models
• Distributed continuous training
• Serve multi model live
• Model storage
• Direct access to distributed
data
• Inference and train

13. Model+Meta
Worker ParameterServer Serving
Algorithm
DocumentsLabels Metadata
EmbeddingVocabulary

14. Simplified example
• User defines labels - ML - BUSINESS
• Asks to train a classification model based on
logistic regression
• A model is saved on the DB (SavedModel)
• User defines a Rule to apply the model (tf serving) -
each time I have a new Mail from Gmail

15. tf.serving pods (c++)
• gRPC from Canonical to inference with mini batch /
model spec
• SourceAdapter (Loader) connector for Canonical
model loading (versions handling of the model)
• ServableAdapter with sharing vocabulary and
multiple loaded models (multiple models on a
serving component)
• k8s scaling and monitoring

16. Dynamic distributed flow
• Allocate variables and workers pods dynamic with k8s python api (there are
limits)
• Packs an Experiment and the Estimator (with Keras model or direct TF)
• input_fn is a feed generator that gets the data from the web socket API to
tfRecord (Guillotina model to Protobuff model)
• Re-train with new documents loading the model last check
• Write the model by the main worker on the canonical
• Another pod runs the validation on the saved canonical for each checkpoint
• Dev can runs the tensorboard to the canonical model endpoint

17. Canonical Ecosystem
Serving TrainingTrainingTrainingTrainingTraining

18. Preguntes ?
Gràcies!
Ramon Navarro Bosch
ramon@onna.com