For real-world ML systems, it is crucial to have scalable and flexible platforms to build ML workflows. In this workshop, we will demonstrate how to build an ML DevOps pipeline using Kubeflow and TensorFlow Extended (TFX). Kubeflow is a flexible environment to implement ML workflows on top of Kubernetes - an open-source platform for managing containerized workloads and services, which can be deployed either on-premises or on a Cloud platform. TFX has a special integration with Kubeflow and provides tools for data pre-processing, model training, evaluation, deployment, and monitoring. In this workshop, we will demonstrate a pipeline for training and deploying an RNN-based Recommender System model using Kubeflow. https://papislatam2019.sched.com/event/OV1M/training-and-deploying-ml-models-with-kubeflow-and-tensorflow-extended-tfx-sponsored-by-cit

1. Training and deploying ML
with Kubeflow and TensorFlow Extended (TFX)

2. About us
Gabriel Moreira
Lead Data Scientist - CI&T
Doctoral Candidate - ITA
@gspmoreira
Rodrigo PereiraFábio Uechi
Data Scientist - CI&T
Master’s Student - UNICAMP
@fabiouechi
ML Engineer - CI&T

3. DRIVEN BY
IMPACT
We are digital transformation agents
for the most valuable brands in the
world, generating business impact for
all projects we lead.

4. Investing in Machine
Learning since 2012
Recognized Expertise
Google ML Specialized Partner
Tensorflow.org Reference
ciandt.com
Cognitive
Solutions
End-to-End
Machine Learning
Capabilities

5. AGENDA
● Motivation
● Kubeflow
● TFX (TensorFlow Extended)
● Demo - News Recommender System
○ Data validation
○ Transform
○ Model training and evaluation
○ Deploy
● Demo - ML models serving and monitoring

6. Building ML Products
Building
a
Model
Logging
Data
Ingestion
Data
Analysis
Data
Transform
-ation
Data
Validation
Data
Splitting
Trainer
Model
Validation
Training
At Scale
Roll-out Serving Monitoring

7. MOTIVATION

8. MOTIVATION
Prototype MVP With Demo In Jupyter
Notebook: 2 Weeks
Demo with front-end mockup with
blog post: +3 Days
Experiments.Github.Com: +3 Months
https://github.com/hamelsmu/code_search https://towardsdatascience.com/semantic-code-se
arch-3cd6d244a39c
https://experiments.github.com/

9. MOTIVATION - Perception
Configuration
Data
Collection
Data
Verification
Feature
Extraction
Process
Management
Tools
Analysis
Tools
Machine
Resource
Management
Serving
Infrastructure
Monitoring
ML Code

10. Reality: ML requires DevOps; lots of it
Configuration
Data Collection
Data
Verification
Feature Extraction Process Management
Tools
Analysis Tools
Machine Resource
Management
Serving
Infrastructure
Monitoring
ML
Code
Source: Sculley et al.: Hidden Technical Debt in
Machine Learning Systems

11. Less devops work
Let data scientists and ML
engineers focus on models & data
Source: Monica Rogatti’s Hierarchy of Needs

12. ML Landscape (https://landscape.lfdl.io)

13. ML platforms
● Uber - Michelangelo
● AirBnB - Bighead
● Facebook - FB Learner
● Lyft - Lyft Learn

14. A curated set of compatible tools and artifacts that lays a
foundation for running production ML apps on top of
Kubernetes

15. What is Kubernetes ?
Greek for “Helmsman”; also the root of the word
“Governor”
● Container orchestrator
● Runs containers
● Supports multiple clouds and bare-metal environments
● Inspired and informed by Google’s experiences and internal
systems
● Open source, written in Go
● kubernetes.io
Manage applications, not machines

16. Kubeflow: A platform for building ML products
● Leverage containers and Kubernetes to solve the challenges of building ML products
○ Reduce the time and effort to get models launched
● Why Kubernetes
○ Kubernetes runs everywhere
○ Enterprises can adopt shared infrastructure and patterns for ML and non ML services
○ Knowledge transfer across the organization
● Kubeflow is open
○ No lock in
○ 120+ Members
○ 20+ Organizations
○ Stats available @ http://devstats.kubeflow.org

17. ML Components
● Goal: components for every stage of ML
● Examples:
○ Experimentation / Data Exploration
■ Jupyter/JupyterHub
○ Training
■ K8s CRDs for distributed training for
PyTorch & TFJob
■ Katib - For HP Tuning
○ Workflows:
■ Pipelines
○ Feature Store
■ Feast (from GOJEK)
○ Serving
■ Seldon, TF and NVIDIA RT

18. Kubeflow
Pipelines

19. ● End-to-end ML workflows
● Orchestration
● Service integration
● Components & sharing
● Job tracking, experimentation,
monitoring
● Notebook integration
KubeFlow Pipelines

20. csv2tfrecord = dsl.ContainerOp(
name='csv2tfrecord',
image='gcr.io/{}/chameleon/preproc:3bb6ab3'.format(project_id),
command=["python3",
"/app/csv2tfrecord_transform.py"],
arguments=[
"--input_clicks_csv_path", input_clicks_csv_path_regex,
"--output_sessions_tfrecords_path", output_sessions_tfrecords_path,
"--tfx_artifacts_dir", output_sessions_artifacts_path,
"--first_n_hours", first_n_hours,
"--runner", "DirectRunner"
],
file_outputs={
"tfrecords_path": "/tfrecords_path.txt",
"tfx_path": "/tfx_path.txt"
}
).apply(gcp.use_gcp_secret('user-gcp-sa'))

21. data_validator = dsl.ContainerOp(
name='data-validator',
image='gcr.io/{}/chameleon/data-validator:b356ed7'.format(project_id),
command=["python", "-m", "data_validator"],
arguments=['--input-data-path', input_clicks_csv_path_regex,
'--top-n', data_validator_top_n,
'--offset', data_validator_hour_offset]
).apply(gcp.use_gcp_secret('user-gcp-sa'))

22. train = dsl.ContainerOp(
name='train',
image='gcr.io/{}/chameleon/train:4f19dbc'.format(project_id),
command=["python", "-m", "nar.nar_trainer_gcom"],
arguments=[
"--model_dir", str(model_dir) + "/{{workflow.uid}}",
"--input_tfrecords_path", csv2tfrecord.outputs["tfrecords_path"],
"--train_files_from", 0,
"--train_files_up_to", first_n_hours,
"--training_hours_for_each_eval", 5,
],
file_outputs={
'saved_model_path': '/saved_model_path.txt'
},
output_artifact_paths={
'mlpipeline-ui-metadata': '/mlpipeline-ui-metadata.json',
'mlpipeline-metrics': '/mlpipeline-metrics.json'
}
).apply(gcp.use_gcp_secret('user-gcp-sa'))
.after(csv2tfrecord)

23. model_pusher = dsl.ContainerOp(
name='pusher',
image='gcr.io/cloud-builders/gsutil',
arguments=[ '-m', 'cp', '-r', saved_model_path + '/*', tfserving_path]
).apply(gcp.use_gcp_secret('user-gcp-sa')).after(train)
Demo: Run the pipe!

24. Use Case:
News Recommender
System

25. Challenges
News Recommender Systems
1. Streaming clicks and news articles
2. Most users are anonymous
3. Users’ preferences shift
4. Accelerated relevance decay
Percentile of clicks Article age
10% up to 4 hours
25% up to 5 hours
50% (Median) up to 8 hours
75% up to 14 hours
90% up to 26 hours

26. Factors affecting news relevance
News Recommender Systems
News
relevance
Topics Entities Publisher
News static properties
Recency Popularity
News dynamic properties
News article
User
TimeLocation Device
User current context
Long-term
interests
Short-term
interests
Global factors
Season-
ality
User interests
Breaking
events
Popular
Topics
Referrer

27. News session-based recommender overview
CHAMELEON
User session clicks
C1
C2
C3
C4
Next-click prediction
(RNN model)
Article B
Article A
Article C
Article D
...
Ranked articles
Candidate (recommendable) articles

28. Article
Context
Article
Content
Embeddings
Next-Article Recommendation (NAR)
Time
Location
Device
User context
User interaction
past read articles
Popularity
Recency
Article context
Users Past
Sessions
candidate next articles
(positive and neg.)
active article
Active
Sessions
When a user reads a news article...
Predicted Next-Article Embedding
Session Representation (SR)
Recommendations Ranking (RR)
User-Personalized Contextual Article Embedding
Contextual Article Representation (CAR)
Active user session
Module Sub-Module EmbeddingInput Output Data repositoryAttributesLegend:
Article
Content
Embedding
28
Recommendations Ranking
(RR) sub-module
Eq. 7 - Loss function (HUANG et al., 2013)
Eq. 4 - Relevance Score of an item for a user session
Eq. 5 - Cosine similarity
Eq. 6 - Softmax over Relevance Score (HUANG et al., 2013)
Recommended
articles
What goes inside the box?CHAMELEON

29. TensorFlow
Extended

30. TensorFlow Extended
TFX is set of libraries that helps you to implement a scalable and high-performance machine learning
pipeline that might includes the steps: data preprocessing, modeling, training, serving inference, and
managing deployments to online, mobile and JavaScript targets.
Main Components:
● TensorFlow Data Validation (TFDV)
● TensorFlow Transform (TFT)
● TensorFlow Model Analysis (TFMA)
OBS: Apache Beam is required to build any TFX pipeline.

31. Demo: Session-based Recommender Systems using TFX components in a Kubeflow pipeline
TensorFlow Extended

32. TFDV - Data
Validation

33. TFDV - TensorFlow Data Validation
TensorFlow Data Validation (TFDV) is a library for data exploration and validation.
TFDV includes:
● Scalable calculation of summary statistics of training and test data.
● Integration with a viewer for data distributions and statistics
● Automated data-schema generation to describe expectations about data like required values, ranges,
and vocabularies
● Anomaly detection to identify anomalies, such as missing features, missing values, out-of-range
values, wrong feature types, distribution skewness

34. def analyse(input_data_list, top_n, offset=24):
logger.info('Infer data schema from first file')
stats = tfdv.generate_statistics_from_csv(
data_location=input_data_list[0])
inferred_schema = tfdv.infer_schema(statistics=stats)
logger.info("Inferred schema n {}".format(inferred_schema))
curr_stats = stats
for file_i in range(offset, top_n, 1):
logger.info('Checking for anomalies between {} and {}'.format(
input_data_list[file_i-offset], input_data_list[file_i]))
future_stats = tfdv.generate_statistics_from_csv(
data_location=input_data_list[file_i])
for feat_name in ["click_article_id",
"session_start",
"click_timestamp",
"click_region",
"click_environment",
"click_country",
"click_os",
"session_size",
"session_id",
"click_deviceGroup",
"user_id",
"click_referrer_type"]:
feature = tfdv.get_feature(inferred_schema, feat_name)
feature.skew_comparator.infinity_norm.threshold = 0.01
feature.drift_comparator.infinity_norm.threshold = 0.01
anomalies = tfdv.validate_statistics(previous_statistics=curr_stats,
statistics=future_stats, schema=inferred_schema)
n_anomalies = len(anomalies.anomaly_info.items())
if n_anomalies == 0:
logger.info('No anomalies found')
else:
logger.warn('{} anomalies found')
for feature_name, anomaly_info in anomalies.anomaly_info.items():
logger.info("Feature {} Anomaly: {}".format(
feature_name, anomaly_info.description))
curr_stats = future_stats

35. TFT -
Tensorflow
Transform

36. TFT - TensorFlow Transform
A library for preprocessing data with TensorFlow. TensorFlow Transform is useful for data that requires a full-
pass transformations, such as:
● Input normalization.
● Convert strings to integers by generating a vocabulary over all input values.
Goal: Write transform function only once and use it both on training and serving.
OBS: Currently FixedLenSequenceFeature are not supported

37. def feature_spec_schema():
""" Feature specification schema
"""
schema_dict = {}
for feat, feat_type in [('user_id', tf.int64),
('session_id', tf.int64),
('session_start', tf.int64),
('session_size', tf.int64),
]:
schema_dict[feat] = tf.FixedLenFeature([], dtype=feat_type)
for feat, feat_type in [('click_timestamp', tf.int64),
('click_article_id', tf.int64),
('click_environment', tf.int64),
('click_deviceGroup', tf.int64),
('click_os', tf.int64),
('click_country', tf.int64),
('click_region', tf.int64),
('click_referrer_type', tf.int64)]:
schema_dict[feat] = tf.VarLenFeature(dtype=feat_type)
schema = dataset_metadata.DatasetMetadata(
dataset_schema.from_feature_spec(schema_dict))
return schema
import apache_beam as beam
import tensorflow_transform as tft
from tensorflow_transform.beam import impl
from tensorflow_transform.tf_metadata import dataset_schema
from tensorflow_transform.tf_metadata import dataset_metadata
from tensorflow_transform.coders import example_proto_coder
from tensorflow_transform.tf_metadata import metadata_io
from tensorflow_transform.beam.tft_beam_io import transform_fn_io

38. dataset_schema = feature_spec_schema()
with beam.Pipeline(options=pipeline_options) as pipeline:
with impl.Context(args.temp_dir):
clicks_hour_files_pc = (
pipeline | "Load Files" >> beam.Create(clicks_hour_files))
sessions_per_hour = (
clicks_hour_files_pc | "Load Sessions Per Hour" >> beam.ParDo(LoadSessionsPerHour()))
transformed_data, transform_fn = ((sessions_per_hour, dataset_schema) |
"Analyze and Transform" >> impl.AnalyzeAndTransformDataset(preprocess_fn))
transformed_dataset, transformed_metadata = transformed_data
timestamped_items = transformed_dataset | 'timestamp' >> beam.ParDo(
AddTimestampDoFn())
fixed_windowed_items = (
timestamped_items | 'window' >> beam.WindowInto(beam.window.FixedWindows(60 * 60), # Window size = 1h
accumulation_mode=beam.trigger.AccumulationMode.DISCARDING
))
_ = (fixed_windowed_items |
'Format' >> beam.ParDo(ExtractTimeWindowStartAsKey()) |
"Aggregate by hour_index" >> beam.GroupByKey() |
"Dump TFRecord per hour" >> beam.Map(
lambda g: DumpTFRecord(g, args.output_sessions_tfrecords_path)))
_ = transform_fn | 'Dump Transform Function Graph' >> transform_fn_io.WriteTransformFn(
args.tfx_artifacts_dir)

39. def preprocess_fn(input_feature):
output_features = {}
# FixedLenFeature
output_features['user_id'] = input_feature['user_id']
output_features['session_id'] = input_feature['session_id']
output_features['session_start'] = input_feature['session_start']
output_features['session_size'] = input_feature['session_size']
# VarLenFeature
output_features['item_clicked'] = input_feature['click_article_id']
output_features['environment'] = input_feature['click_environment']
output_features['deviceGroup'] = input_feature['click_deviceGroup']
output_features['os'] = input_feature['click_os']
output_features['country'] = input_feature['click_country']
output_features['region'] = input_feature['click_region']
output_features['referrer_type'] = input_feature['click_referrer_type']
output_features['event_timestamp'] = input_feature['click_timestamp']
local_weekday = tf.SparseTensor(input_feature['click_timestamp'].indices,
tf.map_fn(fn=utc_timestamp_to_local_weekday,
elems=input_feature['click_timestamp'].values,
back_prop=False,
infer_shape=False,
dtype=tf.int64),
input_feature['click_timestamp'].dense_shape)
output_features['local_weekday'] = local_weekday
...
local_hour_cos = tf.SparseTensor(input_feature['click_timestamp'].indices,
tf.map_fn(fn=utc_timestamp_to_local_hour_cos,
elems=input_feature['click_timestamp'].values,
back_prop=False,
infer_shape=False,
dtype=tf.float32),
input_feature['click_timestamp'].dense_shape)
output_features['local_hour_cos'] = local_hour_cos
return output_features

40. transformed_train, transform_fn = ((raw_train, schema) |
"Analyze and Transform - train" >>
impl.AnalyzeAndTransformDataset(preprocess_fn))
transformed_eval = (((raw_eval, schema), transform_fn) |
"Transform - eval" >> impl.TransformDataset())
What about eval set?

41. Tensorflow
Training
and
Evaluation

43. TF Estimator

44. def build_estimator(model_dir,
content_article_embeddings_matrix,
articles_metadata, articles_features_config,
session_features_config):
"""Build an estimator appropriate for the given model type."""
run_config = tf.estimator.RunConfig(tf_random_seed=RANDOM_SEED,
keep_checkpoint_max=1,
save_checkpoints_secs=1200,
save_summary_steps=100,
log_step_count_steps=100
)
estimator = tf.estimator.Estimator(
config=run_config,
model_dir=model_dir,
model_fn=nar_module_model_fn,
params={
'batch_size': FLAGS.batch_size,
'lr': FLAGS.learning_rate,
'dropout_keep_prob': FLAGS.dropout_keep_prob,
'reg_weight_decay': FLAGS.reg_l2,
...
})
return estimator
Creating the Estimator

45. def nar_module_model_fn(features, labels, mode, params):
model = NARModuleModel(mode, features, labels,… )
hooks = [ItemsStateUpdaterHook(mode, model,… )]
if mode == tf.estimator.ModeKeys.TRAIN:
return tf.estimator.EstimatorSpec(mode, loss=model.total_loss,
train_op=model.train, training_chief_hooks=hooks)
elif mode == tf.estimator.ModeKeys.EVAL:
eval_metrics = {'mrr_at_n': (model.mrr, model.mrr_update_op), ... }
return tf.estimator.EstimatorSpec(mode, loss=model.total_loss,
eval_metric_ops=eval_metrics, evaluation_hooks=hooks)
elif mode == tf.estimator.ModeKeys.PREDICT:
predictions = model.predictions
export_outputs = {'predict_output': tf.estimator.export.PredictOutput(predictions)}
return tf.estimator.EstimatorSpec(mode, predictions=predictions,
export_outputs=export_outputs)
Building the model function

46. tft_metadata = TFTransformOutput(FLAGS.tft_artifacts_dir)
model = build_estimator(model_output_dir, article_embeddings_matrix,
articles_metadata, articles_features_config, ...)
model.train(input_fn=lambda: prepare_dataset_iterator(training_files_chunk,
tft_metadata, batch_size=FLAGS.batch_size, ...))
model.evaluate(input_fn=lambda: prepare_dataset_iterator(eval_file,
tft_metadata, batch_size=FLAGS.batch_size, ...)
predictions = model.predict(input_fn=lambda:
prepare_dataset_iterator(tfrecords_files, tft_metadata,
FLAGS.batch_size, ...)
Training, Evaluating and Predicting with the Estimator

47. def prepare_dataset_iterator(files, tft_metadata, batch_size=128, ...)
feature_spec = tft_metadata.transformed_feature_spec()
# This makes a dataset of raw TFRecords
dataset = tf.data.TFRecordDataset(path, compression_type='GZIP')
dataset = dataset.map(lambda x: tf.io.parse_single_example(x, feature_spec))
dataset = dataset.padded_batch(batch_size, padded_shapes=features_shapes)
# Define an abstract iterator that has the shape and type of our datasets
iterator = ds.make_one_shot_iterator()
# This is an op that gets the next element from the iterator
next_element = iterator.get_next()
return next_element
Defining input function
Features schema come from TFT!

48. def export_saved_model(model, model_output_path, additional_features_info, tft_metadata):
raw_feature_spec = feature_spec_schema()
def serving_input_fn():
raw_input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(
raw_feature_spec, default_batch_size=None)
serving_input_receiver = raw_input_fn()
# Apply the transform function that was used to generate the materialized data.
raw_features = serving_input_receiver.features
transformed_features = tft_metadata.transform_raw_features(raw_features)
for feature_name in transformed_features.keys():
if type(transformed_features[feature_name]) == tf.sparse.SparseTensor
transformed_features[feature_name] = tf.sparse.to_dense(
transformed_features[feature_name])
return tf.estimator.export.ServingInputReceiver(
receiver_tensors=serving_input_receiver.receiver_tensors,
features=transformed_features)
servable_model_path = model.export_savedmodel(
model_output_path, serving_input_fn, strip_default_attrs=True)
return servable_model_path
Defining serving function and exporting SavedModel
Apply transforms
from TFT graph

49. Tensorflow
Model
Analysis

50. TFMA - Model Analysis
TensorFlow Model Analysis allows you to
perform model evaluations in the TFX pipeline,
and view resultant metrics and plots in a
Jupyter notebook. Specifically, it can provide:
● Metrics computed on entire training and
holdout dataset, as well as next-day
evaluations
● Tracking metrics over time
● Model quality performance on different
feature slices
● Supports evaluation on large amounts of
data in the distributed manner

51. TFMA - Model Analysis

52. TFMA - Model Analysis

53. TFMA - Model Analysis

54. Demo
● CI/CD
● Serving Seldon (A/B testing)
● Monitoring with Prometheus and Grafana

55. Machine Learning CI/CD - Gitops style
Data Scientist

56. Tensorflow
Serving
Serving and monitoring: Single model
Model Repository
Model
BModel
A
Model AEngineClients
REST / GRPC
metrics

57. Tensorflow
Serving
Serving and monitoring: A/B testing
Model Repository
Model
BModel
A
Model A
Model B
A/B TestClients
REST / GRPC
metrics

58. midi.as/papislatam
CHAMELEON research code:
http://bit.ly/chameleon_papis
P.s. The code demonstrated in this workshop will be
published soon on GitHub and will be referenced in the
README of the above repo)

59. Thank you!
ciandt.com