TensorFlow Extension (TFX) and Apache Beam

An end-to-end machine learning platform for TensorFlow
Robert Crowe
Developer Advocate, TensorFlow
@robert_crowe

In addition to the actual ML...
ML
Code

...you have to worry about so much more.
Configuration
Data Collection
Data
Verification
Feature Extraction
Process Management
Tools
Analysis Tools
Machine
Resource
Management
Serving
Infrastructure
Monitoring
Source: Sculley et al.: Hidden Technical Debt in Machine Learning Systems
ML
Code

Data
Ingestion
Data
Analysis + Validation
Data
Transformatio
n
Trainer
Model Evaluation
and Validation
Serving Logging
Shared Utilities for Garbage Collection, Data Access Controls
Pipeline Storage
Tuner
Shared Configuration Framework and Job Orchestration
Integrated Frontend for Job Management, Monitoring, Debugging, Data/Model/Evaluation Visualization
TFX is the solution to this problem...

Data
Ingestion
TensorFlow
Data Validation
TensorFlow
Transform
Estimator
or Keras
Model
TensorFlow
Model Analysis
TensorFlow
Serving
Logging
Pipeline Storage
Tuner
…and we are making it available to you.

TFX powers our most important bets and products...
(incl. )
...and more.
Major ProductsAlphaBets

Data
Ingestion
TensorFlow
Data Validation
Logging
Pipeline Storage
Tuner
Data
Transformatio
n
Trainer
Model Evaluation
and Validation
Serving

Why Data Validation is important
ML

garbage in garbage out
ML

Compute and visualize
descriptive statistics
TensorFlow Data Validation Components and Workflows
Infer initial schema
Training/serving
skew detection
Validating data

Compute and visualize descriptive statistics
day1_stats =
tfdv.generate_statistics_from_csv(DAY1)
tfdv.visualize_statistics(day1_stats)
Visualization: Facets Overview (github.com/PAIR-code/facets)
Day 1
generate_statistics
day1_stats

Training/serving skew detection
# Compute statistics over serving data
serving_data_stats =
tfdv.generate_statistics_from_csv(SERVING_DATA)
# Compare how new data conforms to the schema
anomalies = tfdv.validate_statistics(
serving_data_stats,
schema,
environment=”SERVING”)
Day 2
Serving
Data
schema
validate_
statistics
generate_
statistics
day2_stats
validate_
statistics
generate_
statistics
serving_
data_stats
Validation
Report
Validation
Report

TFDV deployment stats
● Data analysis and validation used by hundreds of product teams
● Certain teams set up TFX solely for data analysis and validation
● Petabytes of training/serving data per day
● Several documented ML wins by catching data anomalies

github.com/tensorflow/data-validation
TensorFlow Data Validation
Understand, validate, and monitor your ML data

Data
Ingestion
TensorFlow
Data Validation
Data
Transformatio
n
Trainer
Model Evaluation
and Validation
Serving Logging
Pipeline Storage
Tuner

Data
Ingestion
TensorFlow
Data Validation
TensorFlow
Transform
Trainer
Model Evaluation
and Validation
Serving Logging
Pipeline Storage
Tuner

Using tf.Transform for feature transformations.

Using tf.Transform for feature transformations.
Training
Serving

with beam.Pipeline() as pipeline:
with tft_beam.Context(temp_dir=tempfile.mkdtemp()):
...
raw_data = (
pipeline
| 'ReadTrainData' >> beam.io.ReadFromText(train_data_file)
| 'FixCommasTrainData' >> beam.Map(
lambda line: line.replace(', ', ','))
| 'DecodeTrainData' >> MapAndFilterErrors(converter.decode))
...
_ = (
transformed_data
| 'EncodeTrainData' >> beam.Map(transformed_data_coder.encode)
| 'WriteTrainData' >> beam.io.WriteToTFRecord(
os.path.join(working_dir, TRANSFORMED_TRAIN_DATA_FILEBASE)))
...
25

Common tf.Transform Use-Cases
Scale to ... Bag of Words / N-Grams
Bucketization Feature Crosses
tft.ngrams
tft.string_to_int
tf.string_split
tft.scale_to_z_score
tft.apply_buckets
tft.quantiles
tft.string_to_int
tf.string_join
...
Apply another TensorFlow Model
tft.apply_saved_model

Data
Ingestion
TensorFlow
Data Validation
TensorFlow
Transform
Model
Model Evaluation
and Validation
Serving Logging
Pipeline Storage
Tuner

...
estimator = DNNLinearCombinedClassifier(...)
estimator.train(...)
rcvr_fn_map = {
model_fn_lib.ModeKeys.TRAIN: train_rcvr_fn,
model_fn_lib.ModeKeys.EVAL: eval_rcvr_fn,
model_fn_lib.ModeKeys.PREDICT: serve_rcvr_fn,
}
export_dir = tf.contrib.estimator.export_all_saved_models(
estimator,
export_dir_base='/path/to/savedmodel',
input_receiver_fn_map=rcvr_fn_map)
...
Using TensorFlow Estimators.
DNN{Classifier|Regressor}
Linear{Classifier|Regressor}
DNNLinearCombined{Classifier|Regressor}
BoostedTrees{Classifier|Regressor}
...

...
estimator = DNNLinearCombinedClassifier(...)
estimator.train(...)
rcvr_fn_map = {
model_fn_lib.ModeKeys.TRAIN: train_rcvr_fn,
model_fn_lib.ModeKeys.EVAL: eval_rcvr_fn,
model_fn_lib.ModeKeys.PREDICT: serve_rcvr_fn,
}
export_dir = tf.contrib.estimator.export_all_saved_models(
estimator,
export_dir_base='/path/to/savedmodel',
input_receiver_fn_map=rcvr_fn_map)
...
Using TensorFlow Estimators.
TensorFlow
Serving
TensorFlow
Model Analysis
Train, Eval, and Inference Graphs
SignatureDef
Eval
Metadata
SignatureDef

Data
Ingestion
TensorFlow
Data Validation
TensorFlow
Transform
Estimator
or Keras
Model
Model Evaluation
and Validation
Serving Logging
Pipeline Storage
Tuner

Data
Ingestion
TensorFlow
Data Validation
TensorFlow
Transform
Estimator
or Keras
Model
TensorFlow
Model Analysis
Serving Logging
Pipeline Storage
Tuner

TensorFlow Model Analysis (TFMA)
Deep performance analysis
● Examine subsets of data
● Identify weaker/stronger optimization
● Identify key segments that are underserved
● Model fairness

Using TF Model Analysis to slice your metrics.
OVERALL_SLICE_SPEC = tfma.SingleSliceSpec()
FEATURE_COLUMN_SLICE_SPEC = tfma.SingleSliceSpec(columns=['trip_start_hour'])
ALL_SPECS = [
OVERALL_SLICE_SPEC,
FEATURE_COLUMN_SLICE_SPEC
]
eval_result = tfma.run_model_analysis(
model_location='/path/to/savedmodel',
data_location='/path/to/file/containing/tfrecords',
file_format='tfrecords',
slice_spec=ALL_SPECS)
tfma.view.render_slicing_metrics(
eval_result)
SignatureDef
Eval
Metadata
SignatureDef

Using TF Model Analysis to slice your metrics.
OVERALL_SLICE_SPEC = tfma.SingleSliceSpec()
FEATURE_COLUMN_SLICE_SPEC = tfma.SingleSliceSpec(columns=[‘trip_start_hour’])
ALL_SPECS = [
OVERALL_SLICE_SPEC,
FEATURE_COLUMN_SLICE_SPEC
]
eval_result = tfma.run_model_analysis(
model_location='/path/to/savedmodel',
data_location='/path/to/file/containing/tfrecords',
file_format='tfrecords',
slice_spec=ALL_SPECS)
tfma.view.render_slicing_metrics(
eval_result,
slicing_spec=FEATURE_COLUMN_SLICE_SPEC)
SignatureDef
Eval
Metadata
SignatureDef

Data
Ingestion
TensorFlow
Data Validation
TensorFlow
Transform
Estimator
or Keras
Model
TensorFlow
Model Analysis
TensorFlow
Serving
Logging
Pipeline Storage
Tuner

Using TensorFlow Serving to serve your models.
tensorflow_model_server
--port=9000
--model_name=chicago_taxi
--model_base_path='/path/to/savedmodel'
SignatureDef
Eval
Metadata
SignatureDef

Using TensorFlow Serving to serve your models.
tensorflow_model_server
--rest_api_port=8501
--model_name=chicago_taxi
--model_base_path='/path/to/savedmodel'
SignatureDef
Eval
Metadata
SignatureDef

// Classify Iris flowers from command line!
// More on Iris Model:
// www.tensorflow.org/get_started/get_started_for_beginners
$ curl -d '{"examples": [{"measurements": [7.1, 3.0, 5.9, 2.1]}]}'
-X POST http://localhost:8501/v1/models/iris:classify
{
"results": [[["0", 1.37459e-09], ["1", 0.000219246], ["2", 0.999781]]]
}

Anomalies
TensorFlow
Data Validation
Training
Data
Schema

Anomalies
TensorFlow
Transform
TensorFlow
Data Validation
Training
Data
Schema
SavedModel
Transform
Graph
Transformed
Data

Anomalies
TensorFlow
Transform
TensorFlow
Data Validation
TF Estimator or
Keras Model
Training
Data
Schema
SavedModel
Transform
Graph
Transformed
Data
SavedModel
Eval
Graph
SavedModel
Inference
Graph

Anomalies
TensorFlow
Transform
TensorFlow
Data Validation
TF Estimator or
Keras Model
TensorFlow
Model Analysis
TensorFlow
Serving
Training
Data
Schema
SavedModel
Transform
Graph
Transformed
Data
SavedModel
Eval
Graph
SavedModel
Inference
Graph

Anomalies
TensorFlow
Transform
TensorFlow
Data Validation
TF Estimator or
Keras Model
TensorFlow
Model Analysis
TensorFlow
Serving
Training
Data
Schema
SavedModel
Transform
Graph
Transformed
Data
SavedModel
Eval
Graph
SavedModel
Inference
Graph
Serving
Logs

Use all of this today!
● End-to-end demo works locally (single-node)
● Runner for Flink in prototype phase
● Runner for Spark planned
● Demo runs on Cloud Dataflow for distributed execution
JIRA Ticket: BEAM-2889
goo.gl/4DBvX7

TFX: A TensorFlow-Based Production-Scale Machine Learning Platform. KDD (2017).
https://youtu.be/fPTwLVCq00U

github.com/tensorflow/transform
github.com/tensorflow/model-analysis
github.com/tensorflow/serving
TensorFlow Transform
Consistent in-graph transformations in training and
serving
TensorFlow Model Analysis
Scaleable, sliced, and full-pass metrics
TensorFlow Serving
Serving the world’s machine learning models
github.com/tensorflow/data-validation
TensorFlow Data Validation
Understand, validate, and monitor your ML data

TensorFlow Extension (TFX) and Apache Beam

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to TensorFlow Extension (TFX) and Apache Beam

Similar to TensorFlow Extension (TFX) and Apache Beam (20)

More from markgrover

More from markgrover (20)

Recently uploaded

Recently uploaded (20)

TensorFlow Extension (TFX) and Apache Beam