Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTorch + XGBoost + Airflow + MLflow + Spark ML + Jupyter

2,643 views

Published on

Video: https://youtu.be/T0L0JxDaPkc

RSVP Here: https://www.eventbrite.com/e/full-day-workshop-kubeflow-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-airflow-tickets-63362929227

Description

In this workshop, we build real-world machine learning pipelines using TensorFlow Extended (TFX), KubeFlow, Airflow, and MLflow.

Described in the 2017 paper, TFX is used internally by thousands of Google data scientists and engineers across every major product line within Google.

KubeFlow is a modern, end-to-end pipeline orchestration framework that embraces the latest AI best practices including hyper-parameter tuning, distributed model training, and model tracking.

Airflow is the most-widely used pipeline orchestration framework in machine learning and data engineering.

MLflow is a lightweight experiment-tracking system recently open-sourced by Databricks, the creators of Apache Spark. MLflow supports Python, Java/Scala, and R - and offers native support for TensorFlow, Keras, and Scikit-Learn.


Pre-requisites

Modern browser - and that's it!

Every attendee will receive a cloud instance

Nothing will be installed on your local laptop

Everything can be downloaded at the end of the workshop



Location

Online Workshop

The link will be sent a few hours before the start of the workshop.

Only registered users will receive the link.

If you do not receive the link a few hours before the start of the workshop, please send your Eventbrite registration confirmation to support@pipeline.ai for help.

Agenda

1. Create a Kubernetes cluster

2. Install KubeFlow, Airflow, TFX, and Jupyter

3. Setup ML Training Pipelines with KubeFlow and Airflow

4. Transform Data with TFX Transform

5. Validate Training Data with TFX Data Validation

6. Train Models with Jupyter, Keras/TensorFlow 2.0, PyTorch, XGBoost, and KubeFlow

7. Run a Notebook Directly on Kubernetes Cluster with KubeFlow

8. Analyze Models using TFX Model Analysis and Jupyter

9. Perform Hyper-Parameter Tuning with KubeFlow

10. Select the Best Model using KubeFlow Experiment Tracking

11. Run Multiple Experiments with MLflow Experiment Tracking

12. Reproduce Model Training with TFX Metadata Store

13. Deploy the Model to Production with TensorFlow Serving and Istio

14. Save and Download your Workspace



Key Takeaways

Attendees will gain experience training, analyzing, and serving real-world Keras/TensorFlow 2.0 models in production using model frameworks and open-source tools.

RSVP Here: https://www.eventbrite.com/e/full-day-workshop-kubeflow-kerastensorflow-20-tf-extended-tfx-kubernetes-pytorch-xgboost-airflow-tickets-63362929227

https://youtu.be/T0L0JxDaPkc

Published in: Software

KubeFlow + GPU + Keras/TensorFlow 2.0 + TF Extended (TFX) + Kubernetes + PyTorch + XGBoost + Airflow + MLflow + Spark ML + Jupyter

  1. 1. End-to-End ML Pipelines TFX + Kubeflow + Airflow + MLflow Chris Fregly Founder @ .
  2. 2. Founder @ PipelineAI Continuous Machine Learning in Production Former Databricks, Netflix Apache Spark Contributor O’Reilly Author High Performance TensorFlow in Production Meetup Organizer Advanced Kubeflow Meetup Who Am I? (@cfregly)
  3. 3. Advanced Kubeflow Meetup (Global+Bay Area, Monthly Events) https://meetup.com/Advanced-Kubeflow-Meetup
  4. 4. Full-Day Workshop https://pipeline.ai @cfregly @PipelineAI
  5. 5. 1 OK with Command Line? 2 OK with Python? 3 OK with Linear Algebra? Who are you? 4 OK with Docker? 6 5 OK with Jupyter Notebook?
  6. 6. Recent Poll (July 2019)
  7. 7. 4,000 Stars = $6,000,000 Seed $1,500 per GitHub Star?! (Please star the repo ASAP!!) Recent Comment from Popular VC Investor in Silicon Valley
  8. 8. Community Edition https://community.pipeline.ai
  9. 9. After Pushing Your Model to Production, Your Model is… 1 Already Out of Date – Need to Re-Train Biased – Need to Validate Before Pushing Broken – Need to A/B Test in Production Hacked – Need to Train With Data Privacy Slow – Need to Quantize and Speed Up Predictions 2 3 4 5
  10. 10. 1 Setup Environment with Kubernetes TensorFlow Extended (TFX) ML Pipelines with Airflow and KubeFlow Agenda – Part 1 of 2 Hyper-Parameter Tuning with KubeFlow Deploy Notebook with Kubernetes 2 3 4 5
  11. 11. Agenda – Part 2 of 2 6 MLflow 7 8 9 10 TensorFlow Privacy Model Serving & A/B Tests Model Optimization Papermill
  12. 12. Note #1 of 11 IGNORE WARNINGS & ERRORS (Everything will be OK!)
  13. 13. Note #2 of 11 THERE IS A LOT OF MATERIAL HERE Many opportunities to explore on your own. (Don’t upload sensitive data.)
  14. 14. Note #3 of 11 YOU HAVE YOUR OWN INSTANCE 16 CPU, 104 GB RAM, 200GB SSD (Each with a full Kubernetes Cluster.)
  15. 15. Note #4 of 11 DATASETS Chicago Taxi Dataset (and various others.)
  16. 16. Note #5 of 11 SOME NOTEBOOKS TAKE MINUTES Please be patient. (We are using large datasets)
  17. 17. Note #6 of 11 QUESTIONS? Post questions to Zoom chat or Q&A. (Antje and I will answer soon) Antje >
  18. 18. Note #7 of 11 KUBEFLOW IS NOT A SILVER BULLET There are still gaps in the pipeline. (But gaps are getting smaller)
  19. 19. Note #8 of 11 THIS IS NOT CLOUD DEPENDENT* *Except for 2 small exceptions… (Patches are underway.)
  20. 20. Note #9 of 11 PRIMARILY TENSORFLOW 1.x TF 2.x is not fully supported by TFX (Until Mid-2020.)
  21. 21. Note #10 of 11 SHUTDOWN EACH NOTEBOOK AFTER We are using complex browser voo-doo. (Javascript is a mystery.)
  22. 22. Note #11 of 11 Retrieve 1 Single IP Address Here… <INSERT HERE> (Do not click refresh.)
  23. 23. 1 Setup Environment with Kubernetes TensorFlow Extended (TFX) ML Pipelines with Airflow and KubeFlow Agenda – Part 1 of 2 Hyper-Parameter Tuning with KubeFlow Deploy Notebook with Kubernetes 2 3 4 5
  24. 24. Agenda – Part 2 of 2 6 MLflow 7 8 9 10 TensorFlow Privacy Model Serving & A/B Tests Model Optimization Papermill
  25. 25. Hands On 00_Explore_Environment
  26. 26. 1.1 Kubernetes TensorFlow Extended (TFX) Airflow ML Pipelines 1.0 Environment Overview KubeFlow ML Pipelines 6 Hyper-Parameter Tuning (Katib) Prediction Traffic Router (Istio) 1.2 1.3 1.4 1.6 1.7 MLflow Pipelines1.5
  27. 27. 1.1 Kubernetes Kubernetes NFS Ceph Cassandra MySQL Spark Airflow Tensorflow Caffe TF-Serving Flask+Scikit Operating system (Linux, Windows) CPU Memory DiskSSD GPU FPGA ASIC NIC Jupyter GCP AWS Azure On-prem Namespace Quota Logging Monitoring RBAC
  28. 28. Hands On 01_Explore_Kubernetes_Cluster
  29. 29. System 6 System 5System 4 Training At Scale System 3 System 1 Data Ingestion Data Analysis Data Transform Data Validation System 2 Build Model Model Validation Serving Logging Monitoring Roll-out Data Splitting Ad-Hoc Training Why TFX and Why KubeFlow? Improve Training/Serving Consistency Unify Disparate Systems Manage Pipeline Complexity Improve Portability Wrangle Large Datasets Improve Model Quality Manage Versions Composability Distributed Training Configure
  30. 30. 1.2 TensorFlow Extended (TFX) Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Jenkins?
  31. 31. 1.3 Airflow ML Pipelines
  32. 32. 1.4 KubeFlow ML Pipelines
  33. 33. 1.5 MLflow Experiment Tracking
  34. 34. 1.6 Hyper-Parameter Tuning (Katib)
  35. 35. 1.7 Prediction Traffic Routing (Istio)
  36. 36. 1 Setup Environment with Kubernetes TensorFlow Extended (TFX) ML Pipelines with Airflow and KubeFlow Agenda – Part 1 of 2 Hyper-Parameter Tuning with KubeFlow Deploy Notebook with Kubernetes 2 3 4 5
  37. 37. 2.1 TFX Internals 2.0 TFX Components 6 Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training 2.2 TFX Libraries 2.2 TFX Components
  38. 38. 2.1 TFX Internals Driver/Publisher Moves data to/from Metadata Store Executor Runs the Actual Processing Code Metadata Store Artifact, execution, and lineage Info Track inputs & outputs of all components Stores training run including inputs & outputs Analysis, validation, and versioning results Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training
  39. 39. 2.2 TFX Libraries 2.2.1 TFX Components Use These: TensorFlow Data Validation (TFDV) TensorFlow Transform (TFT) TensorFlow Model Analysis (TFMA) TensorFlow Metadata (TFMD) + ML Metadata (MLMD) 2.2.2 2.2.3 2.2.4 Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training
  40. 40. 2.2.1 TFX Libraries - TFDV TensorFlow Data Validation (TFDV) Find Missing, Redundant & Important Features Identify Features with Unusually-Large Scale `infer_schema()` Generates Schema Describe Feature Ranges Detect Data Drift Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Uniformly Distributed Data è ç Non-Uniformly Distributed Data
  41. 41. Hands On 02_TensorFlow_Data_Validation (TFDV)
  42. 42. 2.2.2 TFX Libraries - TFT TensorFlow Transform (TFT) Preprocess `tf.Example` data with TensorFlow Useful for data that requires a full pass Normalize all inputs by mean and std dev Create vocabulary of strings è integers over all data Bucketize features based on entire data distribution Outputs a TensorFlow graph Re-used across both training and serving Uses Apache Beam (local mode) for Parallel Analysis Can also use distributed mode `preprocessing_fn(inputs)`: Primary Fn to Implement Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training import tensorflow as tf import tensorflow_transform as tft def preprocessing_fn(inputs): x = inputs['x'] y = inputs['y'] s = inputs['s'] x_centered = x - tft.mean(x) y_normalized = tft.scale_to_0_1(y) s_integerized = tft.compute_and_apply_vocabulary(s) x_centered_times_y_normalized = x_centered * y_normalized return { 'x_centered': x_centered, 'y_normalized': y_normalized, 'x_centered_times_y_normalized':x_centered_times_y_normalized, 's_integerized': s_integerized }
  43. 43. Hands On 03_TensorFlow_Transform (TFT)
  44. 44. Hands On 03a_TensorFlow_Transform_Advanced (TFT)
  45. 45. 2.2.3 TFX Libraries - TFMA TensorFlow Model Analysis (TFMA) Analyze Model on Different Slices of Dataset Track Metrics Over Time (“Next Day Eval”) `EvalSavedModel` Contains Slicing Info TFMA Pipeline: Read, Extract, Evaluate, Write ie. Ensure Model Works Fairly Across All Users Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training
  46. 46. Hands On 04_TensorFlow_Model_Analysis (TFMA)
  47. 47. 2.2.4 TFX Libraries – Metadata TensorFlow Metadata (TFMD) ML Metadata (MLMD) Record and Retrieve Experiment Metadata Artifact, Execution, and Lineage Info Track Inputs / Outputs of All TFX Components Stores Training Run Info Analysis and Validation Results Model Versioning Info Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training
  48. 48. 2.3 TFX Components ExampleGen StatisticsGen SchemaGen ExampleValidator Evaluator Transform ModelValidator Trainer Model Pusher2.3.92.3.1 2.3.2 2.3.3 2.3.4 2.3.5 2.3.6 2.3.7 2.3.8 Slack (!!)2.3.10
  49. 49. 2.3.1 ExampleGen Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Load Training Data Into TFX Pipeline Supports External Data Sources Supports CSV and TFRecord Formats Converts Data to tf.Example Note: TFX Pipelines require tf.Example (?!) Difficult to use non-TF models like XGBoost from tfx.utils.dsl_utils import csv_input from tfx.components.example_gen.csv_example_gen.component import CsvExampleGen examples = csv_input(os.path.join(base_dir, 'data/simple')) example_gen = CsvExampleGen(input_base=examples)
  50. 50. 2.3.2 StatisticsGen Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Generates Statistics on Training Data Global `mean` and `stddev` per input feature Consumes tf.Example instances from tfx import components compute_eval_stats = components.StatisticsGen( input_data=examples_gen.outputs.eval_examples, name='compute-eval-stats' )
  51. 51. 2.3.3 SchemaGen Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Schema Needed by Some TFX Components Data Types, Value Ranges, Optional, Required Consumes Data from StatisticsGen Schema used by TFDV, TFT, TFMA Libraries Uses TFDV Library to infer schema Best effort and basic Human should verify feature { name: "age" value_count { min: 1 max: 1 } type: FLOAT presence { min_fraction: 1 min_count: 1 } } from tfx import components infer_schema = components.SchemaGen( stats=compute_training_stats.outputs.output)
  52. 52. 2.3.4 ExampleValidator Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Identifies Anomalies in Training Data Used with serving data to detect drift / skew Uses StatisticsGen and SchemaGen Outputs Produces Validation Results Uses TFDV Library for Input Validation from tfx import components infer_schema = components.SchemaGen( stats=compute_training_stats.outputs.output )
  53. 53. 2.3.5 Transform Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Uses Data from ExampleGen & SchemaGen Transformations Become Part of TF Graph (!!) Helps Avoid Training/Serving Skew Uses TFT Library for Transformations Transformations Require Full Pass Thru Dataset Global Reduction Across All Batches Create Word Embeddings, Normalize, PCA def preprocessing_fn(inputs): # inputs: map from feature keys # to raw not-yet-transformed features # outputs: map from string feature key # to transformed feature operations
  54. 54. 2.3.6 Trainer Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Trains / Validates tf.Examples from Transform Uses schema.proto from SchemaGen Produces SavedModel and EvalSavedModel Uses Core TensorFlow Python API Works with TensorFlow 1.x Estimator API TensorFlow 2.0 Keras Support Coming Soon from tfx import components trainer = components.Trainer( module_file=taxi_pipeline_utils, train_files=transform_training.outputs.output, eval_files=transform_eval.outputs.output, schema=infer_schema.outputs.output, tf_transform_dir=transform_training.outputs.output, train_steps=10000, eval_steps=5000)
  55. 55. 2.3.7 Evaluator Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Uses EvalSavedModel from Trainer Writes Analysis Results to ML Metadata Store Uses TFMA Library for Analysis TFMA Uses Apache Beam to Scale Analysis from tfx import components import tensorflow_model_analysis as tfma taxi_eval_spec = [ tfma.SingleSliceSpec(), tfma.SingleSliceSpec(columns=['trip_start_hour']) ] model_analyzer = components.Evaluator( examples=examples_gen.outputs.eval_examples, eval_spec=taxi_eval_spec, model_exports=trainer.outputs.output)
  56. 56. 2.3.8 ModelValidator Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Validate Models from Trainer Uses Data from SchemaGen & StatisticsGen Compares New Models to Baseline Baseline == current model in production New Model is Good if Meets/Exceeds Metrics If Good, Notify Pusher to Deploy New Model Simulate “Next Day Evaluation” On New Data import tensorflow_model_analysis as tfma taxi_mv_spec = [tfma.SingleSliceSpec()] model_validator = components.ModelValidator( examples=examples_gen.outputs.output, model=trainer.outputs.output)
  57. 57. 2.3.9 Model Pusher (Deployer) Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Push Good Model to Deployment Target Uses Trained SavedModel Writes Version Data to Metadata Store Write to FileSystem or TensorFlow Hub from tfx import components pusher = components.Pusher( model_export=trainer.outputs.output, model_blessing=model_validator.outputs.blessing, serving_model_dir=serving_model_dir)
  58. 58. 2.3.10 Slack Component (!!) Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Runs After ModelValidator Adds Human-in-the-Loop Step to Pipeline TFX Sends Message to Slack with Model URI Asks Human to Review the New Model Respond ‘LGTM’, ‘approve’, ‘decline’, ‘reject’ Requires Slack API Setup / Integration export SLACK_BOT_TOKEN={your_token} _channel_id = 'my-channel-id' _slack_token = os.environ['SLACK_BOT_TOKEN’] slack_validator = SlackComponent( model_export=trainer.outputs.output, model_blessing=model_validator.outputs.blessing, slack_token=_slack_token, channel_id=_channel_id, timeout_sec=3600, ) https://github.com/tensorflow/tfx/tree/master /tfx/examples/custom_components/slack/slack_component
  59. 59. 1 Setup Environment with Kubernetes TensorFlow Extended (TFX) ML Pipelines with Airflow and KubeFlow Agenda – Part 1 of 2 Hyper-Parameter Tuning with KubeFlow Deploy Notebook with Kubernetes 2 3 4 5
  60. 60. 3.0 ML Pipelines with Airflow and KubeFlow Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy 3.1 Airflow KubeFlow3.2
  61. 61. 3.1 Airflow 6 Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Most Widely-Used Workflow Orchestrator Define Execution Graphs in Python Decent UI Good Community Support
  62. 62. Hands On 05_Airflow_ML_Pipelines (Chicago Taxi Dataset)
  63. 63. Hands On 06_Airflow_Feature_Analysis
  64. 64. Hands On 07_Airflow_Model_Analysis
  65. 65. 3.2 KubeFlow 6 Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training Pipelines Based on Argo CI/CD Project from Intuit TFJob & PyTorch Job Supports Distributed Training TensorFlow & PyTorch Jobs KubeFlow Fairing Project (!!) Run a notebook as a production job Deploy training code with dependencies
  66. 66. Hands On 08_Simple_KubeFlow_ML_Pipeline
  67. 67. Hands On 09_Advanced_KubeFlow_ML_Pipeline (Chicago Taxi Dataset)
  68. 68. Hands On 10_Distributed_TensorFlow_Job
  69. 69. Hands On 10a_Distributed_PyTorch_Job
  70. 70. 1 Setup Environment with Kubernetes TensorFlow Extended (TFX) ML Pipelines with Airflow and KubeFlow Agenda – Part 1 of 2 Hyper-Parameter Tuning with KubeFlow Deploy Notebook with Kubernetes 2 3 4 5
  71. 71. 4.0 Hyper-Parameter Tuning 6 Experiment Single Optimization Run Single Objective Function Across Runs Contains Many Trials Trial List of Param Values Suggestion Optimization Algorithm Job Evaluates a Trial Calculates Objective
  72. 72. Hands On 11_Hyper_Parameter_Tuning
  73. 73. 1 Setup Environment with Kubernetes TensorFlow Extended (TFX) ML Pipelines with Airflow and KubeFlow Agenda – Part 1 of 2 Hyper-Parameter Tuning with KubeFlow Deploy Notebook with Kubernetes 2 3 4 5
  74. 74. 5.0 Deploy Notebook as Job Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training 6 5.1 Wrap Model in a Docker Image Deploy Job to Kubernetes5.2
  75. 75. 5.1 Create Docker Image Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training
  76. 76. 5.2 Deploy Notebook as Job Feature Load Feature Analyze Feature Transform Model Train Model Evaluate Model Deploy Reproduce Training
  77. 77. Hands On 12_Deploy_Notebook_Xgboost
  78. 78. Hands On 12a_Deploy_Notebook_TensorFlow
  79. 79. 1 Setup Environment with Kubernetes TensorFlow Extended (TFX) ML Pipelines with TFX, Airflow, and KubeFlow Agenda – Part 1 of 2 Hyper-Parameter Tuning with TFX and KubeFlow Deploy Notebook with Kubernetes 2 3 4 5
  80. 80. Agenda – Part 2 of 2 6 MLflow 7 8 9 10 TensorFlow Privacy Model Serving & A/B Tests Model Optimization Papermill
  81. 81. 7.0 MLflow 7.1 Experiment Tracking Hyper-Parameter Tuning Kubernetes-based Jobs 7.2 7.3
  82. 82. Hands On 14_MLflow_Scikit_Learn
  83. 83. Hands On 14a_MLflow_Keras
  84. 84. Hands On 14b_MLflow_TensorFlow
  85. 85. Agenda – Part 2 of 2 6 MLflow 7 8 9 10 TensorFlow Privacy Model Serving & A/B Tests Model Optimization Papermill
  86. 86. 7.0 Papermill
  87. 87. Hands On 15_Papermill_Notebook_Job
  88. 88. Agenda – Part 2 of 2 6 MLflow 7 8 9 10 TensorFlow Privacy Model Serving & A/B Tests Model Optimization Papermill
  89. 89. 9.0 TensorFlow Privacy (Differential Privacy)
  90. 90. 9.1 Differential Privacy + Stochastic Gradient Descent
  91. 91. 9.1 Differential Privacy + Explainability + Drift Detection
  92. 92. Hands On 16_TF_Privacy
  93. 93. Agenda – Part 2 of 2 6 MLflow 7 8 9 10 TensorFlow Privacy Model Serving & A/B Tests Model Optimization Papermill
  94. 94. 10.0 Model Serving & A/B Tests
  95. 95. Hands On 17_Simple_Serving_REST
  96. 96. Hands On 17a_AB_Test_REST
  97. 97. Hands On 18_Metrics_and_Monitoring
  98. 98. 10.0 Model Optimization
  99. 99. Agenda – Part 2 of 2 6 MLflow 7 8 9 10 TensorFlow Privacy Model Serving & A/B Tests Model Optimization Papermill
  100. 100. Hands On 19_Optimize_Model
  101. 101. Agenda – Part 2 of 2 6 MLflow 7 8 9 10 TensorFlow Privacy Model Serving & A/B Tests Model Optimization Papermill
  102. 102. 1 Setup Environment with Kubernetes TensorFlow Extended (TFX) ML Pipelines with Airflow and KubeFlow Agenda – Part 1 of 2 Hyper-Parameter Tuning with KubeFlow Deploy Notebook with Kubernetes 2 3 4 5
  103. 103. After Pushing Your Model to Production, Your Model is… 1 Already Out of Date – Need to Re-Train Biased – Need to Validate Before Pushing Broken – Need to A/B Test in Production Hacked – Need to Train With Data Privacy Slow – Need to Quantize and Speed Up Predictions 2 3 4 5
  104. 104. Community Edition https://community.pipeline.ai
  105. 105. Thank you! https://pipeline.ai @cfregly @PipelineAI

×