This document discusses machine learning platform lifecycle management. It describes the typical lifecycle of an ML model, including data ingestion, feature engineering, model training, scoring, and development. It emphasizes the importance of managing artifacts like data, models, features, environments and metadata throughout the lifecycle. An effective ML platform should support the entire lifecycle, increase efficiency, and scale to large datasets. It provides examples of how Intuit manages artifacts and deploys models using containers and services.

1. AI NEXTCon Silicon Valley 18
SANTA CLARA | APRIL 10-13
#ainextcon
Hope (Xinwei) Wang
Software Engineer at Intuit

3. • What is a machine learning platform?
• What is the ML platform lifecycle?
• Why ML platform lifecycle management?
• Artifacts and their associations
• Use cases at Intuit
Machine Learning Platform Life-Cycle Management

4. • Manages the entire lifecycle of an ML model
• Includes automating and accelerating the delivery of ML
applications

7. Data
Discovery
Feature
Engineering
Model
Training
Model
Scoring
Model
Development
Data
Ingestion
Life-Cycle Management

8. • Metadata/catalog tool
• Accessible data source
(raw attributes & data lineage)
Metadata/catalog
tool
Data lake

9. • Output: features
• Reproducible
• Reusable
Metadata/catalog
tool
Data lake

10. • Collaborative environment
• Access data lake
Notebooks
Data lake

11. Support ability for:
• Being triggered either manually/via automation
• Creation and management of training sets
• Retraining
• Optimizing hyper parameter tuning through parallelization
of model training execution

12. • Support online/offline (depend on use cases)
• Ability to be triggered either manually/via automation

13. • No central artifact management solution
• Hard to reuse existing features/data/algorithms/toolings
• Inability to scale for large datasets
• Lack of automation/orchestration across the ML lifecycle
• Lack of rigor/discipline in the ML development lifecycle
• Slow down delivery of machine learning applications

14. • Optimizing data scientists’ engineering process
• Tie ML components together into a cohesive platform,
support the lifecycle of ML artifacts end-to-end
• Increase efficiency of delivering ML predictions at scale
• Artifact management

15. • Data artifacts
– Features
– Training sets
• Model artifacts
– Model code
– Trained models
– Performance metrics
– Hyper parameter values
• Environment artifacts
– Languages & language versions
– Packages & package versions

16. Data
Discovery
Feature
Engineering
Model
Training
Model
Scoring
Model
Development
Data
Ingestion
Life-Cycle Management

17. Environment in container

18. • Flexibility: Model has specific environment
• Consistency: Model has same behavior throughout
the lifecycle

19. Environment
metadata
Container
Trained
models
Model
Features
Scheduled
retraining/
performance
benchmarks
Hyper-
parameter
values
Metrics
definition
Performance
metrics
Feature set
definition
Training
datasets
Model artifacts
Data artifacts
Environment artifacts
Model scoring

20. • Developed in notebooks
• Multiple versions
• Each version associated with an externalized
environment artifact

21. • Environment must be consistent for development,
training, scoring
• Externalized as metadata
• Model/execution environments constructed from metadata
and deployed into containers (Docker, Yarn, Conda, etc.)

22. Container/virtual environment Usage
Docker container
• Model development
• Model training
• Online scoring
Yarn container
On Spark cluster
• Distributed training
• Batch offline scoring
Conda environment Model development
• Tailored to the environment (built based on externalized environment metadata)
• Used for model development, training, execution

23. • Used as data input of the model
• Stored in discoverable feature repository
• Metadata defines the model specific feature sets

24. • Serialized, weighted model files
• Associated with a version of model code and training set

25. • Datasets used to train, validate and test the model
• Associated to a trained model

26. • Define what feature sets this model requires

27. • Set up values before learning process
• Model specific

28. • Defines the metrics to collect and thresholds to evaluate
models against.

29. • Metrics to evaluate model effectiveness
• Model metrics including: ROC curve, confusion matrix,
F1 score, precision, recall, etc.

30. § To automate the retraining and deployment of
updated models
§ Model-specific

32. Data
Discovery
Feature
Engineering
Model
Training
Model
Scoring
Model
Development
Data
Ingestion
Life-Cycle Management

34. Personalization
service
Feature service
Online scoring
serviceSpark streamingKafkaBatch source
Offline Spark
job
Online store
Feature
repository
Lake
Clickstream data
Online
features
Offline features
Score data
Get contextual help
Self-help service in QuickBooks

35. Serialized
Model
Environment
Metadata
Deploy
Code
Docker
File
Docker Base
Image
Model Specific
Docker Image
Example: Online scoring service deployment diagram
Model
Deploy
Template
Model Specific
Deploy
Code/metadata
Trained
Model
Serialized
Model

36. Data
Discovery
Feature
Engineering
Model
Training
Model
Scoring
Model
Development
Data
Ingestion
Life-Cycle Management

37. Serialized
Model
Environment
Metadata
Deploy
Code
Docker
File
Docker Base
Image
Model Specific
Docker Image
Model
Deploy
Template
Model Specific
Deploy
Code/metadata
Trained
Model
Serialized
Model
Example: Online scoring service deployment diagram

38. Serialized
Model
Environment
Metadata
Deploy
Code
Docker
File
Docker Base
Image
Model Specific
Docker Image
Model
Deploy
Template
Model Specific
Deploy
Code/metadata
Trained
Model
Serialized
Model
Example: Online scoring service deployment diagram

39. Serialized
Model
Environment
Metadata
Deploy
Code
Docker
File
Docker Base
Image
Model Specific
Docker Image
Model
Deploy
Template
Model Specific
Deploy
Code/metadata
Trained
Model
Serialized
Model
Example: Online scoring service deployment diagram

40. Serialized
Model
Environment
Metadata
Deploy
Code
Docker
File
Docker Base
Image
Model Specific
Docker Image
Model
Deploy
Template
Model Specific
Deploy
Code/metadata
Trained
Model
Serialized
Model
Example: Online scoring service deployment diagram

41. Serialized
Model
Environment
Metadata
Deploy
Code
Docker
File
Docker Base
Image
Model Specific
Docker Image
Model
Deploy
Template
Model Specific
Deploy
Code/metadata
Trained
Model
Serialized
Model
Example: Online scoring service deployment diagram

42. Environment in container

43. Thank you