This document discusses various options for deploying machine learning models including AWS Lambda, Kubernetes, KServe (previously known as Kubeflow Serving), and AWS SageMaker. It provides an overview of deploying models with Kubernetes using Deployments and Services. Specific examples are given for deploying Keras models with TensorFlow Serving and a gateway using gRPC as well as deploying with KServe including the use of transformers. The document concludes that there is no one size fits all solution and different options may be better suited depending on factors like load size, ease of use, transparency, and cost.

1. Model Deployment
Alexey Grigorev
Principal Data Scientist — OLX Group
Founder — DataTalks.Club

2. 2010
2012
2015
2018

3. mlbookcamp.com

4. mlzoomcamp.com

5. Plan
● Different options to deploy a model (Lambda, Kubernetes, SageMaker)
● Kubernetes 101
● Deploying an XGB model with Flask and Kubernetes
● Deploying a Keras model with TF-Serving and Kubernetes
● Deploying a Keras model with KServe (previously known as Kubeflow
Serving)

6. Ways to deploy a model
● Flask + AWS Elastic Beanstalk
● Serverless (AWS Lambda)
● Kubernetes (EKS)
● KServe (EKS)
● AWS SageMaker
● ...
(or their alternatives in other cloud providers)

7. {
"tshirt": 0.9993,
"pants": 0.0005,
"shoes": 0.00004
}
AWS Lambda

8. Kubernetes
Ingress
Client
Node1
Node2
Pod A
Pod B Pod C
Pod D
Pod E
Pod F
Service
1
Service
2
Deployment 1
Deployment 2
Kubernetes Cluster

9. Kubeflow / KServe
Ingress
Client
Node1
Node2
Pod A
Pod B Pod C
Pod D
Pod E
Pod F
Service
1
Service
2
Deployment 1
Deployment 2
Kubernetes Cluster
InferenceService

10. SageMaker
Client
Model
Endpoint
AWS SageMaker

11. SageMaker
AWS

12. Lambda vs SageMaker vs Kubernetes
● Lambda
○ Cheap for small load
○ Easy to manage
○ Not always transparent

13. Lambda vs SageMaker vs Kubernetes
● Lambda
○ Cheap for small load
○ Easy to manage
○ Not always transparent
● SageMaker (serving)
○ Easy to use/manage
○ Needs wrappers
○ Not always transparent
○ Expensive

14. Lambda vs SageMaker vs Kubernetes
● Lambda
○ Cheap for small load
○ Easy to manage
○ Not always transparent
● SageMaker (serving)
○ Easy to use/manage
○ Needs wrappers
○ Not always transparent
○ Expensive
● Kubernetes
○ Complex (for me)
○ More flexible
○ Cloud-agnostic *
○ Requires support
○ Cheaper for high load
* sort of

15. Kubernetes 101

16. Kubernetes glossary
● Pod ~ one instance of your service
● Deployment - a bunch of pods
● HPA - horizontal pod autoscaler
● Node - a server (e.g. EC2 instance)
● Service - an interface to the deployment
● Ingress - an interface to the cluster

17. Kubernetes in one picture
Node 1
Node 2
Service
Internal
Service

18. Kubernetes in one picture
Node 1
Node 2
Flask app with
model
Service
Service

19. Deploying a Flask App

20. import xgboost as xgb
# load the model from the pickle file
@app.route('/predict', methods=['POST'])
def predict():
data = request.get_json()
result = apply_model(data)
return jsonify(result)
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=9696)

21. FROM python:3.9-slim
RUN pip install flask gunicorn xgboost
COPY "model.py" "model.py"
EXPOSE 9696
ENTRYPOINT ["gunicorn", "--bind", "0.0.0.0:9696", "model:app"]

22. apiVersion: apps/v1
kind: Deployment
metadata:
name: xgb-model
labels:
app: xgb-model
spec:
replicas: 1
selector:
matchLabels:
app: xgb-model
template:
metadata:
labels:
app: xgb-model
spec:
containers:
- name: xgb-model
image: XXXXXXXXXXXX.dkr.ecr.eu-west-1.amazonaws.com/xgb-model:v100500
ports:
- containerPort: 9696
env:
- name: MODEL_PATH
value: "s3://models-bucket-pickle/xgboost.bin"

23. apiVersion: v1
kind: Service
metadata:
name: xgb-model
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 9696
protocol: TCP
name: http
selector:
app: xgb-model

24. kubectl apply -f deployment.yaml
kubectl apply -f service.yaml

25. Deploying a Keras Model

26. 🎁

27. 🎁
H5
saved_model

28. import tensorflow as tf
from tensorflow import keras
model = keras.models.load_model('keras-model.h5')
tf.saved_model.save(model, 'tf-model')

29. $ ls -lhR
.:
total 3,1M
4,0K assets
3,1M saved_model.pb
4,0K variables
./assets:
total 0
./variables:
total 83M
83M variables.data-00000-of-00001
15K variables.index

30. saved_model_cli show --dir tf-model --all
MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:
...
signature_def['serving_default']:
The given SavedModel SignatureDef contains the following input(s):
inputs['input_8'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 299, 299, 3)
name: serving_default_input_8:0
The given SavedModel SignatureDef contains the following output(s):
outputs['dense_7'] tensor_info:
dtype: DT_FLOAT
shape: (-1, 10)
name: StatefulPartitionedCall:0
Method name is: tensorflow/serving/predict

31. docker run -it --rm
-p 8500:8500
-v "$(pwd)/tf-model:/models/tf-model/1"
-e MODEL_NAME=tf-model
tensorflow/serving:2.3.0
2021-09-07 21:03:58.579046: I tensorflow_serving/model_servers/server.cc:367]
Running gRPC ModelServer at 0.0.0.0:8500 ...
[evhttp_server.cc : 238] NET_LOG: Entering the event loop ...
2021-09-07 21:03:58.582097: I tensorflow_serving/model_servers/server.cc:387]
Exporting HTTP/REST API at:localhost:8501 ...

32. pip install grpcio==1.32.0
tensorflow-serving-api==2.3.0

33. https://github.com/alexeygrigorev/mlbookcamp-code/blob/master/chapter-09-kubernetes/09-image-preparation.ipynb
def np_to_protobuf(data):
return tf.make_tensor_proto(data, shape=data.shape)
pb_request = predict_pb2.PredictRequest()
pb_request.model_spec.name = 'tf-model'
pb_request.model_spec.signature_name = 'serving_default'
pb_request.inputs['input_8'].CopyFrom(np_to_protobuf(X))
pb_result = stub.Predict(pb_request, timeout=20.0)
pred = pb_result.outputs['dense_7'].float_val

34. Gateway
(Resize and
process image)
Flask
Model
(Make predictions)
TF-Serving
Pants
Raw
predictions
Pre-processed
image

35. Not so fast
def np_to_protobuf(data):
return tf.make_tensor_proto(data, shape=data.shape)
pb_request = predict_pb2.PredictRequest()
pb_request.model_spec.name = 'tf-model'
pb_request.model_spec.signature_name = 'serving_default'
pb_request.inputs['input_8'].CopyFrom(np_to_protobuf(X))
pb_result = stub.Predict(pb_request, timeout=20.0)
pred = pb_result.outputs['dense_7'].float_val

36. 2,0 GB dependency?
Get only the things you need!
https://github.com/alexeygrigorev/tensorflow-protobuf

37. from tensorflow.keras.applications.xception import preprocess_input
https://github.com/alexeygrigorev/keras-image-helper
from keras_image_helper import create_preprocessor
preprocessor = create_preprocessor('xception', target_size=(299, 299))
url = 'http://bit.ly/mlbookcamp-pants'
X = preprocessor.from_url(url)

38. Next steps...
● Bake in the model into the TF-serving image
● Wrap the gRPC calls in a Flask app for the Gateway
● Write a Dockerfile for the Gateway
● Publish the images to ERC

39. Okay!
We’re ready to deploy to K8S

40. apiVersion: apps/v1
kind: Deployment
metadata:
name: tf-serving-model
labels:
app: tf-serving-model
spec:
replicas: 1
selector:
matchLabels:
app: tf-serving-model
template:
metadata:
labels:
app: tf-serving-model
spec:
containers:
- name: tf-serving-model
image: X.dkr.ecr.eu-west-1.amazonaws.com/model-serving:tf-serving-model
ports:
- containerPort: 8500

41. apiVersion: v1
kind: Service
metadata:
name: tf-serving-model
labels:
app: tf-serving-model
spec:
ports:
- port: 8500
targetPort: 8500
protocol: TCP
name: http
selector:
app: tf-serving-model

42. kubectl apply -f tf-serving-deployment.yaml
kubectl apply -f tf-serving-service.yaml

43. apiVersion: apps/v1
kind: Deployment
metadata:
name: serving-gateway
labels:
app: serving-gateway
spec:
replicas: 1
selector:
matchLabels:
app: serving-gateway
template:
metadata:
labels:
app: serving-gateway
spec:
containers:
- name: serving-gateway
image: X.dkr.ecr.eu-west-1.amazonaws.com/model-serving:serving-gateway
ports:
- containerPort: 9696
env:
- name: TF_SERVING_HOST
value: "tf-serving-model.default.svc.cluster.local:8500"

44. apiVersion: v1
kind: Service
metadata:
name: serving-gateway
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 9696
protocol: TCP
name: http
selector:
app: serving-gateway

45. kubectl apply -f gateway-deployment.yaml
kubectl apply -f gateway-service.yaml

46. gRPC load balancing
https://kubernetes.io/blog/2018/11/07/grpc-load-balancing-on-kubernetes-without-tears/

47. Kubeflow / KServe

48. Kubeflow / KServe
Ingress
Client
Node1
Node2
Pod A
Pod B Pod C
Pod D
Pod E
Pod F
Service
1
Service
2
Deployment 1
Deployment 2
Kubernetes Cluster
InferenceService

49. Installing
https://mlbookcamp.com/article/kfserving-eks-install
git clone git@github.com:alexeygrigorev/kubeflow-deep-learning.git
cd kubeflow-deep-learning/install
./install.sh

50. Next...
● Upload the saved_model to S3
● Allow KServe to access S3
https://mlbookcamp.com/article/kfserving-eks-install

51. apiVersion: "serving.kubeflow.org/v1beta1"
kind: "InferenceService"
metadata:
name: "tf-model"
spec:
default:
predictor:
serviceAccountName: sa
tensorflow:
storageUri: "s3://models-bucket-tf/tf-model"

52. kubectl apply -f tf-inference-service.yaml

53. $ kubectl get inferenceservice
NAME URL
flowers-sample http://tf-model.default.kubeflow.mlbookcamp.com/v1/models/tf-model ...

54. url = f'https://{model_url}:predict'
data = {
"instances": X.tolist()
}
resp = requests.post(url, json=data)
results = resp.json()

55. Pre-processing
(resize/process
images)
Post-processing
(transform
predictions)
Transformer
Model
(Make predictions)
KServe
Pants
Raw
predictions
Pre-processed
image

56. apiVersion: "serving.kubeflow.org/v1alpha2"
kind: "InferenceService"
metadata:
name: "tf-model"
spec:
default:
predictor:
serviceAccountName: sa
tensorflow:
storageUri: "s3://models-bucket-tf/tf-model"
transformer:
custom:
container:
image: "agrigorev/kfserving-keras-transformer:0.0.1"
name: user-container
env:
- name: MODEL_INPUT_SIZE
value: "299,299"
- name: KERAS_MODEL_NAME
value: "xception"
- name: MODEL_LABELS
value: "dress,hat,longsleeve,outwear,pants,shirt,shoes,shorts,skirt,t-shirt"

57. https://github.com/alexeygrigorev/kfserving-keras-transformer

58. url = f'https://{model_url}:predict'
data = {
"instances": [
{"url": "http://bit.ly/mlbookcamp-pants"},
]
}
resp = requests.post(url, json=data)
results = resp.json()

59. What’s the catch?
● Kubeflow runs on Kubernetes
● Not easy to run the whole thing locally
● Not easy to debug
● Istio

60. Summary
● AWS SageMaker vs AWS Lambda vs Kubernetes vs Kubeflow

61. Summary
● AWS SageMaker vs AWS Lambda vs Kubernetes vs Kubeflow
● Deploying models with Kubernetes: deployment + service

62. Summary
● AWS SageMaker vs AWS Lambda vs Kubernetes vs Kubeflow
● Deploying models with Kubernetes: deployment + service
● Deploying Keras models: TF-Serving + Gateway (over gRPC)

63. Summary
● AWS SageMaker vs AWS Lambda vs Kubernetes vs Kubeflow
● Deploying models with Kubernetes: deployment + service
● Deploying Keras models: TF-Serving + Gateway (over gRPC)
● KFServing: transformers + model

64. Summary
● AWS SageMaker vs AWS Lambda vs Kubernetes vs Kubeflow
● Deploying models with Kubernetes: deployment + service
● Deploying Keras models: TF-Serving + Gateway (over gRPC)
● KFServing: transformers + model
● No size fits all

65. @Al_Grigor
agrigorev
DataTalks.Club