Ankara Tech Talks Meetup 23.01.2020 https://www.meetup.com/tr-TR/Ankara-Tech-Talks/events/268054959/

1. Managing the Machine
Learning Lifecycle with
MLOps
23.01.2020
Ankara Tech Talks

2. Fatih Baltacı
● Data Scientist @Databoss
○ Supervised Object Detection
● MSc Student in METU Computer Engineering
○ Weakly Supervised Object Detection
● https://github.com/fatihbaltaci

3. What is ML Ops?
● DevOps for ML
● Collaboration & Communication between data scientists and
operations professionals
● Manage production ML/DL Lifecycle

4. What is ML Ops?
● Increase Automation like DevOps
● Improve the quality of production ML
● Started as Best Practices, Evolving into ML Lifecycle Management

5. DEPLOYMENT
TRAINING DATA PREP
RAW DATA
ML
Lifecycle

6. DEPLOYMENT
TRAINING DATA PREP
RAW DATA
ML
Lifecycle

7. DEPLOYMENT
TRAINING DATA PREP
RAW DATA
ML
Lifecycle

8. DEPLOYMENT
TRAINING DATA PREP
RAW DATA
ML
Lifecycle

9. DEPLOYMENT
TRAINING DATA PREP
RAW DATA
ML
Lifecycle

10. https://docs.docker.com/get-started/
● OS-level virtualization
● Build, share, and run applications with containers
● Docker is written in go

13. https://hub.docker.com/r/pytorch/pytorch/tags

14. https://github.com/fatihbaltaci/docker-files/blob/master/jupyter_pytorch/Dockerfile
Dockerfile
FROM pytorch/pytorch:1.4-cuda10.1-cudnn7-runtime
RUN apt-get update && apt-get install -y wget vim unzip
RUN pip --no-cache-dir install mlflow==0.9.1 jupyterlab==1.2.4
…
CMD [ "jupyter", "lab" ]

15. Dockerfile
FROM pytorch/pytorch:1.4-cuda10.1-cudnn7-runtime
RUN apt-get update && apt-get install -y wget vim unzip
RUN pip --no-cache-dir install mlflow==0.9.1 jupyterlab==1.2.4
…
CMD [ "jupyter", "lab" ]
Base Image
https://github.com/fatihbaltaci/docker-files/blob/master/jupyter_pytorch/Dockerfile

16. Dockerfile
FROM pytorch/pytorch:1.4-cuda10.1-cudnn7-runtime
RUN apt-get update && apt-get install -y wget vim unzip
RUN pip --no-cache-dir install mlflow==0.9.1 jupyterlab==1.2.4
…
CMD [ "jupyter", "lab" ]
Base Image
Install
Ubuntu
Packages
https://github.com/fatihbaltaci/docker-files/blob/master/jupyter_pytorch/Dockerfile

17. Dockerfile
FROM pytorch/pytorch:1.4-cuda10.1-cudnn7-runtime
RUN apt-get update && apt-get install -y wget vim unzip
RUN pip --no-cache-dir install mlflow==0.9.1 jupyterlab==1.2.4
…
CMD [ "jupyter", "lab" ]
Base Image
Install
Ubuntu
Packages
Install
Python
Packages
https://github.com/fatihbaltaci/docker-files/blob/master/jupyter_pytorch/Dockerfile

18. Dockerfile
FROM pytorch/pytorch:1.4-cuda10.1-cudnn7-runtime
RUN apt-get update && apt-get install -y wget vim unzip
RUN pip --no-cache-dir install mlflow==0.9.1 jupyterlab==1.2.4
…
CMD [ "jupyter", "lab" ]
Base Image
Install
Ubuntu
Packages
Install
Python
Packages
https://github.com/fatihbaltaci/docker-files/blob/master/jupyter_pytorch/Dockerfile
Container
Starting
Point

19. http://www.markbuckler.com/post/docker-use/

20. Jupyterlab with Single Line
docker run -dit -p 8888:8888 -v /jupyter:/workspace
--restart=always --name=jupyter_lab fatihbaltaci/jupyter
https://hub.docker.com/repository/docker/fatihbaltaci/jupyter

21. https://gist.github.com/fatihbaltaci/d1fa95b9cf166de793d186c1d75f53f6#docker-image-save-and-load
Save & Load Docker Images
docker save my_ubuntu_image > my_ubuntu_image.tar
gzip my_ubuntu_image.tar # (Optional)
gunzip my_ubuntu_image.tar.gz # (Optional)
docker load < my_ubuntu_image.tar

22. https://fatihbaltaci.github.io/docker/2019/10/30/docker-save-load-images-minimal-size.html
Save & Load Docker Images
Image: pytorch/pytorch:1.2-cuda10.0-cudnn7-runtime
Base Image .tar .tar.gz
3.85GB 3.6GB 2.0GB

23. Tensorflow Serving
Flexible, high-performance serving system for machine learning models, designed for production environments.
● Model Discovery
● gRPC and REST Interface
● Separation of API and Model Server
● High Performance Inference
● Limited to Tensorflow

24. docker run -p 8501:8501
--name tfserving_resnet
-v /tmp/resnet:/models/resnet
-e MODEL_NAME=resnet -t tensorflow/serving

25. docker run -p 8501:8501
--name tfserving_resnet
-v /tmp/resnet:/models/resnet
-e MODEL_NAME=resnet -t tensorflow/serving
REST Port
(Host)
REST Port
(Container)

26. docker run -p 8501:8501
--name tfserving_resnet
-v /tmp/resnet:/models/resnet
-e MODEL_NAME=resnet -t tensorflow/serving
Container
Name

27. docker run -p 8501:8501
--name tfserving_resnet
-v /tmp/resnet:/models/resnet
-e MODEL_NAME=resnet -t tensorflow/serving
Bind Volume
(Host)
Bind Volume
(Container)

28. docker run -p 8501:8501
--name tfserving_resnet
-v /tmp/resnet:/models/resnet
-e MODEL_NAME=resnet -t tensorflow/serving
Load the model as
resnet

29. docker run -p 8501:8501
--name tfserving_resnet
-v /tmp/resnet:/models/resnet
-e MODEL_NAME=resnet -t tensorflow/serving
Docker image
name

30. SERVER_URL = 'http://localhost:8501/v1/models/resnet:predict'
...
# Send few actual requests and report average latency.
total_time = 0
num_requests = 10
for _ in range(num_requests):
response = requests.post(SERVER_URL, data=predict_request)
response.raise_for_status()
total_time += response.elapsed.total_seconds()
prediction = response.json()['predictions'][0]
print('Prediction class: {}, avg latency: {} ms'.format(
prediction['classes'], (total_time*1000)/num_requests))
https://raw.githubusercontent.com/tensorflow/serving/master/tensorflow_serving/example/resnet_client.py

31. $ python /tmp/resnet/resnet_client.py
Prediction class: 286, avg latency: 59.0938 ms
cat

32. Slow Inference Time?
... Your CPU supports instructions that this TensorFlow binary was not compiled
to use: AVX2 FMA
● Some optimizations are left out to maximize compatibility
● Build Tensorflow Serving with your CPU architecture

33. Build your own Docker Image
docker build -t $USER/tensorflow-serving-devel
-f Dockerfile.devel .
https://github.com/tensorflow/serving/tree/master/tensorflow_serving/tools/docker

34. ~52 minutes Ryzen 5 2600 3.9GHz
16 GB RAM
M.2 NVME SSD

35. Build your own Docker Image
docker build -t $USER/tensorflow-serving --build-arg
TF_SERVING_BUILD_IMAGE=$USER/tensorflow-serving-devel
.
https://github.com/tensorflow/serving/tree/master/tensorflow_serving/tools/docker

36. docker run -p 8501:8501
--name tfserving_resnet
-v /tmp/resnet:/models/resnet
-e MODEL_NAME=resnet -t
fatih/tensorflow-serving

37. $ python /tmp/resnet/resnet_client.py
Prediction class: 286, avg latency: 57.2427 ms

38. https://github.com/tensorflow/tensorflow/releases/tag/v1.6.0

39. Tensorflow Serving
$ ls /tmp/resnet
1538687457
model
version

40. Tensorflow Serving
$ ls /tmp/resnet
1538687457 1538687458
new model
… Successfully loaded servable version {name:
resnet version: 1538687458}
…
… Unloading servable version {name: resnet
version: 1538687457}

41. MLflow:
Open source ML Platform.
● Works with any ML framework.
● Key Components:
○ MLflow Tracking: experiment tracking
○ MLflow Projects: reproducible runs
○ MLflow Models: model packaging

42. MLflow Tracking:
● git commit id
● REST API
● Supported in Azure Machine Learning

43. MLflow Tracking:
mlflow.log_param(“lr”, 0.001)
mlflow.log_param(“img_size”, 608)
mlflow.log_metric(“accuracy”, 0.85)
mlflow.log_metric(“accuracy”, 0.86)
mlflow.log_metric(“accuracy”, 0.89)
...

44. https://medium.com/analytics-vidhya/tracking-deep-learning-experiments-using-keras-mlflow-and-mongodb-732
fc771266c

45. MLflow:
Model Management Problem ??

46. MLflow:
Model Management Problem ?? classifier_v1.pt
classifier_prod.pt
classifier_v3_23_01_2020.pt
classifier_new.pt
...

47. MLflow:
Model Management Problem ??
● Where can I find the best version of the model?
● What are the hyper-parameters for this trained model?
● How can we review model before production?
Solution: MLflow Registry

48. MLflow Model Registry:

49. MLflow Model Registry:
● Repository of named, versioned models with comments
● Model deploy stages: dev, staging, production, archived

50. MLflow Model Registry:
client = mlflow.tracking.MlflowClient()
client.get_latest_versions(name = "RetinaNet",
stages = ["Staging"])

51. import keras
mlflow.keras.autolog()
mnist = keras.datasets.mnist
…
model.fit(x_train, y_train,
epochs=5)
https://databricks.com/blog/2019/08/19/mlflow-tensorflow-open-source-show.html

52. Polyaxon
A platform for reproducing and managing the whole life cycle of machine learning and deep learning applications
● Kubernetes needed
● Major ML libraries supported
● Open Source
● Tracking & Scheduling between distributed servers

54. Polyaxon
● Alternative to Slurm
● Solves resource allocation problem in organizations
● Create experiments with .yaml files

55. Polyaxon
version: 1
kind: experiment
build:
image: tensorflow/tensorflow:1.4.1-py3
build_steps:
- pip3 install polyaxon-client
run:
cmd: python model.py
polyaxonfile.yaml

56. Polyaxon
version: 1
kind: experiment
build:
image: tensorflow/tensorflow:1.4.1-py3
build_steps:
- pip3 install polyaxon-client
run:
cmd: python model.py
polyaxonfile.yaml
polyaxon run -p quick-start -f polyaxonfile.yaml

57. https://docs.polyaxon.com/concepts/quick-start-external-repo/

58. https://docs.polyaxon.com/concepts/quick-start-external-repo/

59. Other Alternatives
● Kubeflow
● Azure ML Platform
● Google AI Platform

60. Development
● You want to use Docker as remote interpreter
● Pycharm and VScode
● VScode uses configuration file, Pycharm uses UI to add remote docker
interpreter
● Pycharm is not flexible

61. Development - vscode
{
"name": "Existing Dockerfile",
"context": "..",
"dockerFile": "../Dockerfile",
// "image": "ubuntu:18.04",
"runArgs": [
"-v", "/home/fatih/dev/classification:/workspace",
"-p", "1414:1414",
],
"workspaceFolder": "/workspace"
}
devcontainer.json
https://code.visualstudio.com/docs/remote/containers

62. Development - vscode
{
"name": "Existing Dockerfile",
"context": "..",
"dockerFile": "../Dockerfile",
// "image": "ubuntu:18.04",
"runArgs": [
"-v", "/home/fatih/dev/classification:/workspace",
"-p", "1414:1414",
],
"workspaceFolder": "/workspace"
}
devcontainer.json
https://code.visualstudio.com/docs/remote/containers
Build docker file or
Use pre-built image

63. Development - Special Case
● You do not have a GPU on your local machine.
● You can connect to GPU Servers
● You don’t want to use <<Conda>>
● Developing inside a container on a remote Docker host

64. Development - Special Case
{
"name": "Existing Dockerfile",
"context": "..",
"dockerFile": "../Dockerfile",
// "image": "ubuntu:18.04",
"runArgs": [
"-p", "1414:1414",
],
"workspaceMount": "src=/home/server/detection,dst=/workspace,type=bind",
"workspaceFolder": "/workspace"
}
devcontainer.json
https://code.visualstudio.com/docs/remote/containers-advanced#_developing-inside-a-container-on-a-remote-
docker-host

65. Development - Special Case
{
"name": "Existing Dockerfile",
"context": "..",
"dockerFile": "../Dockerfile",
// "image": "ubuntu:18.04",
"runArgs": [
"-p", "1414:1414",
],
"workspaceMount": "src=/home/server/detection,dst=/workspace,type=bind",
"workspaceFolder": "/workspace"
}
devcontainer.json
https://code.visualstudio.com/docs/remote/containers-advanced#_developing-inside-a-container-on-a-remote-
docker-host
Remote Server Path

66. Development - Special Case
{
"name": "Existing Dockerfile",
"context": "..",
"dockerFile": "../Dockerfile",
// "image": "ubuntu:18.04",
"runArgs": [
"-p", "1414:1414",
],
"workspaceMount": "src=/home/server/detection,dst=/workspace,type=bind",
"workspaceFolder": "/workspace"
}
devcontainer.json
https://code.visualstudio.com/docs/remote/containers-advanced#_developing-inside-a-container-on-a-remote-
docker-host
Remote Server Path
Remote Server Docker
Container Path

67. Development - Special Case
{
"python.pythonPath": "/usr/bin/python",
"python.linting.pylintEnabled": true,
"python.linting.pycodestyleEnabled": false,
"python.linting.enabled": true,
...
"docker.host":"ssh://fatihbaltaci@192.168.0.20",
}
.vscode/settings.json
https://code.visualstudio.com/docs/remote/containers-advanced#_developing-inside-a-container-on-a-remote-
docker-host

68. Best Practices
● use gzip after docker save
● install packages with specific versions
● do not use apt upgrade in docker files
● do not use docker commit, update dockerfile instead
● https://docs.docker.com/develop/develop-images/dockerfile_best-pra
ctices/

69. THANK’S
QUESTIONS?

70. References
● https://medium.com/tensorflow/serving-ml-quickly-with-tensorflow-serving-and-docker-7df709
4aa008
● https://papers.nips.cc/paper/5656-hidden-technical-debt-in-machine-learning-systems.pdf
● https://en.wikipedia.org/wiki/MLOps
● https://www.aitrends.com/machine-learning/mlops-not-just-ml-business-new-competitive-fronti
er/
● https://code.visualstudio.com/docs/remote/containers-advanced#_developing-inside-a-container
-on-a-remote-docker-host
● https://devopscube.com/what-is-docker/