Data Science, Machine Learning, and Artificial Intelligence has exploded in popularity in the last five years, but the nagging question remains, “How to put models into production?” Engineers are typically tasked to build one-off systems to serve predictions which must be maintained amid a quickly evolving back-end serving space which has evolved from single-machine, to custom clusters, to “serverless”, to Docker, to Kubernetes. In this talk, we present KubeFlow- an open source project which makes it easy for users to move models from laptop to ML Rig to training cluster to deployment. In this talk we will discuss, “What is KubeFlow?”, “why scalability is so critical for training and model deployment?”, and other topics. Users can deploy models written in Python’s skearn, R, Tensorflow, Spark, and many more. The magic of Kubernetes allows data scientists to write models on their laptop, deploy to an ML-Rig, and then devOps can move that model into production with all of the bells and whistles such as monitoring, A/B tests, multi-arm bandits, and security.

1. Machine Learning:
From Lab to
Production with
Kubeflow
With Special Guests Tensorflow and an
Apache Spark teaser

2. Agenda
● About Us
● Background
● Problem: Building Model is Easy, Serving models in Prod is hard.
● What Is KubeFlow?
● Kubeflow’s Design and Core Components
● …

3. Holden About Me Slides
● Prefered pronouns are she/her
● Developer Advocate at Google
● Apache Spark PMC / ASF member + contributor on lots of other projects
● previously IBM, Alpine, Databricks, Google, Foursquare & Amazon
● co-author of Learning Spark & High Performance Spark
● Twitter: @holdenkarau
● Slide share http://www.slideshare.net/hkarau
● Code review livestreams: https://www.twitch.tv/holdenkarau /
https://www.youtube.com/user/holdenkarau
● Talk Videos http://bit.ly/holdenSparkVideos
● Talk feedback: http://bit.ly/holdenTalkFeedback
● Organizing data track @ IT Next AMS - CFP Open!

4. Trevor Grant
● From Chicago
● Preferred pronouns he/him
● Various odd jobs around IBM (data scientist/evangelist/janitor)
● PMC Apache Mahout, Streams, Community Development
● Apache Roadshow Chicago [1]
● IoT Track at Apache Con North America [2]
● Blog: rawkintrevo.org
● Twitter: @rawkintrevo
● Github: rawkintrevo
● Shameless self promotion: rawkintrevo.org/shameless-self-promotion/
[1] https://www.apachecon.com/chiroadshow19/
[2] https://www.apachecon.com/acna19/index.html

5. Kubeflow Salesman:

6. Kubeflow Salesman:
**Slaps roof of Kubeflow**
THIS BAD BOY CAN FIT SO MANY
BUZZWORDS IN IT

7. Kubeflow Salesman:
**Slaps roof of Kubeflow**
THIS BAD BOY CAN FIT SO MANY
BUZZWORDS IN IT

8. Background
Things we thought you might know.

9. History of Predictive Analytics
Photo: Numerology Sign
Photo: Akash Kataruka
Photo: Hans Splinter

10. What is Statistics?

11. What is Statistics?
Machine Learning?

12. What is Statistics?
Machine Learning?
A.I. (Artificial Intelligence)

13. What is Statistics?
Machine Learning?
A.I. (Artificial Intelligence)
Photo: Andreas Kretschmer
Model Training
Photo: Helen Harrop
Model Serving

14. What is Statistics?
Machine Learning?
A.I. (Artificial Intelligence)

15. What is Statistics?
Machine Learning?
A.I. (Artificial Intelligence)

16. From don’t know shit to know your shit*
Verses # of GPUs required
*Holden’s gut feelings after coffee
NeedGPUs
Knowledge of Shit

17. What is Kubernetes?

18. Kubeflow: Dev on Laptop Deploy in Cloud

19. Deploy to Production

20. So what is Kubeflow?

21. What is Kubeflow?

22. What is Kubeflow?

23. What is Kubeflow?
VIK hotels group

24. Components Buffet
argo
automation
chainer-job
core
credentials-pod-preset
katib
mpi-job
mxnet-job
openmpi
pachyderm
pytorch-job
Seldon
spark
tf-serving
Paul Harrison

25. The (many) kinds of models you can train
● All your favourite Python libraries* (in Jupyter)
○ Different options to parallelize, with more coming (for now MPI or Beam ish)
● PyTorch
● Tensorflow (along with hyper param tuning with katib)
● mxnet
● etc.
Add-ons:
● H2o: CI is failing but you know, it's Wednesday
● And more!

26. But don't forget about data prep friends!
● For now options are a little limited, but other tools like Apache Spark are
in the works
○ https://github.com/kubeflow/kubeflow/pull/1467
● Now with Spark!*
● Pachyderm
● pandas?
● shell scripts?
● Tensorflow Transform in local mode Python 2 only…...
● Yeah I guess we did forget about our dataprep friends
*Where now == what's in master as of Feb 14th 2019

27. Model persistence/deployment/CI
● You need to save your model somewhere
● Your favourite cloud storage provider goes here
○ Why invest in model management when I can make directory?
● ModelDB
● WeaveFlux
● Pachyderm

28. Model Serving
Python Flask
TensorFlow Model Server
Openvino
NVIDIA Inference Server
Seldon Core
● Routers for A/B tests and multi-armed bandits.
● Supports lots of libraries (Python/Spark/R/etc)
● Monitoring / Security
If you can put it in a Docker container, you can use it.

29. So you want to use this?

30. What’s Next?!
Step away from keyboard
Think about type(s) of model
Look at components directory and see what’s a fit tool wise
Don’t know? Choose jupyter deal with the details live
Can’t find it?
^^ New Cat
Content!!!
^._.^

31. What about just tensorflow?*
ks registry add kubeflow
github.com/kubeflow/kubeflow/tree/${VERSION}/kubeflow
ks pkg install kubeflow/core@${VERSION}
ks pkg install kubeflow/tf-serving@${VERSION}
ks pkg install kubeflow/tf-job@${VERSION}

32. Getting the chef's recommend pairing:
kfctl.sh init my_awesome_project --platform {none, gcp, minikube}
cd my_awesome_project
kfctl.sh generate platform && kfctl.sh apply platform
kfctl.sh generate k8s && kfctl.sh apply k8s
# Add spark
cd ks_ap && ks pkg install kubeflow/spark
Douglas O'Brien

33. Connect to the Kubeflow Web UI
kubectl port-forward svc/ambassador -n kubeflow 8080:80 &
# Or use IAP, but that's… another story

34. The chef's recommend pairing is:
● Jupyter Hub
● TF Job & TF Serving
● PyTorch
● Katib (Hyper parameter tuning)
● Ambassador (makes it easier to access the UIs)
● Pipelines (Argo + Magic)
chicoblue

35. Click-to-deploy: get started hella fast* on
GCP
https://deploy.kubeflow.cloud

36. Click-to-deploy continued

37. Click-to-deploy continued

38. Click-to-deploy continued

39. Click-to-deploy continued

40. ro Hasegawa

41. What are those pipelines?
“Kubeflow Pipelines is a platform for building and deploying portable,
scalable machine learning (ML) workflows based on Docker containers.” -
kubeflow.org
Directed Acyclic Graph (DAG) of “pipeline components” (read “docker
containers”) each performing a function.

42. Building that pipeline?

43. Running that pipeline

44. Serving that job (not the only way)

45. When you don't know anything or know a
lot about ML: Hyper Parameter Tuning
● Katib
○ Does not depend on a specific ML tool (e.g. not just TF)
○ Supports a few different search algorithms
○ e.g. "What should I set my L1 regularization too? Idk let's ask the computer"
● Great way to accidently overfit too!* (*if you're not careful)
● As respective cloud providers, we are happy to rent you a lot of
resources
● Seriously, mention our names in the sales call. We're both going for
promo (and that shit is hard)

46. Katib Screenshot

47. But what about [special foo-baz-inator] or
[special-yak-shaving-tool]?
Write a Dockerfile and build an image, use FROM so you’re not starting from
scratch.
FROM gcr.io/kubeflow-images-public/tensorflow-1.6.0-notebook-cpu
RUN pip install py-special-yak-shaving-tool
Then tell set it as a param for your training/serving job as needed:
ks param set tfjob-v1alpha2 image "my-special-image-goes-here”
Now your fortran lives forever!

48. Live streamed demos (recorded on
YouTube)
● Kubeflow intro
https://codelabs.developers.google.com/codelabs/kubeflow-introductio
n/index.html & streamed http://bit.ly/kfIntroStream
● Kubeflow E2E with Github issue
summurizationhttps://codelabs.developers.google.com/codelabs/cloud-
kubeflow-e2e-gis/ & streamed http://bit.ly/kfGHStream
● You can tell they were live streamed by how poorly went, I promise no
video editing has occurred.

49. Limited & Optional: Workshop Demo
● We are doing a workshop @ Strata SF and we'd love to trick offer the
option to do a self-guided pre-ffer you an exciting opportunity to try a
self-guided version free of charge and provide us feedback. We have
some demo accts.
● What you will might learn:
○ Installing Kubeflow
○ Setting up a project
○ Deploying that project to GCP / Azure / IBM
○ Monkeying around with a project and still having it work
● Please come and talk to us after. Holden is wearing a shark dress.
● We'll be around to answer your questions
fionasjournal

50. Why you shouldn't use
this?

51. Downsides to Kubeflow
● Lot's of overhead versus doing it locally
● Active development (look it's 0.4)
● 3 talks on Kubeflow can give you 3 different toolsets

52. Questions? Half-baked Demo?