This document discusses using OpenShift as a cloud platform for data science. It provides an overview of Kubernetes and OpenShift, comparing their key differences. It then discusses how Apache Spark can be used for data science in the cloud and integrated with OpenShift using Oshinko, which provides tools to deploy and manage Spark clusters on OpenShift. It demonstrates running a sample workflow of deploying a Spark cluster, Zeppelin notebook, and running an example notebook on the Spark cluster using Oshinko on OpenShift.

1. OpenShift as a cloud for Data
Science
Yegor Maksymchuk, Soft Serve, Ukraine

2. Agenda
• Kubernetes
• Openshift
• Kubernetes vs Openshift
• Apache Spark
• Radanalitics and OSHINKO

3. whoami
Yegor Maksymchuk
Software engineer,
Soft Serve Ukraine
Telegram: @QAStudy.online
GitHub: YegorMaksymchuk
LinkedIn:ymaksymchuk

4. Problems
Integration with Apache Spark in the Openshift.
Usability on UI and API level, it should be easy to use.
OSHINKO

5. Data Science in the Cloud

6. Kubernetes

7. Kubernetes:POD
apiVersion: v1
kind: Pod
metadata:
name: pod-demo
labels:
name: pod-demo
spec:
containers:
- name: pod-demo
image: yemax/pod-
demo:1
ports:
- containerPort: 8081

8. Kubernetes:Namespace
{
"kind": "Namespace",
"apiVersion": "v1",
"metadata": {
"name":
"development",
"labels": {
"name":
"development"
}
}
}

9. Kubernetes: Replica Sets

10. K8s: Deployment

11. Kubernetes: Ingress

12. K8s: Architecture

13. Openshift

18. Openshift: Deployment

20. Openshift: S2I

22. s2i-lighttpd/
● Dockerfile – This is a standard Dockerfile where we’ll define the builderimage
● Makefile – a helper script for buildingand testing the builderimage
● test/
○ run – test script, testing if the builder image works correctly
○ test-app/ – directory for your test application
● .s2i/bin/
○ assemble – script responsible for buildingthe application
○ run – script responsible for running the application
○ save-artifacts – script responsible for incremental builds, covered in a
future article
○ usage – script responsible for printing the usage of the builderimage

23. Openshift vs Kubernetes
K8s:
Orchestration tool
Ingress based on “Ngnix”
Namespace not “secure”
Openshift:
Platform as a Service
Routes based on HAProxy
Namespace “secure”, and more
understandable.
S2I
Builds new images, after push new
source.
Pool of prepared images

24. Data Science use Spark

25. Data Science in the Cloud

26. Apache Spark

27. Spark on OpenShift

28. OSHINKO: S2I

29. OSHINKO: Spark integrator

30. DEMO

31. DEMO
1. oc cluster up
2. oc new-project devops-stage-demo
3. oc create -f https://radanalytics.io/resources.yaml
4. oc create -f https://radanalytics.io/assets/zeppelin-example/zeppelin-openshift.yaml
5. oc new-app oshinko-webui
6. oshinko create devops-spark-cluster
7. oshinko get devops-spark-cluster
8. oc new-app --template=$namespace/apache-zeppelin-openshift
--param=APPLICATION_NAME=apache-zeppelin
--param=GIT_URI=https://github.com/rimolive/zeppelin-notebooks.git
--param=ZEPPELIN_INTERPRETERS=md

32. Questions ?