The document discusses operating Kubernetes at scale, highlighting challenges in Kubernetes deployments and the advantages of using Kubernetes on DC/OS. It covers Kubernetes architecture, components, service operations, and best practices for container management. Additionally, it outlines the future roadmap for DC/OS Kubernetes integration, emphasizing security and enhanced functionalities.

1. Operating Kubernetes at
Scale
@joerg_schad @dcos @joeljacobson

2. © 2018 Mesosphere, Inc. All Rights Reserved. 2
Jörg Schad
Software Engineer @Mesosphere
@joerg_schad
@joerg.mesosphere
Joel Jacobson
Field Engineer @Mesosphere
@joeljacobson
@jjacobson.mesosphere

3. Operating Kubernetes at
Scale
@joerg_schad @dcos

4. © 2018 Mesosphere, Inc. All Rights Reserved. 4
Florian Niedermaier
Sales Director Central
Europe@Mesosphere
fniedermaier@mesosphere.io
Jörg Schad
Software Engineer @Mesosphere
@joerg_schad
@joerg.mesosphere

5. © 2018 Mesosphere, Inc. All Rights Reserved.
Agenda
● Overview Kubernetes and DC/OS
● Challenges in Kubernetes Deployments
● Why Kubernetes on DC/OS?
● Operating Kubernetes Cluster
● Current Limitations/Future plans
5

6. © 2017 Mesosphere, Inc. All Rights Reserved. 6
In the beginning
there was a big
Monolith

7. © 2016 Mesosphere, Inc. All Rights Reserved. 7

8. © 2018 Mesosphere, Inc. All Rights Reserved.
Hardware
Operating System
Application
8
COMPUTERS

9. © 2018 Mesosphere, Inc. All Rights Reserved.
noun | ˈmīkrō/ /ˈsərvəs/ :
an approach to application development in which a
large application is built as a suite of modular
services. Each module supports a specific business
goal and uses a simple, well-defined interface to
communicate with other modules.*
Microservices are designed to be flexible, resilient,
efficient, robust, and individually scalable.
*From whatis.com
OVERVIEW

10. © 2018 Mesosphere, Inc. All Rights Reserved.
Operating
System
Operating
System
Operating
System
ServiceApp ServiceServiceAppApp
10
MICROSERVI
CES
Polyglot
Single Responsibility
Smaller Teams
Utilization
Machine types/groups
Dependency hell
Machine
Infrastructure
Machine Machine
ServiceService ServiceServiceServiceService

11. © 2017 Mesosphere, Inc. All Rights Reserved. 11
Container …
!=
• container runtime* != container image != container instance
• beyond docker runtime
• Universal Container Runtime
• supports docker images
• CRI{-o}
•…

12. © 2018 Mesosphere, Inc. All Rights Reserved.
ServiceApp ServiceServiceAppApp
OS
12
CONTAINER
S
Rapid deployment
Dependency vendoring
Container image
repositories
Spreadsheet
scheduling
OS OS
Machine
Infrastructure
Machine Machine
Container
Runtime
Container
Runtime
Container
Runtime
ServiceService ServiceServiceServiceService

13. © 2018 Mesosphere, Inc. All Rights Reserved. 13
CONTAINER
SCHEDULING
RESOURCE
MANAGEMENT
SERVICE
MANAGEMENT
- Load Balancing
- Readiness Checking
CONTAINER ORCHESTRATION

14. © 2018 Mesosphere, Inc. All Rights Reserved. 14
CONTAINER
SCHEDULING
- Placement
- Replication/Scaling
- Resurrection
- Rescheduling
- Rolling Deployment
- Upgrades
- Downgrades
- Collocation
RESOURCE
MANAGEMENT
- Memory
- CPU
- GPU
- Volumes
- Ports
- IPs
- Images/Artifacts
SERVICE
MANAGEMENT
- Labels
- Groups/Namespaces
- Dependencies
- Load Balancing
- Readiness Checking
CONTAINER ORCHESTRATION

15. © 2018 Mesosphere, Inc. All Rights Reserved.
Orchestration
15
Machine Infrastructure
Web Apps & Services
Scheduling
Resource Management
Container
Runtime
Machine & OS
Service Management
CONTAINER
ORCHESTRA
TION
Machine & OS Machine & OS
Container
Runtime
Container
Runtime

16. © 2017 Mesosphere, Inc. All Rights Reserved. 16
Kubernetes
Overview

17. © 2018 Mesosphere, Inc. All Rights Reserved.
KUBERNETES Concepts
kind: Deployment
kind: ReplicaSet
kind: Pod
scales
N instances
of Pod
Set of
Containers
Upgrades
deployes
17

18. © 2018 Mesosphere, Inc. All Rights Reserved.
Configuration File
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
18

19. © 2018 Mesosphere, Inc. All Rights Reserved.
Configuration File
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.7.9
ports:
- containerPort: 80
ReplicaSet
Pod
19

20. © 2018 Mesosphere, Inc. All Rights Reserved.
Service Spec & Labels
kind: Service
apiVersion: v1
metadata:
name: my-service
spec:
selector:
app: MyApp
ports:
- protocol: TCP
port: 80
targetPort: 9376
Label
20

21. © 2018 Mesosphere, Inc. All Rights Reserved.
kubectl
$ kubectl create -f https://k8s.io/docs/user-guide/nginx-deployment.yaml
$ kubectl get deployments
$ kubectl decribe
$ kubectl logs --tail=20 nginx-deployment
21

22. © 2018 Mesosphere, Inc. All Rights Reserved.
KUBERNETES ARCHITECTURE
22

23. © 2018 Mesosphere, Inc. All Rights Reserved.
Kubernetes Component Role
etcd Distributed key/value store
kube-apiserver Central API server to interact with the cluster components
kube-controller-manager Reconciles scale events for fault tolerant objects (e.g. Replication
Controllers)
kube-scheduler Schedules containers on the various worker nodes
kubelet Agent running on each worker. Serves as an init daemon to start
containers
kube-proxy Process to configure Linux Netfilter (iptable) rules to route traffic across
the cluster
kubectl CLI to interact with the cluster, and deploy containers
Kubernetes Components and Analogies
23

24. © 2017 Mesosphere, Inc. All Rights Reserved. 24
Challenges

25. © 2018 Mesosphere, Inc. All Rights Reserved. 25
Kubernetes
Machine Infrastructure
Container
Runtime
Machine & OS Machine & OS Machine & OS
Container
Runtime
Container
Runtime
Container Runtime
Machine & OS Machine & OS
Container
Runtime
Orchestration
Scheduling
Resource Management
Service Management
Web Apps & Services
Machine Infrastructure
OrchestrationScheduling
Resource Management
Service Management
Web Apps & Services

26. © 2018 Mesosphere, Inc. All Rights Reserved. 26
Mesos and Kubernetes
Machine Infrastructure
Container
Runtime
Machine & OS Machine & OS Machine & OS
Container
Runtime
Container
Runtime
Container Runtime
Machine & OS Machine & OS
Container
Runtime
Machine Infrastructure
Web Apps & Services
etcd etcdetcd
api-
server
api-
server
api-
server
Kube-
proxy
Kube-
proxy
scheduler
Kube-
proxy
scheduler
schedulercontrollercontroller
controller
kubelet kubelet kubelet
...

27. © 2018 Mesosphere, Inc. All Rights Reserved. 27
Mesos and Kubernetes
Machine Infrastructure
Container
Runtime
Machine & OS Machine & OS Machine & OS
Container
Runtime
Container
Runtime
Container Runtime
Machine & OS Machine & OS
Container
Runtime
Machine Infrastructure
Web Apps & Services
etcd etcdetcd
api-
server
api-
server
api-
server
Kube-
proxy
Kube-
proxy
scheduler
Kube-
proxy
scheduler
schedulercontrollercontroller
controller
kubelet kubelet kubelet
...

28. © 2017 Mesosphere, Inc. All Rights Reserved. 28
Datacenter

29. © 2017 Mesosphere, Inc. All Rights Reserved. 29
Datacenter
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Kubernetes
Jenkins
Kafka
Spark
Kubernetes

30. © 2017 Mesosphere, Inc. All Rights Reserved. 30

31. © 2017 Mesosphere, Inc. All Rights Reserved. 31
Datacenter
Typical Datacenter
siloed, over-provisioned servers,
low utilization
Mesos/ DC/OS
automated schedulers, workload multiplexing onto the
same machines
Tensorflow
Jenkins
Kafka
Spark
Tensorflow

32. © 2018 Mesosphere, Inc. All Rights Reserved. 32
Mesos and Kubernetes

33. © 2018 Mesosphere, Inc. All Rights Reserved. 33
Mesos and Kubernetes
Machine Infrastructure
Container
Runtime
Machine & OS Machine & OS Machine & OS
Container
Runtime
Container
Runtime
Container Runtime
Machine & OS Machine & OS
Container
Runtime
Orchestration
Scheduling
Resource Management
Service Management
Web Apps & Services
Machine Infrastructure
DC/OS / Apache Mesos
OrchestrationScheduling
Resource Management
Service Management
Web Apps & Services

34. © 2018 Mesosphere, Inc. All Rights Reserved. 34
Mesos and Kubernetes
Orchestration
Machine Infrastructure
Web Apps & Services
Scheduling
Resource Management
Container
Runtime
Machine & OS
Service Management
Machine & OS Machine & OS
Container
Runtime
Container
Runtime
Container Runtime
Machine & OS Machine & OS
Container
Runtime
Orchestration
Scheduling
Resource Management
Service Management
Web Apps & Services
Machine Infrastructure
DC/OS / Apache Mesos
Big Data
Services
● Spark
● Flink
● Kafka
● ….

35. © 2018 Mesosphere, Inc. All Rights Reserved. 35
Mesos and Kubernetes
Machine Infrastructure
Container
Runtime
Machine & OS Machine & OS Machine & OS
Container
Runtime
Container
Runtime
Container Runtime
Machine & OS Machine & OS
Container
Runtime
Machine Infrastructure
DC/OS / Apache Mesos
Web Apps & Services
etcd etcdetcd
api-
server
api-
server
api-
server
Kube-
proxy
Kube-
proxy
scheduler
Kube-
proxy
scheduler
schedulercontrollercontroller
controller
kubelet kubelet kubelet
...

36. © 2018 Mesosphere, Inc. All Rights Reserved. 36
Mesos and Kubernetes
Machine Infrastructure
Container
Runtime
Machine & OS Machine & OS Machine & OS
Container
Runtime
Container
Runtime
Container Runtime
Machine & OS Machine & OS
Container
Runtime
Machine Infrastructure
DC/OS / Apache Mesos
Web Apps & Services
etcd etcdetcd
api-
server
api-
server
api-
server
Kube-
proxy
Kube-
proxy
scheduler
Kube-
proxy
scheduler
schedulercontrollercontroller
controller
kubelet kubelet kubelet
...

37. © 2018 Mesosphere, Inc. All Rights Reserved. 37
Reality
Machine Infrastructure
Container
Runtime
Machine & OS Machine & OS Machine & OS
Container
Runtime
Container
Runtime
Container Runtime
Machine & OS Machine & OS
Container
Runtime
Orchestration
Scheduling
Resource Management
Service Management
Web Apps & Services
Machine Infrastructure
DC/OS/Apache
Mesos

38. © 2016 Mesosphere, Inc. All Rights Reserved. 38
MESOS: FUNDAMENTAL ARCHITECTURE
Mesos
Master
Mesos
Master
Mesos
Master
Mesos AgentMesos Agent Service
Cassandra
Executor
Cassandra
Task
Cassandr
a
Scheduler
Container
Scheduler
Spark
Scheduler
Spark
Executor
Spark
Task
Mesos AgentMesos Agent Service
Docker
Executor
Docker
Task
Spark
Executor
Spark
Task
Two-level Scheduling
1. Agents advertise resources to Master
2. Master offers resources to Framework
3. Framework rejects / uses resources
4. Agent reports task status to Master

39. © 2017 Mesosphere, Inc. All Rights Reserved. 40
DC/OS
Kubernetes
Integration

40. © 2018 Mesosphere, Inc. All Rights Reserved.
● Unaltered upstream Kubernetes, unlike OpenShift
● Stand-alone Kubernetes clusters are exposed, not a PaaS using Kubernetes
● Simple install onto a DC/OS cluster
● Integration with DC/OS data services
● HA and Secure by default in EE
● Customers have choice between Marathon and Kubernetes
DC/OS Kubernetes
41

41. © 2018 Mesosphere, Inc. All Rights Reserved.
Kubernetes Component Role DC/OS Component
etcd Distributed key/value store Zookeeper
kube-apiserver Central API server to interact with the cluster components
kube-controller-manager Reconciles scale events for fault tolerant objects (e.g. Replication
Controllers)
kube-scheduler Schedules containers on the various worker nodes Marathon
kubelet Agent running on each worker. Serves as an init daemon to start
containers
Mesos Agent
kube-proxy Process to configure Linux Netfilter (iptable) rules to route traffic across
the cluster
Minuteman
kubectl CLI to interact with the cluster, and deploy containers dcos CLI
Kubernetes Components and Analogies
42

42. © 2018 Mesosphere, Inc. All Rights Reserved. 43
Default Setup

43. © 2018 Mesosphere, Inc. All Rights Reserved. 44
● CNI - Container Networking Interface
● Kube-DNS - Kubernetes DNS server
● Kube-proxy - East-west loadbalancer
● Ingress - North-South loadbalancer
Networking in Kubernetes

44. © 2018 Mesosphere, Inc. All Rights Reserved. 45
● CNI - Container Networking Interface
● Spartan - DC/OS DNS server
● Minuteman - East-west loadbalancer
● Marathon-LB - North-South loadbalancer
Networking in DC/OS

45. © 2018 Mesosphere, Inc. All Rights Reserved. 46
Networking in Kubernetes on DC/OS

46. © 2018 Mesosphere, Inc. All Rights Reserved. 47
Secure TLS Support
OSS EE

47. © 2017 Mesosphere, Inc. All Rights Reserved. 48
Operations

48. © 2017 Mesosphere, Inc. All Rights Reserved. 49
Keep it running!

49. © 2017 Mesosphere, Inc. All Rights Reserved. 50
SERVICE OPERATIONS
● Configuration Updates (ex: Scaling, re-configuration)
● Binary Upgrades
● Cluster Maintenance (ex: Backup, Restore, Restart)
● Monitor progress of operations
● Debug any runtime blockages

50. © 2017 Mesosphere, Inc. All Rights Reserved. 51
SERVICE OPERATIONS
● Configuration Updates (ex: Scaling, re-configuration)
● Binary Upgrades
● Cluster Maintenance (ex: Backup, Restore, Restart)
● Monitor progress of operations
● Debug any runtime blockages

51. © 2016 Mesosphere, Inc. All Rights Reserved. 52
Lab 4
Operations

52. © 2017 Mesosphere, Inc. All Rights Reserved. 53
Kill kubelet process
● dcos task exec -it kube-node-0-kubelet bash
● ps aux | grep "./kubelet "
root 62 2.3 0.6 741984 100580 ? Sl 15:16 3:25
./kubelet --address=10.0.4.5 --hostname-override=kube-node-
1-kubelet.kubernetes.mesos
● kill -9 62

53. © 2017 Mesosphere, Inc. All Rights Reserved. 54
Watch the kubelet being restarted

54. © 2017 Mesosphere, Inc. All Rights Reserved. 55
Watch it being restarted

55. © 2018 Mesosphere, Inc. All Rights Reserved. 56
Deploy Microservices
# Deploy Display
$ kubectl apply -f https://raw.githubusercontent.com/dcos-
labs/dcos-kubernetes-workshop/master/flinkDemo/flink-demo-
actor.yaml
# Deploy Generator
$ kubectl apply -f https://raw.githubusercontent.com/dcos-
labs/dcos-kubernetes-workshop/master/flinkDemo/flink-demo-
generator.yaml
$ kubectl get pods
$ kubectl logs flink-demo-actor-655890656-8d1ls

56. © 2017 Mesosphere, Inc. All Rights Reserved. 57
Demo Time

57. © 2017 Mesosphere, Inc. All Rights Reserved. 58
Future

58. © 2018 Mesosphere, Inc. All Rights Reserved.
DC/OS Kubernetes Roadmap
Beta
● Simple provisioning
● TLS Throughout (EE)
● Monitoring Dashboards
& Alerts
● Add kubelet’s to the
cluster
GA
● Edge-LB Integration
● Tunnel-less kubectl
● Rolling upgrades
● Internal DNS
(DC/OS and K8S)
● OSS (?)
Sept
‘17
Future
Future
● Hybrid-Cloud
● DC/OS Secrets
● Central Logging
● DC/OS Storage
● Disaster Recovery
● Identity Provider
integration
March
‘18
59

59. © 2017 Mesosphere, Inc. All Rights Reserved. 60
THANK YOU!
ANY
QUESTION
S?
@dcos
users@dcos.io
/groups/8295652
/dcos
/dcos/examples
/dcos/demos
chat.dcos.io
https://github.com/mesosphere/dcos-kubernetes-quickstart
https://mesosphere.com/blog/another-kubernetes-service/

60. © 2017 Mesosphere, Inc. All Rights Reserved. 61
Container
Best
Practices

61. © 2017 Mesosphere, Inc. All Rights Reserved. 62
Immutable Container Images
• Use tagged container
images
• Keep tagged images
immutable!

62. © 2017 Mesosphere, Inc. All Rights Reserved.
Dockerhub
works great for
our test
cluster…
Private Container Registries
• Use tagged container
images
• Keep tagged images
immutable!
• Use a private container
registry!

63. © 2017 Mesosphere, Inc. All Rights Reserved.
Repeatable Container Builds
• Use repeatable builds for images
• Including FROM clause
• Keep images minimal
• Multistage build
• From scratch
`docker
commit` is
great*…

64. © 2017 Mesosphere, Inc. All Rights Reserved. 65
Write Once Run Any Where
The (Java) container
was running fine in
testing…
• Java (<9) not groups aware
• # threads for GC
• …
• Set default values carefully