Kubernetes in kubernetes 搭建高可用環境

Kubernetes in Kubernetes：
搭建高可用環境
資訊與通訊研究所蔣是文 Mac Chiang
交通大學資工所鄭偉聖 Sam Zheng

Copyright 2017 ITRI 工業技術研究院
Agenda
• Cluster Management Issues
• Self-hosted Kubernetes
• Service with High Availability
• Summary
2

Microservices Challenges
3
• A lots of microservices/components
• Zero downtime deployment
• Incremental roll out of features faster
• Improves the ability scale efficiently

Kubernetes Cluster
4
• Scheduling
• Deployment
• Healing
• Discovery/Load balancing
• Scaling
But how about operating Kubernetes?

Kubernetes Operation Tasks
5
• Deployment
• Upgrade / rollback
• Scaling
• Monitoring
• Healing
• Security
• …
• A lot of manual/semi-manual work
• No standard way to approach all the problems
Problems!
http://www.infoq.com/cn/presentations/self-hosted-infrastructure-take-kubernetes-as-an-example

What’s Self-hosted Kubernetes?
6
• Running all required and optional components of a
Kubernetes cluster on top of Kubernetes itself
• Kubernetes manages own core components
• Core component deployed as native API objects
https://www.youtube.com/watch?v=EbNxGK9MwN4

Why Self-hosted Kubernetes?
7
• Small Dependencies
▪ Reduce the number of components required (Kubelet and Docker)
• Deployment consistency
▪ Reduce the number of moving parts relying on the host OS
• Introspection
▪ Can be debugged and inspected by users using existing Kubernetes APIs
• Cluster Upgrades
▪ Upgrade the components via Kubernetes APIs
• Easier Highly-Available Configurations
▪ Easier to scale up and monitor an HA environment without complex external tooling
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md

Launching a Self-hosted Cluster
8
• Need an initial control plane to bootstrap a self-hosted
cluster
• Bootkube
▪ Provides a temporary control plane to run a full blown self-hosted control
plane
▪ Run only on very first node, then not needed again
https://www.youtube.com/watch?v=EbNxGK9MwN4

Kubernetes Architecture
9
Master node
etcd
Scheduler
Controller
Manager
API Server
Worker node
Kubelet Kube-proxy
dockerPod Pod
Worker node
Kubelet Kube-proxy
dockerPod Pod

Spectrum of Self-hosted Ways
10
https://github.com/kubernetes/community/blob/master/contributors/design-proposals/self-hosted-kubernetes.md
• 0-4 self-hosted cluster
• kubelet: daemon set
• API server: pod / service
• system installed Kubelet
• everything except etcd self-hosted

How bootkube works?
11
etcd
Bootkube
API Server
Scheduler
Controller
Manager
Kubelet

How bootkube works? (cont.)
12
etcd
Bootkube
API Server
Scheduler
Controller
Manager
Kubelet
Create
• Deployment
• Daemonset
• Service
• Secret
kubectl

13
etcd
Bootkube
API Server
Scheduler
Controller
Manager
Kubelet
Pods
API Server
Scheduler
Controller
Manager
create

14
etcd
Bootkube
API Server
Scheduler
Controller
Manager
Kubelet
Pods
API Server
Scheduler
Controller
Manager
create

Pods
15
etcd
Kubelet
API Server
Scheduler
Controller
Manager

Self-hosted Control Plane
16
[root@centos7 ~]# kubectl get pod -n kube-system
NAME READY STATUS RESTARTS AGE
kube-apiserver-kkxq9 1/1 Running 0 1d
kube-controller-manager-2953862963-t7m1q 1/1 Running 0 1d
kube-controller-manager-2953862963-wlsjp 1/1 Running 0 1d
kube-dns-2431531914-gqnnd 3/3 Running 0 1d
kube-flannel-wnk1j 2/2 Running 0 1d
kube-flannel-xcsx2 2/2 Running 0 1d
kube-flannel-xrksj 2/2 Running 1 1d
kube-proxy-04x11 1/1 Running 0 1d
kube-proxy-11n6t 1/1 Running 0 1d
kube-proxy-1zlgz 1/1 Running 0 1d
kube-scheduler-1873817829-4c7mm 1/1 Running 1 1d
kube-scheduler-1873817829-pmp0n 1/1 Running 0 1d
pod-checkpointer-11q7g 1/1 Running 0 1d
pod-checkpointer-11q7g-10.201.3.6 1/1 Running 0 1d

Self-hosted Control Plane (cont.)
17
[root@centos7 ~]# kubectl get deployment -n kube-system
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-controller-manager 2 2 2 2 2d
kube-dns 1 1 1 1 2d
kube-scheduler 2 2 2 2 2d
[root@centos7 ~]# kubectl get svc -n kube-system
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kube-dns 10.3.0.10 <none> 53/UDP,53/TCP 2d
[root@centos7 ~]# kubectl get ds -n kube-system
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE-SELECTOR AGE
kube-apiserver 1 1 1 1 1 node-role.kubernetes.io/master= 2d
kube-flannel 3 3 3 3 3 <none> 2d
kube-proxy 3 3 3 3 3 <none> 2d
pod-checkpointer 1 1 1 1 1 node-role.kubernetes.io/master= 2d
[root@centos7 ~]# kubectl get secret -n kube-system
NAME TYPE DATA AGE
kube-apiserver Opaque 7 2d
kube-controller-manager Opaque 2 2d

Disaster Recovery
18
• Node failure in HA deployments (Kubernetes)
• Partial loss of control plane components (Kubernetes)
• Power cycling the entire control plane (Kubernetes)
• Permanent loss of control plane (External tool)
http://www.infoq.com/cn/presentations/self-hosted-infrastructure-take-kubernetes-as-an-example

Service with High Availability
19

Kubernetes Networking
20

How do we to export the service IP to
public network on bare metal?
• In kubernetes we have some existed solution
 NodePort
 CloudProvider Load Balancer
 Ingress
• But it is enough?
23

NodePort
24
kubeProxy kubeProxy kubeProxy
NodePort NodePort NodePort
Pod Pod Pod

NodePort
25
Pod Pod Pod

Load Balancer
• Cloud Provider e.g. AWS , GCP, OpenStack
• Load Balancer is created by Cloud Provider, and provide
the external IP to for service
• But it is only for Cloud Provider, the bare metal cannot
do this
26

Load Balancer
27
Cloud
LoadBalancer
Pod Pod Pod

Ingress
29
• HTTP Load Balancing
• SSL Termination
• Content-base Routing
• Not fully for external network

Ingress
30
Ingress
nginx.com echoheaders.com
ServiceA ServiceB
10.0.0.1:80

Ingress
31
Ingress
Controller
Ingress
Resource
Load Balancer
watches configure

Ingress
32
kubeProxy
kubeProxy kubeProxy
Pod PodPodPod PodPod
Momo.foo.com
Jojo.foo.com
yoyo.foo.com
hostnetwork

Ingress
33
kubeProxy
kubeProxy kubeProxy
Pod PodPodPod PodPod
DNS

Keep-Alived VIP
• Real High-Availability
• Virtual IP Address
• IP to Service
• Configmap
• DaemonSet
35
Ref :
- https://github.com/kubernetes/contrib/tree/master/keepalived-vip

Keep-Alived VIP
36
Pod PodPod
Keepalived Keepalived Keepalived
140.113.1.1 140.113.1.2 140.113.1.3
Vip: 140.113.1.50

Keep-Alived VIP
37
PodPodPod
140.113.1.1 140.113.1.2 140.113.1.3
Vip: 140.113.1.50
Pod

Keep-Alived VIP
38
PodPodPod
140.113.1.1 140.113.1.2 140.113.1.3
Vip: 140.113.1.50

Keep-Alived VIP + Ingress
40
PodPod Pod
140.113.1.1 140.113.1.2 140.113.1.3
Vip: 140.113.1.50
Momo.foo.com
Jojo.foo.com

Summary
• Self-hosted K8S
▪ Make K8S operations more easier
▪ Bootkube is still a incubator project
▪ Support disaster recovery
• Service with High Availability
▪ Using DNS to provide your service – Ingress
▪ More vip – keepalived-VIP
▪ Using cloud to build your kubernetes – cloud Load Balancer
▪ Test – NodePort
41

Thank you!
macchiang@itri.org.tw
kweisamx.cs05g@g2.nctu.edu.tw
Kubernetes Taiwan User Group

Kubernetes in kubernetes 搭建高可用環境

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Kubernetes in kubernetes 搭建高可用環境

Similar to Kubernetes in kubernetes 搭建高可用環境 (20)

More from inwin stack

More from inwin stack (20)

Recently uploaded

Recently uploaded (20)

Kubernetes in kubernetes 搭建高可用環境