Container Orchestration 之爭已經落幕，Kubernetes 成為主流，AWS, Azure 跟 GCP 都已提出相對應的解決方案， 但該選擇廠商所提供的服務或是自己架設呢？如何把 Stateless 甚至是 Stateful 應用服務運行於其上呢？部署應用程式到 Kubernetes 之中該如何做比較好？本分享談及多次在公司導入及維運 Kubernetes 的相關經驗，讓有興趣或是剛使用的人可以減少摸索的時間

1. Something About . . .
Operation/Maintenance
Kubernetes

2. Hello!
I am Tom Tsai aka smalltown
- MaiCoin Site Reliability Engineer
- Taipei HashiCorp User Group
Organizer
- AWS User Group Taiwan Staff
2

3. “
3
http://bit.ly/taipei-hug

4. “ Infrastructure as Code
Terraform ❤ AWS EKS
4

5. “ Have You Used Kubernetes
in Production Environment?
5

6. Outline
◂ Where is Your Kubernetes?
◂ High Availability/Scalability
◂ Manage Stateful Service
◂ Log/Monitoring
◂ Continuous Delivery
◂ Your Best Friend: kubectl
◂ Troubleshooting Guide
◂ Q&A
6

7. 1.
Where is Your
Kubernetes?
AWS, Azure, GEK, IDC?

8. Select Where Your Kubernetes is!
8
Public Cloud
No
Yes
Completed
Network Infra.
General Usage
Yes
Yes
No
Shutdown
Suddenly
No Yes
No

9. Secure Your Kubernetes Cluster
No Matter Where it is
9
Private Network
- Kubernetes
- Database
- NAT Gateway
- ...
Public Network
- Load Balancer
- Linux Gateway
- ...
Non-Employee Only
Access Product Service
Through Load Balancer

10. When Should I Self-Hosted Kubernetes
◂ You Should Avoid It! Why?
◂ Unless Company Policy Not Allow
◂ And XKS Really Not Fulfill Your
Requirements (X= A, E, G...)
10

11. 1.
High
Availability/Scalability
Failure is Normal and
Inevitable

12. High Level Design
12
Ref

13. High Availability: Kubernetes Itself
13
- Key/Value Storage - apiserver
- scheduler
- controller-manager
- kubelet
- kube-proxy
- container runtime- HA: 1, 3, 5 …
- Clustering Guide
- General HA Mode

14. What About K8S
Inside?
14

15. 15
Ref

16. 16
Ref

17. High Availability: Application in K8S
◂ Kubernetes Deployments/StatefulSets
Handle it
17
~$ kubectl get deployment
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
Nginx 3 3 3 3 18s

18. Before Talk About
Scalability . . .
18

19. K8S Resource Request & Limit
19
Ref

20. Scalability: Horizontal Pod Autoscaler
20
Ref

21. Scalability: Vertical Pods Autoscaler
21
Ref

22. Scalability: Cluster Autoscaler
22
Ref

23. 1.
Manage
Stateful Service
Running Stateful Service in
Kubernetes Easily

24. Stateful Service
◂ Replication Setup, e.g. MongoDB, MySQL
◂ Cluster Management, e.g. Etcd, Kafka,
ElasticSearch
◂ Scale Out, Scale Up
◂ Upgrade
◂ Data Backup
◂ Other Operation Job...
24

25. Manage Production Service
25
Stateless Stateful
Setup
Scale Out/Up
Backup
Upgrade
Operation

26. CoreOS Brings
K8S Operator . . .
26

27. What is Kubernetes Operator?
◂ An Operator is a Software with Specific
Application Knowledge
◂ Extends the Kubernetes through Custom
Controller/Resource
◂ Enable Users to Create, Configure, and
Manage Applications
27

28. What is Kubernetes Operator?
28
An Operator is a Domain Specific Controller

29. Operation of Stateless Service
29
kubectl
ReplicaSet
app=web, env=prod
Pod
app=web
env=prod
scale up
desired=3 count=1
kubectl
ReplicaSet
app=web, env=prod
Pod
app=web
env=prod
desired=3 count=3

30. Operation of Stateful Service
(Operator)
30
backup
kubectl
etcd Operator
app=3.0.13
size=3
Pod
app=web
env=prod
backup-tool
interval=300s

31. Exist Operators
◂ There is Several Operators Currently
◂ Kubernetes Summit Sharing (Demo)
31

32. 1.
Log/Monitoring
Know Service Health at Any
Time

33. Log Management
. . .
33

34. Log Management: Yes, ELKF
◂ EK: Deploy By Operator
◂ Use Docker Image Without X-Pack
◂ docker.elastic.co/elasticsearch/elastics
earch-oss:6.3.1
34

35. Log Collect
Centralised V.S. Decentralised
◂ Official Suggestion
◂ Ship Log By Each Node
◂ Ship Log By Each Pod
35

36. System/Application
Metric Collect . . .
36

37. Metrics: Yes, Prometheus
◂ Deploy by Operator
37

38. Get Alert When
Service . . .
38

39. Monitoring: Internal
◂ Kubernetes Liveness and Readiness
Probes Cover Auto-Healing
◂ Prometheus Operator Include All
Kubernetes Resource Monitor
39
Ref

40. Monitor: External
◂ Who Can Monitor Prometheus?
◂ Suggest Find Solution to Monitor From
Outside
◂ Access From Different Country
◂ Example: Site24x7, Monitis, ...etc
40

41. 1.
Continuous Delivery
Huge Benefits, But
Challenges Too

42. Common Resource View of K8S Application
42
Deployment YAML
ConfigMap YAML Secret YAML/VaultDocker Image Tag
Service YAML

43. Deploy With kubectl
43
Ref

44. The Key Man: Helm
◂ The Package Manager for Kubernetes
◂ Template Engine + Dependency
Management
◂ Jenkins X and GitOps also Integrate With
It
44

45. Deploy With Helm
45
Ref

46. Good Old Days V.S. Kubernetes
46
GNU/Linux
ELF Binaries Config in /etc
apt, yum, etc
Chef, Puppet, Ansible
Kubernetes
Images K8S Object
Helm
Operator Framework

47. CI/CD Total Solution For K8S
◂ Jenkins X
◂ GitLab
◂ GitOps
◂ Spinnaker
◂ ...
47

48. GitOps Patterns
48
Ref

49. 1.
TroubleShooting
Guide
Where’s Wally?

50. Official TroubleShooting Guide
◂ For Application Debug
◂ Pod, Replication Controller, Service
◂ For Cluster Debug
◂ Master, Node
50

51. Just My Two Cents: Pod
~$ kubectl describe pod #{Pod Name}
◂ Pending: No Node Available, Persistent
Volume Issue
◂ Terminating: Container Cannot be Killed
Normally, Login Node to Check Docker
◂ CrashLoopBackOff: No Docker Image
Exist, or Find Why Container Start Fail
51

52. Just My Two Cents: Service
~$ kubectl describe svc #{SVC Name}
◂ Endpoint is Empty: No Pod is Selected
For This Service
◂ NodePort Conflict With Other Service
◂ Some Modification Need To Delete Then
Re-Create
◂ Ingress Problem
52

53. Just My Two Cents: Node Register
◂ Overlay Network Issue Lead to Register
Fail
◂ Firewall Rule Between Node and Master
◂ ...
◂ Anyway, Login Node to Find Which
Component Encounter Trouble
53

54. 1.
Your Best
Friend: kubectl
Good Craftsmanship Depends
on Use of The Right Tools

55. When You Manage N Clusters N
Namespaces
◂ You Want to Perform Operation For B
Pod in D Cluster U Namespace
◂ Switch Kubeconfig to D Cluster
◂ Switch to U Namspace
◂ Start to Perform Operation For B Pod
◂ Takes Many Steps, Don’t Mention How To
Know In The Right Place?
55

56. There Are Several Helper Tool For You
◂ Setup Kubeconfig For Multiple Cluster
◂ Enable Shell Completion For kubectl
◂ Install kubectx to Switch Cluster or
Namespace
◂ Install kube-ps1 to Know Current Cluster
and Namespace From Terminal
◂ More Helper Tools
56

57. 57
Thanks!
Any questions?
You can find me at
◂ facebook.com/smalltown0110