Many people associate kubernetes with cloud solutions exclusively. I tend to disagree. In this presentation I will explain the rationale and success criteria of this project. I will also describe in details our key findings, tools that we used along the way, incident management process and the benefits analysis. The presentation will also cover all lessons learned as well as pitfalls and drawbacks of this solution. Major incidents will be explained along with the correctional procedures. Moreover this talk will also be focused on the cost effectiveness of the solution in question. No minikube demos, no raspberry pi clusters, no obvious basics. Just the real world application of the tech that works and generates revenue.

1. Kubernetes
on bare metal
after a year in production
MICHAŁ AMBROZIEWICZ, NETSPRINT
@ZATHR November 2019, Warsaw
www.devopsdays.pl

2. 2About me
● Java, Scala
● Devops
● Sci - fi
● Music
● DIY
● Single-family housing

3. 3
MONEY!

4. 4
Why?!

5. 5
MONEY!

6. 6
How much money?
Typical “work horse”:
● CPU: 4 cores / 8 threads
● RAM: 32 GB +
Hetzner OVH AWS AWS (spot) GCE GCE
(preemtible)
Model EX42-NVME Rise-1 t3a.2xlarge t3a.2xlarge n2-standard-8 n2-standard-8
Price € 39,00 $ 235,16 $ 64,94 $ 249,56 $75,49
Price PLN 169,26 249,73 923,58 255,22 980,77 296,67

7. 7
Other costs
• Cluster management (AWS) $144
• Storage
• Network transfer

8. 8
Breaking news
AX41-NVME
• AMD Ryzen 5 3600
• 64 GB DDR4 RAM
• 2 x 512 GB NVME
€ 39,00

9. 9
Pros and cons
• Physical storage
• Hardware control
• No “noisy neighbours”
• Money
• No network storage “out
of the box”
• Hardware and OS
management
• Less elasticity
• No autoscaling

10. 10
Disclaimer
Your case may differ in terms of:
• scale
• workload characteristics
• security concerns

11. 11
The plan

12. 12
Prerequisites
• Experience with dedicated servers
• Experience with vendors like Hetzner and OVH

13. 13
Cluster specs
● 5-7 worker nodes
● 3 master nodes
● Tinted master nodes
● Full monitoring
● Production ready in 6 months

14. 14
Pilot app specs
● Stateless components
● Loose coupling
● HTTP/HTTPS
● Databases outside the cluster
● No autoscaling

15. 15
Security
● Cluster access
○ ssh only for adm/ops
○ local kubectl for others
● API server access control
○ individual token for each person and service
○ token inventory
○ token rotation
○ RBAC
● App security
○ HTTPS
○ ingress
○ exposed services inventory

16. 16
Teams
vs.

17. 17
Teams
Dev
• Pilot app
• CI/CD pipeline
• Ingress config
Ops
• Cluster setup
• Cluster administration
• Procedures

18. 18
Procedures
• Administrative
• Maintenance
• Disaster recovery

19. 19
Training
• Sandbox
• minikube
• physical machines
• Different approach for dev and adm

20. 20
Problems
Challenges

21. 21
Networking and firewall
• VLANs
• VPN/overlay network
• Mangle table

22. 22
Middle layer

23. 23
Proxmox

24. 24
Distributed storage
• Ceph
• Proxmox

25. 25
Logging
• Default
• ELK
• Other way?

26. 26
Staging/production separation
• No separation (a.k.a. YOLO)
• Separate nodes
• Separate clusters

27. 27
Incidents

28. 28
Hardware faults
• Cooling problems
• Power outage
• Disk failure
• Faulty RAM

29. 29
Network issues
• Package loss
• Total outage
• Split brain
• Internal traffic leak

30. 30
Ceph issues
• Repair hickup

31. 31
Proxmox issues
• Automatic update
• VM restart
• Service exposition

32. 32
Bonus: No limits!
• Overprovisioning
• What can go wrong?

33. 33
Future
• Better logging infrastructure
• Distributed tracing
• Stateful apps

34. 34
Summary

35. 35
Questions?

36. MICHAŁ AMBROZIEWICZ, NETSPRINT
@ZATHR
I'm waiting for
your feedback!
You can rate speakers and lectures
using our official conference app