There are several motivations for running kubernetes in custom environments, such as bare-metal, on-premise or in public clouds other than AWS, GKE, and ACS. Operating kubernetes on this kind of infrastructure in production is a challenge. It's not just the initial setup of the cluster itself. Load balancing, monitoring, and persistent storage are crucial for running kubernetes without outages. I will talk about various solutions for each of these problems and will present my project "hetzner-kube", a tool for kubernetes operations on Hetzner Cloud.

1. Getting serious with private kubernetes clusters &
cloud-native storage
David Steiman
Berlin | November 20 - 21, 2018

2. About Me
➢ Name: David Steiman aka. xetys
➢ Owner of github.com/xetys/hetzner-kube
➢ Twitter: @theOnlyScrippi
➢ GitHub: github.com/xetys
➢ Blog: stytex.de
➢ Working @ K-TEL Communications
➢ JHipster core developer

3. Private cluster = ?
➢ Bare-metal
➢ Virtual instances on own infrastructure
➢ Custom cloud provider
➢ Raspberry PI?

4. Motivation

5. Motivation
1. Custom machine providers
2. Own “cloud”
a. IaaS (Block/Object Storage, TLS Certs, VMs, networks)
b. FaaS (OpenFaaS, Kubeless, …)
c. PaaS (fabric8, deis, …)
d. CI/CD (jx, drone, Gitlab)
3. Cost-reduction
a. 82 cores, 328GB RAM, 4TB of resilient storage => ~500 EUR
b. Similar setup at AWS ~12k EUR
4. No GAWA?

6. Three base stands of private clusters
Compute
Network Storage

7. Kubernetes
deployment
strategies
...the hard way?

8. CoreOS
➢ Pure container philosophy
➢ PXE boot provisioning
clusters with matchbox
➢ Enterprise support with
Tectonic
➢ Good for large clusters with
thousands of nodes

9. Ansible powered and driven
Kismatic Enterprise Toolkit
➢ Powered by an ansible
playbook, extended with go
➢ No “real” HA support
➢ Persistent storage with
GlusterFS out-of-the-box
Kube-spray
➢ Full ansible based
➢ Large feature base
○ HA support
○ Self-hosted
○ Many Linux distros
➢ kubernetes-incubator project

10. Rancher 1.x & Rancher 2.0
Rancher 1.x
➢ Focussed on Cattle
➢ k8s as catalog app
➢ Most easy install
➢ Least correct install
➢ User Support!
Rancher 2.0 / RKE
➢ Focussed on k8s
➢ Real HA mode
➢ Yet, quite simple install
➢ User support
➢ Early and little UI

11. kubeadm
➢ In-tree component of k8s
➢ Quite simple install
➢ Ready to production
➢ Simplifies ugly parts

13. hetzner-kube
➢ Go tool for deploying k8s on hetzner cloud
➢ Uses kubeadm under the hood
➢ Ships default with flannel
➢ Bundles addons like helm, ingress, cert-manager, kube-
prometheus, OpenEBS, rook
➢ E2e tested

14. hetzner-kube High Availability
➢ External etcd cluster
➢ Decentralized apiserver proxy using nginx
➢ Tested with evil tools like comcast
○ and Falkenstein DC outtake

16. Network & Load Balancing

17. type: LoadBalancer?
➢ Most commonly not available to private clusters
➢ Exception: Rancher 1.X with cloud-provider Rancher
➢ Should be realized using --cloud-provider=<custom>
➢ MetalLB

18. nginx ingress controller on edge
nodes
➢ Label nodes as edge routers
➢ Deploy nginx-ingress-
controller with
nodeSelector
➢ Multiple A-Records per
domain

19. Node networking
➢ Encrypted private network VPN
tool “Wireguard”
➢ https://github.com/xetys/wgctl
○ Operate wireguard networks

20. Pod Networking
➢ Simple setup: flannel, weave
➢ Simple + Pod Network policies: canal
➢ Integrate with existing network infrastructure: calico
➢ Alternative approaches: kube-router

22. Persistence

23. Needs
➢ Storage Class support
➢ High Availability & Fault resistance
➢ High Performance in Throughput & IOPS
➢ RWO + RWX
➢ Backup tools

24. Kubernetes driven solutions
➢ OpenEBS
○ Uses containers for every storage
○ Uses iSCSI
➢ Rook
○ Leverages ceph as backing storage cluster
○ Simplifies ceph operation via CRD
➢ GlusterFS
○ Supports RWX ootb
○ Supports Storage Class with heketi

25. Dedicated Ceph cluster
➢ Manageable in operation
➢ One ceph for several clusters
➢ Storage Class Support with RWX, and object storage
from kubernetes-incubator/external-storage

27. Demo Time

28. Thanks! Questions?