The slides briefly describe a way of operating kubernetes on various custom environment, such as bare-metal, on-prem and more

1. Getting serious w private
kubernetes clusters &
cloud-native storage
Running kubernetes everywhere with all inclusive

2. About Me
● Name: David Steiman aka. xetys
● Owner of github.com/xetys/hetzner-kube
● Twitter: @theOnlyScrippi
● GitHub: github.com/xetys
● Blog: stytex.de
● Working @ K-TEL Communications
● JHipster core developer

3. Motivation

4. Motivation
1. Custom machine providers
2. Own “cloud”
a. IaaS (Block/Object Storage, TLS Certs, VMs, networks)
b. FaaS (OpenFaaS, Kubeless, …)
c. PaaS (fabric8, deis, …)
d. CI/CD (jx, drone, Gitlab)
3. Cost-reduction

5. Three base stands of private clusters

6. Kubernetes deployment
strategies
...the hard way?

7. CoreOS
● Pure container philosophy
● PXE boot provisioning clusters with
matchbox
● Enterprise support with Tectonic
● Good for large clusters with thousands of
nodes

8. Ansible powered and driven
Kismatic Enterprise Toolkit
● Powered by an ansible playbook, extended
with go
● No “real” HA support
● Persistent storage with GlusterFS
out-of-the-box
Kube-spray
● Full ansible based
● Large feature base
○ HA support
○ Self-hosted
○ Many Linux distros
● kubernetes-incubator project

9. Rancher 1.x & Rancher 2.0
Rancher 1.x
● Focussed on Cattle
● k8s as catalog app
● Most easy install
● Least correct install
● User Support!
Rancher 2.0 / RKE
● Focussed on k8s
● Real HA mode
● Yet, quite simple install
● User support
● Early and little UI

10. kubeadm
● In-tree component of k8s
● Quite simple install
● Ready to production
● Simplifies ugly parts

12. hetzner-kube
● Go tool for deploying k8s on hetzner cloud
● Uses kubeadm under the hood
● Ships default with flannel
● Bundles addons like helm, ingress, cert-manager, kube-prometheus, OpenEBS, rook
● E2e suite incoming

13. hetzner-kube High Availability
● External etcd cluster
● Decentralized apiserver proxy using nginx
● Tested with evil tools like comcast

15. Network & Load Balancing

16. type: LoadBalancer?
● Most commonly not available to private clusters
● Exception: Rancher 1.X with cloud-provider Rancher
● Should be realized using --cloud-provider=<custom>

17. nginx ingress controller on edge nodes
● Label nodes as edge routers
● Deploy nginx-ingress-controller with
nodeSelector
● Multiple A-Records per domain

18. Node networking
● Encrypted private network VPN tool
“Wireguard”
● https://github.com/xetys/wgctl
○ Operate wireguard networks

19. Pod Networking
● Simple setup: flannel, weave
● Simple + Pod Network policies: canal
● Integrate with existing network infrastructure: calico
● Alternative approaches: kube-router

21. Persistence

22. Needs
● Storage Class support
● High Availability & Fault resistence
● High Performance in Throughput & IOPS
● RWO + RWX
● Backup tools

23. Kubernetes driven solutions
● OpenEBS
○ Presented by previous speaker
● Rook
○ Leverages ceph as backing storage cluster
○ Simplifies ceph operation via CRD
● GlusterFS
○ Supports RWX ootb
○ Supports Storage Class with heketi

24. Dedicated Ceph cluster
● Manageable in operation
● One ceph for several clusters
● Storage Class Support with RWX, and object storage from kubernetes-incubator/external-storage

26. Thanks! Questions?