Charlie Drage discussed Kubernetes on bare metal at last week's Kubernetes and Cloud Native meetup in Kitchener-Waterloo. His presentation demonstrated how to deploy Kubernetes on bare metal servers. Charlie is an active Kubernetes maintainer, and his contributions have included fixing some common issues with bare metal servers and using Ansible to build clusters with kubedm.
2. I work on the Developer Tools team at Red Hat
I deal with *a lot* of Kubernetes
I maintain Kompose (Docker Compose to Kubernetes
tool)
I’m frugal and I don’t like using paid Kubernetes services
I work on OpenShift tools (project called Odo)
(short) Introduction
15. Why Wikipedia created a Kubernetes infrastructure
(summary)
- Kubernetes is so good that it only takes 4 people to manage the entire
infrastructure
- Super versatile
- Containers! Containers! Containers!
- Single-node failure management
18. It’s *so* easy to setup a cluster
(if it’s paid for…)
- Using Kops or KubeSpray
kops create cluster
--node-count=2
--node-size=t2.medium
--zones=us-east-1a
--name=${KOPS_CLUSTER_NAME}
- Using Google Kubernetes Engine
gcloud container clusters create
- Using any other paid services
(DigitalOcean, IBM Cloud, Oracle, etc…)
The above will happen if you provide Kubernetes as a
Service
19. Everything is taken care of with the Clouuudddddd
They take of this for you:
● Deployment
● Volumes
● LoadBalancing
● Ingress
● Logging and monitoring
● Automatic Cluster Scaling
● Node Auto-Repair
You pay them so they’ll take care
of the above for you.
23. Easy since 2017!
- Before kubeadm it was a pain in the butt. Now it’s painless!
- Want to know how it used to be? Setup using Kubernetes the Hard Way
(https://github.com/kelseyhightower/kubernetes-the-hard-way)
- Networking sucked before CNI (Container Network Interface) now we can
choose between Flannel, Calico, Canal, etc. without having to worry about
networking
33. Extreme laziness
- Using Ansible!
- https://github.com/kairen/kubeadm-ansible
- As long as you have either CentOS, Fedora, Ubuntu or
Debian it will do it all for you
46. Volumes on Bare Metal
- Volumes provide dynamic storage for containers
- SO MANY OPTIONS TO CHOOSE FROM! (26 options)
- For a home cluster, you’d go for either nfs or hostPath (mounting directly onto the cluster)
- But even after setup… why can’t I dynamically create volumes? Well, only certain ones are
setup for that. Most being Cloud services.
- We’ve got Dynamic NFS Volumes https://github.com/kubernetes-incubator/external-storage
47. Volumes: Install
# On an NFS host
$ docker run
-d
--restart=always
--net=host
--name nfs
--privileged
-v /mnt/storage/k8s:/nfsshare
-e SHARED_DIRECTORY=/nfsshare
cdrage/nfs-server-alpine
# Install nfs support on each node
$ sudo apt-get install nfs-common -y
# Finally, we setup the volumes!
$ helm install stable/nfs-client-provisioner -n nfs-client --set nfs.server=192.168.1.91 --set nfs.path=/
--set storageClass.defaultClass=true
48. Volumes: Usage
$ kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE
data-loopy-hydra-mariadb-0 Bound pvc-ad2d3724-edce-11e8-895e-52540046b08b 8Gi RWO nfs-client 7d
data-wordpress-mariadb-0 Bound pvc-81aeb087-edd1-11e8-895e-52540046b08b 8Gi RWO nfs-client 7d
wordpress-wordpress Bound pvc-81a56a8e-edd1-11e8-895e-52540046b08b 10Gi RWO nfs-client 7d
~
$ kubectl get pv
NAME CAPACITY ACCESSMODES RECLAIMPOLICY STATUS CLAIM STORAGECLASS REASON AGE
pvc-81a56a8e-edd1-11e8-895e-52540046b08b 10Gi RWO Delete Bound default/wordpress-wordpress nfs-client 7d
pvc-81aeb087-edd1-11e8-895e-52540046b08b 8Gi RWO Delete Bound default/data-wordpress-mariadb-0 nfs-client 7d
pvc-ad2d3724-edce-11e8-895e-52540046b08b 8Gi RWO Delete Bound default/data-loopy-hydra-mariadb-0 nfs-client 7d
49. LoadBalancing on Bare Metal
- LoadBalancing assigns an IP Address (ideally a public one) to a service
- If not, you’re forced to use an Ingress, NodePort or ClusterIP (internal IP) instead.
- Really only one option, and that’s MetalLB (https://github.com/google/metallb)
- Uses local IPs (or optionally BGP routers) to distribute IP Addresses
- Seems complicated, but it’s super easy to setup
51. LoadBalancing: Usage
$ kubectl get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes 10.96.0.1 <none> 443/TCP 22d
wordpress-mariadb 10.103.71.121 <none> 3306/TCP 7d
wordpress-wordpress 10.99.189.46 192.168.1.98 80:30295/TCP,443:31509/TCP 7d
52. Ingress on Bare Metal
- Ingress exposes https and http traffic routes
- Kubernetes acts as a master port 80/443 HTTP server and routes traffic
- Most popular implementation is kubernetes/nginx-ingress
59. Automatic Cluster Scaling on Bare Metal
- Haha
- There’s
https://github.com/kubernetes/autoscaler
with support for only cloud providers.
- Please update issue #1060 for me when you
push a PR, it’s been inactive since July, thanks!
60. Node Auto Repair on Bare Metal
- Haha x2
- Nope! But there’s support for it!
- I swear, there is actually support for this
67. We’re getting there! (slowly)
● We’ve got: kubeadm, kubespray, kops with bare metal support to make it easier for us
● Kubernetes has been modularizing / splitting off parts of the ecosystem
● We’ve got Kubernetes SIGs (Special Interest Groups) adding new projects all the time
● Maintainers added support for bare-metal! For example, kops added bare-metal support when
I requested it, but it was then subsequently dropped in favour for kubeadm..
● Ansible is (sometimes) a decent solution for setting up baremetal
● Components are slowly coming out of beta / alpha (nfs AutoProvisioner, MetalLB)