Based on Fabian’s talk, where we learned how to setup a Kubernetes cluster at Hetzner for small environments, Tobias and Alvaro will take a look to the approach to orchestrate and manage one or more cluster by the Cluster API project (a subproject of sig-cluster-lifecycle). The main idea behind it is to use the Kubernetes API itself to manage multiple clusters with there master and worker nodes in same way you would manage your PODs - define the needed resources and the responsible controller will take care for providing it. After an overview about the concepts of cluster API, we will show what’s needed to implement a cluster API conform machine class/deployment on top of the Hetzner Cloud API. There we will see that adding your own provider isn’t that hard as you may aspect. At the end of the day it just requires a simple interface to implement. The corresponding controller is available as open source, so its possible to play around with it. At least a final demo will show how easy it is to spin up and maintain multiple Kubernetes cluster in Hetzner (or other cloud/on-premise providers) over one managing seed cluster. A final wrap up will summarize the current state of the Cluster API project and the advantages of managing clusters as cattles instead of pets.

1. K8s ClusterAPIManaging the infrastructure of
Kubernetes Cluster
Munich

2. K8s Cluster API Folie 2
Who are we? Alvaro & Tobi
Who are we working for? Loodse GmbH
Kubernetes?! Short recap
Kubermatic?! What’s that?
Node Deployment Solutions The old way
Cluster API! Make K8s even greater
Hetzner Machine! Implementation of Hetzner Machine Provider
Demo time! Praise the demo gods!
Review! State, Pro and Cons
Agenda

3. toschneck
@toschneck
Tobias Schneck
&
alvaroaleman
@alvs_versteck
Alvaro Aleman

4. Founders
Awarded by
Partners

5. K8s Cluster API Folie 5
Kubernetes

6. K8s Cluster API Folie 6
Kubernetes - a short recap

7. K8s Cluster API Folie 7
Kubernetes SIG - Cluster API
● Subproject of sig-cluster-lifecycle
○ Focus cluster deployment and upgrades
○ Other Projects: kops, kubeadm, minikube
● Repos
○ (old, in migration) https://github.com/kubernetes/kube-deploy
○ (new) https://github.com/kubernetes-sigs/cluster-api
● Main participants:
○ Google, Redhat, Heptio, VMware, Loodse …
● Open Development: Get involved!

8. K8s Cluster API Folie 8

9. K8s Cluster API Folie 9
Currently Supported Providers
Bare Metal

10. Node Deployment Solution

11. Why we
implemented
Cluster API?

12. K8s Cluster API Folie 12
• Need Ops to scale the cluster
• No generic approach
• Each implementation needs
specific knowledge
• Scaling of new nodes requires
external dependencies
• No generic node auto scaling
• Huge effort to deploy k8s on a
different platform
Node Node Node
Master
Node
Current Approach

13. K8s Cluster API Folie 13
Kubernetes Cluster BKubernetes Cluster B
Kubernetes Cluster A
Cluster API
Infrastructure Providers
Cloud, On-Prem, Bare Metal
Potential Providers =>
bootstraping CLI
upgrade trigger
Provider Specifc Config
autoscaling
Bootstrapping Machine/Cluster
etcd
Cluster
Machine
MachineSet
Cluster API Server
(ext-apiserver)
group: cluster.,k8s.io
Controller Manager
Machine Controler
Prov. Config
Provider Machine
Actuator
Master Machine(s) NodeMachine(s)
kubeadm kubeadm
API
Server etcd
Kubernetes Cluster B
Master Machine(s) NodeMachine(s)
Provisions
Manage Cluster
and Machines
- Creates Machines
- Bootstraps Cluster
- Upgrades Cluster
API Server + Controllers can be
provided after bootstrapping
Kubermatic
Machine Controller

14. K8s Cluster API Folie 14
kubectl create machines
• Generic cluster scaling
• Pets vs cattle for nodes
• Implementation of generic auto
scaling possible
• Very similar setup for different
provider
Node Node Node
Master
Node
Kubernetes Native Integration
Cluster API

15. K8s Cluster API Folie 15
• A "Machine" is the declarative spec for a Node,
as represented in Kubernetes core.
• After provisioning a new Node matching the
Machine spec is registered.
Machines/Nodes

16. Let’s take a look
at the Code!

17. K8s Cluster API Folie 17
How does a machine look like?
apiVersion: "machine.k8s.io/v1alpha1"
kind: Machine
metadata:
name: machine1
spec:
metadata:
name: node1
providerConfig:
sshPublicKeys: []
cloudProvider: "hetzner"
cloudProviderSpec:
token: MY-HZ-TOKEN
serverType: "cx11"
datacenter: ""
location: "fsn1"
operatingSystem: "ubuntu"

18. K8s Cluster API Folie 18
MachineController
Watches for new Machine resources
Provision the machine using provider-specific drivers
• Machine joins the cluster
• Kubelet creates the Node resource
Machine
Resources

19. K8s Cluster API Folie 19
How does it work?

20. K8s Cluster API Folie 20
How does it work?
● Render Userdata
○ CoreOS, Ubuntu & CentOS
● Create instance at Cloudprovider
○ Install Container Runtime, Kubelet, Kubeadm
○ Use Kubeadm + Token to join node
● Match Node to Machine, set ownerRef

21. K8s Cluster API Folie 21
How to implement a new provider?
// Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}

22. K8s Cluster API Folie 22
How to implement a new provider?
// Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}
github.com/kubermatic/machine-controller/../hetzner/provider.go

23. K8s Cluster API Folie 23
Where to get it and how to use it?
github.com/kubermatic/machine-controller
● Apache2 license
● Contains Machine Controllers for:
AWS, Azure, DigitalOcean, Hetzner, OpenStack, vSphere
kubectl apply -f examples/machine-controller.yaml
kubectl apply -f examples/machine-${my_provider}.yaml

24. DEMO TIME

25. K8s Cluster API Folie 25
Pro & Cons Hetzner
Pros:
● Dead cheap
● Very fast
Cons:
● Must use encrypted overlay
● Users need to be conscious about what they do
● Must use custom solution for storage
● No “Real Loadbalancing”, just floating IPs

26. K8s Cluster API Folie 26
ClusterAPI - Quo Vadis?
● Currently in Alpha
● Types do change in a backwards-incompatible way
● Some questions do not have a final answer yet
● Does not solve all problems, e.G. Backup & Restore
● Already allows easy decoupling of general applicable from provider-specific code
● Generic cluster-autoscaler possible
● Participation from Google & RedHat

27. K8s Cluster API Folie 27
Cluster API <> Kubermatic
Kubermatic is based on Cluster API and Machine Controllers and adds:
● Kubermatic Dashboard
● Backup & Restore Concept
● Cluster Upgrade Mechanism
● Secure Communication between Bootstrapping Cluster and User Cluster
● Control of different on-prem/public Cloud Provider over one Master Plane
● Prometheus Meta Monitoring and Alerting
● ELK Stack for Infrastructure
Try it yourself!
(Only AWS, Digital Ocean enabled)
https://cloud.kubermatic.io

28. Questions?

29. We are hiring!
Hamburg, Berlin, Munich, San Francisco, Gdańsk, …
www.loodse.com
@Loodse