Thanks to tools like kubeadm, Terraform or Ansible setting up a Kubernetes cluster on a dedicated environment is getting reachable, but what’s about setting up a bunch of cluster in multiple clouds in automatic way? This is still a challenge. Also if you want to do same in your own datacenter. In this talk we will take a look to the approach to orchestrate and manage a whole set of k8s cluster by the Cluster API project of kubernetes (a subproject of sig-cluster-lifecycle). The main idea behind it is to use the Kubernetes API itself to manage multiple clusters with there master and worker nodes in same way you would manage your PODs - define the needed resources and the responsible controller will take care for providing it.
After an overview about the concepts of cluster API, I will show what’s needed to implement a cluster API conform machine class/deployment. There I will see that adding your own provider isn’t that hard as you may aspect. At the end of the day it just requires a simple interface to implement. The corresponding kubermatic controllers we implemented at Loodse are available as open source, so its possible to play around with it. A live demo will show how easy it is to spin up and maintain multiple Kubernetes cluster at different public and on-premise cloud providers over one managing cluster. A final wrap up will summarize the current state of the Cluster API project and the advantages of managing clusters as cattles instead of pets.
2. K8s Cluster API Folie 2
Who I am? Tobi
Who are we working for? Loodse GmbH
Kubernetes?! Short recap
Kubermatic?! What’s that?
Node Deployment Solutions The old way
Cluster API! Make K8s even greater
Hetzner Machine! Implementation of Machine Provider
Demo time! Praise the demo gods!
Review! State, Pro and Cons
Agenda
4. Sebastian Scheele
Headquarter in
in Hamburg, Germany
Founders
Awarded by
> 25 employees
world wide
Founded in 2016
Julian Hansert
Cloud native
and container
technologies
Software
development
K8s certified
Multi & Hybrid Cloud
Container Engine Partners
Silver Member
& Training Partner
GCP & Training
Partner
12. K8s Cluster API Folie 12
• Need Ops to scale the cluster
• No generic approach
• Each implementation needs
specific knowledge
• Scaling of new nodes requires
external dependencies
• No generic node auto scaling
• Huge effort to deploy k8s on a
different platform
Node Node Node
Master
Node
Current Approach
13. K8s Cluster API Folie 13
Kubernetes Cluster BKubernetes Cluster B
Kubernetes Cluster A
Cluster API
Infrastructure Providers
Cloud, On-Prem, Bare Metal
Potential Providers =>
bootstraping CLI
upgrade trigger
Provider Specifc Config
autoscaling
Bootstrapping Machine/Cluster
etcd
Cluster
Machine
MachineSet
Cluster API Server
(ext-apiserver)
group: cluster.,k8s.io
Controller Manager
Machine Controler
Prov. Config
Provider Machine
Actuator
Master Machine(s) NodeMachine(s)
kubeadm kubeadm
API
Server etcd
Kubernetes Cluster B
Master Machine(s) NodeMachine(s)
Provisions
Manage Cluster
and Machines
- Creates Machines
- Bootstraps Cluster
- Upgrades Cluster
API Server + Controllers can be
provided after bootstrapping
Kubermatic
Machine Controller
14. K8s Cluster API Folie 14
kubectl create machines
• Generic cluster scaling
• Pets vs cattle for nodes
• Implementation of generic auto
scaling possible
• Very similar setup for different
provider
Node Node Node
Master
Node
Kubernetes Native Integration
Cluster API
15. K8s Cluster API Folie 15
• A "Machine" is the declarative spec for a Node,
as represented in Kubernetes core.
• After provisioning a new Node matching the
Machine spec is registered.
Machines/Nodes
17. K8s Cluster API Folie 17
How does a machine look like?
apiVersion: "machine.k8s.io/v1alpha1"
kind: Machine
metadata:
name: machine1
spec:
metadata:
name: node1
providerConfig:
sshPublicKeys: []
cloudProvider: "hetzner"
cloudProviderSpec:
token: MY-HZ-TOKEN
serverType: "cx11"
datacenter: ""
location: "fsn1"
operatingSystem: "ubuntu"
18. K8s Cluster API Folie 18
MachineController
Watches for new Machine resources
Provision the machine using provider-specific drivers
• Machine joins the cluster
• Kubelet creates the Node resource
Machine
Resources
20. K8s Cluster API Folie 20
How does it work?
● Render Userdata
○ CoreOS, Ubuntu & CentOS
● Create instance at Cloudprovider
○ Install Container Runtime, Kubelet, Kubeadm
○ Use Kubeadm + Token to join node
● Match Node to Machine, set ownerRef
21. K8s Cluster API Folie 21
How to implement a new provider?
// Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}
22. K8s Cluster API Folie 22
How to implement a new provider?
// Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}
github.com/kubermatic/machine-controller/../hetzner/provider.go
23. K8s Cluster API Folie 23
Where to get it and how to use it?
github.com/kubermatic/machine-controller
● Apache2 license
● Contains Machine Controllers for:
AWS, Azure, DigitalOcean, Hetzner, OpenStack, vSphere
kubectl apply -f examples/machine-controller.yaml
kubectl apply -f examples/machine-${my_provider}.yaml
26. K8s Cluster API Folie 26
ClusterAPI - Quo Vadis?
● Currently in Alpha
● Types do change in a backwards-incompatible way
● Some questions do not have a final answer yet
● Does not solve all problems, e.G. Backup & Restore
● Already allows easy decoupling of general applicable from provider-specific code
● Generic cluster-autoscaler possible
● Participation from Google & RedHat
27. K8s Cluster API Folie 27
Cluster API <> Kubermatic
Kubermatic is based on Cluster API and Machine Controllers and adds:
● Kubermatic Dashboard
● Backup & Restore Concept
● Cluster Upgrade Mechanism
● Secure Communication between Bootstrapping Cluster and User Cluster
● Control of different on-prem/public Cloud Provider over one Master Plane
● Prometheus Meta Monitoring and Alerting
● ELK Stack for Infrastructure
● User Management (v2.8)
Try it yourself!
(Only AWS, Digital Ocean enabled)
https://cloud.kubermatic.io
