Thanks to tools like kubeadm, Terraform or Ansible setting up a Kubernetes cluster on a dedicated environment is getting reachable, but what’s about setting up a bunch of cluster in multiple clouds in automatic way? This is still a challenge. Also if you want to do same in your own datacenter. In this talk we will take a look to the approach to orchestrate and manage a whole set of k8s cluster by the Cluster API project of kubernetes (a sub-project of sig-cluster-lifecycle). The main idea behind it is to use the Kubernetes API itself to manage multiple clusters with there master and worker nodes in same way you would manage your PODs - define the needed resources and the responsible controller will take care for providing it. After an overview about the concepts of cluster API, I will show what’s needed to implement a cluster API conform machine class/deployment. There I will see that adding your own provider isn’t that hard as you may aspect. At the end of the day it just requires a simple interface to implement. The corresponding Kubermatic machine-controller we implemented at Loodse are available as open source, so its possible to play around with it. A live demo will show how easy it is to spin up and maintain multiple Kubernetes cluster at different public and on-premise cloud providers over one managing cluster. A final wrap up will summarize the current state of the Cluster API project and the advantages of managing clusters with CRDs and Controllers instead of stateful scripts.

1. Welcome to our

2. Get 30% off on the first Ticket!
+ 1 trial day for free!
⇒ Mention the Kubernetes Meetup at the front desk ;-)
Our Host ⇒

3. Our Portfolio
Professional Services
With our tailor-made consulting services, we
are a reliable partner at every step in building,
implementing, running and automating your
containerized infrastructure.
Training
Our experienced instructors will make sure
that your team has the knowledge and skills
needed to successfully adopt container and
cloud native technologies.
Kubermatic Container Engine
Kubermatic empowers you to easily
deploy, manage and run multiple
Kubernetes clusters in the cloud, in
private data centers or hybrid scenarios.

4. Loodse & CNCF

5. Our Conference in 2019
20% OFF
⇒ #CDS19_COMMUNITY

6. Thanks to our sponsors and speakers.
Enjoy the Meetup!

7. K8s ClusterAPI
Creating Kubernetes multi clusters
with ClusterAPI
Vienna

9. toschneck
@toschneck
Tobias Schneck

10. Founders
Awarded by
Partners

13. ● Subproject of sig-cluster-lifecycle
○ Focus cluster deployment and upgrades
○ Other Projects: kops, kubeadm, minikube, kind
● Repos
○ (old) https://github.com/kubernetes/kube-deploy
○ (recent) https://github.com/kubernetes-sigs/cluster-api
● Main participants:
○ Google, Redhat, Heptio, VMware, Loodse …
● Open Development: Get involved!

15. Currently Supported Providers
of Open Source Kubermatic Machine Controller
Bare MetalKubeVirt
Open Telekom Cloud

16. Node Deployment Solution

18. • Need Ops to scale the cluster
• No generic approach
• Each implementation needs
specific knowledge
• Scaling of new nodes requires
external dependencies
• No generic node auto scaling
• Huge effort to deploy k8s on a
different platform
• State management is complex Node Node Node
Master
Node

19. Kubernetes Cluster BKubernetes Cluster B
Kubernetes Cluster A
bootstraping CLI
upgrade trigger
Provider Specifc Config
autoscaling
Bootstrapping Machine/Cluster
etcd
Cluster
Machine
MachineSet
MachineDeployment
Cluster API Server
(ext-apiserver)
group: cluster.k8s.io
Controller Manager
Machine Controler
Prov. Config
Provider Machine
Actuator
Master Machine(s) NodeMachine(s)
kubeadm kubeadm
API
Server etcd
Kubernetes Cluster B
Master Machine(s) NodeMachine(s)
Provisions
Manage Cluster
and Machines
- Creates Machines
- Bootstraps Cluster
- Upgrades Cluster
API Server + Controllers can be
provided after bootstrapping
Kubermatic
Machine Controller

20. kubectl create machines
● Pets vs cattle for nodes
● Implementation of generic auto scaling
possible
● Very similar setup for different provider
● Fast adaptation of new provider
● Managed by Custom Resource Definitions
○ API driven
○ extendable
○ enable 3-rd party tooling,
e.g. generic cluster scaling
Node Node Node
Master
Node
Cluster API

21. • A "Machine" is the declarative spec for a Node,
as represented in Kubernetes core API.
• After provisioning a new Node matching the
Machine spec is registered.

23. apiVersion: "machine.k8s.io/v1alpha1"
kind: Machine
metadata:
name: machine1
spec:
metadata:
name: node1
providerConfig:
sshPublicKeys: []
cloudProvider: "hetzner"
cloudProviderSpec:
token: MY-HZ-TOKEN
serverType: "cx11"
datacenter: ""
location: "fsn1"
operatingSystem: "ubuntu"

24. Watches for new Machine resources
Provision the machine using provider-specific drivers
• Machine joins the cluster
• Kubelet creates the Node resource
Machine
Resources

26. ● Render Userdata
○ CoreOS, Ubuntu, CentOS
● Create instance at cloud provider
○ Apply cloud-init
○ Install Container Runtime, kubelet, kubeadm
○ Use kubeadm + Token to join node
● Match Node to Machine, set ownerRef

27. // Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}

28. // Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}
github.com/kubermatic/machine-controller/../hetzner/provider.go

29. github.com/kubermatic/machine-controller
● Apache2 license
● Contains Machine Controllers for:
AWS, Azure, DigitalOcean, Google Compute Engine, Hetzner,
KubeVirt, Linode, OpenStack, vSphere
kubectl apply -f examples/machine-controller.yaml
kubectl apply -f examples/machine-${my_provider}.yaml

30. apiVersion: "cluster.k8s.io/v1alpha1"
kind: MachineSet
metadata:
name: << MACHINE_NAME >>
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
name: << MACHINE_NAME >>
template:
metadata:
labels:
name: << MACHINE_NAME >>
spec:
providerConfig:
value:
sshPublicKeys:
- "<< YOUR_PUBLIC_KEY >>"
cloudProvider: "vsphere"
cloudProviderSpec:
templateVMName: '<< OS_NAME >>-template'
username: '<< VSPHERE_USERNAME >>'
vsphereURL: '<< VSPHERE_ADDRESS >>'
datacenter: 'Datacenter'
folder: '/Datacenter/vm/e2e-tests'
password: << VSPHERE_PASSWORD >>
cluster: '<< VSPHERE_CLUSTER >>'
datastore: datastore1
allowInsecure: true
cpus: 2
MemoryMB: 2048
operatingSystem: "<< OS_NAME >>"
operatingSystemSpec:
distUpgradeOnBoot: false
containerRuntimeInfo:
name: "<< CONTAINER_RUNTIME >>"
versions:
kubelet: "<< KUBERNETES_VERSION >>"

31. apiVersion: "cluster.k8s.io/v1alpha1"
kind: MachineDeployment
metadata:
name: << MACHINE_DEPLOYMENT_NAME >>
namespace: kube-system
spec:
paused: false
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
minReadySeconds: 0
selector:
matchLabels:
name: << MACHINE_NAME >>r
template:
metadata:
labels:
foo: << MACHINE_NAME >>
spec:
providerConfig:
value:
sshPublicKeys:
- "<< YOUR_PUBLIC_KEY >>"
cloudProvider: "vsphere"
cloudProviderSpec:
templateVMName: '<< OS_NAME >>-template'
username: '<< VSPHERE_USERNAME >>'
vsphereURL: '<< VSPHERE_ADDRESS >>'
datacenter: 'Datacenter'
folder: '/Datacenter/vm/e2e-tests'
password: << VSPHERE_PASSWORD >>
cluster: '<< VSPHERE_CLUSTER >>'
datastore: datastore1
allowInsecure: true
cpus: 2
MemoryMB: 2048
operatingSystem: "<< OS_NAME >>"
operatingSystemSpec:
distUpgradeOnBoot: false
containerRuntimeInfo:
name: "<< CONTAINER_RUNTIME >>"
versions:
kubelet: "<< KUBERNETES_VERSION >>"

33. ●
●
●
●
○
○
●
●
●

34. ●
●
●
●
●
●
●
●

36. ń …