Welcome to our
Get 30% off on the first Ticket!
+ 1 trial day for free!
⇒ Mention the Kubernetes Meetup at the front desk ;-)
Our Host ⇒
Our Portfolio
Professional Services
With our tailor-made consulting services, we
are a reliable partner at every step in building,
implementing, running and automating your
containerized infrastructure.
Training
Our experienced instructors will make sure
that your team has the knowledge and skills
needed to successfully adopt container and
cloud native technologies.
Kubermatic Container Engine
Kubermatic empowers you to easily
deploy, manage and run multiple
Kubernetes clusters in the cloud, in
private data centers or hybrid scenarios.
Loodse & CNCF
Our Conference in 2019
20% OFF
⇒ #CDS19_COMMUNITY
Thanks to our sponsors and speakers.
Enjoy the Meetup!
K8s ClusterAPI
Creating Kubernetes multi clusters
with ClusterAPI
Vienna
toschneck
@toschneck
Tobias Schneck
Founders
Awarded by
Partners
● Subproject of sig-cluster-lifecycle
○ Focus cluster deployment and upgrades
○ Other Projects: kops, kubeadm, minikube, kind
● Repos
○ (old) https://github.com/kubernetes/kube-deploy
○ (recent) https://github.com/kubernetes-sigs/cluster-api
● Main participants:
○ Google, Redhat, Heptio, VMware, Loodse …
● Open Development: Get involved!
Currently Supported Providers
of Open Source Kubermatic Machine Controller
Bare MetalKubeVirt
Open Telekom Cloud
Node Deployment Solution
• Need Ops to scale the cluster
• No generic approach
• Each implementation needs
specific knowledge
• Scaling of new nodes requires
external dependencies
• No generic node auto scaling
• Huge effort to deploy k8s on a
different platform
• State management is complex Node Node Node
Master
Node
Kubernetes Cluster BKubernetes Cluster B
Kubernetes Cluster A
bootstraping CLI
upgrade trigger
Provider Specifc Config
autoscaling
Bootstrapping Machine/Cluster
etcd
Cluster
Machine
MachineSet
MachineDeployment
Cluster API Server
(ext-apiserver)
group: cluster.k8s.io
Controller Manager
Machine Controler
Prov. Config
Provider Machine
Actuator
Master Machine(s) NodeMachine(s)
kubeadm kubeadm
API
Server etcd
Kubernetes Cluster B
Master Machine(s) NodeMachine(s)
Provisions
Manage Cluster
and Machines
- Creates Machines
- Bootstraps Cluster
- Upgrades Cluster
API Server + Controllers can be
provided after bootstrapping
Kubermatic
Machine Controller
kubectl create machines
● Pets vs cattle for nodes
● Implementation of generic auto scaling
possible
● Very similar setup for different provider
● Fast adaptation of new provider
● Managed by Custom Resource Definitions
○ API driven
○ extendable
○ enable 3-rd party tooling,
e.g. generic cluster scaling
Node Node Node
Master
Node
Cluster API
• A "Machine" is the declarative spec for a Node,
as represented in Kubernetes core API.
• After provisioning a new Node matching the
Machine spec is registered.
apiVersion: "machine.k8s.io/v1alpha1"
kind: Machine
metadata:
name: machine1
spec:
metadata:
name: node1
providerConfig:
sshPublicKeys: []
cloudProvider: "hetzner"
cloudProviderSpec:
token: MY-HZ-TOKEN
serverType: "cx11"
datacenter: ""
location: "fsn1"
operatingSystem: "ubuntu"
Watches for new Machine resources
Provision the machine using provider-specific drivers
• Machine joins the cluster
• Kubelet creates the Node resource
Machine
Resources
● Render Userdata
○ CoreOS, Ubuntu, CentOS
● Create instance at cloud provider
○ Apply cloud-init
○ Install Container Runtime, kubelet, kubeadm
○ Use kubeadm + Token to join node
● Match Node to Machine, set ownerRef
// Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}
// Provider exposed all required functions to interact with a cloud provider
type Provider interface {
AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error)
// Validate validates the given machine's specification.
//
// In case of any error a "terminal" error should be set,
// See v1alpha1.MachineStatus for more info
Validate(machinespec v1alpha1.MachineSpec) error
// Get gets a node that is associated with the given machine.
//
// Note that this method can return what we call a "terminal" error,
// which indicates that a manual interaction is required to recover from this state.
// See v1alpha1.MachineStatus for more info and TerminalError type
Get(machine *v1alpha1.Machine) (instance.Instance, error)
GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error)
// Create creates a cloud instance according to the given machine
Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error)
Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error
}
github.com/kubermatic/machine-controller/../hetzner/provider.go
github.com/kubermatic/machine-controller
● Apache2 license
● Contains Machine Controllers for:
AWS, Azure, DigitalOcean, Google Compute Engine, Hetzner,
KubeVirt, Linode, OpenStack, vSphere
kubectl apply -f examples/machine-controller.yaml
kubectl apply -f examples/machine-${my_provider}.yaml
apiVersion: "cluster.k8s.io/v1alpha1"
kind: MachineSet
metadata:
name: << MACHINE_NAME >>
namespace: kube-system
spec:
replicas: 1
selector:
matchLabels:
name: << MACHINE_NAME >>
template:
metadata:
labels:
name: << MACHINE_NAME >>
spec:
providerConfig:
value:
sshPublicKeys:
- "<< YOUR_PUBLIC_KEY >>"
cloudProvider: "vsphere"
cloudProviderSpec:
templateVMName: '<< OS_NAME >>-template'
username: '<< VSPHERE_USERNAME >>'
vsphereURL: '<< VSPHERE_ADDRESS >>'
datacenter: 'Datacenter'
folder: '/Datacenter/vm/e2e-tests'
password: << VSPHERE_PASSWORD >>
cluster: '<< VSPHERE_CLUSTER >>'
datastore: datastore1
allowInsecure: true
cpus: 2
MemoryMB: 2048
operatingSystem: "<< OS_NAME >>"
operatingSystemSpec:
distUpgradeOnBoot: false
containerRuntimeInfo:
name: "<< CONTAINER_RUNTIME >>"
versions:
kubelet: "<< KUBERNETES_VERSION >>"
apiVersion: "cluster.k8s.io/v1alpha1"
kind: MachineDeployment
metadata:
name: << MACHINE_DEPLOYMENT_NAME >>
namespace: kube-system
spec:
paused: false
replicas: 1
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
minReadySeconds: 0
selector:
matchLabels:
name: << MACHINE_NAME >>r
template:
metadata:
labels:
foo: << MACHINE_NAME >>
spec:
providerConfig:
value:
sshPublicKeys:
- "<< YOUR_PUBLIC_KEY >>"
cloudProvider: "vsphere"
cloudProviderSpec:
templateVMName: '<< OS_NAME >>-template'
username: '<< VSPHERE_USERNAME >>'
vsphereURL: '<< VSPHERE_ADDRESS >>'
datacenter: 'Datacenter'
folder: '/Datacenter/vm/e2e-tests'
password: << VSPHERE_PASSWORD >>
cluster: '<< VSPHERE_CLUSTER >>'
datastore: datastore1
allowInsecure: true
cpus: 2
MemoryMB: 2048
operatingSystem: "<< OS_NAME >>"
operatingSystemSpec:
distUpgradeOnBoot: false
containerRuntimeInfo:
name: "<< CONTAINER_RUNTIME >>"
versions:
kubelet: "<< KUBERNETES_VERSION >>"
●
●
●
●
○
○
●
●
●
●
●
●
●
●
●
●
●
ń …

ClusterAPI Overview - Managing multi-cloud Kubernetes Clusters - k8s Meetup@vienna 03-2019

  • 1.
  • 2.
    Get 30% offon the first Ticket! + 1 trial day for free! ⇒ Mention the Kubernetes Meetup at the front desk ;-) Our Host ⇒
  • 3.
    Our Portfolio Professional Services Withour tailor-made consulting services, we are a reliable partner at every step in building, implementing, running and automating your containerized infrastructure. Training Our experienced instructors will make sure that your team has the knowledge and skills needed to successfully adopt container and cloud native technologies. Kubermatic Container Engine Kubermatic empowers you to easily deploy, manage and run multiple Kubernetes clusters in the cloud, in private data centers or hybrid scenarios.
  • 4.
  • 5.
    Our Conference in2019 20% OFF ⇒ #CDS19_COMMUNITY
  • 6.
    Thanks to oursponsors and speakers. Enjoy the Meetup!
  • 7.
    K8s ClusterAPI Creating Kubernetesmulti clusters with ClusterAPI Vienna
  • 9.
  • 10.
  • 13.
    ● Subproject ofsig-cluster-lifecycle ○ Focus cluster deployment and upgrades ○ Other Projects: kops, kubeadm, minikube, kind ● Repos ○ (old) https://github.com/kubernetes/kube-deploy ○ (recent) https://github.com/kubernetes-sigs/cluster-api ● Main participants: ○ Google, Redhat, Heptio, VMware, Loodse … ● Open Development: Get involved!
  • 15.
    Currently Supported Providers ofOpen Source Kubermatic Machine Controller Bare MetalKubeVirt Open Telekom Cloud
  • 16.
  • 18.
    • Need Opsto scale the cluster • No generic approach • Each implementation needs specific knowledge • Scaling of new nodes requires external dependencies • No generic node auto scaling • Huge effort to deploy k8s on a different platform • State management is complex Node Node Node Master Node
  • 19.
    Kubernetes Cluster BKubernetesCluster B Kubernetes Cluster A bootstraping CLI upgrade trigger Provider Specifc Config autoscaling Bootstrapping Machine/Cluster etcd Cluster Machine MachineSet MachineDeployment Cluster API Server (ext-apiserver) group: cluster.k8s.io Controller Manager Machine Controler Prov. Config Provider Machine Actuator Master Machine(s) NodeMachine(s) kubeadm kubeadm API Server etcd Kubernetes Cluster B Master Machine(s) NodeMachine(s) Provisions Manage Cluster and Machines - Creates Machines - Bootstraps Cluster - Upgrades Cluster API Server + Controllers can be provided after bootstrapping Kubermatic Machine Controller
  • 20.
    kubectl create machines ●Pets vs cattle for nodes ● Implementation of generic auto scaling possible ● Very similar setup for different provider ● Fast adaptation of new provider ● Managed by Custom Resource Definitions ○ API driven ○ extendable ○ enable 3-rd party tooling, e.g. generic cluster scaling Node Node Node Master Node Cluster API
  • 21.
    • A "Machine"is the declarative spec for a Node, as represented in Kubernetes core API. • After provisioning a new Node matching the Machine spec is registered.
  • 23.
    apiVersion: "machine.k8s.io/v1alpha1" kind: Machine metadata: name:machine1 spec: metadata: name: node1 providerConfig: sshPublicKeys: [] cloudProvider: "hetzner" cloudProviderSpec: token: MY-HZ-TOKEN serverType: "cx11" datacenter: "" location: "fsn1" operatingSystem: "ubuntu"
  • 24.
    Watches for newMachine resources Provision the machine using provider-specific drivers • Machine joins the cluster • Kubelet creates the Node resource Machine Resources
  • 26.
    ● Render Userdata ○CoreOS, Ubuntu, CentOS ● Create instance at cloud provider ○ Apply cloud-init ○ Install Container Runtime, kubelet, kubeadm ○ Use kubeadm + Token to join node ● Match Node to Machine, set ownerRef
  • 27.
    // Provider exposedall required functions to interact with a cloud provider type Provider interface { AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error) // Validate validates the given machine's specification. // // In case of any error a "terminal" error should be set, // See v1alpha1.MachineStatus for more info Validate(machinespec v1alpha1.MachineSpec) error // Get gets a node that is associated with the given machine. // // Note that this method can return what we call a "terminal" error, // which indicates that a manual interaction is required to recover from this state. // See v1alpha1.MachineStatus for more info and TerminalError type Get(machine *v1alpha1.Machine) (instance.Instance, error) GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error) // Create creates a cloud instance according to the given machine Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error) Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error }
  • 28.
    // Provider exposedall required functions to interact with a cloud provider type Provider interface { AddDefaults(spec v1alpha1.MachineSpec) (v1alpha1.MachineSpec, bool, error) // Validate validates the given machine's specification. // // In case of any error a "terminal" error should be set, // See v1alpha1.MachineStatus for more info Validate(machinespec v1alpha1.MachineSpec) error // Get gets a node that is associated with the given machine. // // Note that this method can return what we call a "terminal" error, // which indicates that a manual interaction is required to recover from this state. // See v1alpha1.MachineStatus for more info and TerminalError type Get(machine *v1alpha1.Machine) (instance.Instance, error) GetCloudConfig(spec v1alpha1.MachineSpec) (config string, name string, err error) // Create creates a cloud instance according to the given machine Create(machine *v1alpha1.Machine, update MachineUpdater, userdata string) (instance.Instance, error) Delete(machine *v1alpha1.Machine, update MachineUpdater, instance instance.Instance) error } github.com/kubermatic/machine-controller/../hetzner/provider.go
  • 29.
    github.com/kubermatic/machine-controller ● Apache2 license ●Contains Machine Controllers for: AWS, Azure, DigitalOcean, Google Compute Engine, Hetzner, KubeVirt, Linode, OpenStack, vSphere kubectl apply -f examples/machine-controller.yaml kubectl apply -f examples/machine-${my_provider}.yaml
  • 30.
    apiVersion: "cluster.k8s.io/v1alpha1" kind: MachineSet metadata: name:<< MACHINE_NAME >> namespace: kube-system spec: replicas: 1 selector: matchLabels: name: << MACHINE_NAME >> template: metadata: labels: name: << MACHINE_NAME >> spec: providerConfig: value: sshPublicKeys: - "<< YOUR_PUBLIC_KEY >>" cloudProvider: "vsphere" cloudProviderSpec: templateVMName: '<< OS_NAME >>-template' username: '<< VSPHERE_USERNAME >>' vsphereURL: '<< VSPHERE_ADDRESS >>' datacenter: 'Datacenter' folder: '/Datacenter/vm/e2e-tests' password: << VSPHERE_PASSWORD >> cluster: '<< VSPHERE_CLUSTER >>' datastore: datastore1 allowInsecure: true cpus: 2 MemoryMB: 2048 operatingSystem: "<< OS_NAME >>" operatingSystemSpec: distUpgradeOnBoot: false containerRuntimeInfo: name: "<< CONTAINER_RUNTIME >>" versions: kubelet: "<< KUBERNETES_VERSION >>"
  • 31.
    apiVersion: "cluster.k8s.io/v1alpha1" kind: MachineDeployment metadata: name:<< MACHINE_DEPLOYMENT_NAME >> namespace: kube-system spec: paused: false replicas: 1 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 minReadySeconds: 0 selector: matchLabels: name: << MACHINE_NAME >>r template: metadata: labels: foo: << MACHINE_NAME >> spec: providerConfig: value: sshPublicKeys: - "<< YOUR_PUBLIC_KEY >>" cloudProvider: "vsphere" cloudProviderSpec: templateVMName: '<< OS_NAME >>-template' username: '<< VSPHERE_USERNAME >>' vsphereURL: '<< VSPHERE_ADDRESS >>' datacenter: 'Datacenter' folder: '/Datacenter/vm/e2e-tests' password: << VSPHERE_PASSWORD >> cluster: '<< VSPHERE_CLUSTER >>' datastore: datastore1 allowInsecure: true cpus: 2 MemoryMB: 2048 operatingSystem: "<< OS_NAME >>" operatingSystemSpec: distUpgradeOnBoot: false containerRuntimeInfo: name: "<< CONTAINER_RUNTIME >>" versions: kubelet: "<< KUBERNETES_VERSION >>"
  • 33.
  • 34.
  • 36.