Usually Kubernetes is deployed on top of virtual machines. In this session, we are going to explain how to deploy Kubernetes on top of a physical infrastructure, including compute, networking, and storage services. Metalkube provides infrastructure management integrated directly with Kubernetes, allowing to host workloads that require running on the host directly: machine learning, VNFs for telcos, or for cases like Edge Computing where you need to have all the infrastructure self-hosted but still leverage the high-availability and fault-tolerance of Kubernetes.

1. Kubernetes Native Infrastructure
Managed Baremetal with Kubernetes Operators
and OpenStack Ironic
Yolanda Robla <yroblamo@redhat.com>
Based on slides from Steven Hardy <shardy@redhat.com>
16 May 2019

2. 2
Introduction
 Who am I?
 Engineeer working on OpenStack since 2014
 Regular contributor on OpenStack infra and Ironic
 Currently working in OpenShift for the Edge use cases
 In this talk
 Why do we need Kubernetes on baremetal
 Kubernetes Infrastructure management
 Introducing Metal3

3. Why do we need it?

4. 4
Why Kubernetes on baremetal?
 Highly depends on use cases - not everyone needs it, but some do:
 Data intensive workloads: big data, machine learning
 Low latency: vnfs for telcos - 5G, RAN
 Direct access to hardware: GPUs, hardware accelerators, network cards
 Edge computing: workloads running closer to the customer
 Increase density, lower costs
 Alternatives: deploy Kubernetes on cloud: AWS, OpenStack...

5. Kubernetes Native?

6. 6
Kubernetes Native Infrastructure
 Define infrastructure in terms of kubernetes resources
 Automate everything: operational knowledge -> code
 Declarative, self-hosting infrastructure management
 Leverage existing k8s SIG APIs (kubernetes-sigs/cluster-api)
 Pluggable abstractions, tooling is an implementation detail
 Manage the entire lifecycle of the k8s cluster, and physical
resources (deployment, upgrades, fault recovery…)
 Integrated management of physical infrastructure and infrastructure
applications, e.g storage

7. 7
TERMS
CRD Custom Resource Definition; Schema Extension to Kubernetes
API
CR Custom Resource; One record/instance/object,
conforming to a CRD
OPERATOR Daemon that watches for changes to resources; built from
framework
using the operator-sdk tools
CONTROLLER Object inside the operator responsible for reconciling changes to
the
resource to make reality match the request
ACTUATOR Driver interface within some controllers used to implement custom
logic while sharing common code

8. 8
CONTROLLER RECONCILIATION LOOP
Handling changes to resources
Kubernetes is built around the controller pattern:
● Controllers monitor a system for deviations between the
user-declared target state and reality and take corrective
actions to reconcile reality with the declared target state.
Controller System
feedback
corrective
actiondeclared
target state Δ
Ctrl.
Ctrl.
Ctrl. Infrastructure
Kubernetes Cluster
Applications / Services
“I want 6 machines provisioned with
CoreOS. Make it so!”
“I want Kubernetes upgraded to
version 1.14. Make it so!”
“I want Kubevirt deployed. Make it
so!”
KNI applies this pattern across the whole Stack:

9. Ironic, Baremetal Deployment

10. 10
OpenStack Ironic
https://docs.openstack.org/ironic/latest/
 Baremetal management service for OpenStack BMaaS
 Mature/stable - diverse community and good vendor support
 Pluggable drivers, ipmi, iLO, drac, redfish, …
 Can be easily run standalone with minimal dependencies
 Go bindings are available in gophercloud (new!)
 Support for introspection and node auto-discovery via ironic-
inspector service (optional)
 Can run in standalone mode or integrated with OpenStack
(nova)

11. 11
Ironic
(Standalone)

12. Metal3

13. 13
COMPONENTS INVOLVED IN PROVISIONING
 A baremetal provisioning component (Ironic in our initial implementation)
 Definition of physical node inventory (MACs, IPMI credentials, represented via a CRD +
CRs per each host)
 An image to deploy on the nodes (OS plus k8s components, initial target os CoreOS)
 A baremetal-operator using Ironic to manage physical server hardware and provision
images.
 An actuator that conforms to the cluster-api provider interface that handles Machine
objects by allocating bare metal hosts from the inventory maintained through the
Kubernetes API.

14. 14
Cluster-api (Cloud vs Metal)

15. 15
WHERE DO HOSTS COME FROM?
User provides BMC
Credentials and IP to create
resource using CR
Baremetal-controller uses
credentials to trigger inventory
inspection via Ironic
Baremetal-controller matches
host resource to a profile and
marks host as ready to be
used
HOW DO HOSTS BECOME NODES?
User creates a new Machine
resource
Host controller uses Ironic to
deploy the image specified by
the actuator to disk and
provide ignition config on boot
Machine controller uses our
actuator to “create” the
backend by finding an
unused host

16. 16
---
apiVersion: v1
kind: Secret
metadata:
name: openshift-worker-0-bmc-secret
type: Opaque
data:
username: YWRtaW4=
password: cGFzc3dvcmQ=
---
apiVersion: metalkube.org/v1alpha1
kind: BareMetalHost
metadata:
name: openshift-worker-0
spec:
online: true
bmc:
address: ipmi://192.168.111.1:6233
credentialsName: openshift-worker-0-bmc-secret
bootMACAddress: 00:ab:4f:d8:9e:fa
$ kubectl get baremetalhosts
NAME STATUS PROVISIONING STATUS MACHINE BMC openshift-master-0 OK
externally provisioned ostest-master-0 ipmi://192.168.111.1:6230
$ kubectl apply -f openshift-worker-crs.yaml
$ kubectl get baremetalhosts
NAME STATUS PROVISIONING STATUS MACHINE BMC openshift-master-0 OK
externally provisioned ostest-master-0 ipmi://192.168.111.1:6230
openshift-worker-0 OK ready
$ kubectl get machinesets
NAME DESIRED CURRENT READY AVAILABLE AGE
ostest-worker-0 0 0 24h
$ kubectl scale machineset ostest-worker-0 –replicas=1
$ kubectl get baremetalhosts
NAME STATUS PROVISIONING STATUS MACHINE BMC
openshift-master-0 OK externally provisioned ostest-master-0 ipmi://192.168.111.1:6230
true
openshift-worker-0 OK provisioned ostest-worker-0-jmhtc ipmi://192.168.111.1:6233
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master-0 Ready master 24h v1.13.4+d4ce02c1d
worker-0 Ready worker 68s v1.13.4+d4ce02c1d
LEARNING BY
EXAMPLE...

17. 17
Conclusion
 Infrastructure/application management is converging
 Kubernetes Operators, manage Applications and Infrastructure
 Metal3, new project leveraging OpenStack Ironic and Operators
 Many opportunities for reuse & cross-community collaboration
 Patches welcome! :)
Want to find out more?
● https://kubernetes-sigs.github.io/cluster-api/
● https://github.com/metal3-io/
● https://github.com/openshift-metal3

18. THANK YOU
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitter.com/RedHat