If you want to provide Kubernetes on your bare-metal infrastructure you need to configure and manage multiple components, like PKI, DHCP, iPXE and some more. Deploying your infrastructure as immutable without SSH access will confront you with some additional challenges, because traditional configuration management tools like Puppet or Ansible require SSH access to each node. Within the scope of this workshop we will present you a practicable way to manage your bare-metal Kubernetes clusters by the use of a CI/CD-driven approach, including Gitlab CI, Terraform, Confd, Vault and some others. As an extra we will keep a fully functional local development setup ready, so you can gain real hands-on experience. The whole configuration is written in code (Infrastructure as Code – IaC) and will automatically update all components the moment you update your configuration. Der komplette Workshop ist hier zu finden: https://github.com/inovex/fluffy-unicorn Speaker: Johannes M. Scheuermann Event: ContainerDays 2018 Mehr Tech-Vorträge: https://www.inovex.de/de/content-pool/vortraege/ Mehr Tech-Artikel: https://www.inovex.de/blog/

1. Manage your bare-metal infrastructure with a
CI/CD-driven approach
Johannes M. Scheuermann
inovex

2. Johannes M. Scheuermann
Cloud Platform Engineer @ inovex
〉 Software-Defined Datacenters
〉 Infrastructure Services
〉 High Availability /Scalability / Resilience
〉 CKA / Linux Foundation Trainer
2
inovex.de/jobs

3. What about you?

4. 4
Agenda
● Immutable infrastructure
● Kubernetes on bare-metal
● Big Picture
● CI/CD for the infrastructure
● Example Gitlab (CI)

5. 5
Exercises
Can be found here:
● Github Link: https://github.com/inovex/fluffy-unicorn

6. 6
What I assume
● You know what Kubernetes is
● You know the basic components of Kubernetes
● You already did some bare-metal installation
● ..?

7. What is immutable infrastructure?

8. 8
“Traditional” infrastructure
● Continuous updates/modification
● Machines get changed after they are created
● Often configuration management is used
● Typically ssh access is required
○ LDAP/Kerberos ...
● Static monitoring

9. An immutable infrastructure is another infrastructure
paradigm in which servers are never modified after
they're deployed. If something needs to be updated,
fixed, or modified in any way, new servers built from a
common image with the appropriate changes are
provisioned to replace the old ones. After they're
validated, they're put into use and the old ones are
decommissioned.
9
https://www.digitalocean.com/community/tutorials/what-is-immutable-infrastruc
ture

10. 10
Immutable infrastructure
● “The” “Golden” image is built
● Never modified after they are deployed
● Update means a new deployment
○ Before you should update the image :)
● No need for ssh access
● Dynamic monitoring
○ Service Discovery needed

11. 11
Building the “golden” image
● Probably most famous: https://www.packer.io
● Create provider specific images
● Multiple provisioners
● Clean up after provisioning
● Use the distro-specific tooling
● OS image as Code
○ Packer configuration
○ OS configuration

12. 12
Challenges
● Image must be kept up-to-date
○ CVE’s
○ CI/CD for the image
● New images must be constantly deployed
● External configuration is needed in most cases
○ Certificates
○ Configuration for different stages
○ dev-prod-parity

13. 13
What we ended up
● Using vanilla Container Linux as base
● Ignition as provisioning tool
○ Only executed once at boot time
● Consul-template + confd for configuration files
● Static certificates are baked into ignition
○ e.g. for secure access to the etcd CMDB

14. 14
Recap immutable infrastructure
● Vanilla Container Linux ✅
● ignition is only executed once during boot ✅
● Updates means new deployment ❌
○ Until now we didn’t deployed anything :)
● No need for ssh access ✅

15. Kubernetes on bare-metal

16. 16
Bare-metal deployments
● “Many roads lead to Rome”
● Actually a (more or less) solved problem
○ OpenStack
○ Foreman
○ Maas
○ Matchbox
○ ….

17. 17
What do we need?
● DHCP/proxyDHCP
● PXE/iPXE
● TFTP
● OS image
● OS configuration
● Role-based configuration (e.g. Master/Worker)

18. 18
Container Linux bare-metal deployment
● Bare-metal deployment with the default toolstack
○ DHCP / TFTP / DNS
● CoreOS Matchbox
○ iPXE Service
○ Deliver assets based on predefined roles
● CoreOS ignition
○ Configure servers on boot-time

19. Source: https://github.com/coreos/matchbox/blob/master/Documentation/network-booting.md#ipxe 19
How does the Setup work

20. Time estimation : 10-20 minutes 20
Exercise 1: Setup Matchbox + PXE boot
● What we do:
○ Setup Wifi/Network (for your laptop)
○ Download: https://github.com/inovex/fluffy-unicorn
○ Setup Matchbox + terraform
○ Setup DHCP
○ Setup DNS
○ Provision 2 Container Linux nodes
○ Look into the general flow
● Example is under /exercise_1

21. 21
Recap: Kubernetes on bare-metal
● Bare-metal provisioning of OS ✅
● OS configuration with ignition ✅
● Always PXE boot ❌
● Role-based configuration ❌
● Kubernetes cluster installed ❌
● Deployment with a pipeline ❌

22. 22
Challenge always PXE boot
● Matchbox is a pretty simple rendering engine
● Doesn’t hold any state
○ e.g. machine X is provisioned
● How to solve this issue?
○ Any ideas?

23. 23
Challenge always PXE boot (our solution)
● No access to IPMI
● External DB that holds the state
○ In our case an etcd cluster
● Machine sets its state to “deployed”
● DHCP server decides what to do
○ Provision from Matchbox
○ Boot from disk
● DHCP config is generated with confd

24. 24
Why etcd?
● Simple to cluster (HA)
● Allows to watch events
● Confd for configuration generation
● Can be used as Vault backend
● … ?

25. Time estimation : 10-20 minutes 25
Exercise 2: Always PXE boot
● What we do (additional):
○ Setup etcd
○ Setup confd
○ Generate DHCP config based on Machine state
● Example is under /exercise_2

26. 26
Recap: Kubernetes on bare-metal
● Bare-metal provisioning of OS ✅
● OS configuration with ignition ✅
● Always PXE boot ✅
● Role-based configuration ❌
● Kubernetes cluster installed ❌
● Deployment with a pipeline ❌

27. Source: https://github.com/coreos/matchbox/blob/master/Documentation/getting-started.md#groups 27
Role-based configuration
● Matchbox groups
● Selectors can be used
○ MAC
○ label (e.g. os=installed)
○ UUID
● Semantic AND (no list support)
○ https://github.com/coreos/matchbox/issues/586

28. Source: https://github.com/coreos/matchbox/blob/master/Documentation/getting-started.md#groups 28
Role-based configuration

29. Time estimation : 10-20 minutes 29
Exercise 3: Role-based provisioning
● What we do (additional):
○ Setup two groups
■ based on the MAC
○ Setup write a file on the host (/etc/my_type)
■ MAC -> content
■ 52:54:00:fb:53:a6 -> “I’m a master”
■ 52:54:00:fb:53:a9 -> “I’m a worker”
● Example is under /exercise_3

30. 30
Recap: Kubernetes on bare-metal
● Bare-metal provisioning of OS ✅
● OS configuration with ignition ✅
● Always PXE boot ✅
● Role-based configuration ✅
● Kubernetes cluster installed ❌
● Deployment with a pipeline ❌

31. Kubernetes what do we need?

32. Read: https://github.com/kelseyhightower/kubernetes-the-hard-way 32
K8s - what do we need?
● etcd cluster
● At least one Master
● Worker nodes
● Certificates for TLS
○ PKI
○ Or manually generated
● Load Balancer in a multi master setup
○ e.g. https://metallb.universe.tf or HAProxy/Nginx/...

33. Read: https://jvns.ca/blog/2017/08/05/how-kubernetes-certificates-work 33
Kubernetes/etcd PKI
● PKI is also needed for the etcd cluster
○ Use TLS everywhere!
● Self-generated certificates
○ Hard to manage the certificates
○ Certificates must be “copied” onto the machines
● Central PKI
○ Auto-generated Certificates
○ Auto-renewal
○ Revocation list

34. Read: https://www.vaultproject.io/docs/secrets/pki/index.html 34
Vault as central PKI
● Simple REST API
● Allows to create multiple CA’s
● Based on policies allow or deny certificate requests
● Can be combined with consul_tempalte
○ Auto-renew certificates (short TTL)

35. Read: https://www.vaultproject.io/docs/auth/approle.html 35
Vault AppRoles
● Authentication mechanism for machines or
applications
● Create AppRole ID and SecretID
○ similar to username + password
● Can be used to fetch an Vault token

36. 36
Big Picture
IPAM
Asset
Mgmt.
Asset
Mgmt.
Asset
Mgmt.etcd
DHCP/
TFTP
Matchbox
confd
Asset
Mgmt.
Asset
Mgmt.Node
Vault
1
2
3 4
5

37. Time estimation : 30-45 minutes 37
Exercise 4: Deploy your K8s cluster
● What we do (additional):
○ Setup Vault
○ Setup consul_template
○ Setup Kubernetes
● Example is under /exercise_4

38. 38
Recap: Kubernetes on bare-metal
● Bare-metal provisioning of OS ✅
● OS configuration with ignition ✅
● Always PXE boot ✅
● Role-based configuration ✅
● Kubernetes cluster installed ✅
● Deployment with a pipeline ❌

39. Who knows what CI/CD is?

40. 40
Continuous integration
● Push into you branch and get Feedback
● Merge changes fast into the master
● Automated build is triggered
● Test run automatically after each push
● Fail fast in the pipeline
● Avoid integration hell → “worked on my machine”

41. 41
Continuous delivery
● Same as continuous integration
● Allow you to release changes quickly
● Release is completely automated
● Deploy is only clicking one button

42. 42
Continuous deployment
● Same as continuous delivery
● Every change that passes all stages will be deployed
● No human interaction needed

43. 43
CI/CD for the infrastructure
● More or less the same as with SE
● Lint/check your configuration files
○ shellcheck / hadolint / …
○ kubeval / kubetest
● Test any transformation steps
● Spin up test infrastructure (virtualized)
● Possibility to update your infrastructure by one click
○ Incremental changes have some benefits

44. https://about.gitlab.com/ 44
Gitlab (CI)
● Free git repository
○ Can be self-hosted
● Integrated CI
○ Different types of runners
○ Supports building pipelines
○ Specified as a YAML file

45. Time estimation : 30-45 minutes 45
Exercise 5: Deploy your K8s cluster over CI/CD
● What we do (additional):
○ Install Gitlab → https://about.gitlab.com/installation/#debian
○ Install a Gitlab runner
○ Create a new Repository and push all files to it
○ Create a dummy pipeline
○ Replace some of the makefile steps
● Example is under /exercise_5

46. 46
Recap: Kubernetes on bare-metal
● Bare-metal provisioning of OS ✅
● OS configuration with ignition ✅
● Always PXE boot ✅
● Role-based configuration ✅
● Kubernetes cluster installed ✅
● Deployment with a pipeline ✅

47. Q&A

48. Johannes M.
Scheuermann
inovex GmbH
jscheuermann@inovex.de
github.com/johcsheuer inovex.de youtube.com/inovexGmbH
CC BY-NC-ND @johscheuer gitlab.com/inovex
Feedback & Questions:
sayat.me/johscheuer

49. 49
Reading list
● https://www.digitalocean.com/community/tutorials/what-is-immutable-
infrastructure
● https://www.oreilly.com/ideas/an-introduction-to-immutable-infrastruc
ture
● https://martinfowler.com/bliki/PhoenixServer.html
● https://maas.io
● https://www.theforeman.org
● https://wiki.openstack.org/wiki/Ironic
● https://github.com/coreos/matchbox
● https://cfssl.org

50. 50
Reading list
● https://blog.digitalocean.com/vault-and-kubernetes
● https://kubernetes.io/docs/concepts/cluster-administration/certificates
● https://github.com/hashicorp/consul-template
● https://about.gitlab.com