Julien Girardin presents metal-k8s, an opinionated Kubernetes distribution designed for bare-metal deployments. Julien explains why we chose certain Kubespray plugins over others for Zenko's needs of scalability and petabyte-scale storage over multiple public and private clouds.

1. featuring
Julien Girardin
R&D Engineer @Scality
Kubernetes Meetup Paris
May 31st, 2018
Metal-k8s:
an opinionated Kubernetes distribution
Zenko

2. Agenda
- Intro: Who is Scality?
- A brief history: transitioning (Zenko) from Docker Swarm to Kubernetes
- Intro to Metal-K8s: why bare-metal?
- Zoom in: Opinionated choices of plugins
- Demo Metal-K8s deployment
- Demo Zenko deployment
- Q&A

3. Who is Scality?
Intro

4. FOR ANY STANDARD x86 SERVER PLATFORM
100% SOFTWARE SOLUTION
PRIVATE CLOUD PUBLIC CLOUDS
……
MULTI-CLOUD DATA CONTROLLER
TO ACCESS AND MANAGE DATA ACROSS CLOUDS
object & file storage in a single system · peer to peer architecture
· unlimited scalability · unbounded scale-out performance · most
adaptive set of robust data protection mechanisms · autonomous
self-healing · designed in close collaboration with the biggest
(cloud-scale) service providers in the world
a single, unified API across all clouds to simplify application
development · the only multi-cloud data management solution
independent of the storage system · stores data in standard cloud
format to make the data consumable directly by native cloud apps and
services · true multi-cloud IT · global search across all managed data
independent of cloud location
……… … …

5. An opinionated Kubernetes distribution
with a focus on long-term on-prem
deployments
- S3 connector (RING product) is based on Docker
- Swarm was the logical extension
- Docker Swarm was THE solution to orchestrate Docker over a cluster in
2016
Why Docker/Docker Swarm ?

6. Why Kubernetes ?
- Run everywhere, on any cloud => provide API abstraction
- Control plane run server side (compared to docker compose)
- Self-healing
- Auto-scaling (of pods, of cluster, of resources requests)
- Huge set of plugins (centralised logging, monitoring, ingress)
- Big community
- Docker announcement to support Kubernetes in 2017
- Customers trust and want it

7. Kubernetes
source https://x-team.com/blog/introduction-kubernetes-architecture/

8. Kubernetes
source https://x-team.com/blog/introduction-kubernetes-architecture/
1) Add new
deployment.yaml

9. Kubernetes
source https://x-team.com/blog/introduction-kubernetes-architecture/
2.a) API OK,
store to ectd
2.b) deployment ↓

10. Kubernetes
source https://x-team.com/blog/introduction-kubernetes-architecture/
3.b) Job for me ?
YES !
3.a) New object
might interest
controller
manager ↓

11. Kubernetes
source https://x-team.com/blog/introduction-kubernetes-architecture/
4.a) It seems that 3
more pods are
needed
4.b) Ok for 3 pods,
store in etcd
4.c) pods * 3 ↓

12. Kubernetes
source https://x-team.com/blog/introduction-kubernetes-architecture/
5.a) This, will
interest ↓scheduler
5.b) Job for me ?
YES !

13. Kubernetes
source https://x-team.com/blog/introduction-kubernetes-architecture/
6.a) Computation:
- 2 pods for first node
- 1 pod for second node
6.b) Affectation OK,
store in etcd
6.c) The list of
pod of each
node is
updated

14. Kubernetes
source https://x-team.com/blog/introduction-kubernetes-architecture/
7.a) Nodes:
your list has
changed
7.b) Jobs for me ?
I have 2 pods to run
7.b) Jobs for me ?
I have 1 pod to run

15. An opinionated Kubernetes distribution with
a focus on long-term on-prem deployments
A commitment to bare-metal
Metal-k8s
● KOPS
● EKS (one day ?)
AWS
● GKE
GCP
● AKS
Azure
● ?
Bare
Metal

16. What is Bare-Metal for us ?
North
Pole
Network ?
VS

17. Metal-k8s
LOAD-BALANCER
Control-Plane/M
aster
Data-Plane/
Nodes
Outside World
etcd, kube-apiserver,
kube-controller-manager,
kube-scheduler, docker,
kubelet, kube-proxy
docker, kubelet,
kube-proxy

18. Deep Dive

19. 1) Inventory precheck
2) ping (connectivity check)
3) precheck about centos (kernel)
4) precheck on storage
5) create lvm vg/lv
6) => call kubespray ! <=
7) register lv into kubespray
8) deploy nginx ingress
9) deploy prometheus + grafana
10) deploy elasticsearch + kibana
Metal-k8s: quality
Inventory
netcd
% 2 = 1
nmaster
> 1
nnode
> 0
Connectivity

20. Docker ?

21. Docker ?
YES

22. Docker !
YES:
● We have experience
● Customer have experience
○ But kubernetes only supports 1.11.2 to
1.13.1 and 17.03.x
■ We decided on 17.03.x… stable but old

23. 1) Inventory precheck
2) ping (connectivity check)
3) precheck about centos (kernel)
4) precheck on storage
5) create lvm vg/lv
6) => call kubespray ! <=
7) register lv into kubespray
8) deploy nginx ingress
9) deploy prometheus + grafana
10) deploy elasticsearch + kibana
Metal-k8s: quality
Storage driver
⇕
OS release
⇕
kernel version

24. Btrfs/Zfs AUFS DEVICE-
MAPPER
OVERLAY OVERLAY2
Require disk
formatting
Not supported by
kernel anymore
since 3.18
Warning : bad
performance for
loopback-lvm
Run out of
inode easily
Require
disabling
selinux, require
Centos7.4
(kernel
3.10.0-693)
Docker storage driver = overlay2
- Best performance/stability with less requirements
- With docker < 18.02, detection over kernel capabilities for overlay2 is
buggy (require force storage driver for docker 17.03)
- Educated bet on future
Docker Storage Driver: which and why?

25. 1) Inventory precheck
2) ping (connectivity check)
3) precheck about centos (kernel)
4) precheck on storage
5) create lvm vg/lv
6) => call kubespray ! <=
7) register lv into kubespray
8) deploy nginx ingress
9) deploy prometheus + grafana
10) deploy elasticsearch + kibana
Metal-k8s: light local storage

26. Efficient storage for small to medium clusters
Centos 7 LVM
LVM
Pre-provisioning of all LV → no need for monitoring of host resources
Dynamic provisioning? Waiting for native Kubernetes support
Local Storage

27. 1) Inventory precheck
2) ping (connectivity check)
3) precheck about centos (kernel)
4) precheck on storage
5) create lvm vg/lv
6) => call kubespray ! <=
7) register lv into kubespray
8) deploy nginx ingress
9) deploy prometheus + grafana
10) deploy elasticsearch + kibana
Metal-k8s: light local storage

28. Ingress: Best friend of an S3 backend
Ingress:
- http routing
- ssl termination
PodPod Pod Pod
USERSIngress: “Map HTTP outside world to kubernetes services”
- Routing based on virtual hostname, path
- Could automatically get certificate (letsencrypt,
Internal CA) with cert-manager
Nginx ingress deployed as a daemonset with hostPort

29. 1) Inventory precheck
2) ping (connectivity check)
3) precheck about centos (kernel)
4) precheck on storage
5) create lvm vg/lv
6) => call kubespray ! <=
7) register lv into kubespray
8) deploy nginx ingress
9) deploy prometheus + grafana
10) deploy elasticsearch + kibana
Metal-k8s: monitoring at scale

30. AWS GKE Kubespray Metal-k8s
Logs X-ray Stackdriver - Fluentd - ES -
Kibana
Metrology Cloudwatch Stackdriver - Prometheus -
Grafana
Storage EBS GCB - Local drive
Monitoring: a matter of choice

31. 1) Inventory precheck
2) ping (connectivity check)
3) precheck about centos (kernel)
4) precheck on storage
5) create lvm vg/lv
6) => call kubespray ! <=
7) register lv into kubespray
8) deploy nginx ingress
9) deploy prometheus
10) deploy elasticsearch
Metal-k8s: a Kubespray backbone
- Opensource
- Battle tested

32. Metal-k8s: How to keep up
- Follow kubespray PR, Contribute
- Follow Storage SIG, and other SIG
- CI for install path, upgrade path

33. Metal-k8s: mais au fait pourquoi?

35. A metal-k8s deployment
A Zenko over metal-k8s deployment
Demos
https://github.com/scality/metal-k8s
https://github.com/Zempashi/metal-k8s-meetup
https://github.com/scality/Zenko

36. An opinionated Kubernetes distribution
with a focus on long-term on-prem
deployments
Too shy?
https://forum.zenko.io
zenko@scality.com
@zenko
@GiorgioRegni
@eikke
Q&A