OSDC 2018 | Self Hosted bare Metal Kubernetes for SMEs by Thomas Hoppe

1Footer
Self Hosted Bare Metal Kubernetes for SMEs
Thomas Hoppe
v1.0

2Self Hosted Bare Metal Kubernetes for SMEs
Who We Are
● n-fuse GmbH
● Since 2001
● ~25 Engineers
● Branches in Stuttgart, Germany and Riga, Latvia
● Accompanying our clients from idea to production
● In hard- and software projects
● Customers ranging from startups to top 10 DAX
www.n-fuse.co

Outline of this Talk
●
What is Kubernetes and why to use it?
●
Installation process and caveats
●
Architecture of our environment
●
Monitoring, backup
●
How we use it, CI/CD
●
Outlook
...and a lot of real world experiences sprinkled over it

What is Kubernetes?
● Open source container orchestrator that automates scale out of container
operations
– Docker in most cases
● Focuses completely on containers and is itself built as a set of containers
● Now maintained by the CNCF which is in-turn a member of the Linux
Foundation

Why to Use K8s?
● All the benfits of containers
● Very good fit for µ-services
● Low overhead
● The ecosystem

How to set up K8s? Just some Ways
●
Developer desktop
– Minikube
●
Turnkey Hosted Kubernetes
– Google Kubernetes Engine
– AWS EKS
– Azure Container Kubernetes Service
●
Custom
– Using IaaS (AWS, Google Computing Engine, Azure)
●
Using Tools like Kubespray, kops
– On Premise/ Bare Metal
●
Using Tectonic, Rancher, SuSE CaaS etc.
●
From Scratch on CoreOS  our pick

CoreOS
●
Aka Container Linux
●
Very minimal Linux distro optimized
– Uses Gentoo’s build system :)
●
Heavy user of Systemd
●
3 release channels (alpha, beta, stable)
●
Automatic updates built-in
– Using two partition strategy like ChromeOS for fault resilience
●
Redhat’s takeover of CoreOS should not interfere as they announced to
keep it as is
– If you don’t trust them, try flatcar Linux, a “friendly fork”

Bare Metal Provisioning of CoreOS with Matchbox
●
Using Matchbox provision the rest of the CoreOS fleet
– Toolbelt to install a whole cluster through network boot (using PXE etc.)
– Running one a “bootstrapping machine” to serve DHCP, images and configs
●
All provisioning configs are described in JSON

Our K8s Environment – Logical View

Our K8s Environment – Physical View

Physical Facts
●
Located in DC of a housing provider in Frankfurt
– Latency matters!
●
Redundant 10 GbE Fibre Network
●
4-Node Server á
– 12 Cores
– 64 GB RAM
– 1x 800 GB DC grade SSD
– 2x 4 TB spinner’s

Networking
●
VLANs and interface bonding on physical level
●
Flannel for K8s/container subnets
●
KubeDNS for service discovery
●
Planned upgrades
– Networking with Project Calico
– Service discovery with CoreDNS

Support of K8s
●
Documentation available online, but doesn’t cover specific configuration
●
Bare metal installation docs aren’t up-to-date
– CoreOS had a installation guide for bare metal, but from one day to the other it was replaced
by a Tectonic installation guide
●
Support channels
– Slack: too crowded; hard to find what you need; nobody really answers when you ask
question
– IRC: land of bots (mostly); no reaction to questions at all; no history at all
– Mailing list: haven’t tried to write ourselves; at least searchable
– GitHub: here you can find proof that you’re not the only one with your problem; some issues
just change milestones, but don’t get resolved
– StackOverflow: by now the best choice; best used together with documentation
– Various blogs: some good recipes, but a lot of them are outdated due to fast evolution of
K8s
– Conference videos: one of good sources of information, but you need to pay attention to
outdated topics

Versions, Versions, Versions...
●
Most annoying: K8s (even the latest version) doesn’t support the latest
Docker version and lags behind for months
●
Not all components (like ETCD, flannel etc.) versions are compatible with
each other and with K8s version of your choice
– Docker
– ETCD
– Flannel
●
CHANGELOG.md of K8s is good source of version compatibility with
external dependencies

Docker Version Support Matrix
●
Currently K8s supports just following Docker versions
●
All supported Docker versions are End-of-Life
●
We’re stuck with older CoreOS Linux releases because they have too
recent Docker versions for K8s
– Watch out for CoreOS auto update!
Docker version 1.10.3 1.11.2 1.12.6 17.03.x 18.05.x
K8s version (latest)
1.6 x x x
1.7 x x x
1.8 x x x x
1.9 x x x
1.10 x x x -
1.11 ? ? ? -

ETCD Version
●
Pay attention to maximum ETCD version supported by K8s release of your
choice
●
Biggest issue: you cannot downgrade ETCD

TLS
●
All communication within ETCD and K8s cluster is secured by TLS
●
For all intra-communication you can and should use self generated CA and
certs
– CFSSL is a good tool to generate CA, certs etc.
●
Only external services – like ingresses – have Let’s Encrypt or commercial
certs

Security and K8s
●
Most important: protect access to the API on master and nodes (via proxy)
– We use a basic IPTables firewall to disabled access from the net
– We use client certificate based authentication
●
RBAC (by default on from K8s v1.8) for resource access control
●
NetworkPolicy resources to control intra-pod and external communication
– Doesn’t work with Flannel, but works with Project Calico

Installation Hints’n’tips
● Use UTC as your timezone everywhere
– Doing so will help later get in sync when reviewing logs etc.
● By default older CoreOS releases uses older flannel version
– That you can easily upgrade by creating respective systemd unit

Workload partitioning
●
No-brainer: cordoning (disabling scheduling) on some nodes
●
Simplest, yet most effective in small clusters: node selection using labels
●
Advanced: affinity and anti-affinity that still uses node labels, but in regex
way
●
More advanced: using taints and tolerations that ensures pods aren’t
scheduled onto inappropriate nodes

CI/CD with Gitlab
●
Running on the K8s cluster
– Deployed via https://github.com/sameersbn/docker-gitlab
– Maybe helm chart in the future (currently in Alpha/Beta)
●
Setting up Docker registry part of it allows to store private Docker images
and use them within K8s clusters
●
Gitlab has K8s integration for CI/CD
– requires some tinkering to get it running on a non-remote cluster
●
For CI/CD to work Gitlab installs the Gitlab-Runner
– Builds/tests your projects or Docker images
– Push them to registry
– Deploy to K8s cluster

Developer Access to K8s
●
Per developer SSH-tunnel
●
kubectl access through tunnel via developer certificate

Backup Strategy
●
Contents of master node as well host-dir based persistent volumes are
rsynced to another host on nightly basis

Monitoring
●
Low-key system tools
– Docker stats
– Linux top
●
Kubernetes Dashboard (relying on data from built-in Heapster)
– No time travel
– No alerting
●
Planned: Prometheus

Updates
●
CoreOS
– In general everything happens automatically out-of-the-box except you can’t use it by
default due to version conflicts
– Instead use CoreRoller tool to run your own image store
– Updates can be scheduled to off-hours and various nodes
●
K8s
– Quite easy within minor releases: just change version numbers in systemd unit as well K8s
system manifest files and restart systemd unit
– Between major releases, carefully study the CHANGELOG to find what has changed in
kubelet startup options (depreciations, additions) and change systemd unit accordingly

Hints’n’tips for Running Stuff on K8s
●
kubectl is your friend
●
Use Deployments instead of ReplicaSets
– Remember to check manifests made by others before deploying them
●
There’s no restarting for pods like “docker restart”
●
Rolling-update is a great thing. Scale up, change deployment settings or
container image and K8s will do the rest
●
Don’t use pure Kube-registry
– It has issues while setting it up as well using it within cluster
– Instead use Gitlab with enabled Docker registry

Outlook
●
Persistent volumes featuring network transparent remote storage – ROOK
– We will migrate all host-dir based PVs to ROOK based PVs
●
Highly available master/ ingress
– Two dedicated master nodes
– HA cluster based pacemaker

OSDC 2018 | Self Hosted bare Metal Kubernetes for SMEs by Thomas Hoppe

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to OSDC 2018 | Self Hosted bare Metal Kubernetes for SMEs by Thomas Hoppe

Similar to OSDC 2018 | Self Hosted bare Metal Kubernetes for SMEs by Thomas Hoppe (20)

Recently uploaded

Recently uploaded (20)

OSDC 2018 | Self Hosted bare Metal Kubernetes for SMEs by Thomas Hoppe