Building Kubernetes for Large Organizations

of 38
Reference: Meetup Kubernetes
a passion for technology
Date:
Reference:
Author(s):
Distribution:
Attendees marked with *
Template date: 06-07-2017
Template PN: 6001-1246-5511
Building a Kubernetes cluster
for a large organisation 101
Meetup Kubernetes
Ed Schouten
Prodrive Tech Talk
2018-06-27

of 38
Prodrive Technologies
- Who are we and what do we do?
- What do I do?
Kubernetes
- What is it?
Kubernetes at Prodrive Technologies
- Why do we want it?
- What kind of automation have we designed to go along?
- What are we going to work on next?
Outline of today’s talk

of 38

of 38
Ready-to-use products
Technology solutions
Manufacturing services
Design of electronics, software
and mechanics
Manufacturing
Added value services
We focus on autonomous growth and a solid preservation of our
company culture

of 38
Company presence
Prodrive Technologies Netherlands (HQ)
- R&D
- Manufacturing
- Service
- Sales
China
- Sales
- Supply Chain
- Manufacturing
- Service
USA
- Sales
- Manufacturing (Q4-2018)
Germany
- Sales
Israel
- Sales
Expansion of NL facility (Q2-2018)
USA manufacturing (2019)
Suzhou facility

of 38
‣ Area- and line-scan cameras
‣ Integrated optics, scintillators, FOPs
‣ VIS, NIR, DUV, LWIR, E-beam
‣ Particle Measurement Systems
‣ PMP (Prodrive Motion Platform)
‣ High Performance Actuators
‣ Various motion drives
‣ Mechatronic systems
‣ Variable frequency drives
‣ Wireless Energy (20kW)
‣ Power Converters (100kW+)
‣ Intelligent Power Distribution
‣ Motion control platforms
‣ Image processing platforms
‣ Industrial PC/server
‣ Home automation systems
‣ Gateways and cloud solutions
‣ Smart Thermostats
‣ Professional Audio/Video Distribution
‣ Controllers and I/O
‣ Displays, touch screens, HMI
‣ Automated production
equipment
‣ AGV’s

of 38
Part of the IT Services team.
• Responsible for Linux-based infrastructure (source control, builds).
• Involved in shaping the future direction of our internal IT.
Part of the High-End Computing (HEC) program.
• Software architecture for our industrial computing products.
What do I do at Prodrive Technologies?

of 38
Kubernetes

of 38
Kubernetes is…
• A cluster management/orchestration tool.
• Based on the design of Borg (Google).
• Capable of scheduling Docker containers.
• Open Source: Apache 2.0 licensed.
Kubernetes

of 38
Kubernetes in a nutshell

of 38
kubectl: command line tool for managing clusters.
• Talks to the API server over a documented REST API.
• Exposes the cluster as objects of different classes.
• Each object has a JSON/YAML configuration/state.
• Easy to learn syntax: kubectl ${verb} ${class} ${instance}
• ‘Bottom half’ is also available as a Golang library.
Managing a Kubernetes cluster: kubectl

of 38
• Node: Linux (or Windows) server that is part of the cluster.
• Pod: Unit of work that is scheduled by Kubernetes on a node.
• Consists of one or more containers. (‘Sidecars’)
• Deployment: Template for starting a fixed number of identical pods.
• Good for starting stateless web frontends.
• Service: Places a (TCP) load balancer address in front of pods.
• Other interesting ones: StatefulSet, DaemonSet, CronJob.
Commonly used object classes

of 38
• List all nodes (i.e., servers) in a cluster:
$ kubectl get nodes
• Start an Nginx pod that is automatically restarted/migrated:
$ kubectl create deployment nginx --image=nginx:1.15
• Edit the deployment to increase the number of replicas:
$ kubectl edit deployment
• Delete a single pod that is misbehaving (and create a new one):
$ kubectl delete pod nginx-9db896598-pj5lz
Example invocations of kubectl

of 38
Objects in the cluster are partitioned into namespaces.
• No referencing across namespaces.
• Deployments start pods in the same namespace.
• Services can only match pods in the same namespace.
• Use case #1: Production vs. development setups.
• kubectl edit deployment -n wiki-prod nginx
• kubectl edit deployment -n wiki-dev nginx
• Reduces risk of accidentally mixing up traffic.
• Use case #2: Multi-tenant clusters.
• Exception: nodes don’t have a namespace.
Namespaces

of 38
From outside of the cluster:
• Basic authentication (username & password).
• SSL client certificates.
• OpenID Connect.
From within the cluster:
• Every pod runs under a ServiceAccount.
• Built-in CA generates SSL client certificates for pods.
• Accessible through /var/run/secrets/… inside containers.
API server authentication

of 38
Kubernetes implements Role-Based Access Controls (RBAC):
• Subject: either an external user or a ServiceAccount.
• Roles: gives a name to a set of rights: ${verb} ${class}.
• ‘release-pusher’ = {‘edit deployments’, ‘get pods’}.
• RoleBinding: grants a subject access to a role in a namespace.
• Grant ‘jenkins’ the role ‘release-pusher’ in namespace ‘wiki-prod’.
• ClusterRoleBinding: grants access to a role in all namespaces.
• Grant ‘prometheus‘ the role ‘node-viewer’.
API server authorisation

of 38
Authorisation class diagram

of 38
Kubernetes at

of 38
Whole fleet of systems to which people currently SSH/X11/….
• Every project has different requirements for development tools.
• Many tickets for IT to install software.
• Separate systems to deal with contradicting version requirements.
• Strong imbalance in system utilisation.
• Lack of reproducible build environments when reviving old projects.
The existing Prodrive development infrastructure

of 38
1. IT creates a Docker image called ‘Prodrivian’.
- Debian with common Prodrive configuration on top.
2. Development team inherits and creates a custom Docker image.
- Adds project specific tools (cross compilers, commercial tools) on top.
3. Development team fires up containers on Kubernetes.
- For interactive use.
- Through CI systems for automated builds.
The future Prodrive development infrastructure?

of 38
Goal:
• Develop a centralised cluster usable by all developers.
• For both batch/interactive use, but also for running services.
Problem:
• Lots of material on using single-tenant Kubernetes clusters.
• Some material on multi-tenant clusters.
• No material on easy, maintainable multi-tenant clusters.
Crux

of 38
Problem: Full Kubernetes RBAC is too hard to get right.
Solution: Subset it to a simplified authorisation model.
• Every employee has a personal namespace.
• Rights for user == rights of namespace’s default service account.
• kubectl on your laptop behaves the same way as on the cluster.
• Employees can create groups.
• Every group automatically gets a namespace in the cluster.
• Group members have read/write access to the namespace.
• Transitive: edsch ➜ it-services-linux ➜ wiki-{prod,qa,dev}
Authorisation

of 38
Group management tool

of 38
$ kubectl describe clusterrole corpdb:group-membership
…
configmaps [create delete get list patch update watch]
cronjobs.batch [create delete get list patch update watch]
deployments.apps [create delete get list patch update watch]
deployments.extensions [create delete get list patch update watch]
…
$ kubectl describe rolebinding -n wiki-prod corpdb:group-membership
…
User https://login.prodrive-technologies.com/#edsch
ServiceAccount default edsch
…
RBAC configuration for a group

of 38
Problem: By default, all pods can connect to each other.
• Ideally, should be restricted by having auth at the application level
(e.g., let everything use credentials or SSL client/server certs).
• Too complex to realise at our scale for now.
• (Interesting project: Istio)
Solution: Also use groups to automatically set up in-cluster firewalling.
• Transitive: edsch ➜ it-services-linux ➜ wiki-{prod,qa,dev}
Network policies

of 38
$ kubectl describe namespace edsch
Labels: …
corpdb-can-access-namespace-wiki-prod=true
…
$ kubectl describe networkpolicy -n wiki-prod corpdb-allow-members-ingress
Spec:
Allowing ingress traffic:
To Port: <any> (traffic allowed to all ports)
From NamespaceSelector: corpdb-can-access-namespace-wiki-prod=true
…
Network policy for a group

of 38
Problem: Nodes in cluster become overloaded due to containers
consuming too much CPU & memory.
Solution: Configure resource limits.
• Give employees a ‘freebie quota’.
• For group namespaces, have a resource allocation procedure.
Resource quotas #1

of 38
$ kubectl describe resourcequota -n edsch corpdb
Name: corpdb
Namespace: edsch
Resource Used Hard
-------- ---- ----
requests.cpu 0 1
requests.memory 0 1Gi
Resource quota for a namespace

of 38
Problem: Providing resource limits for containers is optional.
Omitting them will create containers that are exempt from quotas.
Solution: Automatically inject resource limits for such containers.
• Provided LimitRanger admission controller can do this.
Resource quotas #2

of 38
$ kubectl describe limitrange -n edsch corpdb
Name: corpdb
Namespace: edsch
Type Resource Min Max Default Request Default Limit Max Limit/Request Ratio
---- -------- --- --- --------------- ------------- -----------------------
Container memory 16Mi - 16Mi 64Mi 4
Container cpu 10m - 10m 40m 4
LimitRange configuration

of 38
Problem: We want to allow people to create services that are available
inside the cluster, but oftentimes not ones that are public.
• Type ‘ClusterIP’ vs. ‘LoadBalancer’.
• Kubernetes RBAC is too weak to solve this.
Solution: Set up a ValidatingAdmissionWebhook.
• Lets the API server send ‘kubectl create service’ requests through a
helper process using HTTP calls.
• Helper process can accept/reject the request.
• Groups that should create load balancers can be whitelisted.
Preventing security foot-shooting

of 38
$ kubectl describe validatingwebhookconfiguration corpdb-web-kubernetes-validator
…
Service:
Name: corpdb-web-kubernetes-validator
Namespace: corpdb-prod
Failure Policy: Fail
Rules:
Operations:
*
Resources:
services
…
Validating webhook configuration example

of 38
Problem: How can users easily configure kubectl on their system?
• Setting it up initially (API server hostname, default namespace).
• Obtaining a temporary access token.
Solution: Build a ‘kubeaccess’ web page.
• Generates commands that can be copy-pasted to a terminal.
• Generates OpenID Connect tokens with 20 hour validity on the fly.
• Relies on an OpenID Identity Provider on the same dataset.
External access

of 38
Kubeaccess web page

of 38
CorpDB overview

of 38
• Distributed storage: Ceph/Rook.
• Extend CorpDB to track storage resource limits as well.
• Prodrive globalisation effort: automatically set up replicated storage?
• Continuous Build/Integration: Let Bamboo spawn pods.
• Atlassian offers a per-build-container extension.
• Integrate Kubernetes into High-End Computing products.
• Scale of computing needed in industrial automation is increasing.
• Work along with the Open Source community.
• Contribute in both directions (e.g., Traefik OIDC support).
Future work

of 38
• Lots of awesome projects at Prodrive.
• Nice company culture.
• Chat with us after this talk!
• https://prodrive-technologies.com/careers/
Help!

of 38
a passion for technology
T +31 40 2676200
E contact@prodrive-technologies.com
I www.prodrive-technologies.com

Building Kubernetes for Large Organizations

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Building Kubernetes for Large Organizations

Similar to Building Kubernetes for Large Organizations (20)

Recently uploaded

Recently uploaded (20)

Building Kubernetes for Large Organizations