How we can do Multi-Tenancy on Kubernetes

Multi-Tenancy
on
Jirayut Nimsaeng (Dear)
CEO & Founder, Opsta (Thailand) Co.,Ltd.
TechTalkThai Conference: Kubernetes Trends
September 16, 2021 https://bit.ly/opsta-ttt-k8s-tenancy

Multi-Tenancy on Kubernetes
#whoami
Jirayut Nimsaeng (Dear)
Jirayut has been involved in DevSecOps, Container, Cloud Technology
and Open Source for over 10 years. He has experienced and
succeeded in transforming several companies to deliver greater
values and be more agile.
● He is Founder and CEO of Opsta (Thailand) Co.,Ltd.
● He is Cloud/DevSecOps Transformation Consultant and
Solution Architecture
● He is the first Certified Kubernetes Security Specialist (CKS)
and Certified Kubernetes Administrator (CKA) in Thailand

Agenda
● What is Multi-Tenancy?
● Multi-Tenancy Primitive
● Access Control
● Fair Sharing
● Isolation
● Future

What is Multi-Tenancy?

Single vs Multi-Tenant
https://dev.to/sciencebae/multi-tenant-architecture-and-it-s-issues-h06

https://cloud.google.com/kubernetes-engine/docs/concepts/multitenancy-overview

Why Kubernetes Multi-Tenancy?
● Reduced management overhead
● Reduced resource fragmentation
● Cost eﬃciency

Kubernetes Multi-Tenancy Types
Soft Multi-tenancy
● trust tenants
● may has relation between
tenants
● we believe that they are not
trying to harm other tenants
● focus on preventing accidents
Hard Multi-tenancy
● zero trust tenants
● each tenant has no relation to
each other
● we believe that they are trying
to exploit the system
● focus on securing and
isolating each tenant

Kubernetes Multi-Tenancy Models
● Namespaces as a Service
allows sharing clusters and hence enables resource eﬃciencies with
cluster-wide resources limitation
● Clusters as a Service
better isolation with higher management and resource overhead.
● Control Planes as a Service
virtual cluster where each tenant gets their own dedicated Kubernetes
control plane but share worker node resources

Kubernetes
Multi-Tenancy Primitive

Multi-Tenancy on Kubernetes Overview
https://www.vamsitalkstech.com/architecture/a-deepdive-into-kubernetes-multitenancy-1-2/

Kubernetes Multi-Tenancy Primitive
● Access Control
Use policies to ensure that
tenants can access only what
they should have access to
○ RBAC
● Fair Sharing
Enforce limits per tenant
○ Resource Quota
○ Pod Priority
○ Quality of Service
○ Taints & Tolerations
○ Pod Aﬃnity / Anti-aﬃnity
● Isolation
Ensure tenants cannot access
each others’ workloads,
secrets, etc.
○ Namespace
○ Pod Security Policy
○ Network Policy
○ Sandbox

Access Control

RBAC
Role-based access control (RBAC) is a method of regulating access to computer
or network resources based on the roles of individual users within your organization
https://www.cncf.io/blog/2020/08/28/kubernetes-rbac-101-authorization/

Multi-Tenancy Roles
● [ClusterRoleBinding] Cluster Administrator
● A cluster administrator has access to all cluster resources and can conﬁgure
new tenant namespaces
● [ClusterRoleBinding] Cluster view
Read privileges for all resources in the cluster
● [RoleBinding] Tenant Administrator
● A tenant administrator manages namespaces that belong to the tenant
● [RoleBinding] Tenant User
● Read/write privileges for all resources scoped to that tenant

Sample Role
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: tenant-a-role
namespace: tenant-a
rules:
- apiGroups: ["", "extensions", "apps"]
resources: ["*"]
verbs: ["*"]
- apiGroups: ["batch"]
resources:
- jobs
- cronjobs
verbs: ["*"]

Fair Sharing

Resource Quota
A resource quota, deﬁned by a ResourceQuota
object, provides constraints that limit aggregate
resource consumption per namespace.
apiVersion: v1
kind: ResourceQuota
metadata:
name: mem-cpu-demo
spec:
hard:
requests.cpu : "1"
limits.cpu : "2"
apiVersion: "v1"
kind: "LimitRange"
metadata:
name: "resource-limits"
spec:
limits:
- type: "Container"
max:
cpu: "2"
memory: "1Gi"
min:
cpu: "100m"
memory: "4Mi"
default:
cpu: "300m"
memory: "200Mi"
defaultRequest:
cpu: "200m"
memory: "100Mi"
maxLimitRequestRatio:
cpu: "10"

Pod Priority
Priority indicates the importance of a Pod relative to other Pods. If a Pod cannot be
scheduled, the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the
pending Pod possible. An administrator can use ResourceQuota to prevent users from creating
pods at high priorities.
Priority Class Purpose Value/Priority
Cluster Core Essential services to operate the cluster itself, such as Dex for
authentication
100000
Tenant Critical Services which are business-critical to your tenant, and cannot
easily move to a new machine, such as database back-ends
70000
Administrative Services Dashboards that are important for managing the cluster, but could
tolerate a short outage while moving to a new machine
50000
Best Eﬀort (default) No priority assigned; allows termination in favor of higher priority
workloads
100

Sample Pod Priority
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: tenant-critical
value: 70000
globalDefault: false
description: "This priority class should be used for s ervices
which are business-critical to your tenant, and cannot easily
move to a new machine, such as database back-ends "

Quality of Service for Pods
Quality of Service (QoS) class is a Kubernetes concept that the scheduler uses
for deciding the scheduling and eviction priority of the pods.
https://medium.com/blutv/qos-classes-of-k8s-pods-722238a61c93

Taints & Tolerations
● Taints are applied to node, to repel a set of pods.
● Tolerations are applied to pods, and allow (but do not require) the pods to
schedule onto nodes with matching taints.
● Use cases: Dedicated Nodes and Nodes with Special Hardware

Pod Disruption Budgets
A PDB limits the number of Pods of a replicated application that are down
simultaneously from voluntary disruptions
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: zk-pdb
spec:
maxUnavailable : 1
selector:
matchLabels :
app: zookeeper

Pod Affinity / Anti-affinity
You can use Pod anti-affinity to prevent Pods
from different tenants from being scheduled on the
same node. Anti-affinity constraints are based on
Pod labels. For example high workload shouldn’t
stay on the same node.
apiVersion: v1
kind: Pod
metadata:
name: bar
labels:
team: "billing"
spec:
affinity:
podAntiAffinity :
requiredDuringSchedulingIgnoredD
uringExecution :
- topologyKey :
"kubernetes.io/hostname"
labelSelector :
matchExpressions :
- key : "team"
operator : NotIn
values : ["billing"]

Isolation

Namespace
● System namespaces
Exclusively for system pods. Usually kube-system namespace and manage by
cluster administrator
● Service namespaces
These namespaces should run services or applications that need to be accessed
by services in other namespaces. Usually manage by cluster administrator
● Tenant Namespaces
Tenant namespaces should be spun up to run applications that do not need to
be accessed from other namespaces in the cluster. Usually manage by tenant
administrator

Pod Security Policy
A PodSecurityPolicy is an admission controller resource you create that validates requests to
create and update Pods on your cluster. The PodSecurityPolicy deﬁnes a set of conditions that Pods
must meet to be accepted by the cluster. when a request to create or update a Pod does not meet the
conditions in the PodSecurityPolicy, that request is rejected and an error is returned.
https://rancher.com/blog/2020/pod-security-policies-part-2

Sample PSP
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: restricted
spec:
privileged: false
# Required to prevent escalations to root.
allowPrivilegeEscalation
: false
# The following is redundant with non-root + disallow
privilege
# escalation, but we can provide it for defense in depth.
requiredDropCapabilities
:
- ALL
# Allow core volume types.
volumes:
- 'configMap'
- 'emptyDir'
- 'projected'
- 'secret'
- 'downwardAPI'
# Assume that persistentVolumes set up by the cluster
admin
# are safe to use.
- 'persistentVolumeClaim'
hostNetwork: false
hostIPC: false
hostPID: false
runAsUser:
# Require the container to run without
root privileges.
rule: 'MustRunAsNonRoot'
seLinux:
# Assumes the nodes are using AppArmor
rather than SELinux.
rule: 'RunAsAny'
supplementalGroups
:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535
fsGroup:
rule: 'MustRunAs'
ranges:
# Forbid adding the root group.
- min: 1
max: 65535

PodSecurityPolicy Deprecation
Kubernetes 1.21 starts the deprecation process for PodSecurityPolicy.
The current plan is to remove PSP from Kubernetes in the 1.25 release.
Kubernetes v1.22 as an Alpha feature, Kubernetes oﬀers a built-in Pod
Security admission controller, the successor to PodSecurityPolicies.
● https://github.com/kubernetes/enhancements/issues/2579
● https://kubernetes.io/docs/concepts/security/pod-security-admission/

Open Policy Agent (OPA)
package kubernetes. admission
deny[msg] {
input.request.kind.kind == "Pod"
some i
image := input.request.object.spec.containers[i].image
not startswith (image, "hooli.com/")
msg := sprintf("image '%v' comes from untrusted registry" ,
[image])
}

Network Policy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: deny-all
namespace: tenant-a
spec:
podSelector:
matchLabels :
ingress:
- from:
- podSelector : {}
you should block traﬃc between
namespaces that host diﬀerent
tenants' applications

Sandbox
VM Container gVisor

Future

k8s-sig-multi-tenancy
https://github.com/kubernetes-sigs/multi-tenancy

Benchmarks
https://github.com/kubernetes-sigs/multi-tenancy/blob/master/benchmarks/kubectl-mtb/README.md

The Hierarchical Namespace Controller
$ kubectl hns create my-service -n my-team
$ kubectl hns tree my-team
my-team
└── my-service
https://github.com/kubernetes-sigs/hierarchical-namespaces

VirtualCluster
https://www.cncf.io/blog/2019/06/20/virtual-cluster-extending-namespace-based-multi-tenancy-with-a-cluster-view/

Wrap-up

Key Take-aways
● Use multi-tenancy for improved resource eﬃciency,
cost, and operations
● Multi-tenancy is unavoidable in the future
● Choose your multi-tenancy type and model
● To do multi-tenancy, you need to conﬁg kubernetes
access control, fair sharing, and isolation

More questions?
jirayut@opsta.co.th
Jirayut Nimsaeng
CEO & Founder
Opsta (Thailand)
086-069-4042
Facebook

How we can do Multi-Tenancy on Kubernetes

More Related Content

What's hot

Similar to How we can do Multi-Tenancy on Kubernetes

More from Opsta

Recently uploaded

How we can do Multi-Tenancy on Kubernetes