Here are the slides I presented on 12-12-2017 at the Bay Area Microservices Meeting. I presented some of the best practices to achieve application isolation on Kubernetes

1. Segmentation and Isolation on Kubernetes
for Enterprise Teams
Microservices & Cloud Native Apps,
December 12th 2017

2. 2
Kubernetes cluster sprawl
is the new VM sprawl
Our Goal:
Optimize the number of clusters by
sharing them across users & teams

3. 3
•Developer, VP Engineering, Founder at Nirmata
•Developing large-scale software since the early 90’s.
•Expertise in centralized management for complex
distributed systems.
•Co-founded Nirmata in 2013, for enterprise
management of cloud-native applications.
Damien Toledo

4. 4
ITSM Service Catalog
Public Clouds Private Clouds
Infrastructure-as-a-Service
Containerized Apps
What Nirmata Enables
Benefits:
 Multi-Cluster Policies & Governance
 Integrated Monitoring, Alarms, Remediation
 Automated Cluster & Node Management
CI/CD Tools
Kubernetes Clusters
Use Cases:
 Self-service Environments
 Continuous Delivery
 Adaptive Application Operations

5. 5
• Brief Introduction to Kubernetes
• Namespaces
• Resource Quota
• Role Base Access Control
• Micro Segmentation
• Demo
Agenda

6. Kubernetes

7. 7
• Kubernetes is an open source
container orchestration solution
originally developed by Google now
part of CNCF
• Enable Enterprise DevOps
• Kubernetes is designed for
microservices but can support
stateful applications
Kubernetes
(Greek for “helmsman" or "pilot")

8. 8
Pod
Deployment
Service Ingress
Replica Set
Stateful SetDaemon Set
Network Policy
Job
Container
Secrets
Dude, where’s my app?
Persistent
Volume
Namespace
Ingress Controller
Persistent
Volume Claim
Storage Class
Network
Controller
Cloud
Provider

9. 9
Kubernetes is awesome! But ...
“For newcomers, Kubernetes
can be intimidating….” -- Joe
Beda
“Kubernetes was created by
system engineers, for system
engineers”
-- Craig McLuckie
“Guardrails are important.”
-- Steve Sandke, Salesforce

10. Namespaces

11. 11
• Most obvious construct to create virtual clusters
• Provide a scope for names
• Provide a way to divide cluster resources using quota
• By default namespace does not provide
• Network isolation  Any service can reach any service
• Access Control Anybody can do everything
Namespace

12. 12
Should I use
One namespace per user/developer
One namespace per team
One per service type
One namespace per application type
One namespace per running instance of your application
application
Namespace scope

13. 13
Namespace Per Application Instance
Service
Deployment
Pod
NetworkPolic
y
ReplicaSet
Pod
Service
Deployment
Pod
NetworkPolic
y
ReplicaSet
Pod
Service
Deployment
PodNetworkPolicy
ReplicaSet
Pod
Service
Deployment
Pod
NetworkPolic
y
ReplicaSet
Pod
Service
Deployment
Pod
NetworkPolic
y
ReplicaSet
Pod
Service
Deployment
PodNetworkPolicy
ReplicaSet
Pod
Namespace (shopme-joe) Namespace (shopme-susan)
Cluster

14. 14
Resources Creation & Namespaces
apiVersion: v1
kind: Service
metadata:
name: customer
namespace: shopme-john
spec:
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
selector:
service: customer
All resources belonging to the same
application instance must be created in
the same Namespace
kubectl create –f service-customer.yaml

15. Quota & Limits

16. 16
• Used to provide different level of QoS to Pods
• Containers can specify
• Requests: Guaranteed amount of a specific resource (CPU,
Memory, …)
• Limits: Maximum amount of a resource allowed
• Request <= Limit
• Scheduling is based on Requests, not on Limits
• Sum of limits can exceed system capacity (i.e. oversubscription)
Container Requests & Limits

17. 17
Container Requests & Limits - Example
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: payment
namespace: shopme
spec:
replicas: 1
template:
spec:
containers:
- name: payment
image: nirmata/payment:latest
ports:
- containerPort: 80
resources:
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "128Mi"
cpu: "1"

18. 18
Compressible Resources (CPU)
• Pods are guaranteed to get the amount of CPU requested
• Excess CPU is distributed based on the amount requested
• If a Pod exceeds a limit, the process using the most amount of the resource will be
killed.
• Pods are throttled if they exceed their limit
Incompressible Resources (Memory)
• If a Pod exceeds its request, it can be killed if other Pods needs more memory
• If a Pod exceeds a limit, the process using the most amount of the resource will be
killed.
Container Requests & Limits

19. 19
Resource Quota
• Resource Quota are defined at the namespace level
• Limit aggregate resource consumption
• Limit number of objects (Service, NodePort, Pod, …)
• Limit that total compute resources (CPU, memory, storage)
• Resource Quotas should be created by administrators
• If creating or updating a resource violate a Resource Quota,
the operation is denied (403 FORBIDDEN)
• Users must specify container Requests when Resource
Quotas are in use

20. 20
Resource Quota - Example
In a cluster with 64 GiB RAM, 16 cores, 500 GiB disk
apiVersion: v1
kind: ResourceQuota
metadata:
name: shopme-john-quota
namespace: shopme-john
spec:
hard:
pods: "4"
requests.cpu: “8"
requests.memory: 32Gi
requests.storage: 100Gi
limits.cpu: “16"
limits.memory: 64Gi
shopme-john-quota.yaml
kubectl create –f shopme-john-quota.yaml
apiVersion: v1
kind: ResourceQuota
metadata:
name: shopme-susan-quota
namespace: shopme-susan
spec:
hard:
pods: "4"
requests.cpu: “8"
requests.memory: 32Gi
requests.storage: 200Gi
limits.cpu: “12"
limits.memory: 48Gi
shopme-susan-quota.yaml
kubectl create –f shopme-susan-
quota.yaml

21. 21
Object Count Quota
apiVersion: v1
kind: ResourceQuota
metadata:
name: shopme-quota
namespace: shopme-susan
spec:
hard:
persistentvolumeclaims: "1“
pods: “3"
services.nodeports: "0"
shopme-quota.yaml status:
hard:
persistentvolumeclaims: "1"
services.nodeports: "0"
used:
persistentvolumeclaims: "0“
pods: “2"
services.nodeports: "0“
kubectl create –f shopme-quota.yaml
Kubectl get resourcequota shopme-quota –n
shopme-susan

22. 22
Default CPU Request & Limits
apiVersion: v1
kind: LimitRange
metadata:
name: cpu-limit
spec:
limits:
- default:
cpu: 1
type: Container
shopme-cpu-limit.yaml
kubectl create –f shopme-cpu-limit.yaml –n shopme-susan

23. 23
Default Memory Request & Limits
apiVersion: v1
kind: LimitRange
metadata:
name: cpu-limit
spec:
limits:
- default:
memory: 512Mi
defaultRequest:
memory: 256Mi
type: Container
shopme-cpu-limit.yaml
kubectl create –f shopme-memory-limit.yaml –n shopme-susan

24. RBAC

25. 25
• General availability in K8s 1.8
• Policies controlling permissions within a cluster
• Can provide cluster wide defaults
• Fined grained access control per Namespace
Kubernetes RBAC

26. 26
Service Accounts
• Managed by Kubernetes
• Bound to specific Namespaces
• Tied to a set of credentials stored as Secrets
Normal Users
• Managed outside K8s
• There are no k8s objects representing users
• Can use private keys, file with list of users, user store like keystone, etc
Kubernetes Users

27. 27
Kubernetes RBAC Model
RoleUser
ServiceAccount
RoleBinding
ClusterRole
ClusterRoleBindin
g
All K8s Resources
Pod
Deployment
ReplicaSet
StatefulSet
Ingress
PersistenVolume
Verbs:
update
create
Delete
get
list
watch

28. 28
Role Creation - Example
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: shopme-john
name: pod-reader
rules:
- apiGroups: [""] # "" indicates the core API group
resources: ["pods"]
verbs: ["get", "watch", "list"]
kubectl create –f pod-reader-role.yaml
pod-reader-role.yaml

29. 29
Role Binding Creation - Example
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: read-pods
namespace: shopme-john
subjects:
- kind: User
name: susan
apiGroup: rbac.authorization.k8s.io
roleRef:
kind: Role
name: pod-reader
apiGroup: rbac.authorization.k8s.io
kubectl create –f pod-reader-role-binding.yaml
pod-reader-role-binding.yaml

30. Micro segmentation

31. 31
All Containers can communicates with all containers within a cluster
Best pratices:
1. Enable network policies using a plugin (Calico, Cilium, Kuke-router, Romana,
Weave Net, ..)
2. By default, deny all traffic to your services
3. Create Network Policies to selectively enable traffic from other services
Kubernetes Networking is Great But …

32. 32
Kubernetes Microsegmentation
Service
payment
Pod
NetworkPolicy
Namespace
shopme-john
Service
customers
Pod
NetworkPolicy
Service
Recommendation
s
Pod
NetworkPolicy
Allow
Deny
Service
Pod
Service
Pod
Service
customers
Pod
NetworkPolicy
Namespace
shopme-susan
Deny
Deny

33. 33
Network Policies Creation
apiVersion:
networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: shopme-john
spec:
podSelector:
policyType:
- Ingress
kubectl create –f default-deny-all.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: payment-network-policy
namespace: shopme-john
spec:
podSelector:
matchLabels:
service: customer
policyTypes:
- Ingress
ingress:
- from:
- namespaceSelector:
matchLabels:
name: shopme-john
- podSelector:
matchLabels:
service: customers
ports:
- protocol: TCP
name: payment-port
kubectl create –f payment-policy.yaml

34. Additional Constructs

35. 35
• Storage
• ConfigMaps
• Labels
• Pod Selector
• Secrets
Additional Constructs

36. Demo

37. Conclusion

38. 38
• Kubernetes is a powerful toolbox
• Provides many building blocks that can be used to implement application
isolation
• Developers shouldn’t have to worry about this
• Admins need solutions to simplify their work …
Conclusion

39. 39
• Single management plane
across multiple clusters
• Secure and scalable
multi-cloud management
• Seamless integrations for
continuous delivery
Nirmata simplifies Kubernetes
forenterpriseDevOpsteams

40. Thank-you!
http://try.nirmata.io