Slides from SCaLE 18x. Abstract: https://www.socallinuxexpo.org/scale/18x/presentations/kubernetes-concrete-abstraction-layer

1. Kubernetes as a Concrete Abstraction Layer
SCaLE 18x

2. 2
WHO AM I
• Karen Bruner
• Recovering SRE
• Tech Evangelist at StackRox
• Totally Sane Cat Lady
• Twitter @fuzzyKB

3. 3
WHAT IS KUBERNETES
• Container Orchestration Platform

4. 4
WHAT IS KUBERNETES
• Self-Driving Car for ContainerWorkloads and Services
• Except More Like "Assisted Driving"
• And Everything is in a Different Place in Every Model
• And If It Catches Fire, It Burns for Hours

5. 5
WHAT WE'LL COVER
• Kubernetes Releases and Conformance
• Storage
• Service Load Balancers
• Nodeless
• Security Options
• Other Functional Differences
• Mitigation Plans

6. 6
Kubernetes Releases and Conformance

7. 7
Kubernetes Release Cycle
• Release cadence: every 3 months
• 3 versions supported at a time
• Interactive release notes: relnotes.k8s.io

8. 8
Version EKS AKS GKE Kubernetes
1.18 beta
1.17 preview ✔
1.16 preview beta ✔
1.15 ✔ ✔ ✔
1.14 default default default
1.13 ✔ ✔ ✔
1.12 ✔
Current Support Matrix

9. 9
CNCF Certified Kubernetes Conformance
• Covered
• kubelet behavior
• Most Core API resources
• App API resources
• Batch API resources
• Pod-to-pod and service networking
• Not Covered
• StorageClass, PersistentVolumes
• Service of type LoadBalancer
• APIs requiring plugins, controllers (usually), or device drivers
• Nodeless platforms

10. 10
Conformance Examples (Illustrated)
Covered: Pods Not Covered: Load Balancers

11. 11
Abstract vs Concrete
Examples FromWhere Portability Ends

12. 12
Platform Kubernetes
Version
Cloud Provider ToolVersion Notes
AKS 1.14.8 Azure az v2.0.77 kubenet network
EKS 1.14.9 AWS eksctl v0.13.0
GKE 1.14.10 Google Cloud gcloud v283.0.0
Kops 1.16.7 AWS kops v1.16.0 flannel CNI
Kubespray 1.16.6 AWS
master branch
2019-02-27
OpenShift 1.14.6 AWS
openshift-installer
v4.3.2
OpenShift
v4.2.21
Evaluation Environments

13. 13
Platform Default StorageClass Supports Resize
AKS Standard Azure Storage (HDD) Yes
EKS Elastic Block Storage gp2 (SSD) Yes
GKE Standard Persistent Disk (HDD) Yes
Kops Elastic Block Storage gp2 (SSD) Yes
Kubespray None N/A
OpenShift Elastic Block Storage gp2 (SSD) Yes
PersistentVolumes

14. 14
Managing Differences in Storage
• Create standard default StorageClasses on all platforms
• Use consistent StorageClass names across platforms (ssd, hdd)
• Avoid provider special snowflake storage products
• Or use a portable storage service like Rook/Ceph

15. 15
Platform Default Load Balancer
Type
Source IP
Preserved
Default Idle
Timeout (K8s)
Supported
Timeout Range
AKS
Azure Standard Load
Balancer
Yes 30 m 4-30 m
EKS
Elastic Load Balancer
(Classic ELB)
No 60 s 1-4000 s
GKE
ExternalTCP/UDP
Network Load Balancer
Yes Not documented Not documented
Kops
Elastic Load Balancer
(Classic ELB)
No 60 s 1-4000 s
Kubespray None N/A N/A N/A
OpenShift
Elastic Load Balancer
(Classic ELB)
No 60 s 1-4000 s
Service Load Balancers

16. 16
Managing Differences in Load Balancers
• For source IP, set externalTrafficPolicyto Local
• LB must support IP SNAT
• Requires DaemonSet for backend service unless LB does health checking
• For LBs with no source IP NAT
• Check provider for alternative LB types
• Use Proxy Protocol
• Examples
• Annotation to use AWS Network Load Balancer:
service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
• Annotation to enable Proxy Protocol on AWS Classic ELB:
service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"

17. 17
Load Balancer Demo

18. 18
"Nodeless" Kubernetes
• virtual-kubelet
• Currently at 70% CNCF Conformance
• Main portability limitation: no DaemonSet support
• Cloud provider "nodeless"
• AKS on Azure Container Instances
• EKS on Fargate (not based on virtual-kubelet)
• GKE - none
• Common limitations on managed providers
• No DaemonSet support (run as sidecars)
• No privileged containers
• No node-level visibility

19. 19
Platform RBAC Network Policy Pod Security Policy
AKS Optional; enabled by default Optional Optional
EKS Always on User-managed Always on
GKE Optional; enabled by default Optional Optional
Kops Optional; enabled by default
Depends on CNI
(default no: kubenet)
Optional
Kubespray Optional; enabled by default
Depends on CNI
(default yes: Calico)
Optional
OpenShift Always on
Optional; enabled by
default
Not supported
Kubernetes Security Features

20. 20
Other Potential Problem Areas
• Supported Admission Controllers
• Feature gates
• Enabled API groups
• Container runtimes

21. 21
Maintaining Portability
• Create self-managed clusters
• Create your own sonobuoy plugins for organizational conformance
• Enable RBAC and Network Policy everywhere
• Use consistent resource manifests across platforms
• Open Policy Agent Gatekeeper to enforce security and resource conformance

22. 22
References

23. 23
Thanks and References
Thanks to Brendan Cox and MadhuriYechuri for virtual-kubelet info
Conformance
• https://github.com/cncf/k8s-conformance/blob/master/docs/KubeConformance-1.15.md
• https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.17/
• https://sonobuoy.io/
Platforms
• https://docs.microsoft.com/en-us/azure/aks/
• https://docs.aws.amazon.com/eks/latest/userguide/what-is-eks.html
• https://cloud.google.com/kubernetes-engine/docs/concepts/kubernetes-engine-overview
• https://github.com/kubernetes/kops
• https://github.com/kubernetes-sigs/kubespray
• https://docs.openshift.com/container-platform/4.2/welcome/index.html
Storage
• https://kubernetes.io/docs/concepts/storage/storage-classes/
• https://kubernetes.io/docs/concepts/storage/persistent-volumes/
• https://github.com/rook/rook
Load balancing
• https://kubernetes.io/docs/concepts/architecture/cloud-controller/
• https://kubernetes.io/docs/concepts/services-networking/service/#loadbalancer
• https://aws.amazon.com/elasticloadbalancing/
• https://docs.microsoft.com/en-us/azure/load-balancer/load-balancer-overview
• https://cloud.google.com/load-balancing/docs/network

24. 24
References (cont.)
Nodeless
• https://virtual-kubelet.io/
• https://docs.microsoft.com/en-us/azure/aks/virtual-nodes-portal
• https://docs.aws.amazon.com/eks/latest/userguide/fargate.html
• https://cloud.google.com/run/docs
Security
• https://kubernetes.io/docs/reference/access-authn-authz/rbac/
• https://kubernetes.io/docs/concepts/services-networking/network-policies/
• https://kubernetes.io/docs/concepts/policy/pod-security-policy/
• https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet-authentication-authorization/
Other potential problem areas
• https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/
• https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/
• https://kubernetes.io/docs/reference/using-api/api-overview/
Maintaining portability
• https://kubernetes.io/docs/concepts/cluster-administration/cloud-providers/
• https://github.com/kubernetes-sigs/kubespray/blob/master/docs/comparisons.md
• https://github.com/open-policy-agent/gatekeeper

25. 25
Questions?
• Twitter @fuzzyKB
• LinkedIn www.linkedin.com/in/kmbruner/
• Blog posts
• www.stackrox.com/post/ (paycheck)
• nightmare-before-devops.xyz (personal)