Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kubernetes day 2 Operations

87 views

Published on

Installing and Using Kubernetes is hard, but Operating Kubernetes is even harder! This BOF is for Kubernetes Operators to get together and discuss our day to day Operations, and for people new to Kubernetes to learn more about how to operate it.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Kubernetes day 2 Operations

  1. 1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Version 1.0 Paul Czarkowski Principal Technologist @pczarkowski github.com/paulczar Day 2 Kubernetes Beyond the install
  2. 2. Who is using Linux Containers ?
  3. 3. Who is using Kubernetes ?
  4. 4. Who is operating Kubernetes ?
  5. 5. > 1 year Kubernetes experience?
  6. 6. People build Platforms People build Apps Apps run on Platforms People are the most important component of any platform.
  7. 7. “Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.” - Conways Law
  8. 8. https://www.slideshare.net/SatnamSingh67/2015-0605-cluster- management-with-kubernetes
  9. 9. Infra Services App Platform Evolve your IT teams! Platform Team Application Team Build common services for App Teams Take business requirements and turn them into features IaaS Virtual Infrastructure Physical Infrastructure Abstract infrastructure complexity with easy consumption DBaaSELK App2App1 App3 Middleware ML Creds/CertsMessaging ??? Container Services Container Hosts | Kubernetes Infrastructure Team
  10. 10. “In general, taking something that’s already working somewhere and expanding its usage (capabilities) is far more likely to succeed than building these capabilities from scratch”
  11. 11. Patches Patching App and System components as CVEs occur Scaling Seamlessly scale platform components to accommodate changing demand. Upgrades. How do you roll out new versions of the platform with the lights on? Operating Effort Operating the platform should require very few resources and minimum manual intervention. Otherwise, you will be spending lots on operational support! Development The team can make progress in developing new features for the platform CI/CD CI/CD pipelines drive the testing and promotion of artifacts Consistency Provide a consistent setup experience, across different environment configurations. Setup time How long does it take to setup a real world working environment? Think hours, not weeks. Day 1 - Build Day 2 - Operate & Enhance
  12. 12. Architecture
  13. 13. API Server Kube Scheduler K8s Master Controller Manager Etcd K8s Worker Pod Pod Pod CNI Kubelet Kube-proxy Docker
  14. 14. API Server Kube Scheduler K8s Master Controller Manager Kubelet Kube-proxy Etcd K8s Worker Pod Pod Pod CNI Docker Etcd
  15. 15. API Server Kube Scheduler K8s Master Controller Manager Etcd K8s Worker Pod Pod Pod CNI Kubelet Kube-proxy Docker
  16. 16. API Server Kube Scheduler K8s Master Controller Manager Etcd Kubelet Kube-proxy K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master Controller Manager
  17. 17. API Server Kube Scheduler K8s Master Controller Manager Etcd Kubelet Kube-proxy K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master Controller Manager API Server Kube Scheduler K8s Master Controller Manager EtcdEtcd us-tirefire-1a us-tirefire-1b us-tirefire-1c
  18. 18. High Availability
  19. 19. API Server Kube Scheduler K8s Master Controller Manager Etcd Kubelet Kube-proxy K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker
  20. 20. API Server Kube Scheduler K8s Master Controller Manager Etcd Kubelet Kube-proxy K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master Controller Manager
  21. 21. API Server Kube Scheduler K8s Master Controller Manager Etcd Kubelet Kube-proxy K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master Controller Manager EtcdEtcd
  22. 22. API Server Kube Scheduler K8s Master Controller Manager Etcd Kubelet Kube-proxy K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master Controller Manager
  23. 23. API Server Kube Scheduler K8s Master Controller Manager Etcd Kubelet Kube-proxy K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master Controller Manager EtcdEtcd
  24. 24. http://www.bsielearning.com.au/keep-simple-stupid/
  25. 25. Kubelet Kube-proxy K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod K8s Worker Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master Controller Manager Etcd Etcd
  26. 26. Deployment
  27. 27. How to Get an Kubernetes Are you in the “cloud”? yes Which cloud ? GKEAKS EKS Azure Google Amazon Do you want help? no GLHF Pivotal Container Service … ... https://kubernetes.io/partners no yes Other A laptop ? minikube no yes
  28. 28. Infrastructure Compute Network Monitoring Security Storage Kubernetes Cluster vSphere NSX Wavefront NSX Datastores Load Balancer Storage Requirements Availability Zone Security Policy Application Metrics
  29. 29. https://docs-cfcr.cfapps.io/ https://github.com/openshift/origin https://github.com/kubernetes-incubator/kubespray
  30. 30. Kubespray https://github.com/kubernetes-incubator/kubespray ● Ansible based, so very approachable ● An official Kubernetes (incubator) project ● Good support for CNIs and Cloud Providers ● Combine with one of the Ansible Hardening projects ○ https://github.com/dev-sec/ansible-os-hardening ○ https://github.com/openstack/ansible-hardening
  31. 31. gitops ● Deployed Platform == code repo + environment repo ○ Ansible - Playbook + Inventory ○ Bosh - Release + Manifest ● Keep it all in git! ○ Fork upstream repo… if only to ensure it doesn’t get changed from under you ○ Inventory/Manifest is probably YAML … perfect to be stored in git. ○ One repo for all envs, or a repo per env … either is fine. ● Consider using a gitops focussed wrapper around ansible ○ Ursula-cli (https://github.com/blueboxgroup/ursula-cli) ○ Gosible (https://github.com/paulczar/gosible) ○ Molecule (https://github.com/metacloud/molecule) ● Use Jenkins or similar to run tests, deploy test envs, push to prod??? ○ But probably not full on Continuous Delivery … risks are very high!
  32. 32. Validate and Backup Validate your Kubernetes cluster is conformant! https://github.com/heptio/sonobuoy Backup your Kubernetes cluster state! https://github.com/heptio/ark
  33. 33. Upgrades
  34. 34. API Server Kube Scheduler K8s Master Controller Manager Etcd K8s Worker Pod Pod Pod CNI Kubelet Kube-proxy Docker
  35. 35. API Server Kube Scheduler K8s Master 1.10 Controller Manager Etcd Kubelet Kube-proxy K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master 1.10 Controller Manager API Server Kube Scheduler K8s Master 1.10 Controller Manager EtcdEtcd us-tirefire-1a us-tirefire-1b us-tirefire-1c
  36. 36. API Server Kube Scheduler K8s Master 1.10 Controller Manager Etcd Kubelet Kube-proxy K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master 1.10 Controller Manager API Server Kube Scheduler K8s Master 1.10 Controller Manager EtcdEtcd us-tirefire-1a us-tirefire-1b us-tirefire-1c API Server Kube Scheduler K8s Master 1.11 Controller Manager
  37. 37. API Server Kube Scheduler K8s Master 1.11 Controller Manager Etcd Kubelet Kube-proxy K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master 1.10 Controller Manager API Server Kube Scheduler K8s Master 1.10 Controller Manager EtcdEtcd us-tirefire-1a us-tirefire-1b us-tirefire-1c
  38. 38. API Server Kube Scheduler K8s Master 1.11 Controller Manager Etcd Kubelet Kube-proxy K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master 1.11 Controller Manager API Server Kube Scheduler K8s Master 1.11 Controller Manager EtcdEtcd us-tirefire-1a us-tirefire-1b us-tirefire-1c
  39. 39. API Server Kube Scheduler K8s Master 1.11 Controller Manager Etcd Kubelet Kube-proxy K8s Worker 1.11 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master 1.11 Controller Manager API Server Kube Scheduler K8s Master 1.11 Controller Manager EtcdEtcd us-tirefire-1a us-tirefire-1b us-tirefire-1c
  40. 40. API Server Kube Scheduler K8s Master 1.11 Controller Manager Etcd Kubelet Kube-proxy K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod K8s Worker 1.10 Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master 1.11 Controller Manager API Server Kube Scheduler K8s Master 1.11 Controller Manager EtcdEtcd us-tirefire-1a us-tirefire-1b us-tirefire-1c
  41. 41. API Server Kube Scheduler K8s Master 1.11 Controller Manager Etcd Kubelet Kube-proxy K8s Worker 1.11 Pod Pod Pod K8s Worker 1.11 Pod Pod Pod K8s Worker 1.11 Pod Pod Pod CNI CNI CNI Docker Kubelet Kube-proxy Docker Kubelet Kube-proxy Docker API Server Kube Scheduler K8s Master 1.11 Controller Manager API Server Kube Scheduler K8s Master 1.11 Controller Manager EtcdEtcd us-tirefire-1a us-tirefire-1b us-tirefire-1c
  42. 42. Operations
  43. 43. Monitoring / Logging - The Platform Server Agents ● Install as binaries / containers on the underlying OS ● No chicken and egg problems ● Extra devops toil (config management etc) ● Direct access to system metrics and logs ● Can use existing tools / processes Daemonsets ● Run in Kubernetes on each node as daemonset ● If Kubernetes is broken, will the monitoring daemonset be broken ? ● Have to be able to dockerize the agent ● Privileged containers / host volumes to access system metrics and logs ● Masters also have to be workers or can’t run daemonset on them.
  44. 44. Monitoring / Logging - Workloads Kubernetes Metrics API ● Basic point in time pod/node metrics ● $ kubectl top {node,pod} ● Adaptors for Prometheus / Graphite / etc Kubernetes logging ● Kubernetes configures docker to log all Pod stdout/sterr to a file ● $ kubectl logs <name-of-pod> ● Need daemonset or agent to read k8s logs from filesystem ● EFK - Elastic, Fluent, Kibana
  45. 45. Authentication / Access Control ● Node vs ABAC vs RBAC ● Service Accounts - managed inside kubernetes ● User Accounts - managed outside kubernetes ○ OpenID Connect ■ Ldap / AD ■ Oauth2 ■ Etc ● Secure your Kubernetes Dashboard silly! ○ Everything is TLS encrypted …. Right ? $ kubectl auth can-i create deployments --namespace dev yes $ kubectl auth can-i create deployments --namespace prod no
  46. 46. Value!!
  47. 47. ● Leverage features in modern cloud platforms ○ Blue/Green deploys ○ Auto-healing ○ Auto-scaling ○ Advanced routing/networking automation ● Design and build based on known Cloud Native patterns ● Longer term investment in the application ● Likey you need access to the code ● Plus everything mentioned in “replatforming” ● Lift and Shift with “just enough modernization” ● You may not have access to the code ● Revisit decisions made in Greenfield time ○ Around CI/CD process ● Get some quick wins through platform capabilities ○ Reduced operating and infrastructure cost ○ Improved speed to deploy & scale ○ Faster patching of kernel level vulnerabilities Replatforming vs Modernization for PKS Lift & Shift / Replatforming Modernization
  48. 48. TIME Methodology TECHNICALQUALITY BUSINESS VALUEWORSE BETTER WORSEBETTER Tolerate Invest MigrateEliminate * Gartner’s TIME methodology for Application Portfolio Rationalization TECHNICAL QUALITY - Technical Debt Level BUSINESS VALUE - Revenue / Cost Impact Identify top 10s list
  49. 49. APP APP APP APP 1 Identify 5-10 apps confirmed as suitable to run on PKS 2 Work on a short project to push a few apps all the way to prod and measure the ROI metrics SampleToolChain Gitlab Concourse
  50. 50. PLATFORM VALUE STREAM AND METRICS REPLATFORM > MODERNIZE > OPTIMIZE ESTABLISH, MEASURE AND UPDATE KEY OBJECTIVES AND RESULTS (OKRs) SPEED & AGILITY STABILITY SCALABILITY SAVINGS $SECURITY 40-60%* More Projects With Same Staff Millions Annual Savings on HW, SW and Support 25-50%* Fewer Support Incidents 40%* Faster Patching Delivery @ Zero Downtime -90%* Time to Scale $ $ % How We Think about the Business Case
  51. 51. Transforming How The World Builds Software © Copyright 2018 Pivotal Software, Inc. All rights Reserved.

×