Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From Rollercoasters to Meerkats: 3 Generations of Production Kubernetes Clusters

423 views

Published on

A walk through the various generations of Kubernetes deployments by Jetstack, and a reasoning for the development and launch of Tarmak the open-source toolkit for Kubernetes cluster lifecycle management.

Published in: Software
  • Be the first to comment

From Rollercoasters to Meerkats: 3 Generations of Production Kubernetes Clusters

  1. 1. jetstack.io From Rollercoasters to Meerkats: 3 Generations of Production Kubernetes Clusters Presented by Matthew Bates / Christian Simon
  2. 2. jetstack.io WHO ARE WE? Matt Bates Co-founder twitter.com/mattbates25 github.com/mattbates Christian Simon Senior Solution Engineer twitter.com/simonswine github.com/simonswine
  3. 3. jetstack.io KUBERNETES What is Kubernetes? Kubernetes Master Node Node Node Declarative description of application workloads Workloads scheduled onto nodes and actively managed ● Declarative system description using application abstractions ● Abstracts away the servers so developers can concentrate on code ● Infrastructure-agnostic ● Pro-actively monitors, scales, auto-heals and updates
  4. 4. jetstack.io KUBERNETES Exploring the cluster map kubectl storage (etcd) API server scheduler controller-manager Node kubelet proxy docker Node kubelet proxy docker Control Plane Other clients
  5. 5. jetstack.io REPLICA SETS Actual vs. Desired state apiVersion: core/v1 kind: ReplicaSet metadata: name: nginx spec: replicas: 1 template: [...] API server kubectl Desired state Actual state nginx
  6. 6. jetstack.io REPLICA SETS Actual vs. Desired state apiVersion: core/v1 kind: ReplicaSet metadata: name: nginx spec: replicas: 1 template: [...] status: replicas: 0 API server controllers Desired state Actual state nginx nginx
  7. 7. jetstack.io REPLICA SETS Actual vs. Desired state apiVersion: core/v1 kind: ReplicaSet metadata: name: nginx spec: replicas: 1 template: [...] status: replicas: 1 Desired state Actual state nginx nginx API server controllers
  8. 8. jetstack.io REPLICA SETS Actual vs. Desired state apiVersion: core/v1 kind: ReplicaSet metadata: name: nginx spec: replicas: 3 template: [...] status: replicas: 1 API server kubectl Desired state Actual state nginx nginx nginx nginx
  9. 9. jetstack.io REPLICA SETS Actual vs. Desired state apiVersion: core/v1 kind: ReplicaSet metadata: name: nginx spec: replicas: 3 template: [...] status: replicas: 1 API server controllers Desired state Actual state nginx nginx nginx nginx nginx nginx
  10. 10. jetstack.io REPLICA SETS Actual vs. Desired state apiVersion: core/v1 kind: ReplicaSet metadata: name: nginx spec: replicas: 3 template: [...] status: replicas: 3 API server controllers Desired state Actual state nginx nginx nginx nginx nginx nginx
  11. 11. jetstack.io CLUSTER PROVISIONING Provisioning Kubernetes clusters storage (etcd) API server scheduler controller-manager Node kubelet proxy docker Node kubelet proxy docker Control Plane
  12. 12. jetstack.io FIRST GENERATION ● CoreOS ● Infrastructure: CloudFormation ● Config: Cloud-Init Bash ● Ruby scripts to drive CloudFormation “Rollercoasters”
  13. 13. jetstack.io SECOND GENERATION ● CoreOS ● Infrastructure: Terraform ● Config: Cloud-Init Bash ● Shell script/Makefile to wrap Terraform ● All wrapped-up in Docker “Elevation”
  14. 14. jetstack.io LESSONS LEARNED ● Immutable infrastructure isn’t always that desireable ● Testing and debugging ability is critical for development - and operations ● All dependencies need to be versioned ● Cluster PKI management is not easy ● Advantages/disadvantages of declarative infrastructure (Terraform et al) abstraction Lessons learned through the first generations of cluster deployment
  15. 15. jetstack.io NEXT-GEN MOTIVATIONS ● Quick feedback loop for developers ○ Testability (unit, integration) ● Reusability of code ○ Follow DRY, KISS principles ● Continuous roll-out of changes ● Dry-run capability and meaningful output ● Tried and tested, well-understood tools Developer/operator experience
  16. 16. jetstack.io NEXT-GEN MOTIVATIONS ● Immutable infrastructure updates can be expensive and slow ● Desired vs actual state for configuration updates ● Stateful applications (caches, risks of migration) Less disruptive upgrades
  17. 17. jetstack.io NEXT-GEN MOTIVATIONS ● All Kubernetes solutions behave differently ● Abstractions ○ Provider/Infrastructure ○ Operating system ○ Version of the components Consistency across different infrastructure
  18. 18. jetstack.io NEXT-GEN MOTIVATIONS Consistency across different Infrastructures
  19. 19. jetstack.io NEXT-GEN MOTIVATIONS Consistency across different Infrastructures
  20. 20. jetstack.io NEXT-GEN MOTIVATIONS Consistency across different Infrastructures
  21. 21. jetstack.io “Tarmak is an open-source toolkit for Kubernetes cluster lifecycle management. It focuses on best-practice cluster security and operations. It’s built from the ground-up to be cloud provider-agnostic, and give consistent and reliable cluster deployment and management, across cloud and on-premises environments.”
  22. 22. jetstack.io ● Build (cloud) infrastructure with Terraform ● Instances types have roles ● Puppet manifests artifact uploaded to object storage TARMAK Infrastructure
  23. 23. jetstack.io ● Instance Cloud-Init runs Wing agent on startup ● Wing downloads Puppet artifact ● Wing runs the puppet-agent TARMAK Configuration management
  24. 24. jetstack.io WING Deep-dive ● Using puppet apply means we run masterless ○ reduces complexity ○ reduces insights ● Wing API Server as single source of truth, stores actual and desired state ● Wing Agent on every instance, evaluates states and acts accordingly
  25. 25. jetstack.io WING Architecture tarmak storage (etcd) wing-server instance-a wing puppet instance-b wing puppet Wing server
  26. 26. jetstack.io WING Actual vs. Desired state apiVersion: wing/v1alpha1 kind: Instance metadata: name: instance-a spec: converge: hash: sha16:ffaa tarmak
  27. 27. jetstack.io WING Actual vs. Desired state apiVersion: wing/v1alpha1 kind: Instance metadata: name: instance-a spec: converge: hash: sha16:ffaa status: converge: hash: sha16:ffaa state: converging wing (instance-a) puppet apply
  28. 28. jetstack.io WING Actual vs. Desired state apiVersion: wing/v1alpha1 kind: Instance metadata: name: instance-a spec: converge: hash: sha16:ffaa status: converge: hash: sha16:ffaa state: converged wing (instance-a)
  29. 29. Demo time jetstack.io
  30. 30. jetstack.io PUPPET MODULES Open source modules on PuppetForge ● aws_ebs attach, format, mount EBS volumes ● calico calico overlay network ● etcd etcd kv-store ● kubernetes kubernetes core components ● kubernetes_addons additional kubernetes components ● prometheus monitoring using prometheus ● tarmak tarmak kubernetes cluster ● vault_client get secrets, TLS credentials from vault
  31. 31. jetstack.io MEERKATS What about them?
  32. 32. github.com/jetstack/tarmak docs.tarmak.io @JetstackHQ Thanks! Get involved jetstack.io
  33. 33. jetstack.io TARMAK RESOURCES Providers, Environments and Clusters
  34. 34. WHO ARE JETSTACK? jetstack.io We are a UK-based company that help enterprises in their path to modern cloud-native infrastructure. We also develop tooling and integrations for Kubernetes to improve user experience and operations for customers and end-users alike. UK/EU customers include major e-commerce websites, telcos, media and publishers and government departments.

×