Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Zero-downtime deployment of Micro-services with Kubernetes

103 views

Published on


Talk on deployment strategies with Kubernetes covering kubernetes configuration files and the actual implementation of your service in Golang.

You will find demos for recreate, rolling updates, blue-green, and canary deployments.

Source and demos, you will find on github: https://github.com/wojciech12/talk_zero_downtime_deployment_with_kubernetes

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Zero-downtime deployment of Micro-services with Kubernetes

  1. 1. DEPLOYMENT STRATEGIES FOR YOUR MICRO-SERVICE WITH K8S + HOW TO PREPARE YOUR SERVICE Wojciech Barczynski - SMACC.io | Hypatos.ai Listopad 2018
  2. 2. WOJCIECH BARCZYŃSKI Lead So ware Engineer & System Engineer Interests: working so ware Hobby: teaching so ware engineering
  3. 3. BACKGROUND ML FinTech ➡ micro-services and k8s Before: 1 z 10 Indonesian mobile e-commerce (Rocket Internet) Spent 3.5y with Openstack, 1000+ nodes, 21 data centers I do not like INFRA :D
  4. 4. STORY Lyke - [12.2016 - 07.2017] SMACC - [10.2017 - present]
  5. 5. AGENDA. Mikroserwises Dlaczego kubernetes? Strategie deploymentu Jak przygotować mikroserwis?
  6. 6. MICRO-SERVICES Scalling the team ... Scalling the products
  7. 7. MICRO-SERVICES Independent Following 12factor app Self-aware
  8. 8. 12FACTOR APP Heroku - 2011 App easy to run in production Low TCO Easy to manage
  9. 9. MICRO-SERVICES /healthz /metrics /readiness /info
  10. 10. MICRO-SERVICES Do not need to share code Share the same conventions Every git repos looks familiar Might be in different tech
  11. 11. MICRO-SERVICES Not a silver bullet :)
  12. 12. KUBERNETES
  13. 13. WHY? Admistracja jest trudna i kosztowna Virtualne Maszyny, ansible, salt, etc. Za dużo ruchomych części Nie kończąca się standaryzacja
  14. 14. MIKROSERWISY AAA!
  15. 15. WHY? Cloud is not so cheap - $$$
  16. 16. IMAGINE do not need to think about IaaS no login on a VM less gold plating your CI / CD ... DC as a black box
  17. 17. KUBERNETES Container management Service and application mindset Simple Semantic* Independent from IaaS provider
  18. 18. KUBERNETES Batteries for your 12factory apps Service discovery, meta-data support Utilize resources to nearly 100%
  19. 19. KUBERNETES Kubernetes Node Node Node Node App Ingress Controller Repository make docker_push; kubectl create -f app-srv-dpl.yaml
  20. 20. SCALE UP! SCALE DOWN! Kubernetes Node Node Node Node App Ingress Controller App Appscale 3x kubectl --replicas=3 -f app-srv-dpl.yaml
  21. 21. SCALE UP! SCALE DOWN! Kubernetes Node Node Node Node Ingress Controller scale 1x App kubectl --replicas=1 -f app-srv-dpl.yaml
  22. 22. ROLLING UPDATES! Kubernetes Node Node Node Node App Ingress Controller App App kubectl set image deployment/app app=app:v2.0.0
  23. 23. ROLLING UPDATES! Kubernetes Node Node Node Node App Ingress Controller App App kubectl set image deployment/app app=app:v2.0.0
  24. 24. ROLLING UPDATES! Kubernetes Node Node Node Node App Ingress Controller App App kubectl set image deployment/app app=app:v2.0.0
  25. 25. RESISTANCE! Kubernetes Node Node Node Node App Ingress Controller App App
  26. 26. RESISTANCE! Kubernetes Node Node Node Node App Ingress Controller App App
  27. 27. RESISTANCE! Kubernetes Node Node Node Node App Ingress Controller App App
  28. 28. HOW GET USER REQUESTS? API BACKOFFICE 1 DATA WEB ADMIN BACKOFFICE 2 BACKOFFICE 3 API.DOMAIN.COM DOMAIN.COM/WEB BACKOFFICE.DOMAIN.COM ORCHESTRATOR PRIVATE NETWORKINTERNET API LISTEN (DOCKER, SWARM, MESOS...) Ingress Controller
  29. 29. INGRESS Pattern Target App Service api.smacc.io/v1/users users-v1 api.smacc.io/v2/users users-v2 smacc.io web
  30. 30. LOAD BALANCING Kubernetes Worker Kubernetes Worker Kubernetes Worker Node Port 30000 Node Node Kubernetes Worker Node user-232 <<Requests>> B users Port 30000 Port 30000 Port 30000 Load Balancer user-12F user-32F
  31. 31. SERVICE DISCOVERY names in DNS: curl labels: name=value annotations: prometheus.io/scrape: "true" http://users/list
  32. 32. SERVICE DISCOVERY loosely couple components auto-wiring with logging and monitoring
  33. 33. DROP-IN traefik / Ingress / Envoy prometheus audit checks ...
  34. 34. THE BEST PART All live in git: all in Yaml integration with monitoring, alarming integration with ingress-controller ... Devs can forget about infrastructure... almost DevOps Culture Dream!
  35. 35. DEPLOYMENT STRATEGIES
  36. 36. STRATEGIES We will see: Replace (downtime visible) Rolling updates Blue Green Canary
  37. 37. OTHER We will not cover: Feature toggles A/B like Shadow deployment
  38. 38. FIRST THE HOMEWORK Need to support: liveness - am I dead? readiness - can I serve requests?
  39. 39. KUBE LIVENESS PROBE livenessProbe: httpGet: path: /model port: 8000 httpHeaders: - name: X-Custom-Header value: Awesome initialDelaySeconds: 600 periodSeconds: 5 timeoutSeconds: 18 successThreshold: 1 failureThreshold: 3
  40. 40. LIVENESS PROBE our pod gets restarted too many restarts -> CrashLoop
  41. 41. KUBE READINESS PROBE readinessProbe: exec: command: - cat - /tmp/healthy initialDelaySeconds: 5 periodSeconds: 5
  42. 42. YOUR APP SHOULD ON STOP when we get SIGTERM signal app gives 500 on readinessProbe app does not receive new requests app graceful shutdown kuberenetes forces kill if 30 limit exceeded
  43. 43. ALWAYS Implement readiness for: ML Model-based components slow starting time
  44. 44. DEMO - RECREATE Service Pods Labels Service Pods Labels v1 v1 v1 v2 v2 v2
  45. 45. DEMO - RECREATE spec: replicas: 3 strategy: type: Recreate kubectl set image deployment/demo-api app=wojciech11/api-status:2.0.0
  46. 46. DEMO - RECREATE quick downtime visible
  47. 47. DEMO - ROLLING UPDATES Service Pods Labels Service Pods Labels v1 v1 v1 v1 v1 v2
  48. 48. DEMO - ROLLING UPDATES strategy: type: RollingUpdate rollingUpdate: maxSurge: 2 maxUnavailable: 0 docs
  49. 49. DEMO - ROLLING UPDATES kubectl set image deployment/demo-api app=wojciech11/api-status:2.0.0
  50. 50. DEMO - ROLLING UPDATES the most popular
  51. 51. DEMO - GREEN/BLUE Pods Service Pods Blue Green
  52. 52. DEMO - GREEN/BLUE Pods Service Pods Blue Green
  53. 53. DEMO - GREEN/BLUE kubectl patch service api-status -p '{"spec":{"selector": {"label": "green"} }}'
  54. 54. DEMO - GREEN/BLUE For big changes Less common Might be implemented with Ingress
  55. 55. DEMO - CANARY Pods Service Pods Production Canary
  56. 56. DEMO - CANARY Pods Service Pods Production Canary
  57. 57. DEMO - CANARY Pods Service Pods Production Canary
  58. 58. DEMO - CANARY Pods Service Pods Production Canary
  59. 59. DEMO - CANARY Pods Service Pods Production Canary
  60. 60. kubectl scale --replicas=3 deploy/api-status-nginx-blue kubectl scale --replicas=1 deploy/api-status-nginx-green # no errors, let's continoue kubectl scale --replicas=2 deploy/api-status-nginx-blue kubectl scale --replicas=2 deploy/api-status-nginx-green
  61. 61. DEMO - CANARY manually with help of Traefik / Istio / ...
  62. 62. SUMMARY kubernetes simple semantic easy deployment of your applications will work for any application type
  63. 63. DZIĘKUJĘ. PYTANIA? ps. We are hiring.
  64. 64. BACKUP SLIDES
  65. 65. STORY Lyke - [12.2016 - 07.2017] SMACC - [10.2017 - present]
  66. 66. LYKE Now JollyChic Indonesia E-commerce Mobile-only 50k+ users 2M downloads Top 10 Fashion Apps w Google Play Store http://www.news.getlyke.com/single- post/2016/12/02/Introducing-the- New-Beautiful-LYKE
  67. 67. GOOD PARTS Fast Growth A/B Testing Data-driven Product Manager, UI Designer, Mobile Dev, and tester - one body
  68. 68. CHALLENGES 50+ VMs in Amazon, 1 VM - 1 App, idle machine Puppet, hilarious (manual) deployment process Fear Forgotten components sometimes performance issues
  69. 69. SMACC Machine Learning FinTech SaaS and API platform From Enterprise (Deutsche Bank, AoK) to SME Well-known FinTech Startup in Germany
  70. 70. STORY Legacy on AWS, experiments with AWS ECS :/ Self-hosted K8S on ProfitBricks Get to Microso ScaleUp, welcome Azure Luckily - Azure-Kubernetes-Service
  71. 71. DIFFERENCE ☠ Two teams in Berlin and Warsaw Me in Warsaw
  72. 72. APPROACH Simplify, Simplify Hide K8S magic git tag driven Continoues Deployment
  73. 73. KUBERNETES CONCEPTS
  74. 74. Node Master Deployment Docker containers Node Docker Containers Pod Schedule Schedule
  75. 75. Node Deployment Pod Node Pod services Fixed virtual address Fixed DNS entry
  76. 76. PODS See each other on localhost Live and die together Can expose multiple ports Pod nginx WebFiles ENV: HOSTNAME=0.0.0.0 LISTENPORT=8080
  77. 77. SIDE-CARS Pod memcached prometheus exporter Pod app swagger-ui 8080 80 9150 11211
  78. 78. BASIC CONCEPTS Name Purpose Service Interface Entry point (Service Name) Deployment Factory How many pods, which pods Pod Implementation 1+ docker running

×