Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Azure Kubernetes Service - benefits and challenges


Published on

Working with AKS for more then 3 months, I want to share my experience. I discuss benefits of AKS and some issues you might have. K8S is damn close to a silver bullet in regards of the simplicity to work with.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Azure Kubernetes Service - benefits and challenges

  1. 1. AZURE KUBERNETES SERVICE Wojciech Barczynski - | Wrzesień 2018
  2. 2. WOJCIECH BARCZYŃSKI Lead So ware Engineer & System Engineer Interests: working so ware Hobby: teaching so ware engineering
  3. 3. BACKGROUND ML FinTech ➡ microservices and k8s Before: 1 z 10 Indonesian mobile ecommerce (Rocket Internet) Spent 3.5y with Openstack, 1000+ nodes, 21 data centers I do not like INFRA :D
  4. 4. DLACZEGO? Admistracja jest trudna i kosztowna Virtualne Maszyny, ansible, salt, etc. Za dużo ruchomych części Nie kończąca się standaryzacja
  6. 6. DLACZEGO? Chmura jednak $$$
  7. 7. IMAGINE Świat bez wiedzy o IaaS żadnego konfigurowania na nodzie mniej dyskusji o CI / CD ... Środowisko jak czarna skrzynka
  8. 8. KUBERNETES Simple Semantic* Batteries for your 12factory apps Service discovery, meta-data support Independent from IaaS provider
  9. 9. GOALS Utilzie resources to early 100% Application and services mindset
  10. 10. KUBERNETES
  11. 11. KUBERNETES Kubernetes Node Node Node Node App Ingress Controller Repository make docker_push; kubectl create -f app-srv-dpl.yaml
  12. 12. SCALE UP! SCALE DOWN! Kubernetes Node Node Node Node App Ingress Controller App Appscale 3x kubectl --replicas=3 -f app-srv-dpl.yaml
  13. 13. SCALE UP! SCALE DOWN! Kubernetes Node Node Node Node Ingress Controller scale 1x App kubectl --replicas=1 -f app-srv-dpl.yaml
  14. 14. ROLLING UPDATES! Kubernetes Node Node Node Node App Ingress Controller App App kubectl set image deployment/app app=app:v2.0.0
  15. 15. ROLLING UPDATES! Kubernetes Node Node Node Node App Ingress Controller App App
  16. 16. ROLLING UPDATES! Kubernetes Node Node Node Node App Ingress Controller App App
  18. 18. INGRESS Pattern Target App Service users-v1 users-v2 web
  19. 19. SERVICE DISCOVERY names in DNS: curl labels: name=value annotations: "true" http://users/list
  20. 20. SERVICE DISCOVERY loosely couple components auto-wiring with logging and monitoring
  21. 21. LOAD BALANCING Kubernetes Worker Kubernetes Worker Kubernetes Worker Node Port 30000 Node Node Kubernetes Worker Node user-232 <<Requests>> B users Port 30000 Port 30000 Port 30000 Load Balancer user-12F user-32F
  22. 22. RESISTANCE! Kubernetes Node Node Node Node App Ingress Controller App App
  23. 23. RESISTANCE! Kubernetes Node Node Node Node App Ingress Controller App App
  24. 24. RESISTANCE! Kubernetes Node Node Node Node App Ingress Controller App App
  25. 25. RESISTANCE! When a node dies When other apps eats all memory Draining nodes before upgrade You can easily scale up, create machine, and join it to cluster
  26. 26. FEDERATION Global LoadBalancer App App App On-premise Poland Amazon eu-west-1 Google asia-southeast1
  27. 27. CONFIGURATION FILES Yaml easy to generate and work with
  29. 29. OPTIONS AKS - managed ACS - installation wizard Your own installation with Installer
  30. 30. AKS GKE for Google EKS or Fargate for Amazon
  31. 31. AKS Independent from IaaS Our OnPrem = Our OnCloud Consolidation of our micro-services Plug and play, e.g., monitoring
  32. 32. BEGINNINGS my experience
  33. 33. AKS You: k8s workers Azure: k8s masters
  34. 34. AKS You: upgrade your k8s Azure: update your kube-system pods, k8s config, and nodes
  35. 35. AZURE UPDATES Bumpy road ahead Kube-Systen pods Kubernetes configuration - ☠ System: on the node restart applied System: Memory-preserving updates - ☠
  36. 36. AZURE UPDATES - NODES kubectl get nodes -o wide NAME VERSION OS-IMAGE KERN aks-nodepool1-27173880-0 v1.10.3 Ubuntu 16.04.4 LTS 4.15 aks-nodepool1-27173880-1 v1.10.3 Ubuntu 16.04.4 LTS 4.15 aks-nodepool1-27173880-2 v1.10.3 Ubuntu 16.04.4 LTS 4.15 aks-nodepool1-27173880-3 v1.10.3 Ubuntu 16.04.4 LTS 4.15 aks-nodepool1-27173880-5 v1.10.3 Ubuntu 16.04.4 LTS 4.15
  37. 37. AZURE UPDATES - K8S Scalling down / up your cluster applies the newest k8s config changes
  38. 38. AZURE K8S INTEGRATION Load Balancers Persistence Volumes Graphic Cards Support Authentication with oauth Monitoring?
  39. 39. LIMITS No node-pool support RBAC? No limited centralized logging Federation support
  40. 40. PAIN POINTS Memory preserving updates AKS team changes configuration
  41. 41. ANNOYING Slow deletes Slow attaching and detaching volumes You are not able to delete a pod without --force
  42. 42. LOVE Openess on github: AKS issues
  43. 43. CREATE az aks create --name portal-production --resource-group MYCOMPANY --node-vm-size 'Standard_D4_v2' --node-count 4 --generate-ssh-keys
  44. 44. CREATE GO TO PORTAL ssh to nodes
  45. 45. READY TO GO! az aks get-credentials -g MYCOMP -n portal-prod kubectl get pods kubectl get pods -n kube-system
  46. 46. UPDATE Name MasterVersion NodePoolVersion Upgrades ------- --------------- ----------------- ----------------- default 1.10.3 1.10.3 1.10.5, az aks get-upgrades -g MYCOMP --name portal-dev -o table
  47. 47. UPGRADE Do not rush! az aks upgrade --name portal-dev --resource-group MYCOMP --output table --kubernetes-version 1.10.3
  48. 48. UPGRADE Do not rush! az aks upgrade --name portal-dev --resource-group MYCOMP --output table --kubernetes-version 1.10.3
  49. 49. DATABASES Usually outside the cluster Mysql Postgres Mongodb You migth consider Azure Database
  50. 50. AKS @ SMACC
  51. 51. SETUP AZURE az aks CLI for setting k8s Terraform for everything else TF also sets our AWS
  52. 52. TECH Golang dla mikroserwisów Python dla wszystkiego Machine Learning JS i Emberjs dla webui OpenAPI ML Pipeline - evaluate kubeflow i patchyderm
  53. 53. KUBERNETES Pure, generated, kubernetes config 2x kubernetes operators
  54. 54. CONTINUOUS DEPLOYMENT Github TravisCI Kubernetes In spirit similar to the Kelsey Hightower approach
  55. 55. BACKUP Ark CronJobs for some components
  56. 56. ENVIRONMENTS Env Number of Nodes Prod 7 Staging 5 Dev 4 Tools 1 We also have short-living ML clusters
  57. 57. SUMMARY Kubernetes not a silver bullet, but damn close AKS the easiest way to start with k8s in Azure Still bumpy period - see github issues
  58. 58. DZIĘKUJĘ. PYTANIA? ps. We are hiring.
  60. 60. HIRING Senior Polyglot So ware Engineers Experienced System Engineers Frontend Engineers 1 Data-Driven Product Manager Apply:, Questions?, or We will teach you Go if needed. No k8s or ML, we will take care of that. FB LI
  61. 61. 0.1 ➡ 1.0
  62. 62. 1. CLEAN UP Single script for repo - Makefile [1] Resurrect the README [1] With zsh or bash auto-completion plugin in your terminal.
  63. 63. 2. GET BACK ALL THE KNOWLEDGE Puppet, Chef, ... ➡ Dockerfile Check the instances ➡ Dockerfile, README.rst Nagios, ... ➡ README.rst, checks/
  64. 64. 3. INTRODUCE RUN_LOCAL make run_local A nice section on how to run in README.rst Use: docker-compose The most crucial point.
  65. 65. 4. GET TO KUBERNETES make kube_create_config make kube_apply Generate the yaml files if your envs differ
  66. 66. 5. CONTINUOUS DEPLOYMENT Travis: test code, build docker, push to docker repo only run the rolling update: kubectl set image deployment/api- status nginx=nginx:1.9.1 did not create any kubernetes artificats [*] [*]
  67. 67. 6. KEEP IT RUNNING Bridge the new with old: Use external services in Kubernetes Add Kubernetes services to your Service Discovery [1] [1] I evaluated feeding K8S events to HashiCorp consul
  68. 68. Service <External> Rabbit Prometheus Exporter RabbitMQ Google AWS Kubernetes EC2 http://rabbitmq:15672 IP 9090
  69. 69. 7. INTRODUCE SMOKE-TEST TARGET_URL=127.0.0 make smoke_test make smoke_test
  70. 70. 8. GOT FIRST MICRO-SERVICES To offload the biggest components: Keep the light on of the old components New functionality delegated to micro-services
  71. 71. 9. GET PERFORMANCE TESTING introduce wrk for evaluating performance (more like a check for dockers) load test the real system
  72. 72. SERVICE SELF-CONSCIOUSNESS Add to old services: 1. metrics/ 2. health/ 3. info/ 4. alertrules/ - PoC
  73. 73. CHANGE THE WORK ORGANIZATION From Scrum To Kanban For the next talk
  75. 75. Node Master Deployment Docker containers Node Docker Containers Pod Schedule Schedule
  76. 76. Node Deployment Pod Node Pod services Fixed virtual address Fixed DNS entry
  77. 77. PODS See each other on localhost Live and die together Can expose multiple ports Pod nginx WebFiles ENV: HOSTNAME= LISTENPORT=8080
  78. 78. SIDE-CARS Pod memcached prometheus exporter Pod app swagger-ui 8080 80 9150 11211
  79. 79. BASIC CONCEPTS Name Purpose Service Interface Entry point (Service Name) Deployment Factory How many pods, which pods Pod Implementation 1+ docker running
  80. 80. ROLLING RELEASE WITH DEPLOYMENTS Service Pods Labels Deployment Deployment << Creates >> << Creates >> Also possible