Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis

347 views

Published on

Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis @Agile Israel 2019

Published in: Leadership & Management
  • Be the first to comment

  • Be the first to like this

Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis

  1. 1. Copyright @ 2019 JFrog - All rights reserved Kubernetes is hard! Lessons learned taking our apps to Kubernetes Eldad Assis | DevOps Architect | JFrog
  2. 2. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Me Email: eldada@jfrog.com Twitter: @eldadak LinkedIn: Eldad Assis What I do - I (try to) solve problems @ JFrog - DevOps Architect - Bringing Dev and Ops closer for over 15 years - Doing CI/CD since the turn of the century
  3. 3. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Docker Containers ● Kubernetes An open-source system for automating deployment, scaling, and management of containerized applications. No deep diving. No Demo… sorry Come see me after the talk for more! For this talk, I assume you know a bit Recommended knowledge
  4. 4. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! 1. I need a running environment. Now! a. Developers, QA, Support, Product Managers, Solution… anyone! 2. Per branch CI/CD a. Full CI to Integrate my branch with other products development branches 3. Better utilize our resources for dev and production 4. Support a new distribution for JFrog products Why? (The problems we wanted to solve) The journey to Kubernetes begins...
  5. 5. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Why? (Taking containers to production) Docker in development vs production Docker container Docker container Networking Security Monitoring Logging Auto scaling Auto healing Development Production
  6. 6. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Start small ● Get the application ready before jumping into Kubernetes ● Security - control what’s running in your cluster ● Limits ● Health probes ● Visibility - usable and accessible monitoring and logging systems ● Spread the word and work with the community! The End What you should take from here
  7. 7. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Kubernetes - the myth ZZZZ….
  8. 8. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Kubernetes - the reality Hmmm….
  9. 9. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! 1. See a tweet on a new technology! (WOW - THIS IS COOL!) 2. See video showing how easy it is (LOOKS EASY!) 3. Try running it yourself (HMM… ) 4. Fail miserably (WOW - THIS IS HARD!) 5. Feel stupid 6. Try again (GO TO 3) How I started A recipe for a successful new technology adoption
  10. 10. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Start with a small application example (nginx hello world) ● Use existing demos ○ Understand it. What each line in the yaml actually means ● Set a minimal goal for getting your app to run in Kubernetes ● Get comfortable before you move on ○ Start with managed K8s for easy setup (AKS, EKS, GKE) Where should I start? Start small!
  11. 11. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! A lot has to be done on your application before you can comfortably run it in Kubernetes Here are some key points to consider when planning your move to Kubernetes Start with the application! Is your application ready to run in Kubernetes?
  12. 12. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Logging ○ STDOUT/STDERR ○ Handling multiple log files ● Data Persistency ○ What kind of data needs persistency (if at all)? ● Proper handling of termination signals to init a proper shutdown ○ Controlled shutdown of the application ○ Easier recovery (after a controlled shutdown) ● Survive a restart ○ Managing leftovers from previous run Is your app ready for Kubernetes? It’s not just pushing it to Docker...
  13. 13. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Improve durability and availability ● Support running multiple instances of your application with load balancing between them ○ Scaling up and down will be easy (horizontal scaling) ● Rolling upgrades for zero downtime ○ Backward compatibility ● Zero service unavailability due to pod rescheduling ○ Cluster scaling (down) ○ Pod evicted or crashed ○ Unplanned outage of a node Is your app ready for Kubernetes? High availability as the new default!
  14. 14. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! I. Codebase - One codebase tracked in revision control, many deploys II. Dependencies - Explicitly declare and isolate dependencies III. Config - Store config in the environment IV. Backing services - Treat backing services as attached resources V. Build, release, run - Strictly separate build and run stages VI. Processes - Execute the app as one or more stateless processes Is your app ready for Kubernetes? The Twelve-Factor App (https://12factor.net/) VII. Port binding - Export services via port binding VIII. Concurrency - Scale out via the process model IX. Disposability - Maximize robustness with fast startup and graceful shutdown X. Dev/prod parity - Keep development, staging, and production as similar as possible XI. Logs - Treat logs as event streams XII. Admin processes - Run admin/management tasks as one-off processes
  15. 15. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Your application is rarely the only thing in its pod ○ OS packages, OSS libs, 3rd party processes ● Security vulnerabilities ● Do you use public images from unknown sources? ● FOSS licenses compliance Security What’s running in your cluster?
  16. 16. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Once your application is ready, let’s talk about to how to properly manage your applications run-time and configuration in Kubernetes And now - Kubernetes Properly running in Kubernetes
  17. 17. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Pod limits. Always! ● The app inside might need its own limits. For example: ○ Java apps ■ Limit the java process with `-Xms=1g -Xmx=2g` ■ Pod memory limit should be higher than Xmx ○ RabbitMQ ■ [rabbitmq.conf] total_memory_available_override_value = 1GB ○ MongoDB ■ --wiredTigerCacheSizeGB=1 Know your limits There might be more than you know…. … resources: requests: memory: "1Gi" cpu: "100m" limits: memory: "2Gi" cpu: "250m" ...
  18. 18. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● The total sum of limits on a node can be way over 100% ● This allows for resource sharing between pods ● Pods will crash! Nodes will crash! ● See out of resources handling by Kubernetes ● Be prepared ○ Think about the requested and limits values ○ Spread your application across nodes using multiple replicas (HA) ○ Use pods priority and preemption to take control Know your limits - don’t overload There might be more than you know…. … resources: requests: memory: "64Mi" cpu: "20m" limits: memory: "2Gi" cpu: "4" ...
  19. 19. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Your application must have a reliable metric for health ● readinessProbe ○ Is the app ready to start serving ● livenessProbe ○ Is the app good to continue serving ● Types ○ Exec - return 0 on success ○ Http - return < 400 on success ○ Tcp - succeed to open a socket on a given port ● For complex checks, write a script and use the exec probe Know your app’s health Application’s readiness and health … readinessProbe: httpGet: path: /api/system/health port: 8080 … livenessProbe: exec: command: - mongo - --eval - "db.adminCommand('ping')" … livenessProbe: tcpSocket: port: 5672 …
  20. 20. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Init Containers - run before main container ○ Prepare storage ○ Setup configuration ● Sidecar containers - run alongside main container ○ Maintenance ○ Log collection ○ Monitoring ○ Proxy Multiple containers in a Pod When a single container is not enough Application pod example. Multiple artifactory logs forwarded by a Fluentbit container to a log aggregator. Pod Application container Fluentbit container Logs Log collector
  21. 21. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Your application is made up of multiple components ○ Each represented as a yaml file or snippet ● Versioning of an application that’s made up of several yaml files is challenging ● Having all configuration in the yaml files is great, but ○ How can I use the same yaml for different environments? ■ Local (minikube) ■ Dev cluster ■ Production ● Rolling back to earlier versions of the application Managing an application’s lifecycle The static nature of the yaml descriptors is challenging
  22. 22. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● https://helm.sh/ ● Helm is the package manager for Kubernetes. Like ‘yum’ for CentOS/RedHat ● Describes the whole application in a single package - helm chart (template yamls) ● Default configuration values (values.yaml) ● Single version for every chart (Chart.yaml) Here comes Helm! Dynamic control over your application’s deployment
  23. 23. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Use same chart for dev, staging and production! ○ Each environment’s config managed in its own copy of values.yaml that is version controlled (values-stg.yaml, values-prod.yaml) ○ The default values.yaml should be for dev or local, so a developer can use it locally without hassle ○ Everything is configurable! ● External charts as requirements (dependencies) ○ 3rd parties like databases Helm - YES! Use same chart in all environments
  24. 24. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Helm - YES! YES! Useful helm commands for understanding what the helm is going on... # Lint your chart for errors and recommendations $ helm lint <chart path> # Download a chart for local viewing $ helm fetch <chart> # Get a release (application already deployed with helm) actual configuration $ helm get <release> # Get the status of all the resources included in a release $ helm status <release> # Get the actual resolved configuration without deploying anything $ helm template <chart> $ helm install --debug --dry-run <chart> # Test your release (need to write test pods in your chart) $ helm test <release>
  25. 25. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! More Helm charts Find existing charts Official Helm charts hub https://hub.helm.sh/
  26. 26. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● No more “ssh to the server and get me the logs”! ● Developers should not need kubectl access to debug their applications ● Provide your Dev and Ops easy and reliable data ○ Monitoring ○ Logging ● Managed solutions provide OOB tooling Visibility in Kubernetes? We are not in Kansas anymore...
  27. 27. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Dashboard for nodes and pods * Prometheus * Grafana Visibility in Kubernetes? Monitoring
  28. 28. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Logs from all pods EFK stack * Fluentd * ElasticSearch * Kibana Visibility in Kubernetes? Logging
  29. 29. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Use with your CI Plug your CI/CD to a Kubernetes cluster for easy environment deployment
  30. 30. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● CI/CD for our products using Kubernetes ○ Using internal dev versions with our official Helm charts ○ On demand, fully isolated environment per branch per developer ○ 100’s of custom testing environments a week made up of ~50+ services ● Some of our cloud based applications already running on K8s ○ Internal and external ● Official JFrog Helm charts for all our products JFrog and Kubernetes What are we doing with Kubernetes?
  31. 31. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● You are not the first one to stumble on that particular problem ● It’s as if you have more developers in your team ● Fast turnaround ● Examples ○ RabbitMQ HA ○ MongoDB (Bitnami) The community Use already tested and managed helm charts
  32. 32. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! ● Start small ● Get the application ready before jumping into Kubernetes ● Security - control what’s running in your cluster ● Limits ● Health probes ● Visibility - usable and accessible monitoring and logging systems ● Spread the word and work with the community! The (real) End What you should take from here
  33. 33. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Smooth sailing... Copyright © 2018 JFrog. All Rights Reserved | www.swampup.jfrog.com
  34. 34. Copyright @ 2019 JFrog - All rights reserved. ALL TOGETHER NOW! Be Good! Questions?

×