Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modern CI/CD in the microservices world with Kubernetes

In this talk, we will go through the design process of modern CI/CD for the microservices-based system with Kubernetes support. We will discuss how to verify consistency between microservices, apply different levels of quality gates and promote artifacts between environments. Thanks to Kubernetes we will review different approaches of environment resources optimization for development needs during CI/CD cycles.

Modern CI/CD in the microservices world with Kubernetes

  1. 1. Modern CI/CD in the microservices world with Kubernetes Mikalai Alimenkou @xpinjection https://t.me/xpinjection_channel https://xpinjection.com
  2. 2. Telegram channel https://t.me/xpinjection_channel
  3. 3. Disclaimer This talk is based on personal experience
  4. 4. Hello world samples on conferences
  5. 5. Real world is a little bit more complex
  6. 6. CI/CD of the son of your mother’s friend After moving to its own cloud, Amazon engineers deploy code every 11.7 seconds, on average reducing both the number and duration of outages at the same time.
  7. 7. Why this is not your case? Product is developed in iterative way (Scrum, 2 weeks) Engineering culture is far from perfect Level of developers is not so high No big investments in test automation at all levels Engineering practices are not used on regular basis Legacy services or dependencies Business is not ready to continuous delivery … many more reasons
  8. 8. Chaos deployment in DEV/QA QA/DEV environment always contain latest versions of all microservices Deployment on QA/DEV is done continuously by different dev teams when something is ‘READY’ Compatibility of microservices is not verified There is no way to rollback on specific system version QA life looks like a hell (random issues, non- reproducible results, 500 HTTP codes everywhere, chaos testing)
  9. 9. Does it look familiar to you?
  10. 10. Old monolith mindset doesn’t work
  11. 11. Microservices has their own lifecycles
  12. 12. Difference between CI and CD
  13. 13. Step #1: Dockerize your microservices Immutable infrastructure principle Artifacts promotion with tags Unified blocks for CD stage, no dependency on technical stack Simplified deployment and environment management Common configuration approach Efficient resources utilization
  14. 14. Step #2: Specify versioning policy Semantic versioning as a basis Major version for API breaking changes or major service release Minor version corresponds to current dev iteration Patch version is reserved for hot fixes Commit hash as suffix to snapshot versioning and tracing Build date as an additional optional suffix
  15. 15. CI pipeline for each microservice QUALITY GATES: - Unit tests - Integration tests - Static code analysis - API tests - Contract tests - Security checks - … FULL VERSION: 2.3.0-350fca-20200218
  16. 16. What is shippable version of the system?
  17. 17. Step #3: Introduce compatibility sets Store full list of compatible microservices as dedicated artifact VCS provides historical timeline, rollback, full tracing Used as primary artifact for Continuous Delivery Promoted between different environments in CD pipeline (dev, qa, stage, prod) Add middleware versions for full compatibility set
  18. 18. Compatibility set in properties file Clear logical versions management independent from concrete deployment technologies Flexible usage for different purposes Separation between deployment artifacts versioning Customization step before deployment
  19. 19. Compatibility set in Helm/helmfile Single versioned storage for environment configuration Ability to specify compatibility on configuration level The same toolset is used for all environments Additional level of abstraction around K8S manifests Helm is needed as an additional component
  20. 20. Step #4: Implement system level pipeline Triggered when new candidate is built Uses latest compatible set as a baseline Overrides versions for ALL pending candidates Creates new environment for system deployment Deploys all microservices according to their versions Waits for all health checks to become OK Runs system level quality gates Stores new compatibility set in case of SUCCESS Retag and promote artifacts for the next stage
  21. 21. System pipeline for compatibility checks QUALITY GATES: - Health checks - Smoke API tests - Smoke UI tests - Smoke system tests - Smoke security tests - … ADDITIONAL TAGS: qa-ready, set-283gpt
  22. 22. Example from the real life USER SERVICE 1.0.3-abc EVENT SERVICE 1.0.0-cde SCHEDULE SERVICE 0.9.1-fgh COMPATIBLE SET 1.0.3-prs NEW CANDIDATE 1.0.0-cde 1.0.3-prs 0.9.1-fgh FAILED SYSTEM BUILD 1.0.1-tuv FIX APPLIED 1.0.3-prs 0.9.1-fgh SUCCESS SYSTEM BUILD 1.0.1-tuv 1.0.3-prs 1.0.1-tuv 0.9.1-fgh COMPATIBLE SET
  23. 23. System pipeline: important notes If pipeline fails, then candidate is not approved and it's artifact is invisible for other environments, but pending for the next pipeline run Every build takes ALL available pending candidates Different concurrency strategies may be applied: • single build with lock on compatibility set • concurrent builds with “last win” approach • concurrent builds with optimistic locking in VSC
  24. 24. We are almost GitOps ready now!
  25. 25. New tools become available for CD
  26. 26. Step #5: Implement deployment pipeline Triggered manually, automatically by successful system pipeline build or by specific schedule Predefined set of environments (dev, qa, stage, prod) New environment may be created with specified or generated name Only appropriate compatible sets may be deployed Environment TTL could be set manually or smart default value is applied
  27. 27. Full CI/CD cycle
  28. 28. FEATURE BRANCH PIPELINE: - Environment name equals to feature branch name - Reused to speed up development - DEV compatibility set is used as a baseline - Artifacts on microservices pipeline are tagged with branch name - For each microservice latest available version from the branch with the same name is used as candidate PULL REQUEST PIPELINE: - Triggered on each PR - Special flag to activate system level quality gates - DEV compatibility set is used as a baseline - Only light system level quality gates are run - No compatibility sets updated
  29. 29. It is time for Kubernetes to show up!
  30. 30. Step #6: Speed up environments in K8S Create dedicated namespace for each environment Configure minimal limits and scale factors everywhere Share common external heavy resources (DB, Elasticsearch, Redis, Kafka, etc.) Invest in performant K8S nodes Use prepopulated images with data to speed up Configure common observability tools Reuse environments where possible with smart TTLs Destroy environments asynchronously
  31. 31. Use full power of K8S namespaces COMMON Glossary Service Redis Cache FB-243 User Service Reuse from DEV STAGE User Service Event Service Schedule Service Reuse from DEV QA User Service Event Service Schedule Service DEV User Service Event Service Schedule Service RDBMS Kafka Cluster Elasticsearch
  32. 32. How to share common resources? It is really important to follow cloud-native 12 factors! Redis via key prefixes or databases RDBMS via databases or schemas Kafka via common topics prefix Elasticsearch via indexes
  33. 33. Now developers could use Skaffold continuously QC is shifted as left as possible Several teams could develop in common environment
  34. 34. Step #7: Start using K8S native CD tools Dockerize tests with required test environment Execute tests as K8S jobs Manage test infrastructure in K8S Choose suitable K8S native CD tool
  35. 35. Summary and take aways Dockerize your microservices Specify versioning policy Introduce compatibility sets Implement system level pipeline Implement deployment pipeline Speed up environments in K8S Start using K8S native CD tools Enjoy your life in amazing CI/CD world!
  36. 36. @xpinjection https://xpinjection.com https://t.me/xpinjection_channel

×