In this talk, we will go through the design process of modern CI/CD for the microservices-based system with Kubernetes support. We will discuss how to verify consistency between microservices, apply different levels of quality gates and promote artifacts between environments. Thanks to Kubernetes we will review different approaches of environment resources optimization for development needs during CI/CD cycles.

1. Modern CI/CD in the
microservices world with
Kubernetes
Mikalai Alimenkou
@xpinjection
https://t.me/xpinjection_channel
https://xpinjection.com

3. Telegram channel
https://t.me/xpinjection_channel

4. Disclaimer
This talk is based on
personal experience

5. Hello world samples on conferences

6. Real world is a little bit more complex

7. CI/CD of the son of your mother’s friend
After moving to its own cloud, Amazon engineers
deploy code every 11.7 seconds, on average
reducing both the number and duration of outages
at the same time.

8. Why this is not your case?
Product is developed in iterative way (Scrum, 2 weeks)
Engineering culture is far from perfect
Level of developers is not so high
No big investments in test automation at all levels
Engineering practices are not used on regular basis
Legacy services or dependencies
Business is not ready to continuous delivery
… many more reasons

9. Chaos deployment in DEV/QA
QA/DEV environment always contain latest versions of
all microservices
Deployment on QA/DEV is done continuously by
different dev teams when something is ‘READY’
Compatibility of microservices is not verified
There is no way to rollback on specific system version
QA life looks like a hell (random issues, non-
reproducible results, 500 HTTP codes everywhere,
chaos testing)

10. Does it look familiar to you?

11. Old monolith mindset doesn’t work

12. Microservices has their own lifecycles

13. Difference between CI and CD

14. Step #1: Dockerize your microservices
Immutable infrastructure principle
Artifacts promotion with tags
Unified blocks for CD stage, no dependency on
technical stack
Simplified deployment and environment management
Common configuration approach
Efficient resources utilization

16. Step #2: Specify versioning policy
Semantic versioning as a basis
Major version for API breaking changes or major
service release
Minor version corresponds to current dev iteration
Patch version is reserved for hot fixes
Commit hash as suffix to snapshot versioning and
tracing
Build date as an additional optional suffix

17. CI pipeline for each microservice
QUALITY GATES:
- Unit tests
- Integration tests
- Static code analysis
- API tests
- Contract tests
- Security checks
- … FULL VERSION: 2.3.0-350fca-20200218

18. What is shippable version of the system?

19. Step #3: Introduce compatibility sets
Store full list of compatible microservices as dedicated
artifact
VCS provides historical timeline, rollback, full tracing
Used as primary artifact for Continuous Delivery
Promoted between different environments in CD
pipeline (dev, qa, stage, prod)
Add middleware versions for full compatibility set

20. Compatibility set in properties file
Clear logical versions management independent from
concrete deployment technologies
Flexible usage for different purposes
Separation between deployment artifacts versioning
Customization step before deployment

21. Compatibility set in Helm/helmfile
Single versioned storage for environment configuration
Ability to specify compatibility on configuration level
The same toolset is used for all environments
Additional level of abstraction around K8S manifests
Helm is needed as an additional component

22. Step #4: Implement system level pipeline
Triggered when new candidate is built
Uses latest compatible set as a baseline
Overrides versions for ALL pending candidates
Creates new environment for system deployment
Deploys all microservices according to their versions
Waits for all health checks to become OK
Runs system level quality gates
Stores new compatibility set in case of SUCCESS
Retag and promote artifacts for the next stage

23. System pipeline for compatibility checks
QUALITY GATES:
- Health checks
- Smoke API tests
- Smoke UI tests
- Smoke system tests
- Smoke security tests
- …
ADDITIONAL TAGS: qa-ready, set-283gpt

24. Example from the real life
USER SERVICE
1.0.3-abc
EVENT SERVICE
1.0.0-cde
SCHEDULE SERVICE
0.9.1-fgh
COMPATIBLE SET
1.0.3-prs
NEW CANDIDATE
1.0.0-cde
1.0.3-prs
0.9.1-fgh
FAILED SYSTEM BUILD
1.0.1-tuv
FIX APPLIED
1.0.3-prs
0.9.1-fgh
SUCCESS SYSTEM BUILD
1.0.1-tuv
1.0.3-prs
1.0.1-tuv
0.9.1-fgh
COMPATIBLE SET

25. System pipeline: important notes
If pipeline fails, then candidate is not approved and it's
artifact is invisible for other environments, but pending
for the next pipeline run
Every build takes ALL available pending candidates
Different concurrency strategies may be applied:
• single build with lock on compatibility set
• concurrent builds with “last win” approach
• concurrent builds with optimistic locking in VSC

26. We are almost GitOps ready now!

27. New tools become available for CD

28. Step #5: Implement deployment pipeline
Triggered manually, automatically by successful system
pipeline build or by specific schedule
Predefined set of environments (dev, qa, stage, prod)
New environment may be created with specified or
generated name
Only appropriate compatible sets may be deployed
Environment TTL could be set manually or smart
default value is applied

29. Full CI/CD
cycle

30. FEATURE BRANCH PIPELINE:
- Environment name equals to feature branch name
- Reused to speed up development
- DEV compatibility set is used as a baseline
- Artifacts on microservices pipeline are tagged with branch name
- For each microservice latest available version from the branch
with the same name is used as candidate
PULL REQUEST PIPELINE:
- Triggered on each PR
- Special flag to activate system
level quality gates
- DEV compatibility set is used
as a baseline
- Only light system level quality
gates are run
- No compatibility sets updated

32. It is time for Kubernetes to show up!

33. Step #6: Speed up environments in K8S
Create dedicated namespace for each environment
Configure minimal limits and scale factors everywhere
Share common external heavy resources (DB,
Elasticsearch, Redis, Kafka, etc.)
Invest in performant K8S nodes
Use prepopulated images with data to speed up
Configure common observability tools
Reuse environments where possible with smart TTLs
Destroy environments asynchronously

34. Use full power of K8S namespaces
COMMON
Glossary
Service
Redis
Cache
FB-243
User
Service
Reuse
from DEV
STAGE
User
Service
Event
Service
Schedule
Service
Reuse
from DEV
QA
User
Service
Event
Service
Schedule
Service
DEV
User
Service
Event
Service
Schedule
Service
RDBMS
Kafka
Cluster Elasticsearch

35. How to share common resources?
It is really important to follow cloud-native 12 factors!
Redis via key prefixes or databases
RDBMS via databases or schemas
Kafka via common topics prefix
Elasticsearch via indexes

36. Now developers could use Skaffold continuously
QC is shifted as left as possible
Several teams could develop in common environment

37. Step #7: Start using K8S native CD tools
Dockerize tests with required test environment
Execute tests as K8S jobs
Manage test infrastructure in K8S
Choose suitable K8S native CD tool

38. Summary and take aways
Dockerize your microservices
Specify versioning policy
Introduce compatibility sets
Implement system level pipeline
Implement deployment pipeline
Speed up environments in K8S
Start using K8S native CD tools
Enjoy your life in amazing CI/CD world!

39. @xpinjection
https://xpinjection.com
https://t.me/xpinjection_channel