The document outlines the automation of Continuous Integration (CI) and Continuous Deployment (CD) at Conductor, discussing its objectives, tools, and processes involved. It highlights the challenges faced, such as flaky tests and environment discrepancies, along with their mitigations. Additionally, it emphasizes the shift towards microservices to enhance scalability and deployment efficiency.

1. CONDUCTOR - AUTOMATING CI AND CD

2. 2
 Introduction
 Continuous Integration Overview
 Reforming Continuous Integration at Conductor
 Common Issues and Mitigations
 Deployment Overview
 Microservices at Conductor
 Q and A
AGENDA

3. 3
WHO WE ARE

4. 4
WHO WE ARE

5. 5
“The practice of integrating and testing source code continuously
in an automated fashion”
WHAT IS CONTINUOUS INTEGRATION?

6. 6
 Minimize the duration and effort required by each integration
episode
 Be able to deliver a product version suitable for release at any
moment
OBJECTIVES

7. 7
 Maintain a single source repository
 Automate the build
 Make Your build self-testing
 Keep the build Fast
 Test in a clone of the production environment
 Automate deployment
PRACTICES OF CONTINUOUS INTEGRATION

8. 8
QUALITY GATES
Static Code
Analysis
Unit Tests
Integration
Tests
Functional UI
Tests
Performance
Tests

9. 9
REFORMING CI AT CONDUCTOR

10. 10
· Choose and setup a CI tool
· Build and configure Testing Environment
· Build a Continuous Integration pipeline
· Collect statistics and metrics
MILESTONES / GOALS

11. 11
 Choose and setup a CI tool
· Build and configure Testing Environment
· Build a Continuous Integration pipeline
· Collect statistics and metrics
MILESTONES / GOALS

12. 12
CONTINUOUS INTEGRATION TOOLS

13. 13
CONTINUOUS INTEGRATION TOOLS

14. 14
Choose and setup a CI tool
 Build and configure Testing Environment
· Build a Continuous Integration pipeline
· Collect statistics and metrics
MILESTONES / GOALS

15. 15
 Easy to spin up and maintain
 Immutable and disposable Infrastructure
 Production-like
 Cost effective
 Environment agnostic
TEST ENVIRONMENT REQUIREMENTS

16. 16
ORIGINAL TEST ENVIRONMENT

17. 17
CURRENT TEST ENVIRONMENT
PBE
Searchlight
ZooKeeper
WireMock MySQL
Thrift
Selenium Grid
Node
Node
NodeHub

18. 18
CONTINUOUS INTEGRATION ARCHITECTURE
GitHub
Docker
Registry
TeamCity Master
Nexus
TeamCity Agent Selenium
Grid
PBE

19. 19
Choose and setup a CI tool
Build and configure Testing Environment
 Build a Continuous Integration pipeline
· Collect statistics and metrics
MILESTONES / GOALS

20. 20
PULL REQUEST WORKFLOW
Open PR
Unit and
Integration
Tests
Jasmine and
Karma Tests
Linting
Launch
Selenium Grid
Launch PBE Feature Tests
Merge PR

21. 21
PULL REQUEST IN TC

22. 22
Choose and setup a CI tool
Build and configure Testing Environment
Build a Continuous Integration pipeline
 Collect statistics and metrics
MILESTONES / GOALS

23. 23
 “require branches to be up to date before merging” is disabled in Github
 To mitigate conflicts we run a “heartbeat” workflow against master every 4 hours
 Posts build status into Slack dev channels
 PRs are paused in the case of failure
PR HEARTBEAT

24. 24
PR STATISTICS

25. 25
PR STATISTICS

26. 26
Choose and setup a CI tool
Build and configure Testing Environment
Build a Continuous Integration pipeline
Collect statistics and metrics
MILESTONES / GOALS

27. 27
COMMON ISSUES AND MITIGATIONS

28. 28
 Inconsistent behavior (flaky tests, code, infrastructure, etc.)
 Differences in production and PBE environments
 AWS is having a bad day
COMMON BUILD PROBLEMS

29. 29
FLAKY TESTS AND HOW WE MITIGATE THEM
Validate
artifacts
Launch
Selenium
Grid
Launch PBE
Rerun
Failures
Terminate
Resources
Green Check
PR

30. 30
 Kubernetes is an open-source
system for automating deployment,
scaling, and management of
containerized applications.
 It groups containers that make up
an application into logical units for
easy management and discovery.
TEST ENVIRONMENTS ON KUBERNETES
Kubernetes Cluster
Selenium Grid
Hub
Node
Node
Node
Searchlight
ZooKeeper
WireMock MySQL
Thrift
PBE Selenium Grid
Hub
Node
Node
Node
Searchlight
ZooKeeper
WireMock MySQL
Thrift
PBE

31. 31
TC CLOUD AGENTS
Configured TeamCity with AWS to start and stop images with TeamCity agents on-
demand based on the queued builds.

32. 32
DEPLOYMENT OVERVIEW

33. 33
 Tests similar to those run for PRs are used
 Runs on a production-like beta environment
 Test results are reviewed
 Jenkins job deploys in rolling fashion with zero downtime
DEPLOY TESTING

34. 34
PROD DEPLOY PIPELINE
Checkout
SCM
Schedule
Icinga
Downtime
DB
Migrations
Deploy UIs
Deploy
Thrift
Post
Actions
Instance
Auto Scaling Group
Auto Scaling Group
Instance InstanceInstance
Instance InstanceInstance

35. 35
MONITORING

36. 36
 Integrate performance tests into the PR process
 Run security scanning as part of PRs
 Remove reliance on release testing
 Microservices!
WHAT’S NEXT?

37. 37
 Large Monolithic applications are complicated!
 They create unnecessary dependencies
 They are hard to scale vertically and horizontally
 Overall velocity and speed is hindered
WHY MICROSERVICES?

38. 38
 Define organizational standards for designing, developing, testing and
deploying microservices
 Use out of the box solutions as much as possible
 Build new features as microservices alongside the monolith
 Migrate existing features to microservices when necessary
GREAT, NOW WHAT?

39. 39
MICROSERVICE CI/CD
Jenkins
Dev Kubernetes Cluster
(1) a PR is opened against
a branch in GitHub
(2) Jenkins runs tests against that branch, if
successful changes are merged
(3) Jenkins runs tests against the head of features, if successful
artifacts are published to Nexus and the Docker registry
(4) Engineers can deploy their service manually to the development
Kubernetes cluster for manual validation during development
Prod Kubernetes Cluster
(5) Deployments to the production Kubernetes cluster are
triggered manually
E2E TESTS PERF TESTS
SCALE
TESTS
(6) E2E, performance and scale tests are run in a "blue"
environment and if successful the service is switched to "green"
Nexus
Docker
Registry
CD Tool
LINTING UNIT TESTS
CONTRACT
TESTS
CODE
COVERAGE
PUBLISH
ARTIFACTS
GitHub
Jenkins
LINTING UNIT TESTS
CONTRACT
TESTS
CODE
COVERAGE
OPEN PR

40. 40
· Daily monolith deploys
· Frequent microservice deploys
· Deliver bug fixes and new features faster
· Fewer bugs in production
· Continued adoption of microservices
WHERE WE ARE TODAY

41. 41
QUESTIONS?