Tomas Riha discusses the challenges of scaling continuous delivery. Initially, their process worked well with a small team but failed as they grew. Key lessons include: making environments portable across infrastructure to allow scaling; building quality through an architecture and test strategy that optimize lead time; and empowering cross-functional teams through consensus on practices and autonomous environments.

1. Scaling
Continuous Delivery
Tomas Riha

2. Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvement
Horrible at following instructions and performing repetitive tasks
MAJOR Project Liability
mail: triha74@gmail.com
twitter: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.com
slides: www.slideshare.net/TomasRiha

3. Three Years ago
New Telematics Delivery Platform
Multiple Stakeholders
Continuous Regression Testing
How hard can it be?

4. Regression Test
Unit Testing
Component Testing
System Testing
Rollback/Compatibility Testing
Failover Testing
Performance Testing

5. At first it was super easy!
Small team of just product owner architect and scrum master.
Huge productivity, natural test driven development.
Fast return on investment
All by the book.... we were doing
Continuous Delivery!

6. Then we tried to scale it...
... and we failed in every possible way.
We lost our test driven development
We lost the individual responsibility
We more or less became a automated waterfall unable to delivery daily
We lost our ability to Continuously Improve the process

7. Three components of
Continuous Delivery
Process & Implementation
Definition of the Software Delivery process and its lead time optimization
Product Integrity
The Architecture & Test Strategy to optimize product integrity and lead time.
People & Organization
Behavioural change to the individuals and the organization needed to optimize
lead time and increase product integrity.

8. Process & Implementation
“Dedicated test servers”
First Continuous Delivery implementation relied on standard corporate test
environment with one system test server, one integration test server and one
pre prod server per production delivery
Only one commit tested at the time.
Build Release Deploy Test
DB
Test
Server

9. Process & Implementation
“Single threaded process”
Stacking jobs became a problem at 5 code committers, almost killed us at 20
Feedback time == Stack Depth * Pipe Exec Time
Variable feedback time results in less responsible developers
We were somewhat helped by component based architecture

10. Process & Implementation
“First effort to scale”
Forced to use our internal Operations as infrastructure provider, no cloud and
very rudimentary virtualization
Built a Server Pool based on Jenkins slaves and “sticky pipes”
Primitive but multi threaded pipes
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker

11. Process & Implementation
“Crowded Environment”
Visibility was bad due to hundreds of jobs on same build system.
Teams blocking each other due to still insufficient server resources.
Traceability hard due to all server clutter.
Changes to one teams Test Environment affected the other teams.
Overall performance was horrible at peak with Jenkins totally dieing

12. Process & Implementation
“Continuous Delivery of Continuous Delivery”
Live what you teach!
Our ability to Continuously Improve was seriously slowed by lack of
Continuous Delivery
Deployment Scripts didn't have test automation.
No test environment for Continuous Delivery Process
Build pipes where manually maintained.
Jenkins servers where manually maintained.

13. Process & Implementation
“Autonomous Environments for Autonomous Teams”
Started to do Continuous Delivery of Continuous Delivery
Automated provisioning of Continuous Delivery Environment
Created template based pipeline generator
Built Autonomous Continuous Delivery environments for each team
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker

14. Process & Implementation
“Portability is Scalability”
Portability in the Runtime Environments
Dev
Test
Load
Test
UAT
PROD
INT
Test
Complexity
Scale

15. Process & Implementation
“Portability is Scalability”
If the Continuous Delivery process can run without a Build Server then it
can scale to X number of build servers
By ensuring that anything above the portability line can run local or remote to
the orchestration we ensure portability
Orchestration
ProvisionBuild Deploy Test Release
Test EnvironmentSCM
Artifact
Repo
Portability Line

16. Process & Implementation
“Portability is Scalability”
All environments provisioned and deployed in the same way
Orchestration
ProvisionBuild Deploy Test Commit
Orchestration
ProvisionBuild Deploy Test Release
Orchestration
Provision Deploy Test Accept
Orchestration
Provision Deploy Test
Local Dev Continuous Delivery Pipe
UAT Prod

17. Process & Implementation
“Portability is Scalability”
Dev Test
Load
Test
UAT
INT
Test
Provisioning Interface
Vagrant Impl Cloud Provider Impl PDF Document Impl
Topology
Spec Env Spec
Legacy
PROD
With the help of Cloud Technology and Vagrant we manage to create test
and dev environments with the same complexity as our production
environments.

18. Process & Implementation
“Portability is Scalability”
To make the Continuous Delivery Process portable we also need to move all
reporting and monitoring out of the test servers and the build servers
Orchestration
ProvisionBuild Deploy Test Release
Test EnvironmentSCM
Artifact
Repo
Logging & Metrics Services
Build Data Repository

19. Process & Implementation
“Portability is Scalability”
Orchestration Worker
Log, Metrics, Build
Data Repositories
WorkerWorker
Build Env
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Team Environment Pre Cloud
One Server Per Pipe
Shared Database for all pipes
Team Environment in Cloud
One Environment Per Pipe
Including Load Balance, App Servers & DB
Separating Data from Process Implementation improves the traceability increases dramatically.
Portability in Test Environments increases Quality and Scalability by removing bottlenecks in
shared infrastructure.
Worker
WorkerWorker
Test Env
Worker
Worker
Worker
WorkerWorker
Test Env
Worker
Test EnvTest Env

20. Process & Implementation
“Provisioning, Install and Dependency Management”
If one or more environments is provisioned manually it becomes a
bottleneck
Application artifacts need dependency management to infrastructure.
Infrastructure has to be provisioned and installed with the application.
Databases have to be upgraded and migrated as part of the automated
process.
An automated process doesn't pause for DBAs or SysAdmins to do manual
work.

21. People & Organization
“The Tools Team”
Initially we did development of the Continuous Delivery process as
skunkworkz.
Quickly realized this was a bottleneck.
Added a CM to our team with disastrous result.
Finally created a Tools Team.

22. People & Organization
“The Tools Team is a Bottleneck”
Tools Team a huge bottleneck, simplest issues took ages to fix.
Managed the Continuous Delivery process manually.
Only ones that understood the process.
Became responsible for every teams ability to interface with the process.
Only had time to do reactive maintenance.
Short on resources.
Became yet another IT department.

23. People & Organization
“Build Police, please dont!”
Only Tools Team and few others understood the process.
Tried to help the few others by adding a rotational “Build Police” Role, as bad
idea as it sounds.

24. People & Organization
“The Tools Team matures”
Continuous Delivery as a Service
Tools Team responsible for the process implementation, its interfaces and
the infrastructure it runs on.
The Development Teams responsible for integrating with these interfaces,
configuring the pipe generator and maintaining green state of pipes.
Tools Team supports and helps if there are bugs in the mechanisms.
Tools Team consists of a core team of Developers, Testers and Operations
specialists but is extended with part time resources from Development
Teams.

25. People & Organization
“Lack of Consensus”
Disagreements on shared responsibility for the release.
Disagreements on what test automation is and how it should be used.
Disagreement on who implements tests and how.
A Continuous Delivery Engine without the consensus of the people that use
it is just an overpriced CI System.

26. Product Integrity
“Build Quality In”
SYSTEM
GUI

27. Product Integrity
“Build Quality In”
Internal
Service
GUI
Internal
Service
Internal
Service
Service API

28. Product Integrity
“Build Quality In”
Internal
Service
Internal
Service
Internal
Service
Each Service Tested as Black Box in isolation
and with high detail level

29. Product Integrity
“Build Quality In”
GUI
Service API Mock Impl
GUI Test are much faster and robust if they test the GUI in isolation.
High detail level.

30. Product Integrity
“Build Quality In”
Internal
Service
GUI
Internal
Service
Internal
Service
Service API
Verifying Use Case
acceptance criterias.
Low level of detail

31. Product Integrity
“Build Quality In” Optimizes Lead Time
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Assemble Release Deploy Test
Component Pipe
1000s of tests run in parallel testing
System Pipe
10s Use Case Requirements Verified

32. Product Integrity
“Build Quality In”
An Architecture & Test Strategy that goes hand in hand increases product
integrity and optimizes the lead time and allows the Continuous Delivery
process to scale

33. People & Organization
“Empower the Team!”
Increase Team responsibility.
Team is responsible for everything within the Continuous Delivery process
Analyzing Requirements
Defining Requirement Verification
Automating Requirement Verification
Implementing the System
Shipping the System
Supporting the System
Create Team consensus on what its responsibility and delivery is.

34. People & Organization
“Empower the Team!”
To help empower teams we have
Autonomous Team Environments where teams are responsible for their own
Continuous Delivery increased the teams understanding of the process.
Teams that needed extra help sent one or more individuals to work part time
on Tools Team.
Architecture & Test Strategy that works well with Continuous Delivery
Cross functional communities created to help increase consensus on Test
Driven Development, Continuous Delivery and Architecture.

35. People & Organization
“Test Driven Development Community”
Developers need to take more responsibility.
Code has to work all the time.
Test Driven Development requires developers to participate in specification
of requirement verifications.
Test Automation is Code, Developers need to code much more tests.
Continuous Regression Testing means Developers get instant feedback and
have to act on it.

36. People & Organization
“Test Driven Development Community”
Test Profession changes.
Two main type of Test activities Test Automation and Exploratory Testing.
Test Automation is Code.
Let developers test.
Test Driven Development is Proactive.
Exploratory testing is NOT manual regression testing.

37. People & Organization
“Continuous Delivery Community”
Agreement on Practices of Continuous Delivery
Agreement on Practices of Provisioning & Deployment
Competence Development
Infrastructure as Code
Testing Infrastructure as Code
Operations Specialists need to work with Developer tools and languages

38. People & Organization
“Organizational Support”
Continuous Delivery is organizational change.
Scaling Continuous Delivery is as much creating a scalable agile
organization as it is creating a scalable implementation.
Organization has to fully support Continuous Delivery through re-organization.
Organization has to fully support and invest in infrastructure, architecture,
automation, process and test.
Organization needs to start measuring key values, such as cycle time, early.

39. Summary
Process & Implementation
Portability is Scalability.
Product Integrity
The Architecture & Test Strategy go hand in hand.
People & Organization
Empowered Teams and Consensus.

40. Thats it!
Feedback & Any questions you forgot to ask?
http://continuous-delivery-and-more.blogspot.se
@TomasRihaSE
or by
the Coffee stand!
And the slides are at
http://www.slideshare.net/TomasRiha