Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Scaling
Continuous Delivery
Tomas Riha
Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvement
Horrible at following instructi...
Three Years ago
New Telematics Delivery Platform
Multiple Stakeholders
Continuous Regression Testing
How hard can it be?
Regression Test
Unit Testing
Component Testing
System Testing
Rollback/Compatibility Testing
Failover Testing
Performance ...
At first it was super easy!
Small team of just product owner architect and scrum master.
Huge productivity, natural test d...
Then we tried to scale it...
... and we failed in every possible way.
We lost our test driven development
We lost the indi...
Three components of
Continuous Delivery
Process & Implementation
Definition of the Software Delivery process and its lead ...
Process & Implementation
“Dedicated test servers”
First Continuous Delivery implementation relied on standard corporate te...
Process & Implementation
“Single threaded process”
Stacking jobs became a problem at 5 code committers, almost killed us a...
Process & Implementation
“First effort to scale”
Forced to use our internal Operations as infrastructure provider, no clou...
Process & Implementation
“Crowded Environment”
Visibility was bad due to hundreds of jobs on same build system.
Teams bloc...
Process & Implementation
“Continuous Delivery of Continuous Delivery”
Live what you teach!
Our ability to Continuously Imp...
Process & Implementation
“Autonomous Environments for Autonomous Teams”
Started to do Continuous Delivery of Continuous De...
Process & Implementation
“Portability is Scalability”
Portability in the Runtime Environments
Dev
Test
Load
Test
UAT
PROD
...
Process & Implementation
“Portability is Scalability”
If the Continuous Delivery process can run without a Build Server th...
Process & Implementation
“Portability is Scalability”
All environments provisioned and deployed in the same way
Orchestrat...
Process & Implementation
“Portability is Scalability”
Dev Test
Load
Test
UAT
INT
Test
Provisioning Interface
Vagrant Impl ...
Process & Implementation
“Portability is Scalability”
To make the Continuous Delivery Process portable we also need to mov...
Process & Implementation
“Portability is Scalability”
Orchestration Worker
Log, Metrics, Build
Data Repositories
WorkerWor...
Process & Implementation
“Provisioning, Install and Dependency Management”
If one or more environments is provisioned manu...
People & Organization
“The Tools Team”
Initially we did development of the Continuous Delivery process as
skunkworkz.
Quic...
People & Organization
“The Tools Team is a Bottleneck”
Tools Team a huge bottleneck, simplest issues took ages to fix.
Man...
People & Organization
“Build Police, please dont!”
Only Tools Team and few others understood the process.
Tried to help th...
People & Organization
“The Tools Team matures”
Continuous Delivery as a Service
Tools Team responsible for the process imp...
People & Organization
“Lack of Consensus”
Disagreements on shared responsibility for the release.
Disagreements on what te...
Product Integrity
“Build Quality In”
SYSTEM
GUI
Product Integrity
“Build Quality In”
Internal
Service
GUI
Internal
Service
Internal
Service
Service API
Product Integrity
“Build Quality In”
Internal
Service
Internal
Service
Internal
Service
Each Service Tested as Black Box i...
Product Integrity
“Build Quality In”
GUI
Service API Mock Impl
GUI Test are much faster and robust if they test the GUI in...
Product Integrity
“Build Quality In”
Internal
Service
GUI
Internal
Service
Internal
Service
Service API
Verifying Use Case...
Product Integrity
“Build Quality In” Optimizes Lead Time
Build Release Deploy Test
Build Release Deploy Test
Build Release...
Product Integrity
“Build Quality In”
An Architecture & Test Strategy that goes hand in hand increases product
integrity an...
People & Organization
“Empower the Team!”
Increase Team responsibility.
Team is responsible for everything within the Cont...
People & Organization
“Empower the Team!”
To help empower teams we have
Autonomous Team Environments where teams are respo...
People & Organization
“Test Driven Development Community”
Developers need to take more responsibility.
Code has to work al...
People & Organization
“Test Driven Development Community”
Test Profession changes.
Two main type of Test activities Test A...
People & Organization
“Continuous Delivery Community”
Agreement on Practices of Continuous Delivery
Agreement on Practices...
People & Organization
“Organizational Support”
Continuous Delivery is organizational change.
Scaling Continuous Delivery i...
Summary
Process & Implementation
Portability is Scalability.
Product Integrity
The Architecture & Test Strategy go hand in...
Thats it!
Feedback & Any questions you forgot to ask?
http://continuous-delivery-and-more.blogspot.se
@TomasRihaSE
or by
t...
Upcoming SlideShare
Loading in …5
×

Scaling continuous delivery @ GeeCon 2014

1,967 views

Published on

The slides from my presentation at GeeCon 2014 in Krakow

Published in: Technology
  • Very useful presentation for our engineering team, thanks! Scaling is something we have just started to dabble in. Perhaps you may find our service useful for your presentation tomorrow, also!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Scaling continuous delivery @ GeeCon 2014

  1. 1. Scaling Continuous Delivery Tomas Riha
  2. 2. Tomas Riha Architect @ VGT/WirelessCar Passionate about creativity, change and improvement Horrible at following instructions and performing repetitive tasks MAJOR Project Liability mail: triha74@gmail.com twitter: @TomasRihaSE blog: continuous-delivery-and-more.blogspot.com slides: www.slideshare.net/TomasRiha
  3. 3. Three Years ago New Telematics Delivery Platform Multiple Stakeholders Continuous Regression Testing How hard can it be?
  4. 4. Regression Test Unit Testing Component Testing System Testing Rollback/Compatibility Testing Failover Testing Performance Testing
  5. 5. At first it was super easy! Small team of just product owner architect and scrum master. Huge productivity, natural test driven development. Fast return on investment All by the book.... we were doing Continuous Delivery!
  6. 6. Then we tried to scale it... ... and we failed in every possible way. We lost our test driven development We lost the individual responsibility We more or less became a automated waterfall unable to delivery daily We lost our ability to Continuously Improve the process
  7. 7. Three components of Continuous Delivery Process & Implementation Definition of the Software Delivery process and its lead time optimization Product Integrity The Architecture & Test Strategy to optimize product integrity and lead time. People & Organization Behavioural change to the individuals and the organization needed to optimize lead time and increase product integrity.
  8. 8. Process & Implementation “Dedicated test servers” First Continuous Delivery implementation relied on standard corporate test environment with one system test server, one integration test server and one pre prod server per production delivery Only one commit tested at the time. Build Release Deploy Test DB Test Server
  9. 9. Process & Implementation “Single threaded process” Stacking jobs became a problem at 5 code committers, almost killed us at 20 Feedback time == Stack Depth * Pipe Exec Time Variable feedback time results in less responsible developers We were somewhat helped by component based architecture
  10. 10. Process & Implementation “First effort to scale” Forced to use our internal Operations as infrastructure provider, no cloud and very rudimentary virtualization Built a Server Pool based on Jenkins slaves and “sticky pipes” Primitive but multi threaded pipes Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker
  11. 11. Process & Implementation “Crowded Environment” Visibility was bad due to hundreds of jobs on same build system. Teams blocking each other due to still insufficient server resources. Traceability hard due to all server clutter. Changes to one teams Test Environment affected the other teams. Overall performance was horrible at peak with Jenkins totally dieing
  12. 12. Process & Implementation “Continuous Delivery of Continuous Delivery” Live what you teach! Our ability to Continuously Improve was seriously slowed by lack of Continuous Delivery Deployment Scripts didn't have test automation. No test environment for Continuous Delivery Process Build pipes where manually maintained. Jenkins servers where manually maintained.
  13. 13. Process & Implementation “Autonomous Environments for Autonomous Teams” Started to do Continuous Delivery of Continuous Delivery Automated provisioning of Continuous Delivery Environment Created template based pipeline generator Built Autonomous Continuous Delivery environments for each team Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker
  14. 14. Process & Implementation “Portability is Scalability” Portability in the Runtime Environments Dev Test Load Test UAT PROD INT Test Complexity Scale
  15. 15. Process & Implementation “Portability is Scalability” If the Continuous Delivery process can run without a Build Server then it can scale to X number of build servers By ensuring that anything above the portability line can run local or remote to the orchestration we ensure portability Orchestration ProvisionBuild Deploy Test Release Test EnvironmentSCM Artifact Repo Portability Line
  16. 16. Process & Implementation “Portability is Scalability” All environments provisioned and deployed in the same way Orchestration ProvisionBuild Deploy Test Commit Orchestration ProvisionBuild Deploy Test Release Orchestration Provision Deploy Test Accept Orchestration Provision Deploy Test Local Dev Continuous Delivery Pipe UAT Prod
  17. 17. Process & Implementation “Portability is Scalability” Dev Test Load Test UAT INT Test Provisioning Interface Vagrant Impl Cloud Provider Impl PDF Document Impl Topology Spec Env Spec Legacy PROD With the help of Cloud Technology and Vagrant we manage to create test and dev environments with the same complexity as our production environments.
  18. 18. Process & Implementation “Portability is Scalability” To make the Continuous Delivery Process portable we also need to move all reporting and monitoring out of the test servers and the build servers Orchestration ProvisionBuild Deploy Test Release Test EnvironmentSCM Artifact Repo Logging & Metrics Services Build Data Repository
  19. 19. Process & Implementation “Portability is Scalability” Orchestration Worker Log, Metrics, Build Data Repositories WorkerWorker Build Env Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Team Environment Pre Cloud One Server Per Pipe Shared Database for all pipes Team Environment in Cloud One Environment Per Pipe Including Load Balance, App Servers & DB Separating Data from Process Implementation improves the traceability increases dramatically. Portability in Test Environments increases Quality and Scalability by removing bottlenecks in shared infrastructure. Worker WorkerWorker Test Env Worker Worker Worker WorkerWorker Test Env Worker Test EnvTest Env
  20. 20. Process & Implementation “Provisioning, Install and Dependency Management” If one or more environments is provisioned manually it becomes a bottleneck Application artifacts need dependency management to infrastructure. Infrastructure has to be provisioned and installed with the application. Databases have to be upgraded and migrated as part of the automated process. An automated process doesn't pause for DBAs or SysAdmins to do manual work.
  21. 21. People & Organization “The Tools Team” Initially we did development of the Continuous Delivery process as skunkworkz. Quickly realized this was a bottleneck. Added a CM to our team with disastrous result. Finally created a Tools Team.
  22. 22. People & Organization “The Tools Team is a Bottleneck” Tools Team a huge bottleneck, simplest issues took ages to fix. Managed the Continuous Delivery process manually. Only ones that understood the process. Became responsible for every teams ability to interface with the process. Only had time to do reactive maintenance. Short on resources. Became yet another IT department.
  23. 23. People & Organization “Build Police, please dont!” Only Tools Team and few others understood the process. Tried to help the few others by adding a rotational “Build Police” Role, as bad idea as it sounds.
  24. 24. People & Organization “The Tools Team matures” Continuous Delivery as a Service Tools Team responsible for the process implementation, its interfaces and the infrastructure it runs on. The Development Teams responsible for integrating with these interfaces, configuring the pipe generator and maintaining green state of pipes. Tools Team supports and helps if there are bugs in the mechanisms. Tools Team consists of a core team of Developers, Testers and Operations specialists but is extended with part time resources from Development Teams.
  25. 25. People & Organization “Lack of Consensus” Disagreements on shared responsibility for the release. Disagreements on what test automation is and how it should be used. Disagreement on who implements tests and how. A Continuous Delivery Engine without the consensus of the people that use it is just an overpriced CI System.
  26. 26. Product Integrity “Build Quality In” SYSTEM GUI
  27. 27. Product Integrity “Build Quality In” Internal Service GUI Internal Service Internal Service Service API
  28. 28. Product Integrity “Build Quality In” Internal Service Internal Service Internal Service Each Service Tested as Black Box in isolation and with high detail level
  29. 29. Product Integrity “Build Quality In” GUI Service API Mock Impl GUI Test are much faster and robust if they test the GUI in isolation. High detail level.
  30. 30. Product Integrity “Build Quality In” Internal Service GUI Internal Service Internal Service Service API Verifying Use Case acceptance criterias. Low level of detail
  31. 31. Product Integrity “Build Quality In” Optimizes Lead Time Build Release Deploy Test Build Release Deploy Test Build Release Deploy Test Build Release Deploy Test Build Release Deploy Test Assemble Release Deploy Test Component Pipe 1000s of tests run in parallel testing System Pipe 10s Use Case Requirements Verified
  32. 32. Product Integrity “Build Quality In” An Architecture & Test Strategy that goes hand in hand increases product integrity and optimizes the lead time and allows the Continuous Delivery process to scale
  33. 33. People & Organization “Empower the Team!” Increase Team responsibility. Team is responsible for everything within the Continuous Delivery process Analyzing Requirements Defining Requirement Verification Automating Requirement Verification Implementing the System Shipping the System Supporting the System Create Team consensus on what its responsibility and delivery is.
  34. 34. People & Organization “Empower the Team!” To help empower teams we have Autonomous Team Environments where teams are responsible for their own Continuous Delivery increased the teams understanding of the process. Teams that needed extra help sent one or more individuals to work part time on Tools Team. Architecture & Test Strategy that works well with Continuous Delivery Cross functional communities created to help increase consensus on Test Driven Development, Continuous Delivery and Architecture.
  35. 35. People & Organization “Test Driven Development Community” Developers need to take more responsibility. Code has to work all the time. Test Driven Development requires developers to participate in specification of requirement verifications. Test Automation is Code, Developers need to code much more tests. Continuous Regression Testing means Developers get instant feedback and have to act on it.
  36. 36. People & Organization “Test Driven Development Community” Test Profession changes. Two main type of Test activities Test Automation and Exploratory Testing. Test Automation is Code. Let developers test. Test Driven Development is Proactive. Exploratory testing is NOT manual regression testing.
  37. 37. People & Organization “Continuous Delivery Community” Agreement on Practices of Continuous Delivery Agreement on Practices of Provisioning & Deployment Competence Development Infrastructure as Code Testing Infrastructure as Code Operations Specialists need to work with Developer tools and languages
  38. 38. People & Organization “Organizational Support” Continuous Delivery is organizational change. Scaling Continuous Delivery is as much creating a scalable agile organization as it is creating a scalable implementation. Organization has to fully support Continuous Delivery through re-organization. Organization has to fully support and invest in infrastructure, architecture, automation, process and test. Organization needs to start measuring key values, such as cycle time, early.
  39. 39. Summary Process & Implementation Portability is Scalability. Product Integrity The Architecture & Test Strategy go hand in hand. People & Organization Empowered Teams and Consensus.
  40. 40. Thats it! Feedback & Any questions you forgot to ask? http://continuous-delivery-and-more.blogspot.se @TomasRihaSE or by the Coffee stand! And the slides are at http://www.slideshare.net/TomasRiha

×