SlideShare a Scribd company logo
1 of 40
Download to read offline
Scaling
Continuous Delivery
Tomas Riha
Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvement
Horrible at following instructions and performing repetitive tasks
MAJOR Project Liability
mail: triha74@gmail.com
twitter: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.com
slides: www.slideshare.net/TomasRiha
Three Years ago
New Telematics Delivery Platform
Multiple Stakeholders
Continuous Regression Testing
How hard can it be?
Regression Test
Unit Testing
Component Testing
System Testing
Rollback/Compatibility Testing
Failover Testing
Performance Testing
At first it was super easy!
Small team of just product owner architect and scrum master.
Huge productivity, natural test driven development.
Fast return on investment
All by the book.... we were doing
Continuous Delivery!
Then we tried to scale it...
... and we failed in every possible way.
We lost our test driven development
We lost the individual responsibility
We more or less became a automated waterfall unable to delivery daily
We lost our ability to Continuously Improve the process
Three components of
Continuous Delivery
Process & Implementation
Definition of the Software Delivery process and its lead time optimization
Product Integrity
The Architecture & Test Strategy to optimize product integrity and lead time.
People & Organization
Behavioural change to the individuals and the organization needed to optimize
lead time and increase product integrity.
Process & Implementation
“Dedicated test servers”
First Continuous Delivery implementation relied on standard corporate test
environment with one system test server, one integration test server and one
pre prod server per production delivery
Only one commit tested at the time.
Build Release Deploy Test
DB
Test
Server
Process & Implementation
“Single threaded process”
Stacking jobs became a problem at 5 code committers, almost killed us at 20
Feedback time == Stack Depth * Pipe Exec Time
Variable feedback time results in less responsible developers
We were somewhat helped by component based architecture
Process & Implementation
“First effort to scale”
Forced to use our internal Operations as infrastructure provider, no cloud and
very rudimentary virtualization
Built a Server Pool based on Jenkins slaves and “sticky pipes”
Primitive but multi threaded pipes
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker Worker Worker
Process & Implementation
“Crowded Environment”
Visibility was bad due to hundreds of jobs on same build system.
Teams blocking each other due to still insufficient server resources.
Traceability hard due to all server clutter.
Changes to one teams Test Environment affected the other teams.
Overall performance was horrible at peak with Jenkins totally dieing
Process & Implementation
“Continuous Delivery of Continuous Delivery”
Live what you teach!
Our ability to Continuously Improve was seriously slowed by lack of
Continuous Delivery
Deployment Scripts didn't have test automation.
No test environment for Continuous Delivery Process
Build pipes where manually maintained.
Jenkins servers where manually maintained.
Process & Implementation
“Autonomous Environments for Autonomous Teams”
Started to do Continuous Delivery of Continuous Delivery
Automated provisioning of Continuous Delivery Environment
Created template based pipeline generator
Built Autonomous Continuous Delivery environments for each team
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Process & Implementation
“Portability is Scalability”
Portability in the Runtime Environments
Dev
Test
Load
Test
UAT
PROD
INT
Test
Complexity
Scale
Process & Implementation
“Portability is Scalability”
If the Continuous Delivery process can run without a Build Server then it
can scale to X number of build servers
By ensuring that anything above the portability line can run local or remote to
the orchestration we ensure portability
Orchestration
ProvisionBuild Deploy Test Release
Test EnvironmentSCM
Artifact
Repo
Portability Line
Process & Implementation
“Portability is Scalability”
All environments provisioned and deployed in the same way
Orchestration
ProvisionBuild Deploy Test Commit
Orchestration
ProvisionBuild Deploy Test Release
Orchestration
Provision Deploy Test Accept
Orchestration
Provision Deploy Test
Local Dev Continuous Delivery Pipe
UAT Prod
Process & Implementation
“Portability is Scalability”
Dev Test
Load
Test
UAT
INT
Test
Provisioning Interface
Vagrant Impl Cloud Provider Impl PDF Document Impl
Topology
Spec Env Spec
Legacy
PROD
With the help of Cloud Technology and Vagrant we manage to create test
and dev environments with the same complexity as our production
environments.
Process & Implementation
“Portability is Scalability”
To make the Continuous Delivery Process portable we also need to move all
reporting and monitoring out of the test servers and the build servers
Orchestration
ProvisionBuild Deploy Test Release
Test EnvironmentSCM
Artifact
Repo
Logging & Metrics Services
Build Data Repository
Process & Implementation
“Portability is Scalability”
Orchestration Worker
Log, Metrics, Build
Data Repositories
WorkerWorker
Build Env
Orchestration Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Worker
Team Environment Pre Cloud
One Server Per Pipe
Shared Database for all pipes
Team Environment in Cloud
One Environment Per Pipe
Including Load Balance, App Servers & DB
Separating Data from Process Implementation improves the traceability increases dramatically.
Portability in Test Environments increases Quality and Scalability by removing bottlenecks in
shared infrastructure.
Worker
WorkerWorker
Test Env
Worker
Worker
Worker
WorkerWorker
Test Env
Worker
Test EnvTest Env
Process & Implementation
“Provisioning, Install and Dependency Management”
If one or more environments is provisioned manually it becomes a
bottleneck
Application artifacts need dependency management to infrastructure.
Infrastructure has to be provisioned and installed with the application.
Databases have to be upgraded and migrated as part of the automated
process.
An automated process doesn't pause for DBAs or SysAdmins to do manual
work.
People & Organization
“The Tools Team”
Initially we did development of the Continuous Delivery process as
skunkworkz.
Quickly realized this was a bottleneck.
Added a CM to our team with disastrous result.
Finally created a Tools Team.
People & Organization
“The Tools Team is a Bottleneck”
Tools Team a huge bottleneck, simplest issues took ages to fix.
Managed the Continuous Delivery process manually.
Only ones that understood the process.
Became responsible for every teams ability to interface with the process.
Only had time to do reactive maintenance.
Short on resources.
Became yet another IT department.
People & Organization
“Build Police, please dont!”
Only Tools Team and few others understood the process.
Tried to help the few others by adding a rotational “Build Police” Role, as bad
idea as it sounds.
People & Organization
“The Tools Team matures”
Continuous Delivery as a Service
Tools Team responsible for the process implementation, its interfaces and
the infrastructure it runs on.
The Development Teams responsible for integrating with these interfaces,
configuring the pipe generator and maintaining green state of pipes.
Tools Team supports and helps if there are bugs in the mechanisms.
Tools Team consists of a core team of Developers, Testers and Operations
specialists but is extended with part time resources from Development
Teams.
People & Organization
“Lack of Consensus”
Disagreements on shared responsibility for the release.
Disagreements on what test automation is and how it should be used.
Disagreement on who implements tests and how.
A Continuous Delivery Engine without the consensus of the people that use
it is just an overpriced CI System.
Product Integrity
“Build Quality In”
SYSTEM
GUI
Product Integrity
“Build Quality In”
Internal
Service
GUI
Internal
Service
Internal
Service
Service API
Product Integrity
“Build Quality In”
Internal
Service
Internal
Service
Internal
Service
Each Service Tested as Black Box in isolation
and with high detail level
Product Integrity
“Build Quality In”
GUI
Service API Mock Impl
GUI Test are much faster and robust if they test the GUI in isolation.
High detail level.
Product Integrity
“Build Quality In”
Internal
Service
GUI
Internal
Service
Internal
Service
Service API
Verifying Use Case
acceptance criterias.
Low level of detail
Product Integrity
“Build Quality In” Optimizes Lead Time
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Assemble Release Deploy Test
Component Pipe
1000s of tests run in parallel testing
System Pipe
10s Use Case Requirements Verified
Product Integrity
“Build Quality In”
An Architecture & Test Strategy that goes hand in hand increases product
integrity and optimizes the lead time and allows the Continuous Delivery
process to scale
People & Organization
“Empower the Team!”
Increase Team responsibility.
Team is responsible for everything within the Continuous Delivery process
Analyzing Requirements
Defining Requirement Verification
Automating Requirement Verification
Implementing the System
Shipping the System
Supporting the System
Create Team consensus on what its responsibility and delivery is.
People & Organization
“Empower the Team!”
To help empower teams we have
Autonomous Team Environments where teams are responsible for their own
Continuous Delivery increased the teams understanding of the process.
Teams that needed extra help sent one or more individuals to work part time
on Tools Team.
Architecture & Test Strategy that works well with Continuous Delivery
Cross functional communities created to help increase consensus on Test
Driven Development, Continuous Delivery and Architecture.
People & Organization
“Test Driven Development Community”
Developers need to take more responsibility.
Code has to work all the time.
Test Driven Development requires developers to participate in specification
of requirement verifications.
Test Automation is Code, Developers need to code much more tests.
Continuous Regression Testing means Developers get instant feedback and
have to act on it.
People & Organization
“Test Driven Development Community”
Test Profession changes.
Two main type of Test activities Test Automation and Exploratory Testing.
Test Automation is Code.
Let developers test.
Test Driven Development is Proactive.
Exploratory testing is NOT manual regression testing.
People & Organization
“Continuous Delivery Community”
Agreement on Practices of Continuous Delivery
Agreement on Practices of Provisioning & Deployment
Competence Development
Infrastructure as Code
Testing Infrastructure as Code
Operations Specialists need to work with Developer tools and languages
People & Organization
“Organizational Support”
Continuous Delivery is organizational change.
Scaling Continuous Delivery is as much creating a scalable agile
organization as it is creating a scalable implementation.
Organization has to fully support Continuous Delivery through re-organization.
Organization has to fully support and invest in infrastructure, architecture,
automation, process and test.
Organization needs to start measuring key values, such as cycle time, early.
Summary
Process & Implementation
Portability is Scalability.
Product Integrity
The Architecture & Test Strategy go hand in hand.
People & Organization
Empowered Teams and Consensus.
Thats it!
Feedback & Any questions you forgot to ask?
http://continuous-delivery-and-more.blogspot.se
@TomasRihaSE
or by
the Coffee stand!
And the slides are at
http://www.slideshare.net/TomasRiha

More Related Content

What's hot

Tech Mahindra ADOPT©: Accelerate DevOps Transformation
Tech Mahindra ADOPT©: Accelerate DevOps TransformationTech Mahindra ADOPT©: Accelerate DevOps Transformation
Tech Mahindra ADOPT©: Accelerate DevOps TransformationCA Technologies
 
DevOps Introduction
DevOps IntroductionDevOps Introduction
DevOps IntroductionRobert Sell
 
XebiaLabs Enterprise DevOps
XebiaLabs Enterprise DevOpsXebiaLabs Enterprise DevOps
XebiaLabs Enterprise DevOpsBob Sokol
 
DevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesDevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesSlideTeam
 
Deployment Strategies Powerpoint Presentation Slides
Deployment Strategies Powerpoint Presentation SlidesDeployment Strategies Powerpoint Presentation Slides
Deployment Strategies Powerpoint Presentation SlidesSlideTeam
 
Building a DevOps Team that isn't Evil
Building a DevOps Team that isn't EvilBuilding a DevOps Team that isn't Evil
Building a DevOps Team that isn't EvilIBM UrbanCode Products
 
DevOps Challenges and Best Practices
DevOps Challenges and Best PracticesDevOps Challenges and Best Practices
DevOps Challenges and Best PracticesBrian Chorba
 
Evolving Team Structure in DevOps
Evolving Team Structure in DevOpsEvolving Team Structure in DevOps
Evolving Team Structure in DevOpsSherry Chang
 
Engineering DevOps to meet Business Goals
 Engineering DevOps to meet Business Goals Engineering DevOps to meet Business Goals
Engineering DevOps to meet Business GoalsMarc Hornbeek
 
DevOps - A Gentle Introduction
DevOps - A Gentle IntroductionDevOps - A Gentle Introduction
DevOps - A Gentle IntroductionGanesh Samarthyam
 
The Journey to DevOps #MFSummit2017
The Journey to DevOps #MFSummit2017The Journey to DevOps #MFSummit2017
The Journey to DevOps #MFSummit2017Micro Focus
 
What is DevOps? - ITSM Academy Webinar
What is DevOps?  - ITSM Academy Webinar What is DevOps?  - ITSM Academy Webinar
What is DevOps? - ITSM Academy Webinar ITSM Academy, Inc.
 
DevOps - an Agile Perspective (at Scale)
DevOps - an Agile Perspective (at Scale)DevOps - an Agile Perspective (at Scale)
DevOps - an Agile Perspective (at Scale)Brad Appleton
 
DevOps Transformation - Another View
DevOps Transformation - Another ViewDevOps Transformation - Another View
DevOps Transformation - Another ViewAgron Fazliu
 
DevOps Overview
DevOps OverviewDevOps Overview
DevOps OverviewSagar Mody
 

What's hot (20)

Tech Mahindra ADOPT©: Accelerate DevOps Transformation
Tech Mahindra ADOPT©: Accelerate DevOps TransformationTech Mahindra ADOPT©: Accelerate DevOps Transformation
Tech Mahindra ADOPT©: Accelerate DevOps Transformation
 
Bn1006 demo ppt devops
Bn1006 demo ppt devopsBn1006 demo ppt devops
Bn1006 demo ppt devops
 
DevOps for beginners
DevOps for beginnersDevOps for beginners
DevOps for beginners
 
DevOps Introduction
DevOps IntroductionDevOps Introduction
DevOps Introduction
 
XebiaLabs Enterprise DevOps
XebiaLabs Enterprise DevOpsXebiaLabs Enterprise DevOps
XebiaLabs Enterprise DevOps
 
DevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation SlidesDevOps Powerpoint Presentation Slides
DevOps Powerpoint Presentation Slides
 
Deployment Strategies Powerpoint Presentation Slides
Deployment Strategies Powerpoint Presentation SlidesDeployment Strategies Powerpoint Presentation Slides
Deployment Strategies Powerpoint Presentation Slides
 
Building a DevOps Team that isn't Evil
Building a DevOps Team that isn't EvilBuilding a DevOps Team that isn't Evil
Building a DevOps Team that isn't Evil
 
Introduction to DevOps
Introduction to DevOpsIntroduction to DevOps
Introduction to DevOps
 
DevOps Challenges and Best Practices
DevOps Challenges and Best PracticesDevOps Challenges and Best Practices
DevOps Challenges and Best Practices
 
Evolving Team Structure in DevOps
Evolving Team Structure in DevOpsEvolving Team Structure in DevOps
Evolving Team Structure in DevOps
 
Engineering DevOps to meet Business Goals
 Engineering DevOps to meet Business Goals Engineering DevOps to meet Business Goals
Engineering DevOps to meet Business Goals
 
DevOps - A Gentle Introduction
DevOps - A Gentle IntroductionDevOps - A Gentle Introduction
DevOps - A Gentle Introduction
 
The Journey to DevOps #MFSummit2017
The Journey to DevOps #MFSummit2017The Journey to DevOps #MFSummit2017
The Journey to DevOps #MFSummit2017
 
What is DevOps? - ITSM Academy Webinar
What is DevOps?  - ITSM Academy Webinar What is DevOps?  - ITSM Academy Webinar
What is DevOps? - ITSM Academy Webinar
 
DevOps101 (version 2)
DevOps101 (version 2)DevOps101 (version 2)
DevOps101 (version 2)
 
DevOps - an Agile Perspective (at Scale)
DevOps - an Agile Perspective (at Scale)DevOps - an Agile Perspective (at Scale)
DevOps - an Agile Perspective (at Scale)
 
DevOps Transformation - Another View
DevOps Transformation - Another ViewDevOps Transformation - Another View
DevOps Transformation - Another View
 
DevOps 2020: How Enterprise Strategy has Matured
DevOps 2020: How Enterprise Strategy has MaturedDevOps 2020: How Enterprise Strategy has Matured
DevOps 2020: How Enterprise Strategy has Matured
 
DevOps Overview
DevOps OverviewDevOps Overview
DevOps Overview
 

Similar to Scaling continuous delivery @ GeeCon 2014

Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyMike Brittain
 
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCContinuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCMike Brittain
 
Functional Continuous Integration with Selenium and Hudson
Functional Continuous Integration with Selenium and HudsonFunctional Continuous Integration with Selenium and Hudson
Functional Continuous Integration with Selenium and HudsonDavid Jellison
 
Continuous delivery @ hi q
Continuous delivery @ hi qContinuous delivery @ hi q
Continuous delivery @ hi qTomas Riha
 
Continuous Delivery Testing @HiQ
Continuous Delivery Testing @HiQContinuous Delivery Testing @HiQ
Continuous Delivery Testing @HiQTomas Riha
 
Continuous delivery its not about the technology, its about the people. @pipe...
Continuous delivery its not about the technology, its about the people. @pipe...Continuous delivery its not about the technology, its about the people. @pipe...
Continuous delivery its not about the technology, its about the people. @pipe...Tomas Riha
 
Testing Microservices
Testing MicroservicesTesting Microservices
Testing MicroservicesNathan Jones
 
From Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auFrom Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auevanbottcher
 
Continuous delivery its not about the technology, its about the people. @sats...
Continuous delivery its not about the technology, its about the people. @sats...Continuous delivery its not about the technology, its about the people. @sats...
Continuous delivery its not about the technology, its about the people. @sats...Tomas Riha
 
Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S...
 Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S... Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S...
Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S...Skytap Cloud
 
Performance Test Slideshow Recent
Performance Test Slideshow RecentPerformance Test Slideshow Recent
Performance Test Slideshow RecentFuture Simmons
 
Performance Test Slideshow R E C E N T
Performance Test Slideshow R E C E N TPerformance Test Slideshow R E C E N T
Performance Test Slideshow R E C E N TFuture Simmons
 
AfterTest Madrid March 2016 - DevOps and Testing Introduction
AfterTest Madrid March 2016 - DevOps and Testing IntroductionAfterTest Madrid March 2016 - DevOps and Testing Introduction
AfterTest Madrid March 2016 - DevOps and Testing IntroductionPeter Marshall
 
Continuous Delivery in a Complex S.O.A.
Continuous Delivery in a Complex S.O.A.Continuous Delivery in a Complex S.O.A.
Continuous Delivery in a Complex S.O.A.Richard Lennox
 
Cloud continuous integration- A distributed approach using distinct services
Cloud continuous integration- A distributed approach using distinct servicesCloud continuous integration- A distributed approach using distinct services
Cloud continuous integration- A distributed approach using distinct servicesAndré Agostinho
 
Continuous testing & devops with @petemar5hall
Continuous testing & devops with @petemar5hallContinuous testing & devops with @petemar5hall
Continuous testing & devops with @petemar5hallPeter Marshall
 
Soasta | CloudBees webinar 11/12/2015
Soasta | CloudBees webinar 11/12/2015Soasta | CloudBees webinar 11/12/2015
Soasta | CloudBees webinar 11/12/2015SOASTA
 
Continous integration and delivery for single page applications
Continous integration and delivery for single page applicationsContinous integration and delivery for single page applications
Continous integration and delivery for single page applicationsSunil Dalal
 
Kevin Slade - CV
Kevin Slade - CVKevin Slade - CV
Kevin Slade - CVKevin Slade
 
Karim Fanadka
Karim FanadkaKarim Fanadka
Karim FanadkaCodeFest
 

Similar to Scaling continuous delivery @ GeeCon 2014 (20)

Principles and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at EtsyPrinciples and Practices in Continuous Deployment at Etsy
Principles and Practices in Continuous Deployment at Etsy
 
Continuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYCContinuous Deployment at Etsy — TimesOpen NYC
Continuous Deployment at Etsy — TimesOpen NYC
 
Functional Continuous Integration with Selenium and Hudson
Functional Continuous Integration with Selenium and HudsonFunctional Continuous Integration with Selenium and Hudson
Functional Continuous Integration with Selenium and Hudson
 
Continuous delivery @ hi q
Continuous delivery @ hi qContinuous delivery @ hi q
Continuous delivery @ hi q
 
Continuous Delivery Testing @HiQ
Continuous Delivery Testing @HiQContinuous Delivery Testing @HiQ
Continuous Delivery Testing @HiQ
 
Continuous delivery its not about the technology, its about the people. @pipe...
Continuous delivery its not about the technology, its about the people. @pipe...Continuous delivery its not about the technology, its about the people. @pipe...
Continuous delivery its not about the technology, its about the people. @pipe...
 
Testing Microservices
Testing MicroservicesTesting Microservices
Testing Microservices
 
From Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.auFrom Monoliths to Microservices at Realestate.com.au
From Monoliths to Microservices at Realestate.com.au
 
Continuous delivery its not about the technology, its about the people. @sats...
Continuous delivery its not about the technology, its about the people. @sats...Continuous delivery its not about the technology, its about the people. @sats...
Continuous delivery its not about the technology, its about the people. @sats...
 
Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S...
 Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S... Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S...
Tips to achieve continuous integration/delivery using HP ALM, Jenkins, and S...
 
Performance Test Slideshow Recent
Performance Test Slideshow RecentPerformance Test Slideshow Recent
Performance Test Slideshow Recent
 
Performance Test Slideshow R E C E N T
Performance Test Slideshow R E C E N TPerformance Test Slideshow R E C E N T
Performance Test Slideshow R E C E N T
 
AfterTest Madrid March 2016 - DevOps and Testing Introduction
AfterTest Madrid March 2016 - DevOps and Testing IntroductionAfterTest Madrid March 2016 - DevOps and Testing Introduction
AfterTest Madrid March 2016 - DevOps and Testing Introduction
 
Continuous Delivery in a Complex S.O.A.
Continuous Delivery in a Complex S.O.A.Continuous Delivery in a Complex S.O.A.
Continuous Delivery in a Complex S.O.A.
 
Cloud continuous integration- A distributed approach using distinct services
Cloud continuous integration- A distributed approach using distinct servicesCloud continuous integration- A distributed approach using distinct services
Cloud continuous integration- A distributed approach using distinct services
 
Continuous testing & devops with @petemar5hall
Continuous testing & devops with @petemar5hallContinuous testing & devops with @petemar5hall
Continuous testing & devops with @petemar5hall
 
Soasta | CloudBees webinar 11/12/2015
Soasta | CloudBees webinar 11/12/2015Soasta | CloudBees webinar 11/12/2015
Soasta | CloudBees webinar 11/12/2015
 
Continous integration and delivery for single page applications
Continous integration and delivery for single page applicationsContinous integration and delivery for single page applications
Continous integration and delivery for single page applications
 
Kevin Slade - CV
Kevin Slade - CVKevin Slade - CV
Kevin Slade - CV
 
Karim Fanadka
Karim FanadkaKarim Fanadka
Karim Fanadka
 

More from Tomas Riha

Driving change
Driving changeDriving change
Driving changeTomas Riha
 
DevOps - Its just Agile done right
DevOps - Its just Agile done rightDevOps - Its just Agile done right
DevOps - Its just Agile done rightTomas Riha
 
Test Automation
Test AutomationTest Automation
Test AutomationTomas Riha
 
Show me the money!
Show me the money!Show me the money!
Show me the money!Tomas Riha
 
Show me the money! - Draft
Show me the money! - DraftShow me the money! - Draft
Show me the money! - DraftTomas Riha
 
Into the cloud
Into the cloudInto the cloud
Into the cloudTomas Riha
 
Continuous delivery @ Diabol
Continuous delivery @ DiabolContinuous delivery @ Diabol
Continuous delivery @ DiabolTomas Riha
 
Continuous delivery its not about the technology, its about the people.
Continuous delivery its not about the technology, its about the people.Continuous delivery its not about the technology, its about the people.
Continuous delivery its not about the technology, its about the people.Tomas Riha
 
Continuous delivery @åf consult
Continuous delivery @åf consultContinuous delivery @åf consult
Continuous delivery @åf consultTomas Riha
 

More from Tomas Riha (10)

Driving change
Driving changeDriving change
Driving change
 
DevOps - Its just Agile done right
DevOps - Its just Agile done rightDevOps - Its just Agile done right
DevOps - Its just Agile done right
 
Test Automation
Test AutomationTest Automation
Test Automation
 
Show me the money!
Show me the money!Show me the money!
Show me the money!
 
Show me the money! - Draft
Show me the money! - DraftShow me the money! - Draft
Show me the money! - Draft
 
Dev ops
Dev opsDev ops
Dev ops
 
Into the cloud
Into the cloudInto the cloud
Into the cloud
 
Continuous delivery @ Diabol
Continuous delivery @ DiabolContinuous delivery @ Diabol
Continuous delivery @ Diabol
 
Continuous delivery its not about the technology, its about the people.
Continuous delivery its not about the technology, its about the people.Continuous delivery its not about the technology, its about the people.
Continuous delivery its not about the technology, its about the people.
 
Continuous delivery @åf consult
Continuous delivery @åf consultContinuous delivery @åf consult
Continuous delivery @åf consult
 

Recently uploaded

Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfalexjohnson7307
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptxFIDO Alliance
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Paige Cruz
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Skynet Technologies
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfSrushith Repakula
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...panagenda
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard37
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxFIDO Alliance
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxjbellis
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)Samir Dash
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfdanishmna97
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...FIDO Alliance
 
Microsoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdfMicrosoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdfOverkill Security
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsLeah Henrickson
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireExakis Nelite
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistandanishmna97
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewDianaGray10
 

Recently uploaded (20)

Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdfHow we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
Overview of Hyperledger Foundation
Overview of Hyperledger FoundationOverview of Hyperledger Foundation
Overview of Hyperledger Foundation
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Vector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptxVector Search @ sw2con for slideshare.pptx
Vector Search @ sw2con for slideshare.pptx
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Microsoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdfMicrosoft BitLocker Bypass Attack Method.pdf
Microsoft BitLocker Bypass Attack Method.pdf
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
How to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in PakistanHow to Check GPS Location with a Live Tracker in Pakistan
How to Check GPS Location with a Live Tracker in Pakistan
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 

Scaling continuous delivery @ GeeCon 2014

  • 2. Tomas Riha Architect @ VGT/WirelessCar Passionate about creativity, change and improvement Horrible at following instructions and performing repetitive tasks MAJOR Project Liability mail: triha74@gmail.com twitter: @TomasRihaSE blog: continuous-delivery-and-more.blogspot.com slides: www.slideshare.net/TomasRiha
  • 3. Three Years ago New Telematics Delivery Platform Multiple Stakeholders Continuous Regression Testing How hard can it be?
  • 4. Regression Test Unit Testing Component Testing System Testing Rollback/Compatibility Testing Failover Testing Performance Testing
  • 5. At first it was super easy! Small team of just product owner architect and scrum master. Huge productivity, natural test driven development. Fast return on investment All by the book.... we were doing Continuous Delivery!
  • 6. Then we tried to scale it... ... and we failed in every possible way. We lost our test driven development We lost the individual responsibility We more or less became a automated waterfall unable to delivery daily We lost our ability to Continuously Improve the process
  • 7. Three components of Continuous Delivery Process & Implementation Definition of the Software Delivery process and its lead time optimization Product Integrity The Architecture & Test Strategy to optimize product integrity and lead time. People & Organization Behavioural change to the individuals and the organization needed to optimize lead time and increase product integrity.
  • 8. Process & Implementation “Dedicated test servers” First Continuous Delivery implementation relied on standard corporate test environment with one system test server, one integration test server and one pre prod server per production delivery Only one commit tested at the time. Build Release Deploy Test DB Test Server
  • 9. Process & Implementation “Single threaded process” Stacking jobs became a problem at 5 code committers, almost killed us at 20 Feedback time == Stack Depth * Pipe Exec Time Variable feedback time results in less responsible developers We were somewhat helped by component based architecture
  • 10. Process & Implementation “First effort to scale” Forced to use our internal Operations as infrastructure provider, no cloud and very rudimentary virtualization Built a Server Pool based on Jenkins slaves and “sticky pipes” Primitive but multi threaded pipes Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker
  • 11. Process & Implementation “Crowded Environment” Visibility was bad due to hundreds of jobs on same build system. Teams blocking each other due to still insufficient server resources. Traceability hard due to all server clutter. Changes to one teams Test Environment affected the other teams. Overall performance was horrible at peak with Jenkins totally dieing
  • 12. Process & Implementation “Continuous Delivery of Continuous Delivery” Live what you teach! Our ability to Continuously Improve was seriously slowed by lack of Continuous Delivery Deployment Scripts didn't have test automation. No test environment for Continuous Delivery Process Build pipes where manually maintained. Jenkins servers where manually maintained.
  • 13. Process & Implementation “Autonomous Environments for Autonomous Teams” Started to do Continuous Delivery of Continuous Delivery Automated provisioning of Continuous Delivery Environment Created template based pipeline generator Built Autonomous Continuous Delivery environments for each team Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker
  • 14. Process & Implementation “Portability is Scalability” Portability in the Runtime Environments Dev Test Load Test UAT PROD INT Test Complexity Scale
  • 15. Process & Implementation “Portability is Scalability” If the Continuous Delivery process can run without a Build Server then it can scale to X number of build servers By ensuring that anything above the portability line can run local or remote to the orchestration we ensure portability Orchestration ProvisionBuild Deploy Test Release Test EnvironmentSCM Artifact Repo Portability Line
  • 16. Process & Implementation “Portability is Scalability” All environments provisioned and deployed in the same way Orchestration ProvisionBuild Deploy Test Commit Orchestration ProvisionBuild Deploy Test Release Orchestration Provision Deploy Test Accept Orchestration Provision Deploy Test Local Dev Continuous Delivery Pipe UAT Prod
  • 17. Process & Implementation “Portability is Scalability” Dev Test Load Test UAT INT Test Provisioning Interface Vagrant Impl Cloud Provider Impl PDF Document Impl Topology Spec Env Spec Legacy PROD With the help of Cloud Technology and Vagrant we manage to create test and dev environments with the same complexity as our production environments.
  • 18. Process & Implementation “Portability is Scalability” To make the Continuous Delivery Process portable we also need to move all reporting and monitoring out of the test servers and the build servers Orchestration ProvisionBuild Deploy Test Release Test EnvironmentSCM Artifact Repo Logging & Metrics Services Build Data Repository
  • 19. Process & Implementation “Portability is Scalability” Orchestration Worker Log, Metrics, Build Data Repositories WorkerWorker Build Env Orchestration Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Worker Team Environment Pre Cloud One Server Per Pipe Shared Database for all pipes Team Environment in Cloud One Environment Per Pipe Including Load Balance, App Servers & DB Separating Data from Process Implementation improves the traceability increases dramatically. Portability in Test Environments increases Quality and Scalability by removing bottlenecks in shared infrastructure. Worker WorkerWorker Test Env Worker Worker Worker WorkerWorker Test Env Worker Test EnvTest Env
  • 20. Process & Implementation “Provisioning, Install and Dependency Management” If one or more environments is provisioned manually it becomes a bottleneck Application artifacts need dependency management to infrastructure. Infrastructure has to be provisioned and installed with the application. Databases have to be upgraded and migrated as part of the automated process. An automated process doesn't pause for DBAs or SysAdmins to do manual work.
  • 21. People & Organization “The Tools Team” Initially we did development of the Continuous Delivery process as skunkworkz. Quickly realized this was a bottleneck. Added a CM to our team with disastrous result. Finally created a Tools Team.
  • 22. People & Organization “The Tools Team is a Bottleneck” Tools Team a huge bottleneck, simplest issues took ages to fix. Managed the Continuous Delivery process manually. Only ones that understood the process. Became responsible for every teams ability to interface with the process. Only had time to do reactive maintenance. Short on resources. Became yet another IT department.
  • 23. People & Organization “Build Police, please dont!” Only Tools Team and few others understood the process. Tried to help the few others by adding a rotational “Build Police” Role, as bad idea as it sounds.
  • 24. People & Organization “The Tools Team matures” Continuous Delivery as a Service Tools Team responsible for the process implementation, its interfaces and the infrastructure it runs on. The Development Teams responsible for integrating with these interfaces, configuring the pipe generator and maintaining green state of pipes. Tools Team supports and helps if there are bugs in the mechanisms. Tools Team consists of a core team of Developers, Testers and Operations specialists but is extended with part time resources from Development Teams.
  • 25. People & Organization “Lack of Consensus” Disagreements on shared responsibility for the release. Disagreements on what test automation is and how it should be used. Disagreement on who implements tests and how. A Continuous Delivery Engine without the consensus of the people that use it is just an overpriced CI System.
  • 27. Product Integrity “Build Quality In” Internal Service GUI Internal Service Internal Service Service API
  • 28. Product Integrity “Build Quality In” Internal Service Internal Service Internal Service Each Service Tested as Black Box in isolation and with high detail level
  • 29. Product Integrity “Build Quality In” GUI Service API Mock Impl GUI Test are much faster and robust if they test the GUI in isolation. High detail level.
  • 30. Product Integrity “Build Quality In” Internal Service GUI Internal Service Internal Service Service API Verifying Use Case acceptance criterias. Low level of detail
  • 31. Product Integrity “Build Quality In” Optimizes Lead Time Build Release Deploy Test Build Release Deploy Test Build Release Deploy Test Build Release Deploy Test Build Release Deploy Test Assemble Release Deploy Test Component Pipe 1000s of tests run in parallel testing System Pipe 10s Use Case Requirements Verified
  • 32. Product Integrity “Build Quality In” An Architecture & Test Strategy that goes hand in hand increases product integrity and optimizes the lead time and allows the Continuous Delivery process to scale
  • 33. People & Organization “Empower the Team!” Increase Team responsibility. Team is responsible for everything within the Continuous Delivery process Analyzing Requirements Defining Requirement Verification Automating Requirement Verification Implementing the System Shipping the System Supporting the System Create Team consensus on what its responsibility and delivery is.
  • 34. People & Organization “Empower the Team!” To help empower teams we have Autonomous Team Environments where teams are responsible for their own Continuous Delivery increased the teams understanding of the process. Teams that needed extra help sent one or more individuals to work part time on Tools Team. Architecture & Test Strategy that works well with Continuous Delivery Cross functional communities created to help increase consensus on Test Driven Development, Continuous Delivery and Architecture.
  • 35. People & Organization “Test Driven Development Community” Developers need to take more responsibility. Code has to work all the time. Test Driven Development requires developers to participate in specification of requirement verifications. Test Automation is Code, Developers need to code much more tests. Continuous Regression Testing means Developers get instant feedback and have to act on it.
  • 36. People & Organization “Test Driven Development Community” Test Profession changes. Two main type of Test activities Test Automation and Exploratory Testing. Test Automation is Code. Let developers test. Test Driven Development is Proactive. Exploratory testing is NOT manual regression testing.
  • 37. People & Organization “Continuous Delivery Community” Agreement on Practices of Continuous Delivery Agreement on Practices of Provisioning & Deployment Competence Development Infrastructure as Code Testing Infrastructure as Code Operations Specialists need to work with Developer tools and languages
  • 38. People & Organization “Organizational Support” Continuous Delivery is organizational change. Scaling Continuous Delivery is as much creating a scalable agile organization as it is creating a scalable implementation. Organization has to fully support Continuous Delivery through re-organization. Organization has to fully support and invest in infrastructure, architecture, automation, process and test. Organization needs to start measuring key values, such as cycle time, early.
  • 39. Summary Process & Implementation Portability is Scalability. Product Integrity The Architecture & Test Strategy go hand in hand. People & Organization Empowered Teams and Consensus.
  • 40. Thats it! Feedback & Any questions you forgot to ask? http://continuous-delivery-and-more.blogspot.se @TomasRihaSE or by the Coffee stand! And the slides are at http://www.slideshare.net/TomasRiha