This document discusses the challenges of implementing continuous delivery at scale. It began with a small, highly productive team practicing test-driven development and continuous regression testing. However, when the team scaled up, they lost these practices and their quality dropped. The document outlines changes needed in development processes, testing approaches, and team roles to successfully adopt continuous delivery at an enterprise scale. Developers must take more responsibility for testing and quality. Testers need skills in test automation and working with developers. Teams must be cross-functional and take full ownership of delivery. Continuous delivery requires transforming the entire organization, not just the technology.

1. Continuous Delivery
It’s not about the technology it’s about the
people.

2. Tomas Riha
Architect @ VGT/WirelessCar
Passionate about creativity, change and improvement
Horrible at following instructions and performing repetitive tasks
MAJOR Project Liability
mail: triha74@gmail.com
twitter: @TomasRihaSE
blog: continuous-delivery-and-more.blogspot.com

3. Four Years ago
New Telematics Delivery Platform
Multiple Stakeholders
Continuous Regression Testing
How hard can it be?

4. Regression Test
Unit Testing
Component Testing
System Testing
Rollback/Compatibility Testing
Failover Testing
Performance Testing

5. At first it was super easy!
Small team of just product owner, architects and scrum
master.
Huge productivity, natural acceptance test driven
development.
Fast return on investment
All by the book.... we were doing
Continuous Delivery!

6. Then we tried to scale it...
... and we failed in every possible way.
We lost our test driven development
We lost the individual responsibility
We more or less became a automated waterfall unable to
deliver daily

7. Continuous Delivery changes our behaviour
Changing the behaviour of 5 people is easy.
Changing the behaviour of 50 is super hard.
You have to understand the changes!
Continuous Delivery changes the entire organization.

8. Developers
Everything has to work all the time!
Pre
Planning
Dev Sys Test Reg Test
Pre
Planning
Dev Sys Test Reg Test
In traditional scrummerfall projects system only works once every iteration.
No one cares if you check something in before lunch just to “save” your work.
Sprint

9. Developers
Pre
Planning
Dev
Test
Automation
Reg Test
Thats an awful lot of testing... YES!
Test driven development requires that developers work with testing.
Pre
Planning
Reg Test Reg Test
Verification
Verification
Acceptance Test Driven Development

10. Developers
Pre
Planning
Dev
Test
Automation
Reg Test
This creates context switching but its up to the developer to step it up!
Pre
Planning
Reg Test Reg Test
Verification
Verification
Regression has priority! ALWAYS!

11. Developers
Developers hate branches, or so they say
Hate merging because it mixes their code with others
Love feature branches because they don't have to integrate their work
Developers LOVE BRANCHES!!
Continuous Delivery is done on the trunk.

12. Developers
Developers need to take more responsibility for
Regression test
Writing test code
Testing the system
Integrating their code
Understanding Production Runtime
In order to always have a working system
Need to take more responsibility

13. Developers
So it’s just up to the developers to shape up?
The Continuous Delivery process needs to provide visibility and traceability to
provide means to take responsibility.
Developers need to be coached to increase responsibility and to mitigate fear.

14. Testers
When we scaled up we added testers
Developers were happy because they could go back to
handing over code for testing.
Adding testers we lost all test driven development.
Our QUALITY DROPPED!!

15. Testers
Why did this happen?
Handover developer to test
Lost consensus
What is test automation for
How is test automation built
Who does test automation
Test as Code

16. Testers
Pre
Planning
Dev Sys Test Reg Test
Pre
Planning
Dev Sys Test Reg Test
Sprint
Recruiting the Role
A tester is a tester is a tester
Manual System & Regression Testing is a reactive repetitive task.

17. Testers
Pre
Planning
Dev
Test
Automation
Reg Test
Acceptance Test Driven Development is NOT Test Driven
its driven by Automated Requirement Verification, checking.
Not all testers have the skill set or mind set
to work with Automated Requirement Verification!
Pre
Planning
Reg Test Reg Test
Verification
Verification
Acceptance Test Driven Development is proactive!

18. Testing in the past
How its been done
SYSTEM
GUI

19. Building testable systems
Internal
Service
GUI
Internal
Service
Internal
Service
Service API

20. Component Tests
Internal
Service
Internal
Service
Internal
Service
Each Service Tested as Black Box in isolation
and with high detail level

21. Continuous Delivery
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Build Release Deploy Test
Assemble Release Deploy Test
Component Pipe
100s of tests run in parallel testing
System Pipe
10s Use Case Requirements Verified

22. Testers
Exploratory Testing
Requirement Verification/Checking ensures we deliver what we promise
Exploratory Testing ensures that we improve our application
Exploratory testing done outside the release cycle
Never hide manual regression testing behind Exploratory Testing
Exploratory testing can NEVER block a production deploy

23. Testers
Agreement is key
Reliable and Repeatable Test Automation
has to be done on a black box.
Never verify on system files, databases or code.
Build quality in
Verify on interfaces
Tests as Code

24. Testers
Managing the change to the tester profession.
Developers have an expanded role
Testers have a changed role which is much harder to manage
Testers must let developers work with test
The Team creates the Tests as Code with the knowledge and experience of the
Tester and the Developer.

25. Testers
Our most sought after profile
Testers with strong technical skills
Developers with strong testing skills
Be either and earn $$$$
Be either to have a job

26. Project Management
Pre
Planning
Dev Sys Test Reg Test
Pre
Planning
Dev Sys Test Reg Test
Sprint
Scrum really doesn't fit all that well
Scrum builds up to a end of sprint release, but why not release daily?

27. Project Management
Kanban inspired feature development better fit
Building and releasing one feature “at the time” is a much better fit.
Feature cycle
Pre Planning
Dev
Test Automation
Pre Planning
Verification
Verification
Pre
Planning
Dev
Test
Automation
Pre
Planning
Verification
Verification
Feature cycle

28. The Team
The Team only takes responsibility to the next hand over
The Team will never reach out to an external party for help
The Team is only aware of its weaknesses
if its responsible for its deliveries
DevOps is a must in order to take production responsibility

29. The Team
The Team
Requirements
Architecture
Code
Deployment
Automation
Middleware
Databases Infrastructure
Monitoring
Logging
Testing
Verification
Automation

30. The Team
The Team
Requirements
Architecture
Code
Deployment
Automation
Middleware
Databases Infrastructure
Monitoring
Logging
Testing
Verification
Automation
A
TD
We recruited by roles and this is what we got!

31. The Team
The Team
Requirements
Architecture
Code
Deployment
Automation
Middleware
Databases Infrastructure
Monitoring
Logging
Testing
Verification
Automation
Recruiting by Competence and T shape

32. How did it go?
Organization changed to delivery micro service based platform through
Continuous Delivery
Team dedicated to building and delivering Continuous Delivery as a Service for
all the development teams.
Continuous Delivery as a Service used by 100+ Team Members
and
Over 200 deployable applications and libraries
We still suck at Continuous Delivery!

33. Thats it!
Feedback & Any questions you forgot to ask?
http://continuous-delivery-and-more.blogspot.se
@TomasRihaSE
triha74@gmail.com
or at
Coffee machine