Picture this scene from three years ago: Employing the corporately mandated processes, a software engineering team is delivering system updates about once every nine months. When their senior user suddenly demands the next delivery in twenty-two weeks-half the current cycle duration-the team realize that they must quickly change development practices. Mathew Bissett describes how Her Majesty's Government did precisely that-and much, much more. First, they reduced delivery cycles from unpredictable dates every nine months to predictable releases every six weeks. Then, they cut releases cycle time to once every week. By identifying and mitigating risks early in the work intake process, enforcing quality gates, executing multiple test levels concurrently-and more-they dramatically increased throughput with the same or better quality. Today, these new processes provide their teams the best balance of structure versus agility. Join Mathew to see if what works for Her Majesty's Government might just work for you and your company.

1.
BW5
Concurrent Session
11/7/2012 2:15 PM
"Reduce Release Cycle Time:
Nine Months to a Week Is Jolly Good!"
Presented by:
Mathew Bissett
UK Government
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888‐268‐8770 ∙ 904‐278‐0524 ∙ sqeinfo@sqe.com ∙ www.sqe.com

2. Mathew Bissett
UK Government
Mathew Bissett has been working for Her Majesty’s Government for more than six years, having been
recruited straight from University. Deployed into the integration and test team of his department’s main
engineering group, Mathew demonstrated strong leadership qualities to quickly progress from test analyst
through test team lead to test management. Currently the test manager responsible for the integration and
testing of his area’s flagship system, he has driven through delivery process improvements to enable weekly
deliveries. Mathew has delivered many talks within his government department and has presented at a
British Computer Society Special Interest Group in Software Testing conference.

3. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
1
Reduce Release Cycle Time
From Nine Months to a Week – Nice!
BSC East 2012 v0.13
03/09/2012
Mathew Bissett
M h Bi
Test Manager
Her Majesty’s Government
mathew.bissett@gmail.com
2
BSC East 2012 v0.13
03/09/2012
Introduction
• I joined Her Majesty’s
Government in late
2006 from University
• I worked my way up to
Test Management
within our Integration &
Test Team
• Have b
been (
(E2E) Test
)
Manager for my
Department’s flagship
product since early 2011
• I also run a bit
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
1

4. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
3
BSC East 2012 v0.13
03/09/2012
Show of Hands
• Would you call yourself a:
▫
▫
▫
▫
▫
Developer
Tester
Project Manager
Business Analyst
Other
h
© UK Crown Copyright 2012
4
BSC East 2012 v0.13
03/09/2012
Context Setting
Delivery Cycles
Conclusions & Future
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
2

5. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
5
BSC East 2012 v0.13
03/09/2012
Context Setting
Processed Data Flow
© UK Crown Copyright 2012
6
BSC East 2012 v0.13
03/09/2012
Context Setting
C
Command & Control
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
3

6. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
7
BSC East 2012 v0.13
03/09/2012
Context Setting
System Under Test
© UK Crown Copyright 2012
8
BSC East 2012 v0.13
03/09/2012
Context Setting
• “[Testing’s] role changes from verifying to
prevention by concurrent engineering with both
acceptance- and unit- test-driven development –
and that blurs the distinction between test,
requirements analysis, and design, so
that…testing is
th t t ti i no l
longer t ti ”
testing.”
[LV10]
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
4

7. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
9
BSC East 2012 v0.13
03/09/2012
Context Setting
Delivery Cycles
Conclusions & Future
© UK Crown Copyright 2012
10
BSC East 2012 v0.13
03/09/2012
9 Months
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
5

8. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
11
BSC East 2012 v0.13
03/09/2012
9 Months - Process
• Based on RUP & CMMI
▫ Heavyweight documentation up front
• Implemented more like Waterfall
▫ Defined by serial workflows
• User Acceptance left until the end
▫ Once product was nearing completion
© UK Crown Copyright 2012
12
BSC East 2012 v0.13
03/09/2012
9 Months - Process
D e v e lo p m e n t
In te g ra tio n
I& T R e fe re n c e
1
O p e ra tio n s
2
D e v e lo p c o d e
Q u a lity R e v ie w
Q u a lity R e v ie w
A p p ly (D E V ) la b e l
(C h e c k lis t)
(C h e c k lis t)
T est
In s ta ll R e le a s e
In s ta ll R e le a s e
B u ild a p p lic a tio n
B u ild V e rific a tio n
T est
A c c e p ta n c e T e s t
B u ild V e rific a tio n
T est
1
No
F it fo r
R e le a s e
Yes
F it fo r
R e le a s e
No
2
T e s tin g C o m p le te
2
S y s te m T e s t
T est
A p p ly (IN T ) la b e l
O peval
Dliv ryP c s
e e roes
2
T e s t B u ild
C re a te R e le a s e
1
Dliv ryP c s
e e roes
Dliv ryP c s
e e roes
1
C re a te R e le a s e
A p p ly (R E L ) la b e l
T e s tin g C o m p le te
Yes
C re a te R e le a s e
3
3
Ite ra tiv e ly
Q u a lity R e v ie w
(C h e c k lis t)
E la b o ra tio n /C o n s tru c tio n
T ra n s is tio n
Key
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
D e fa u lt P a th
D e v /I& T P ro c e s s
F a u lt P a th
T T O P ro c e s s
6

9. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
13
BSC East 2012 v0.13
03/09/2012
9 Months – Process
Planning
Testing
Developing
© UK Crown Copyright 2012
14
BSC East 2012 v0.13
03/09/2012
9 Months - Pros
• Time-boxed process
▫ Iterations were a fixed duration
• Code freezes
▫ After which all functionality should have been
delivered
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
7

10. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
15
BSC East 2012 v0.13
03/09/2012
9 Months - Cons
• Feature-bound delivery
▫ More iterations were added until the product was
fit for delivery - resulting in endless Transition
phases
• Subsystems were produced independently
▫ It was not possible to test major integration risks
p
j
g
until late in the delivery cycle
▫ Manual regression testing meant that integrating
and re-testing fixes was very costly
• Users not consulted until the product neared
completion
© UK Crown Copyright 2012
16
BSC East 2012 v0.13
03/09/2012
6 Weeks – The Heartbeat
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
8

11. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
17
BSC East 2012 v0.13
03/09/2012
9 Months – 6 Weeks
• Intermediary ‘Stepping Stones’ process
▫ Heavy Systems Engineering involvement
▫ Defined integration points
• Challenge lead by Senior User
▫ Changed risk appetite in exchange for a delivery
within 22 weeks
i hi
k
▫ Successfully met challenge after 20 weeks but
wasted final 2 weeks
© UK Crown Copyright 2012
18
BSC East 2012 v0.13
03/09/2012
6 Weeks - Process
• Backlog driven & ‘light-weight RUP’ based
▫ Customer lead prioritisation
▫ Planned for Product rather than Subsystems
• Delivery scope was advertised up front on a Wiki
▫ Based on customer priority
• Introduced concurrent test phases
▫ Start of automated regression testing
▫ Continuous Subsystem Integration
▫ User Trials undertaken in parallel to development
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
9

12. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
19
BSC East 2012 v0.13
03/09/2012
6 Weeks - Process
© UK Crown Copyright 2012
20
BSC East 2012 v0.13
03/09/2012
6 Weeks – Process
Planning
Developing
Testing
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
10

13. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
21
BSC East 2012 v0.13
03/09/2012
6 Weeks - Pros
• Time-boxed process
▫ Delivery cycles were fixed to 6 week
• Integration risks assessed faster
▫ Invested in Subsystem and Product automated
regression testing
• User Trials h l d User Acceptance
i l helped
▫ Prior-proven business benefit
© UK Crown Copyright 2012
22
BSC East 2012 v0.13
03/09/2012
6 Weeks - Cons
• Feature-bound delivery
▫ Great pressure to deliver desired functionality
• Requirements/Designs were often not produced
▫ Lack of upfront Systems Engineering effort
• Subsystems still produced independently
▫ Usually only managed one or two integration
points
▫ Often made to work by I&T in the final week
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
11

14. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
23
BSC East 2012 v0.13
03/09/2012
1 Week – The Pulse
© UK Crown Copyright 2012
24
BSC East 2012 v0.13
03/09/2012
6 Weeks – 1 Week
• Challenge lead by Integration & Test Team
▫ Needed to assess integration risks earlier
▫ Enabled by previous investment in automation
▫ Intention was to remove the pressure from the
end of the delivery cycle
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
12

15. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
25
BSC East 2012 v0.13
03/09/2012
1 Week - Process
• Customer Backlog and Integration Logic driven
▫ Technical lead prioritisation
• Delivery scope was more flexible
▫ Reduced the planning horizon to ~3 weeks
• Removed team independence by introducing CI
▫ Subsystem integration responsibility moved into
Development Teams
▫ Enforced by Quality Gates
© UK Crown Copyright 2012
26
BSC East 2012 v0.13
03/09/2012
1 Week - Process
CDD’s
Daily CIE Regression
Build
Logic
Test
Reports
Test
Reports
Weekly E2E & UAT
Weekly Regression
Weekly DQ Regression
Builds
VDDs
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
Builds
& Docs
13

16. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
27
BSC East 2012 v0.13
03/09/2012
1 Week - Process
Wednesday
Thursday
TRR
CIE
Friday
TRR
CIE
Sat/Sun
Monday
TRR
Tuesday
TRR
CIE
Wednesday
TRR
CIE
Thursday
TRR
CIE
CIE
TRR
CIE
TRR
Friday
Sat/Sun
Monday
TRR
Tuesday
TRR
CIE
CIE
TRR
CIE
CIE
TRR
E2E Agility
E2E Agility
E2E Regression
E2E Regression
DQ Regression
E2E
DQ Regression
TRR
TRR
Build
Build
VDD
VDDs
Test Reports
Transition
to
Deployment
Test Reports
PRM
GO
GO
PRM
© UK Crown Copyright 2012
28
BSC East 2012 v0.13
03/09/2012
1 Week - Process
Mon
Tue
Wed
Thu
Fri
Sat/Sun
Mon
Tue
Wed
B
1
3
Thu
Fri
Sat/Sun
Mon
Planning
2
2
2
A
2
2
G
2
E
E
Tue
Wed
Thu
B
Test Rpt
4
H
D
CIE
CIE
CIE
CIE
CIE
CIE
3
C
E2E
E2E Agility
1
2
3
Supplier #1
Supplier #2
4
- Pulse Planning Meeting (PPM)
- Daily Test Readiness Review (TRR) (1200hrs)
- Product Handover Meeting (1230hrs)
- Pulse Go/No Go meeting (1630hrs)
C
- Create RTC Tickets
- Update RTC Ticket
- Build and Test
D - Test results analysis and report to TRR
E - Possible new builds following TRR
F - Continuous analysis of test results and report defects
G - Produce Test Report (1400hrs)
H - Review Problem Tickets (1500hrs)
A
B
C
F
E2E Regression
F
DQ Regression
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
14

17. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
29
BSC East 2012 v0.13
03/09/2012
1 Week – Process
Planning
Developing
Testing
© UK Crown Copyright 2012
30
BSC East 2012 v0.13
03/09/2012
1 Week - Pros
• Non-feature bound time-boxed process
▫ Delivery cycles were fixed to 1 week and delivered
only what was ready
▫ Regular release candidates based on Product
integration logic
• Risk identified up front
p
▫ Testing defined in the take-on-work process
▫ Enforced by quality gates between delivery phases
• User Acceptance became an input into delivery
quality gate
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
15

18. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
31
BSC East 2012 v0.13
03/09/2012
1 Week – Cons
• Software is constantly changing
▫ Manual continuous integration is expensive
▫ Manual configuration management is hard across
many integration environments
▫ Delivery process required improvement to meet 1
week d li i
k deliveries
© UK Crown Copyright 2012
32
BSC East 2012 v0.13
03/09/2012
Context Setting
Delivery Cycles
Conclusions & Future
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
16

19. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
33
BSC East 2012 v0.13
03/09/2012
Conclusions
• Identify Risks Early
• Concurrency
• Planning Horizon
© UK Crown Copyright 2012
34
BSC East 2012 v0.13
03/09/2012
Identify Risks Early
• Introduce risk identification into your take on
work process
• Have many risk assessment opportunities and
assess as soon as practical
▫ Share mitigation of risk between teams
• Introduce Quality Gates and enforce them
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
17

20. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
35
BSC East 2012 v0.13
03/09/2012
Identify Risks Early
Phase
Who
Environment
Exit Criteria
Test Tool Development
Automation Team
Tool Development Env.
Test tools available
Development Testing
Development Team
Development Env.
x% code coverage
System Testing
Test Team
System Test Env.
x% feature coverage
Feature Trials
Project Team
Trial Environment
Initial User validation
Initial E2E verification
System Integration
Testing
Test Team
Quality Gate #1
System Integration
Environment
Verify integration to
delivery baseline
Quality Gate #2
End-to-End Testing
Test Team
End-to-End Test
Environment
End-to-End data flow
verification successful
User Acceptance
Testing
Test Team
End-to-End Test
Environment
Final User validation
successful
© UK Crown Copyright 2012
36
BSC East 2012 v0.13
03/09/2012
Concurrency
• Introduce concurrency where practical
▫ Which test phases can be performed in parallel?
▫ Use different test environments for different
purposes
• Give yourself more opportunities to succeed
▫ Regular release candidates
l
l
did
• Enforce process when pulling through bugfixes
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
18

21. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
37
BSC East 2012 v0.13
03/09/2012
Concurrency
Time to Market
20
18
16
14
12
10
8
6
4
2
0
18.5
Weekly Delivery
y
y
6-Weekly Delivery
7
6
2.5
2
1
0
P12
P13
P14
P15
P16
P17 HB21+
© UK Crown Copyright 2012
38
BSC East 2012 v0.13
03/09/2012
Concurrency
Delivery Capacity
25
23.5
20
20
15
17.5 17.5
13.5
11.5
10
5
8.5
85
12.5 12.5
9.5
18.5
14.5
11
8
5
0
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
19

22. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
39
BSC East 2012 v0.13
03/09/2012
Planning Horizon
• Only plan as far as practical
▫ Let the technical community define the best path
to meet your customers’ requirements
▫ Reduce your planning horizon to save re-work
• Advertise your planning horizon to your
customers
▫ Negotiate flexibility with your customers
• Be flexible to unexpected change
▫ Embrace the opportunity to deliver fixes quickly
© UK Crown Copyright 2012
40
BSC East 2012 v0.13
Known
Better Software Conference East 2012 v0.13
03/09/2012
Predicted Copyright 2012 Make Believe
© UK Crown
20

23. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
41
BSC East 2012 v0.13
03/09/2012
Future
• Automated CI & Build tool
• Replace E2E Environment with OpEval
Environment
• Empower Users to drive & test system and
enable direct feedback loops
© UK Crown Copyright 2012
42
BSC East 2012 v0.13
03/09/2012
Future
Package
Sys Test
Dev
Dev
Component
Component Component
Test
Test
Int. Test
Package
Sys Test
Build & Test
Queue
Component
Component Component
Test
Test
Int. Test
Component Component
Package
Test
Int. Test
Dev
Component
Test
Capability Platform
Team
Team
Sys Test
Op
Release
Static
Review
Static
Review
CI Tools
Op
Evaluation
Fu
unctional
Re
egression
Dev
Supplier
Volumetric
Regression
Dev
Dev
Delivery
Team
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
21

24. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
43
BSC East 2012 v0.13
03/09/2012
Context Setting
Delivery Cycles
Conclusions & Future
© UK Crown Copyright 2012
44
BSC East 2012 v0.13
03/09/2012
References
• Bibliography
▫ Ambler10a
Ambler, S., 2010. Agile Testing and Quality
Strategies: Discipline Over Rhetoric
▫ Ambler10b
Ambler, S 2010
Ambler S., 2010. Introduction to Disciplined Agile
Delivery
▫ CG09
Crispin, L., Gregory, J., 2009. Agile Testing: A
Practical Guide for Testers and Agile Teams
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
22

25. Reduce Release Cycle Time From Nine Months
to a Week ‐ Nice!
45
BSC East 2012 v0.13
03/09/2012
References
• Bibliography
▫ Leffingwell10
Leffingwell, D., 2010. Scaling Software Agility:
Rearchitecting Enterprise Class systems – An Agile
Enterprise Trifecta
▫ LV10
Larman, C., Vodde, B., 2010. Practices for Scaling
Lean & Agile Development: Large, Multisite, and
Offshore Product Development with Large-Scale
Scrum
© UK Crown Copyright 2012
46
BSC East 2012 v0.13
03/09/2012
Mathew Bissett
Test Manager
Her Majesty’s Government
mathew.bissett@gmail.com
© UK Crown Copyright 2012
Better Software Conference East 2012 v0.13
23