Continuous Delivery practices have increasingly become “table stakes” for effective software delivery since the publication of the book Continuous Delivery by Jez Humble and Dave Farley in 2010. The combination of sound technical practices and a scientific approach to testing and feature development has led to significant success with software delivery with organisations around the world. However, in large organisations with many teams and many different suppliers, there are also many ideas about what Continuous Delivery is. This variety of approaches and assumptions can lead to conflicts around deployments, testing, releasing, and operations, resulting in a macro-level sub-optimal flow of change. In this talk, Matthew Skelton will share some insights from his time spent as Engineering Lead at a large GOV.UK department during most of 2018. There he championed Continuous Delivery practices across 70+ teams and 7 locations, helping to raise standards for software operability, Developer Experience (DevEx), testing, deployments, and inter-team communications. He will share some practical techniques for getting Continuous Delivery working at scale.

1. 1
Continuous Delivery at scale
in the public sector
Matthew Skelton, Conflux
@matthewpskelton
confluxdigital.net
NHS Digital Agile Community of Practice
Leeds, 27 March 2019

2. Continuous Delivery at scale
Context and challenges
“Continuous Delivery”
Things we did to help
Results so far
2
@matthewpskelton

3. Context and challenges
3

4. GOV.UK departmental division
4
700+ people in total
70+ engineering teams, 7 locations
Time-critical delivery (#Brexit)
Most of 2018

5. Complicated environment
5
Increase speed & safety of delivery
Multi-year programme
New Brexit-related work
Track & control infra costs

6. Technology landscape
6
(Trad DC + VMs + JVM)
AWS + Kubernetes + JVM
ELK for logs and transaction traces
[ ]

7. 7

8. Scale existing
approach to 7x size
8

9. Spread awareness of
effective practices
9

10. Cross internal and
external boundaries
10

11. “Continuous Delivery”
11

12. Avoid “one-size-fits-all”
12

13. Define the Platform
13

14. Improve logging quality
14

15. Multi-supplier delivery model
15

16. “Continuous Delivery”
16

17. 17
Reliable Software Releases
Through Build, Test, and
Deployment Automation
Jez Humble & Dave Farley, 2010
Continuous
Delivery

18. Continuous Delivery needs...
18
Good engineering practices (TDD, ...)
Fast feedback from deployment pipelines
Re-aligned architecture
Team ownership of software & services

19. Continuous Delivery overview
19
Continuous Delivery with
Windows and .NET
Matthew Skelton & Chris O’Dell
O’Reilly, 2016
** Now a key text for MSc in
Software Engineering at UCL **
cdwithwindows.net

20. Things we did to help
20

21. New flow-based
operating model
21

22. Focus on the
"operator experience"
22

23. Experts on site
23

24. Engineering Guild
Engineering Working Group
24

25. Weekly lunchtime tech talks
25

26. Continuous Delivery for
people who don’t write code
26

27. Team-first techniques
Example: Run Book Dialogue Sheet
27

28. 28
runbooktemplate.infoRun Book dialogue sheets

29. Engineering Standards
29

30. 30

31. Engineering Standards
31
Minimal but important constraints
Dynamic for reviewing and updating
Emphasise team engagement

32. Engineering Assessments
32

33. Engineering Assessments
33
6 dimensions:
Team Health, Deployment, Continuous
Delivery, Flow, Operability, Testing
SoftwareDeliveryAssessment.com

34. Engineering Assessments
34
Criteria taken from existing sources:
Spotify Team Health Check, DevOps for the Modern
Enterprise, Continuous Delivery, Accelerate, Team
Guide to Software Operability, Team Guide to Software
Testability

35. 35

36. 36

37. 37

38. Engineering Assessments
38
Team self-assessment
Facilitated to guide discussions
“Viral” facilitation model
Results collated across teams

39. Results so far
39

40. 40
Software releases
From 2 successful release candidate
builds per week to 7 or 8 builds per day
Multiple, independent routes to live

41. 41
Operability
Ops teams love the new "operator
experience" with logging & dashboards
No major operational problems with new
releases

42. 42
Collaboration
Logging beginning to be seen as a key
observability tool
Better cross-team awareness of how
systems work through telemetry

43. 43
(Many other people also involved)

44. Key Takeaways:
Continuous Delivery at scale
44

45. 45
Make space for learning/sharing
Promote good work
Help to develop skills in speaking
Lunchtime talks (food)
Interplay of working group + guild
@matthewpskelton

46. 46
Define the Platform
Service Level Expectations
Who owns what? What's missing?
Use DevEx to drive improvements
Thinnest Viable Platform
@matthewpskelton

47. 47
Address operability early on
Operational aspects are also features
Add operational checks into the
deployment pipeline
Team-first: RunBookTemplate.info
@matthewpskelton

48. 48
Good logging is foundational
Use a well-defined event space
Searchable EventIDs increase awareness
Enables collaboration between teams
Invest in logging infra (Saas/people/&c.)
@matthewpskelton

49. 49
Involve teams in improvements
Co-create the engineering standards
Look for organisational blockers
“No-blame” is paramount
SoftwareDeliveryAssessment.com
@matthewpskelton

50. Continuous Delivery at scale
50
Make space for learning/sharing
Define the Platform
Address operability early on
Good logging is foundational
Involve teams in improvements
@matthewpskelton

51. Further reading & information
51
TechTalksForBeginners.com
RunBookTemplate.info
SoftwareDeliveryAssessment.com
OperabilityBook.com

52. Team Topologies
52
Organizing business and
technology teams for fast flow
Matthew Skelton & Manuel Pais
Sept 2019, IT Revolution Press
teamtopologies.com

53. thank you
53
@matthewpskelton
@ConfluxHQ / confluxdigital.net