The document discusses quality and how to build it into software development processes. It begins by noting that quality is subjective and depends on many factors. It then recommends starting by creating a quality vision, selecting some metrics to measure, and evolving those metrics over time based on learnings. Common mistakes like adding extensive testing late or creating separate quality teams are noted. The document provides suggestions for areas to measure like releases, product quality, and process quality to improve software delivery performance. It emphasizes the importance of culture and psychological safety.

1. What is quality, and how
do we build it in?
@maryamumar | @jezhumble | #DeliveryConf

2. Agenda
● Introduce Thought Machine
● How to get started with improving quality
● Selecting metrics
● Evolving metrics based on what we learn
@jezhumble
@maryamumar

3. TLDR
● Quality is subjective - what to focus on depends on many factors
● Start by creating a vision, selecting some metrics, and measuring
● What you measure will evolve, and will determine what teams work on
@jezhumble
@maryamumar

4. Please fix
Quality!
Clients are unhappy
Issues are found late
Releases are delayed
Team feels stressed

6. Common mistakes we have observed
● Stop work and build a comprehensive automated test suite
● Add new process and approvals to try and prevent escaped defects
● Create a new “quality” function which sits downstream of development
● Underinvestment in infrastructure and tools
@jezhumble
@maryamumar

7. Create a Plan
But where do you start?*
*Leading Quality - Ronald Cummings-John, Owais Peer

8. Vision Statement
As an <engineering team/customer>
When <there are any changes to production code>
Then <every merge to the codebase is releasable>
And <there are no known regressions in the system>
@maryamumar

9. Assuring Quality enables teams to drive
for customer satisfaction at a sustainable
pace.

10. Who are your Stakeholders?
● CEO/CTO
● Tech leads
● Testing team
● Incident Management
● Release Management
Questions to ask:
● How do you know quality was bad?
● Escaped defects?
● What’s your code coverage like?
● Do you have the right processes in place?
@maryamumar

11. Software Delivery Performance
@jezhumble https://cloud.google.com/devops/state-of-devops/

12. How to get better
https://bit.ly/dora-bfd@jezhumble

13. Areas to Improve
Releases Product Quality Process Quality
@maryamumar

14. Releases
- Create visibility landscape for releases
- Monitor issues found post-release vs pre-release
- How good is your regression cycle?
- Release notes
- Changes
- Known issues; when to expect fixes
- How easy is it to create deployables?
- How quickly can you deploy?
@jezhumble
@maryamumar

16. Product Quality
@jezhumble
@maryamumar
MTTG Build FailuresBuild Time Deploy Time
Average Time
in Status
PR Review PR commit rate
per Sprint
%age of Flaky
Tests
Ticket
Fluctuation
from QA to Dev
Test Coverage

17. Product Quality (contd.)
@jezhumble
@maryamumar
Defects found
in Sprint vs
Escaped
Defects
Team FeedbackDefects found
via Automation
via Exploratory
testing per
Feature
Bug resolution
time per
Severity level

20. Time from check-in to deployed in test environment

21. SATs vs Defects per Team

22. High trust culture
Westrum, “A Typology of Organizational Cultures” | http://bmj.co/1BRGh5q
how organizations process information

23. Psychological safety
“Wherever there is fear, you will get wrong figures”
— W Edwards Deming
@jezhumble
@maryamumar

24. https://rework.withgoogle.com/blog/five-keys-to-a-successful-google-team/
Culture of psychological safety
● Predicts software delivery performance
and organizational performance
● Implement by adopting continuous
delivery and lean product management
practices
@jezhumble

25. 2018 State of DevOps Report: cloud.google.com/devops
SDO Performance
Trust
Software
Delivery
Performance
Availability
Organizational
Performance
Voice
Westrum organizational culture
Autonomy
Retrospectives
Climate for Learning
@jezhumble

26. How to make metrics safe
● Teams involved in creating vision
● Consult with teams when deciding what to measure
● Testing team makes quality visible
● Let teams set their own targets
● Regular retrospectives on what is being measured
@jezhumble

27. Conclusion
● Quality is subjective - what to focus on depends on many factors
● Start by creating a vision, selecting some metrics, and measuring
● What you measure will evolve, and will determine what teams work on
© 2020 Google LLC & Thought Machine Group Ltd

28. Parking lot

29. Continuous testing
● Continuously reviewing and improving test suites to better find defects and keep complexity and cost
under control
● Allowing testers to work alongside developers throughout the software development and delivery
process
● Performing manual test activities such as exploratory testing, usability testing, and acceptance testing
throughout the delivery process
● Having developers practice test-driven development by writing unit tests before writing production
code for all changes to the codebase
● Being able to get feedback from automated tests in less than ten minutes both on local workstations
and from a CI server
2018 State of DevOps Report | cloud.google.com/devops

30. Continuous delivery
The ability to get changes—features,
configuration changes, bug fixes,
experiments—into production or into the hands of
users safely and quickly in a sustainable way.
2018 State of DevOps Report | cloud.google.com/devops

31. Technical debt quadrant
https://martinfowler.com/bliki/TechnicalDebtQuadrant.html