This document discusses the importance of defining a quality strategy in order to balance quality and velocity. It provides examples of key elements to include in a quality strategy such as understanding requirements, system models, quality activities, tools, release criteria and staffing. Quality strategies should consider both functional and non-functional quality. The document advocates planning for quality from the start of a project and aligning the quality strategy with the delivery strategy. It argues that having a defined quality strategy can help organizations effectively respond to issues and meet business and customer expectations.

1. Defining your Quality
Strategy
Managing Risk on the road to Continuous Delivery

2. Some of the Fun places I’ve
worked
Sr. Director Engineering Director of Engineering
Corporate Quality Leader
Management Consultant
Executive Project Manager VP Engineering
Head of Engineering
Software Engineer
Division Chief Technology Officer
Most of my experience is with very large scale

3. Think for a minute about why
agile has become so popular
• Incremental Delivery reduces risk
• Lets the business quickly test new ideas
• Rapid response to changed environments
• Agile is how we achieve those outcomes
• The business could care less about how we
execute as long as they get what they want
Business only cares about Velocity and Quality

4. CI and CD are powerful tools to
achieve velocity and quality
Local
Inspection
Inspection
SONAR
Quality
Portal
Source Code
Repository
Build Unit Tests
Integration Test
Environment
Automated
Integration Tests
Staging
Violations
Project Specific Rules
Inspection Results
Coverage Data
&
Test Results
ready4customers
Next.ready4Customers
Coverage Data & Results
Developers
Development
Manager
Code Reviews Features that are stillunder
development will be here
for Product Managers and
Owners to see
Development managers
specify therules they want
enforced
The Local Inspection gives
the engineer a list of
violations prior to checkin
Dev manager only
notified of
violations engineer
chose to ignore
Allenvironments will be
deployed to using the same
tool
This is where all our “done
Done” features are deployed
Sonar will capture code and test
quality metrics throughout the
process
Powerful tools have to be used carefully though or they
can do more harm than good

5. It takes a lot more than just CI
to get Velocity and Quality right
• Understanding the business expectations
• Understanding the risks you are willing to take
• Planning to meet the expectations
• Execution
• Continuous Learning and Improvement
Most importantly we have to always remember
Velocity AND Quality not Velocity OR Quality

6. By defining a
Quality
Strategy we
create the
foundation for
balancing
Quality and
Velocity

7. Some context about Quality
Strategies
Quality Strategies ≠ Test Strategies
If your Quality Strategy is testing
you don’t have one!
Most teams focus on functional
when they think quality
Non-Functional is equally important

8. Imagine this nightmare
scenario

9. Imagine this nightmare
scenario
20,000 units
4PM Pacific Time
On a Sunday
2 weeks before
Christmas
3 days to get ready
We only found out because someone saw the cnet post

10. Imagine this nightmare
scenario
20,000 units
4PM Pacific Time
On a Sunday
2 weeks before
Christmas
3 days to get ready
We only found out because someone saw the cnet post

12. There was no panic because we
knew what we had to do and how
• We knew what our risks were
• We knew our system limits and capacity
• We knew what we needed to validate
• We knew how we needed to validate
• We had the right tools in place

13. We had Quality Strategy defined and
were prepared for almost anything
We were prepared because we thought about
things in advance, knew what was important
and had a strategy in place to allow us to make
emergency changes with very little risk

14. So what is a Quality Strategy?
A Quality Strategy describes how you are
going to organize and execute to ensure your
team delivers the quality your customers
deserve with the velocity your business
demands and does so as efficiently as
possible
Engineering owns the creation but stakeholder
agreement is essential to managing expectations

15. When I say Quality I am by no
means saying Perfection!
Part of your Quality strategy is defining the right level
Cost of Quality
Cost of Poor Quality

16. A deep understanding of your system
is needed to define your strategy
• The logical or functional view
• The physical view
• Non-functional requirements
• System KPIs

17. Most of you have seen the
ebay SRP but did you know
• Above the fold rendering has
a massive impact on clicks
• Clicks = $$$$$
• Above the fold rendering is a
major KPI

18. Ideally you define your Quality
Strategy at the start of your project
• We are going to declare
• Who does what where
• What tools you are going to use
• Expectations at each stage
• Show Stoppers
Construction Validation Operation

19. In defining your Quality Strategy
you will need to consider
• Key User Scenarios
• Quality Activities
• Tools you will use
• Release Criteria
• Staffing models
• Live Site Monitoring

20. Keep in mind that a Quality
Strategy is definitely not a test plan
• Probably drives multiple test plans
• Functional and non-functional aspects
• Aligns with and informs DoD
• Must align with delivery strategy
• Is it Continuous?
• Is it Automatic
• MOST functional testing ignores QoS
Sometimes your quality strategy might be to not have
any testers at all!

21. The first thing you have to do
is understand your system
• What are the KPIs
• What is going to make users Love it ?
• What is going to make the business Love You ?
• What is the system’ competitive advantage ?
• Functional and non-functional requirements

22. The next step is to document
the logical flow of your system
• Functional Context Diagram
• Ecosystem Boundaries
• Next Neighbor Interfaces
It can be very helpful to actually conduct an FMEA as
part of developing your quality Strategy

23. The Physical Model is the next
thing you need to consider
• High Availability
• Disaster Recovery
• Scalability
• Monitoring
• Maintainability
The physical looks at how you system is deployed
• VIPS
• Firewalls
• Multi Colo

24. Quality of Service in production is
essential in any Quality Strategy
• External Failures cost more than internal ones
• You actually lose revenue
• And Reputation
• Performance failures take longer to diagnose
and fix than functional failures
• They can also be really hard to detect
• Time To Mitigate incidents is a quality KPI

25. For each of your stages you want
to define tools and objectives
Construction Validation Operation
What activities are you going to do ?
What tools are you going to use ?
What are your release criteria ?
Its ok to say there are different stages or no
criteria but you want to agree to all that up front

26. Some actual artifacts from a
Quality Strategy
How we were planning to test How we would report our results

27. The most interesting Quality
Strategy I ever learned of was

28. In Case anyone was wondering
> 6 million users
at 4PM Pacific Time
On that Sunday
2 weeks before
Christmas
had a few failures
And… we learned a lot
over in 15 minutes

29. A Funny story about executing
on our quality strategy at ebay
Not just 1
> 2,000,000

30. A Quality Strategy won’t solve
all problems but prevents a lot

31. If you have any questions or want
to discuss this in more detail
email: john.belbute@gmail.com
Twitter: @jbelbute
Linkedin: John Belbute
mobile: +351 919 014 911