This presentation explains how testing activities constitute the main bottleneck to flow in most continuous delivery pipelines. Continuous Testing strategies are designed to reduce testing bottlenecks, and accelerate time-to-quality. A blueprint is presented for Continuous Testing. Specific strategies are presented including Continuous Test Tenets, Leadership and Culture practices, Test strategies and Plans, Test Management, Test Automation, Test Tools, Test Environment Management and Test Results Analysis. A Continuous Testing Assessment approach is described to help assess current state of of continuous testing. A phased implementation approach is explained.

1. The Quest
for
Quality at Speed
Continuous Testing strategies overcome bottlenecks.
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.

2. Marc Hornbeek
a.k.a. DevOps_the_Gray esq.
CEO and Principal Consultant
Engineering DevOps Consulting
Author – Engineering DevOps
mhornbeek@engineeringdevops.com

3. Testing is THE major bottleneck for most DevOps value streams.
Testing activities span the end-to-end value stream and often account for >50% of bottlenecks.
Source of Bottlenecks
Culture
Test Planning
Test Management
Test Automation
Test Tools
Test Environments
Test Results
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.

4. Test
Management
Test
Tools
Test
Automation
Leadership
and Culture
Continuous Testing Blueprint
Test Strategies and
Plans
Continuous Test is a quality assessment strategy in which most tests are automated and integrated as a
core and essential part of a DevOps value stream.
Continuous testing is much more than simply “automating tests”.
Test Results
Analysis
Test Environment
Management
Continuous Testing
Tenets
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.

5. Continuous Testing Tenets
Be Relevant: Focus on the
most important tests and
results.
Shift Left: Conduct each test as early
in the pipeline as possible.
Fail Early: Arrange the tests so
that the most likely problems
are found early.
Fail Often: Run tests frequently and
with many different conditions.
Test Fast: Arrange tests
to run in quick cycles.
Continuous
Testing Tenets
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
These five Continuous Testing tenets guide practices for accelerating DevOps testing
and quality assurance and reduce bottlenecks caused by testing.

6. Continuous Testing – Leadership and Culture
Bottlenecks to CT
• Inadequate funding of
testing due to lack of
quality vision and goals
at executive level.
• Bottlenecks caused by
distrust and lack of co-
operations between
Dev, QA, & Ops.
CT Strategies
• Executives
communicate vision and
goals for quality.
• Incentives for
collaborative behavior.
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
Leadership and culture practices have a major impact on influencing behaviors and
habits of everyone involved in testing across the DevOps value stream.
Executive
Definition of
Quality.
Importance of
Quality.

7. Continuous Testing – Test Strategies and Plans
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
Bottlenecks to CT
• Confusion cause by lack
of product/service test
strategies.
• Unplanned testing
activities and resources
not included sufficiently
in planning process.
CT Strategies
• Coverage defined.
• Types of tests.
• Tests required at input
gates.
• Test results required at
output gates.
• Test tasks in planning
backlog and sprints.
Test strategies and plans are needed to guide testing approaches for each
product/service pipeline.
Coverage standard.
Test case standard.
Gate criterion.
Planning process.

8. Continuous Testing – Test Management
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
Bottlenecks to CT
• Poor utilization of test
resources due to lack of
visibility
• Schedule bottlenecks
due to lack of co-
ordination of test
schedules
CT Strategies
• Version management of
tests and resources.
• Inventory of test
resources.
• Dynamic test
scheduling.
Test management is needed to co-ordinate testing resources and test schedules.
Database of tests
and test
resources.
Test scheduling
tools.

9. Continuous Testing – Test Automation
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
Bottlenecks to CT
• Slow test creation.
• Slow test execution.
• Slow time to verdicts.
CT Strategies
• Test design templates
• Advanced test creation
tools – TDD, keyword,
behavior, model-based.
• Dynamic test selection.
• Design tests for fast
verdicts.
• Horizontal and vertical
scaling tests.
Designing, creating, selecting and executing tests in an automated fashion to get to
a verdict as fast as possible is the primary strategy to accelerate testing.
Test design.
Test creation.
Test selection.
Test execution.

10. Continuous Testing – Test Tools
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
Bottlenecks to CT
• Test tools not designed to
scale limit test
configurations.
• Lack of complete APIs
make integration with
DevOps platforms
inefficient and error
prone.
CT Strategies
• Limit selection of test
tools to ones that are
DevOps-ready or
DevOps-native and that
have well-defined stable
APIs.
Test tools are designed to automate specific types of tests and integrate with DevOps
platforms through APIs for orchestration, execution & reporting.
Scalable
test tools with
good APIs.

11. Continuous Testing – Test Environment Management
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
Bottlenecks to CT
• Lack of test resource
scheduling causes
conflicts for resources.
• Manual test setup is time
consuming causing long
times to wait for testing.
• Delayed retesting.
CT Strategies
• Test environment
inventory tools.
• Test environment
sharing tools.
• Test environment
orchestration.
• Monitor test
environment utilization.
Scheduling and setup of resources into topologies needed for running different
types of tests for developers and testers can be a major bottleneck if not managed
efficiently.
Test resources
scheduling.
Orchestration.
Utilization
management.

12. Continuous Testing – Test Results
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
Bottlenecks to CT
• Poorly written tests can
report confusing results
that take a lot time to
analyze.
CT Strategies
• Test verdict design in
test cases.
• Common test reporting
tools.
• Test analysis tools.
Reporting and analyzing the results of tests can be a major bottleneck after test
execution is completed.
Test results design
standards.
Test results
reporting
standards.

13. Continuous Testing – Gap Assessment
eLearning Course - Fundamentals of SRE Copyright © 2021 by Marc Hornbeek All rights reserved.
A Gap Assessment is a great way to determine the current state of CT practices
which will help focus the implementation plan.

14. Start with one or two services, and a small set of critical SRE practices, then expand to
more services and more advanced CT practices.
Steps to Continuous Testing
Beginner CT
• Strategy sponsor.
• Define Quality goals.
• Inventory test
resources and
requirements.
• Version management
for tests.
• Test standards.
• Tool selection.
• Refactor tests.
Intermediate CT
• Tool integrations
• Test scheduling
• Orchestrate test
Environments
• Test result reporting
optimizations
Advanced CT
• Advanced test creation
strategies
• Accelerated test
execution
• Test Environment
Management
• Test results analysis
optimizations

15. Marc Hornbeek
a.k.a. DevOps_the_Gray esq.
CEO and Principal Consultant
Engineering DevOps Consulting
Author – Engineering DevOps
mhornbeek@engineeringdevops.com
Learn More