The document outlines the essential components and benefits of continuous testing within the realms of quality engineering, DevOps, DevSecOps, and SRE. It presents a maturity model for continuous testing practices, emphasizing the need for automated testing and collaboration across teams to enhance software delivery and minimize bottlenecks. Key takeaways include the significant ROI from test automation and the role of continuous testing as fundamental to achieving quality in software development processes.

1. Continuous Testing
How to define, assess and create a roadmap for Continuous
Testing for Quality Engineering, DevOps, DevSecOps and SRE.
Continuous Testing Copyright © 2023 by Marc Hornbeek All rights reserved.

2. Marc Hornbeek
DevOps_the_Gray
DevOps Institute Ambassador
CEO Engineering DevOps Consulting
www.engineeringdevops.com
mhornbeek@engineeringdevops.com
- Essential for Continuous Quality Engineering
Continuous Testing

3. Agenda:
• What is Continuous Testing
• Why Continuous Testing is Important
• Continuous Testing Blueprint and Maturity
• Pillars of Practice for Continuous Testing
• Transformation Roadmap
• Strategic Considerations
• CT Benefits
• Key Take-ways

4. • Testing is about verifying and finding defects.
• Quality Engineering is about process,
preventing defects, satisfying customers and
stakeholders.
Continuous Quality Engineering includes
Continuous Testing.
Continuous Testing
in relation to
Continuous Quality Engineering
Dr. Edwards Deming emphasized
systems thinking, continuous
improvement, statistical control,
and other principles that are
foundational to the concept of
quality engineering.
"Quality is everyone's responsibility."

5. Test
Management
Test
Tools
Test
Automation
Leadership
and Culture
Continuous Testing Blueprint
Test Strategies and
Plans
Continuous Test is a quality assessment strategy in which most tests are automated and integrated as a
core and essential part of a DevOps value stream.
CT is much more than simply “automating tests”. There are nine pillars of practices.
Test Results
Analysis
Test Environment
Management
Continuous Testing
Tenets
Continuous Testing Blueprint
Artifacts
Development (Dev)
Artifacts Artifacts Artifacts Artifacts
Integration (CI) Delivery (CD) Deployment (Deploy) Production (Prod)
Planning (Plan)
Test Data
Management

6. Testing is THE major bottleneck for most value streams.
Testing activities often account for >50% of bottlenecks.
Test
Management
Leadership
and Culture
Test Strategies and
Plans
Continuous Testing
Tenets
Test Data
Management
Test
Tools
Test
Automation
Test Results
Analysis
Test Environment
Management
Contract
Tests
Conformance
Tests
Regulatory
Tests
Compliance
Tests
Chaos
Tests
Exploratory
Tests
End-to-End
Tests
Usability
Tests
Static Code
Tests
Dynamic Code
Tests
Interface
Tests
Compatibility
Tests
Configuration
Tests
Data
Tests
Integration
Tests
Regression
Tests
Acceptance
Tests
Deployment
Tests
Penetration
Tests
Functional
Tests
System
Tests
Administration
Tests
Backup Restore
Tests
White-Box
Tests
Gray-Box
Tests
Black-Box
Tests
API
Tests
Container
Tests
Packaging
Tests
Capacity
Tests
Latency
Tests
Access Security
Tests
Canary
Tests
Build
Tests
Artifact
Tests
Release Acceptance
Tests
A/B
Tests
Feature Flags
Tests
Unit
Tests
Protocol
Tests
Usability
Tests
Reliability
Tests
Security
Tests
Support Materials
Tests
Documentation
Tests
Production Readiness
Tests
Database
Tests

7. Scope of Continuous Testing
SRE
Agile, DevOps, DevSecOps
Requirement
Production
Plan Develop CI/CD
In-Prod Ops
& SecOps
CT scope includes testing practices for end-to-end value streams including
planning, development, delivery and production operations.
It is an essential part of Continuous Quality Engineering
Pre-Prod Continuous CT
Deploy
In-Production Continuous CT

8. CT is important for DevOps, DevSecOps and SRE
CT Systems and Practices Optimizations
People: Promote collaboration and monitoring for tests amongst teams.
Process: Efficiently orchestrate and automate appropriate level of test coverage.
Technologies: Platforms and tool-chains architected to enable sharing test artifacts.
DevOps and DevSecOps SRE
Accelerate continuous delivery
and deployment safely and
securely.
Assure reliability, quality
and security of services
in production, at scale.
Testability = Controllability + Observability

9. Continuous Testing Failure Cases !
Plan
Manual tests
cause CI/CD
delays
Develop CI/CD Operate
Insufficient Dev Testing
causes high CI/CD failures
and rejects.
Deploy
Too slow to
manually validate
production deploy
decisions.
Unplanned quality
cause
delays, cost overruns
and production
problems.
Too slow to verify
restores and
recovery of
production issues.

10. State of Software Testing
Quality requirements
left out of plan cause
delays, cost overruns
and production
problems.
79% positive ROI in
the first year of
adopting test
automation.
Challenges

11. Benefits of Continuous Testing

12. Level 1: Unknown
Level 2: Initial
Continuous Testing Maturity Model*
Unmeasured
implementations,
different tools and
frameworks used
across different teams
and applications
Level 3: Automated
Level 4: Advanced
Level 5: Elite
Initial CICD pipeline
CICD tasks are partially
automated, and
metrics are reported.
Automated CICD
pipeline.
Most CICD tasks are
automated up to the
point of release
acceptance. Release
management decisions
use CICD metrics.
Continuous Delivery
CICD tasks are fully
automated up to the
point of release
acceptance. Metrics are
automated sufficiently
to automatically
approve a release.
Continuous
Deployment
CICD Tasks and metrics
are fully automated
sufficiently to
automatically deploy to
production.
Culture of continuous
improvement.
Metrics: Not reported
• Lead Time: ?
• Release Frequency: ?
• Failure Rate: ?
• Time to Restore: ?
DORA Metrics:
• Lead Time: < six months
• Frequency: > monthly
• Failure Rate: < 45%
• Time to Restore: < 1 month
DORA Metrics:
• Lead Time: < 1 month
• Frequency: weekly
• Failure Rate: < 30%
• Time to Restore: < 1 week
DORA Metrics:
• Lead Time: < 1 week
• Frequency: daily
• Failure Rate: < 15%
• Time to Restore: < 1 day
DORA Metrics:
• Lead Time: < 1 day
• Frequency: hourly
• Failure Rate: < 15%
• Time to Restore : < 1 hour
* Defined for each organization

13. Build Test Metrics - # of automated tests prioritized by risk, successful code
builds, Unit test verdicts, # defects and code coverage.
Continuous
Testing
Metrics
Improved understanding
of business risk and user
experience
Functional Verification - Requirements covered, Critical defects, Pass/Fail Rates,
Defect density and Risk Coverage
Integration Testing - Requirements covered, New Defects, Defect Density,
Pass/Fail Rates, and Code/Risk Coverage.
End-to-End Regression - % automated tests, Requirements Covered,
Total # defects, # test executed and Test Case Coverage

14. Vulnerabilities – depends on measures of impact of vulnerability exploitations
and vulnerability exploitation probabilities.
DevSecOps
Security Metrics
DevSecOps metrics
provide insights into the
current state of
application security and
indicate continuous
improvement of software
security posture over
time.
Software Asset Security – Measure of DevSecOps controls such as scans over
applications categorized as Mission Critical, Business Critical, Business
Operational and Office Productivity.
Codebase Security – Volume of code changes
Software Security Risk – security technical debt includes unresolved
vulnerabilities deployed to production categorized by severity and
mean vulnerability age and security risk density per code base.
Application Security Risk – based on application size, business importance,
potential impact and probability of exploitations.

15. Continuous Testing Practices Self-Assessment
Continuous Testing
Maturity Level
Selection
(choose
one)
People
Practices
Selection
(choose
one)
Process
Practices
Selection
(choose
one)
Technology
Practices
Level 1
Unmeasured
Silo team organization, Little knowledge
of continuous testing, Blame, finger-
pointing
CT metrics not reported. Testing not part
of planning, No test standards, Few
automated tests
No CI/CD pipeline. Missing tools to
test performance of applications,
pipelines, and infrastructures
Level 2
Initial Pipeline
Limited knowledge of Continuous
Testing, Ad hoc training, some Dev/QA
co-ordination
1
CT Maturity levels and metrics defined.
Some CT metrics and tasks are
automated. Most tests other than build
tests are manual, minimal test version
management
Initial CI/CD pipeline in place.
Version Management, Automated
build tests, Painful but repeatable
releases
Level 3
Automated Pipeline
1
CT skills and training program, risk
management, Dev/QA joint team
CT metrics are reported. E2E CI/CD
pipeline, tests visible, test/release
standards, test management. Release
Management decisions use QE metrics.
1
CICD pipeline complete. Most test
execution tasks automated for app,
infra and pipeline.
Level 4
Advanced Continuous
Delivery
Collaboration across multifunctinal
teams using shared CT metrics, Agreed
goals are in place : SLOs, Mentors, and
Guilds
CT metrics and automated and used for
predictive test analytics.Sufficient to
automate release approvals. E2E
performance trends drive test design.
Focus on removing test bottlenecks.
CT test test environment
orchestration and test execution
tasks are mostly automated.
Level 5
Elite Continuous
Deployment
Continuous experimentation, Learning
from Failure, Integrated Dev/QA. E2E
user experience focus
Metrics and analysis are automated
sufficiently to automtically approve a
release. Risk based test design.
CT tasks including test creation,
environment orchestration, test
execution and results analysis are
highly automated E2E. Deploy to
production automated.
Practices Scores 3 People Practices Score (Calculated) 2 Process Practices Score (Calculated) 3 Tech Practices Score (Calculated)
2
Instructions: for each of the three Practices columns enter a "1" in the row with the description that best matches your environment.
Overall Continuous Testing
Maturity Score (Calculated)
www.engineeringdevops.com

16. CT Roadmap from Level 1 to Level 2
• Value Stream Map current and possible future states.
• Define CT Maturity levels and metrics
• Select a platform and initial tools for CT orchestration and
automation.
• Inventory infrastructure and data provisioning
requirements.
• Implement initial CI/CD pipeline.
• Define and Implement initial metrics and monitoring
including lead time.
• Train team on use of platform, tools and metrics.
• Include CT requirements in Backlog.
• Cross-functional communications, Some CT knowledge
sharing, Ad hoc CT training.

17. CT Roadmap Level 2 to Level 3
• Value Stream Map current and possible future states.
• Complete the CI/CD toolchain including quality and
security scanners and test tools.
• Automate test infrastructure and data provisioning.
• Implement sufficient CT and security metrics needed for
release decisions.
• Leadership sponsors DevOps and CT .
• Collaboration between cross-functional DevOps, Security
and SRE teams.
• DevOps, DevSecOps and SRE training program.

18. CT Roadmap Level 3 to Level 4
• Value Stream Map current and possible future states.
• E2E CI/CD pipeline fully automated end-to-end to the point
of delivery but not deployment, a sufficient number of
quality and security scans and tests are automated.
• Results are used to automatically determine if a release
meets CT, Security and SRE requirements for deployment.
• Culture of Collaboration and Learning from failures, shared
DevOps, Security and SRE metrics focus on removing
bottlenecks, SLIs, SLOs, and SLAs, DevOps Mentors and
Guilds.

19. CT Roadmap Level 4 to Level 5
• Value Stream Map current and possible future
states.
• E2E CI/CD pipeline fully automated including
automated deployment, most quality and security
scans and tests are automated.
• CT Results are used to automatically deploy an
approved release to production.
• Culture of continuous experimentation and learning
from failure exists for DevOps, Security and SRE.

20. AI/ML Game-Changer for CT
“Smart Quality Technologies”
“From Test Strategy to Test Results reduced to hours instead of days.”

21. Value Stream Management
Value Stream Management platforms provide
enterprise-scale QA integration, orchestration,
automation and analysis over multiple value
streams.
• Unified data model integrates APIs.
• Unified maintenance practices
• Unified orchestration and automation
• Unified resource management
• Unified analysis and dashboard

22. Continuous Testing projects
realize
‘instant” ROI.
$

23. CT/CQE Benefits
Accelerating Software
Delivery and Innovation
ROI, Competitiveness,
Survival

24. Take-Aways
• Continuous Testing, as an integral part of Continuous Quality
Engineering, is foundational for DevOps, DevSecOps and SRE success.
CT projects realize “instant” ROI.
• CT Capability maturity levels and Self-Assessment tools help you build
your own CT implementation roadmap to improve capabilities.
• CT strategies should consider use of AI tools and VSM platforms to
further improve quality, speed and scale of continuous deliveries and
production operations.
• Make Continuous Testing and Continuous Quality Engineering part of
your Continuous Learning program.

25. Marc Hornbeek
DevOps_the_Gray esq.
DevOps Institute Ambassador
CEO Engineering DevOps Consulting
www.engineeringdevops.com
mhornbeek@engineeringdevops.com
- Essential for Continuous Quality Engineering
Continuous Testing