The document discusses DevOps test engineering, emphasizing its importance in ensuring continuous testing and quality across the software development pipeline. It outlines various strategies, best practices, and benefits of DevOps testing, such as automation and integrated test environments. Additionally, it explores different testing methodologies, including canary and blue-green testing, along with the potential consequences of inadequate testing practices.

1. DevOps Test Engineering
Marc Hornbeek
1
http://devopsinstitute.com

2. Marc Hornbeek DevOps_The_Gray
Marc Hornbeek “DevOps_The_Gray”
2
https://www.linkedin.com/in/marchornbeek
Trace3.com DevOpsInstitute.com DevOps.com www.ieee.org
Principal Consultant
DevOps
Course
Author
IEEE Region 6
Engineer of 2016
Blogger
https://devops.com/author/marc-hornbeek/

3. Marc Hornbeek DevOps_The_Gray
3
• What is DevOps test engineering?
• Why is DevOps test engineering important?
• DevOps test engineering strategies and best practices
• Benefits of DevOps testing engineering
• References to learn more

4. Marc Hornbeek DevOps_The_Gray
“Culture is the door to DevOps…
…but ...
Continuous Testing is the primary key...
to Continuous Delivery ”

5. Marc Hornbeek DevOps_The_Gray
Test Engineering is the “magical”
ingredient that enables DevOps
Test verdict data enables assessment for
artifact stage promotions
Best practices are the secret to success

6. Marc Hornbeek DevOps_The_Gray
Testing maturity is a key
differentiator of DevOps
maturity.
DevOps testing is different
than traditional QA !

7. Marc Hornbeek DevOps_The_Gray
Testing scope is
end-to-end,
and continuous
across the
pipeline
DevOps testing
is not a phase
between
integration and
delivery

8. Marc Hornbeek DevOps_The_Gray
DevOps Testing Differentiators
• Continuous quality assessment for each pipeline stage
• Most tests are automated from the beginning
• Tests are integrated within all stages of the DevOps toolchain
• A production equivalent test environment is used for each stage
Pre-Flight CI CD
DevOps Testing

9. Marc Hornbeek DevOps_The_Gray
Assessments performed “continuously” on incremental changes
determine if artifacts from each stage are accepted or rejected.

10. Marc Hornbeek DevOps_The_Gray
Rejected Artifacts
Changed artifacts are “rejected” according to test acceptance criterion
Rejected changes are immediately analyzed and
remediation is scheduled
Serious changes: halt the pipeline until the rejected
change is either removed (called a roll-back) or
remedied (i.e. resolved)
Less serious changes: promote to the next stage and remedy later

11. Marc Hornbeek DevOps_The_Gray
Pre-Flight Testing
Pre-Flight testing refers to assessment of design
and software code prior to the trunk integration
stage
Consequences if not done correctly:
• Errors integrated into the DevOps pipeline have
negative “ripple” effects on the later DevOps
pipeline stages
• Interruptions with urgent corrective work
required to remedy problems
Integrate to main
trunk branch
Test
Test
Test

12. Marc Hornbeek DevOps_The_Gray
Production Equivalent Test Environment
The test environment refers to
combinations of use cases, environment,
systems and configurations in which tests
are performed.
DevOps tests, including Pre-Flight tests,
should be performed in a test
environment that is equivalent to the
production environment of customer
deployments.
Use Cases
(types of
users and
workflows)
Environment
(external
conditions)
Systems
(Operating
systems and
versions)
Configuration
(system
topologies,
data)

13. Marc Hornbeek DevOps_The_Gray
Culture is the #1 critical success factor
• “Quality is everyone’s
responsibility.”
• W. Edwards Deming Ph.D.
• In 2009 Patrick Dubois, a Quality
Assurance (QA) consultant, conceived
DevOps as a way to bring Dev and
Ops teams closer together
• Roles for individuals are not
prescribed
• Cross-functional team owns testing
• Team agrees who will perform test
tasks

14. Marc Hornbeek DevOps_The_Gray
DevOps Testing Strategy
• Testing is integrated into the toolchain
• Test environment is orchestrated
• Tests are automated as much as
possible
• Accelerate tests as much as possible
Integrat
e
Orchestrat
e
Automate Accelerate

15. Marc Hornbeek DevOps_The_Gray
Five Tenets of DevOps Testing
• Shift Left: Conduct tests as early as
possible
• Fail Early: Test for most likely problems
first
• Fail Often: Test frequently with different
conditions
• Test Fast: Arrange tests to run in quick
cycles
Test
fast
DevOps
Test
Tenets

16. Marc Hornbeek DevOps_The_Gray
Integrated DevOps test infrastructures
DevOps testing is fully integrated to
support relevant and timely
assessments for each DevOps stage
For each DevOps stage tests are
automatically triggered from the
DevOps toolchain
DevOps test assessment for each
stage provides feedback to support
DevOps workflow decisions

17. Marc Hornbeek DevOps_The_Gray
Test Environment Orchestration
and Test Automation
Environment orchestration:
Automatically setup the test
environment and resources
(physical and virtual) to match the
requirements of a test, and
release resources after tests are
done.
Test automation: Execute test
tasks without manual work
required

18. Marc Hornbeek DevOps_The_Gray
A/B Test Strategy
Some test cases such as
usability testing require a
human user to assess a
change.
Feature toggles allow
changes to be selectively
exposed to predetermined
“trial” customers, thereby
enabling controlled
customer evaluations of
features and alternative
designs.

19. Marc Hornbeek DevOps_The_Gray
Blue - Green Test Strategy
• Two identical production
environments
• At any given time, one of them is
live
• If blue is live, final stage testing
should be done in the green
environment (or vice versa)
• When the software works in the
green environment, switch the
router so that all incoming requests
go to the green environment. The
blue environment is now idle
• Blue-green deployment provides a
rapid way to rollback the router to
Taking software from the
final stage of testing to live
production needs to be done
quickly to minimize downtime
Reference:
http://martinfowler.com/bliki/BlueGreenDeployment.html

20. Marc Hornbeek DevOps_The_Gray
Canary Test Strategy
• A canary test is a push
of code changes to a small
number of unsuspecting end
users
• The goal is to ensure
changes are transparent and
work in the real environment
• Canary tests offer limited
production level testing and
early warning of possible
errors
The term “canary” refers to an old practice that used
canaries to alert miners to toxic gas levels.
Reference: http://whatis.techtarget.com/definition/canary-canary

21. Marc Hornbeek DevOps_The_Gray
DevSecOps
• Develop expertise in
DevSecOps testing
• Include security experts in
DevOps testing
• Partner with security experts to
train others
• Use security-ready test tools
which support automation
• Static code analysis during pre-
flight and continuous
integration builds
• Penetration testing during
integration and functional
Integrate security
testing throughout the
DevOps pipeline

22. Marc Hornbeek DevOps_The_Gray
Microservices Test Strategy
What to test:
• APIs, Contracts
• Dependencies
• Cascading failures
• Changes that affect the
allocation of
responsibilities between
microservices
• Backwards and forwards
compatibility policies for
microservices

23. Marc Hornbeek DevOps_The_Gray
Test Container
• Does not contain application code
• Contains testing tools, testing tool runtime images, test
environment configuration, environment variables, config files,
bootstrap scripts, tests, test data and a test startup script
• Works in all environments: pre-flight/Dev, integration, staging
and production
• Deployed as micros-services
• Reduces footprint and dependencies between microservices
• Can always go back in time and rerun a specific version

24. Marc Hornbeek DevOps_The_Gray
Test Acceleration Strategy
• Organize the team for speed
• Select powerful resources (Vertical scaling)
• Run more tests in parallel (Horizontal scaling)
• Choose test tools designed for fast execution
• Organize test schedules and test suites to fail
early
• Design test cases to fail early
• Arrange test results to minimize wait times
https://devops.com/continuous-
testing-accelerated/
Testing needs to keep up
with the ever increasing
demand for speed.

25. Marc Hornbeek DevOps_The_Gray
Test Results Analysis Strategy
• Determine clear responsibilities and
SLAs
• Design test scripts for efficient results
analysis
• Choose tools designed for fast results
analysis
• Configure tools to process results as
available
• Configure workflows for optimum
analysis
https://devops.com/test-
results-analysis-speed-devops/
Ensure test results analysis keeps pace
with accelerated continuous testing

26. Marc Hornbeek DevOps_The_Gray
Test Coverage Strategy
Strategies:
• Requirements coverage: % of requirements
• Test plan coverage*: % of planned test cases
• Code coverage: % of code lines or paths
• Function coverage: % of functions
• Artifacts coverage: % of build artifacts
• Code module coverage: % of source code
modules
• Dependencies coverage: % of dependencies
% of tests run and passed,
relative to a coverage
strategy indicates
readiness for release.
*Test plan coverage is the most popular,
flexible and simplest to measure.

27. Marc Hornbeek DevOps_The_Gray
DevOps-Ready Test Tools
• RESTful APIs (configure, control,
report)
• Framework plug-ins (e.g. Jenkins)
• Virtual / cloud compatible
• Dynamically scalable
• Test automation features
Easy to integrate into a DevOps
testing framework and
dynamically scale using cloud
computing techniques and
automate tests.
Warning!

28. Marc Hornbeek DevOps_The_Gray
DevOps Test Planning Best Practices
Yes, test plans are needed for
DevOps
• Separate test plans need not be
written for every change
• Test plans provide overall guidance
for product test strategies and
policies
• Define the strategy/contract for
measuring test coverage

29. Marc Hornbeek DevOps_The_Gray
DevOps Pipeline Test Strategy

30. Marc Hornbeek DevOps_The_Gray
DevOps Regression Test Best Practices
• Regression tests are identified during test
planning
• Test selection is dynamic. Only tests that
are relevant to the changes are selected
• Test reporting is dynamic. Only test
results that are relevant to a stakeholder
are reported to the stakeholder
• New or updated regression tests are
determined for each customer found
defect test escape
• Tests are maintained in a version
management system
• Regression tests that report false failures

31. Marc Hornbeek DevOps_The_Gray
Test Automation
Creation Strategies
There are a variety of
methods for creating
automated tests:
• Scripting automated tests
• Keyword-driven tests
• Interactive Capture/replay
tests
• Test driven development
(TDD)
• Model-based tests
• Behavior-based tests

32. Marc Hornbeek DevOps_The_Gray
Test Management Database (TMDB)
Best configured as a microservice so
the database can be used for test
selection and analysis application
within a DevOps pipeline
Test scripts are usually not kept in
this database. They are usually kept in
a source code management system
Repository for meta data that describes tests, test schedules and test results. Examples of
meta data may include description of the test, attributes or tags that describe the purpose
and version of a test and test results, the time to execute a test, and verdict history.

33. Marc Hornbeek DevOps_The_Gray
Benefits of DevOps Testing
Business
• Reduce time-to-market
• Improved quality
• Reduced costs
• Improved innovation
• Team benefits
Team
• More interesting and creative work
• Pride of workmanship
• Timely recognition

34. Copyright 123RF Stock
Photo
Consequences of
Not Doing
DevOps Testing
Properly
• Culture conflicts
• Test escapes, false
positives
• False negative test results
• Increased competition

35. DevOps Testing Engineer Certificate Course
35
http://devopsinstitute.com/wp-content/uploads/2017/07/DOI-DevOps-
Test-Engineering-170712.pdf
• The purpose, benefits, concepts and vocabulary of
DevOps testing
• How DevOps testing differs from other types of testing
• DevOps testing strategies, test management and results
analysis strategies for selecting test tools and
implementing test automation
• Integration of DevOps testing into Continuous
Integration and Continuous Delivery workflows
• How DevOps testers fit with a DevOps culture,
organization and roles

36. Marc Hornbeek DevOps_The_Gray
Marc Hornbeek “DevOps_The_Gray”
36
https://www.linkedin.com/in/marchornbeek
https://devops.com/author/marc-hornbeek/
THANK-YOU
Trace3.com DevOpsInstitute.com DevOps.com www.ieee.org