Continuous testing is one of the most effective ways of building quality into the continuous delivery pipeline; yet it is difficult to implement in practice. Continuous testing involves more than test automation. Although test automation is a must; continuous risk analysis and optimizing the test suite is critical so that test automation doesn’t become a bottleneck in the DevOps pipeline. In this presentation, you’ll learn how to implement an effective continuous test strategy throughout the continuous delivery pipeline.

1. Test Automation and Beyond:
Developing an Effective Continuous
Testing Strategy
November 2, 2018

2. About Me
|Gerie Owen
• VP, Knowledge & Innovation--US at QualiTest
Group
• Speaker and Writer on Testing topics
• Experienced Tester, Test Lead, & Test
Architect
• Gerie can be reached:
• gowen@qualitestgroup.com
• www.gerieowen.com,
• her blog, Testing in the Trenches, is at
https://testinggirl.wordpress.com/
• @GerieOwen on Twitter and on
• http://www.linkedin.com/in/gerieowen
2

3. Agenda
| Continuous Testing Explained
| Implementing Continuous Testing
| Developing a Test Strategy for Continuous Testing
| Monitoring in Production

4. Continuous Testing
| A core capability in Agile and DevOps where all testing activities run continuously in an
integrated fashion with development and delivery. Immediate bug fixing is enabled,
test environments are provisioned instantly, and unit, functional, and nonfunctional
tests are run in an automated way, orchestrated by continuous integration and
continuous delivery tools. The Forrester Wave™: Modern Application Functional Test Automation Tools, Q4
2016
4

5. Why Implement Continuous Testing?
| DevOps is about Continuous Delivery
| Testing Cycles must be shortened yet quality must be increased
| Increase in microservices architectures require high-quality functional testing including
automated API testing
| Builds, Integrations and deployments are automated; testing must be integrated into the pipeline
5

6. Attributes of Continuous Testing
| Assesses business risk coverage as its primary goal
| Establishes a safety net that helps the team protect the user experience
| Requires a stable test environment to be available on demand
| Integrates seamlessly into the software delivery pipeline and DevOps toolchain
| Delivers actionable feedback appropriate for each stage of the delivery pipeline
6

7. Continuous Testing vs Test Automation
| Continuous testing requires automation, however, it encompasses much more.
| Continuous testing is an approach to managing risk by focusing on not only on
improving testing efficiency but more importantly, increasing the effectiveness of our
test processes.
7

8. Implementing Continuous Testing
| Engage in continuous risk analysis and process improvement
| Increase the velocity of current quality assurance process
| Implement automated testing through the entire software development process
| Develop a culture in which the entire team is responsible for quality.
8

9. Key Practices of Continuous Testing
| Defect Management
| Discuss their “observations” prior to logging defects.
| Fix defects immediately rather than logging and tracking.
| Test Case Management and Optimization
| Analyze individual test cases for for their quality and effectiveness.
| Evaluate test suites to determine if they are providing the maximum amount of coverage with
the fewest number of test cases.
| Advanced Analytics
| Using code change impact analysis for test optimization to determine:
• what new tests are needed,
• what old tests need updating or replacement,
• what other areas of the application have been affected and require regression testing.

10. Key Practices of Continuous Testing
| Environment Access and Availability
| Requires easy access to production-like environments.
| Automated processes for creating and verifying these environments.
| Service Virtualization and Data Management
| Processes include deployment automation, test data creation. and service virtualization.
Embraces shift left because integration testing can begin even though all the integrations may
not be completed or available
| Automated Testing
| Includes automating not only all layers of testing, but also, automated testing of the deployment
process, including automated health checks and feedback at each step
10

11. Continuous Testing Strategy
| Components
| Streamline the Test Process
| Champion Team Responsibility for Quality
| Increase Velocity AND Improve Quality
| Testing throughout the Build and Delivery Pipeline
| Implementing Automated Testing Checkpoints
| “Shift-Left” both Functional and Non-Functional Testing
| Continuous Monitoring in Test and Production
11

12. Streamlining the Test Process
| Focus on Mitigating Business Risk
| Optimize Test Scope, Test Cases & Test Suites
| Run Automated Tests In Parallel
| Identify and Mitigate Bottlenecks

13. Focus on Mitigating Business Risk
| Focus on the customer; aspects of the applications that are most important to the
customer are areas of highest risk
| Understand the competitive climate of the organization; how important is it to be first
to market.
| Map risks to application components, requirements and tests
| Optimize the test suite to provide the greatest risk coverage with the fewest number of
test cases
| Always report status in a way that highlights the business risk including, technical,
performance, and compliance perspectives
13

14. Optimize Test Scope
| Select end-to-end tests that focus on the critical and high-risk features of the
application
| Understand all integrations involved in critical user workflows including the
technologies employed in those applications (web, mobile, message/API-layer etc.)
| Implement service virtualization for integrations and components that aren’t readily
available in order to run end-to-end test continuously
| Use the most realistic data possible for each test, include synthetic data generation
where needed
| Include exploratory testing to find user-experience and other bugs not easily found by
automated tests
14

15. Optimizing Test Cases
| Tests should be focused on the functionality they are validating.
| Tests should be as simple as possible; the more complex the test, the more validation is required for the test code itself.
| Even though service virtualization can be used to fill in for missing components in integration testing, tests should as independent as
possible.

16. Optimizing Test Suites
| An ongoing process

17. Optimizing Test Suites
| Include test cases that validate high risk areas of code
| Include test cases that validate customer-centric features
| Include test cases that validate critical business workflows
| Use Functional Automation tools that create multi-purpose scripts
| Cross-Platform Testing
| Scripts should run on Desktop, Web, Mobile & Server
| Cross-Browser Testing
| Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required
| Performance Testing
| Scripts should collect performance statistics including page loads etc.

18. Run Automated Tests In Parallel
| Speed up Automated Test Suites by running them in parallel on multiple virtual
machines; best practices include:
| Atomic Tests
• 1 test per test script
| Autonomous Tests
• Test should not rely on the results of previous test to run
| Quick-running Tests
• Keep execution time to one minute on the local environment
| Data Management
• Each Test should create and delete its own data
18

19. Why Identify and Mitigate Bottlenecks
| Bottlenecks impede velocity!
19

20. Identify and Mitigate Bottlenecks
| Constraints to the test process can be found throughout from requirements through
post production checkout. Some Examples:
| Testers aren’t included in backlog refinement so they don’t have the opportunity help build the
acceptance criteria.
| Testers aren’t included in sprint planning, story points may not include testing which affects
velocity.
| Defects may not be addressed on a timely basis
| Regression suite runs too long or is ineffective
| Post production checkout is still done manually.
| Constraints impact velocity.
20

21. Championing Team Responsibility for Quality
21

22. Increase Velocity AND Improve Quality
| Increase velocity
| “Shift Left”; Push automation to the base of the testing pyramid
• TDD, BDD
• Pair testers and developers
| Include risked-based manual and exploratory testing
| Implement continuous non-functional testing including:
• Performance
• Security
• Availability
• Accessibility
| Amplify automated feedback loops to enable risk-based decision-making
| Improve quality
| Developers need to embrace testing
| Entire team is responsibility for quality
| Implement post-production monitoring
| Implement testing in production, based on application risk
22

23. Integrate Testing Throughout the Delivery Pipeline
| Integrate automated tests in each stage the Continuous Integration Pipeline
23

24. Implement Automated Testing Checkpoints
| Automated Testing Checkpoints
| Unit tests verify code
| Component tests verify key features
| Integration test verify workflows involving multiple applications
| Risk-based, Optimized Regression tests run prior to each deployment
| Post production health checks verify applications are up and running after deployment
| Production monitoring finds defects and performance issues before they impact the customer
24

25. “Shift Left” Testing
| Begin Performance testing as early as possible in order to correct design flaws more
easily
| Use Functional Automation tools that create multi-purpose scripts
| Cross-Platform Testing
• Scripts should run on Desktop, Web, Mobile & Server
| Cross-Browser Testing
• Scripts should run on multiple browsers including Chrome, Edge, Safari and any others required
| Performance Testing
• Scripts should collect performance statistics including page loads etc.
| Security Testing
• Automated tests to ensure that code written in a secure manner
• Test execution security as early as possible
25

26. Continuous Monitoring in Test and Production
| Continuous Monitoring
| Comes from the DevOps principle of amplifying feedback loops in both test and production.
| Enables collection and analysis of data and metrics throughout the lifecycle that are used to
optimized user experience.
| The most important data is about how users are interacting with the application and feedback
from customers. This data allows stakeholders and DevOps teams to make targeted adjustments
such as:
• Functional enhancements to remove customer pain points
• Upgrades to environments to improve application performance
| Testing in production includes:
• Monitoring data and transactions
• Simulating transactions
• Experimentation with real users
• A/B Testing

27. Summary
| Continuous testing is an approach to managing risk by focusing not only on eliminating
testing bottlenecks but more importantly, increasing the effectiveness of our test
processes.
| Automated Testing is a subset of Continuous Testing; it is a key practice.
| A Continuous Testing Strategy involves:
| Streamline the Test Process
| Champion Team Responsibility for Quality
| Increase Velocity AND Improve Quality
| Testing throughout the Build and Delivery Pipeline
| Implementing Automated Testing Checkpoints
| “Shift-Left” Functional and Non-Functional Testing
| Continuous Monitoring in Test and Production

28. References
| Continuous Testing for IT Leaders, Wayne Ariola and Cynthia Dunlop
| A Practical Guide to Testing in DevOps, Katrina Clokie
| What Testers Need To Know About Continuous Testing, Wayne Ariola
| The Forrester Wave™: Modern Application Functional Test Automation Tools, Q4 2016,
Diego Lo Guidice
28

29. www.QualiTestGroup.com