This document discusses continuous testing in the context of DevOps. It defines continuous testing as how software is tested and outlines a three step process for increasing quality while also increasing speed of development and delivery. The first step is to understand what causes quality to decrease with increased speed. The second step is to find or develop solutions to identified root problems like lack of test coverage or quality holes. The third step is to integrate the solutions into continuous integration/continuous delivery workflows. It provides an example case study of one company's technical stack and CI/CD workflow that incorporates continuous testing.

1. Continuous Testing
What QA means for DevOps

2. Eran Sher
Co-founder & CEO of SeaLights
Nolio (Acquired by CA Technologies)
Mercury Interactive (Acquired by HP)
Conduct (Acquired by Mercury)

3. The State of QA & DevOps
“managing and implementing quality and test measures represent a challenge for
teams trying to increase productivity (or release velocity) with acceptable quality”
WORLD QUALITY REPORT 2016-2017
Increasing
Release speed
Decreasing
Release quality
Continuous Delivery is becoming the de-facto standard

4. The Industry Reaction

5. The Industry Reaction
Organizational People Skills Tools Practices Communication
Now that the hysteria has passed

6. What QA Means for DevOps
Continuous Integration
How we build software
Continuous Deployment
How we deploy software
Continuous Testing
How we test software
Continuous Delivery

7. Continuous Testing
Increase quality while increasing speed

8. Step 1
Understand what is causing quality to
decrease when speed is increased

9. Understanding Test Quality
How do you measure test quality?

10. Quality Holes
How do you stop untested code changes
from reaching production?

11. Centralized Quality Dashboard
How do you collect quality metrics across
environments and tools?

12. Step 2
Find/ develop a solution for the identified
root cause problems

13. Uncover Code Coverage
Unit Integration Functional UI API Exploratory
? ? ? ? ?

14. Identify Quality Holes
Scan every build and
analyze code changes
Track every test and
analyze footprints

15. Build Quality Dashboard
Quality report per build Collect statuses and
coverage across tools,
environments and silos
Track executed tests

16. Step 3
Integrate solutions into CI/CD

17. Case Study
Our Technical Stack:
1. Node.js, Java, and Python
2. Microservices architecture, Mongo DB
3. CI: Jenkins
4. SCM: Github
5. Test automation frameworks: Mocha, TestNG, JUnit, and Py.test
6. Infrastructure: AWS
7. CD: homegrown

18. Case Study
Our CI/CD Workflow
1. Pull request
2. Build + unit test
3. Verify that all tests passed, no decrease in coverage, and no quality holes
4. Integration build
5. Setup and deploy build to integration lab (AWS)
6. Run integration tests
7. Verify that all tests passed, no decrease in coverage, and no quality holes
8. Ready for production » Deploy

19. Copyright © 2016 Sealights. All rights reserved.
Q&A

20. Start testing with the lights on
Copyright © 2016 Sealights. All rights reserved.