The document discusses the importance of continuous testing (CT) in DevOps, highlighting its role as a critical component for achieving continuous delivery and improved software quality. It outlines the principles of continuous testing, key implementation strategies, and the necessity of integrating CT with automation and cloud environments. Additionally, it emphasizes the organizational and cultural changes required to successfully adopt continuous testing practices.

2. 2
Shashi Kiran
CMO
Quali
Marc Hornbeek
“DevOps-the-Gray”
Principal DevOps Consultant,
Trace3
Author, DevOps Test
Engineering Course
The DevOps Institute
HOST EXPERT PANEL
Pascal Joly
Director, Technology
Partnerships
Quali

3. Agenda
3
1 Introduction
2 Principles of Continuous Testing
3 Implementing CT
4 Demo: Continuous Testing
5 Q&A Continuous Testing
The “Missing Link”

4. 01
02
03
Venture-funded start-up focused on Cloud-centric
DevOps and BizOps automation
Delivers “Sandboxes” - Replicas of production
environments
Strong Innovation DNA – Trusted by customers
worldwide
About Quali
4
300+ customers – Cloud providers, Telcos,
Enterprises – across FSI, Retail, Healthcare and
Government
~115 employees in USA, Israel and Europe
focused on serving customers worldwide300+

6. Infographic
Download here: http://bit.ly/2nY4FTB

7. The Rising Importance of (Continuous) Testing
SPEED QUALITY
TEST

8. Traditional Testing
Source: www.saucelabs.com
• Siloed development and testing
• Waterfall approach
• Long lead times between
identifying and solving bugs
• High risk when bugs are missed
• Rework and delays

9. What is Continuous Testing (CT)?
• Quality assessment strategy
• Incremental changes tested
“continuously”
• Automated tests (large %)
• DevOps Toolchain integration
• Production equivalent test
environment stages
• End-to-end testing across the
pipeline

10. CT – A sign of DevOps Maturity
1
2
3
4
5
Processes characterized for specific
projects. Organization is often reactive
Automated Test Execution
Projects tailor processes from
organizations development
methodology
Test Framework pipeline
Processes are measured and controlled
Test-driven Release Tools
Focus on process improvement
Dynamic Test and Environment
Blueprints
Process is unpredictable,
poorly controlled and
reactive
Manual Adhoc Tests

11. Hurdles to Implement Continuous Testing
Low % test automation
Budget
Missing/fragmented
toolchain
Production equivalent test
environment stages
Product
Leadership
Culture
Training and Expertise
Collaboration – Dev/Sec/Ops
Budget
Rewards and Incentives
People
Continuous Test planning
End-to-end Strategy
End-to-end testing across
pipeline
Process
CI
CT
CD

12. CollaborativeCultureisthedoortoDevOps,but
ContinuousTestingisthekeytoContinuous
Delivery
“
”
DevOps-the-Gray

13. CT provides Assessment of CD Stage Artifacts

14. Continuous Testing – Implementation Strategy
Adopt
5 test
Tenets
Org. and
Culture
Streamline
Process
and
Workflow
Integrating
CT Infra.
Environment
Orchestration
and Test
Automation
1 2 3 4 5

15. 1. Five Tenets of Continuous Testing
2
Arrange tests so most
likely problems are
found in earlier stage of
pipeline.
Fail Early
1 Shift Left
Conduct each test as
early in the pipeline as
possible.
4
Focus on most
important tests and
results
Relevance
3 Test Fast
Arrange tests to run in
quick cycles
5 Fail Often
Run tests frequently
and with many different
conditions

16. 2. Build Organizations and Culture
• DevOps does not mandate roles
• Teams decide role assignments
• Checklist help ensure tasks are
covered.
• Multiple checklists may be needed to
cover different stages of the pipeline
such as Pre-Flight, Integration,
Delivery, Deployment, etc.

17. 3. Process and Workflow Practices
• Streamline Procurement, install and support:
o Infrastructure
o Tools
o Integration of test tools into tool frameworks
o Operating the DevOps pipeline
• Align Rewards and Incentives
• Train on Automation Skills and Tools

18. 4. Integrated DevOps Test Infrastructure
PLAN
Collaboratively
plan integration
into each stage of
DevOps
Infrastructure
TEST
Automatically
trigger tests from
master DevOps
framework.
ASSESS
Run DevOps test
assessment for
each stage to
provide feedback
and support
workflow decisions.
Stage 2

19. 4A. Testing Triggers – Infrastructure “Glue”
• Continuous Test framework is integrated with
the DevOps master framework.
• DevOps test automation can be performed by
the DevOps master framework directly or may
be implemented using special purpose DevOps
test framework tools.
• Example: DevOps testing for the build stage is
triggered when the DevOps master framework
triggers the DevOps test framework.
TEST

20. 4B.Asssessment Actions Processing
• CT test executions produce verdicts.
• Verdicts provide decision support data
for analytics that determine if a change
is good enough to proceed to the next
stage in the DevOps pipeline or needs to
be remediated before proceeding.
• Analytics may be basic verdict reports or
more sophisticated verdict trend analysis.
Verdicts
Verdict Trends
Analytics
ASSESS

21. Automatically setup the test
environment and resources
(physical and virtual) to match
the requirements of a test.
Environment Orchestration
5. Test Environment Orchestration and
Automation
Execute test tasks without
manual work required.
Test Automation

22. 5A. Environment Orchestration
• Test resources vary depending on
the complexity and risk of change
being tested.
• A flexible lab environment
employs orchestration tools to
elastically scale physical and
virtual resources for a specific
test.

23. 5B. Test Automation
• Create new automated tests at the
same time changes are created.
• Run automated tests matched to
specific content being changed.
• Produce test results automatically
to support in-process DevOps test
assessments.
Continuous testing uses automation to execute tests quickly and consistently and
to keep up with the frequent changes that are processed through DevOps pipelines.

24. DevOps Orchestration Stack
Testing
Orchestration
simplifies
management of
test resource
topologies through
test environment
blueprints.
Test Orchestration
(Dev)
Build Automation Packaging Config Mgmt.
Dev & Integration DeliveryPackaging
Test Orchestration
(Staging and
Production)
Test Orchestration
(Acceptance)
Test Resources Cloud
CI/CD Pipeline: Application Release Automation & ALM Systems

25. CI/CD with CT – Leveraging Cloud Sandboxes
Pipeline Automic
Quali
Sandbox
Software
Load Security Integration
Staging/
Production
Dev
CI/CD Pipeline: Application Release Automation & ALM Systems

26. Quali CloudShell -Sandboxes
Blueprint Modeling Automation & Orchestration
Reservation & Scheduling Self-Service Catalog
Built-In Networking BI & Analytics
REST API
EMBEDDED GUI
DevOps Plug-Ins
BUILD ANY ENVIRONMENT. DEPLOY AS-A-SERVICE
Bare Metal
TO ANY CLOUD*
ENVIRONMENTFEATURESINTERACE
Applications Data Test Tools
Physical Infra. Virtual Infra. Services
Network Storage
Compute Security
CLOUD SANDBOX

27. Demo
Continuous Integration
and Testing
with Cloud Sandboxes

28. Test Strategies for Continuous Deployment
Reference: https://vwo.com/ab-testing/
A/B Testing Canary Testing Blue/Green Testing

29. Continuous Testing - Benefits
Reduced test
Time and test
wait time
Improved
Test Results
Analysis
Cost
Efficiency of
Shared
Resources
Improved
SLAs
CONTINUOUS
TESTING
Rapid
Test
Environment
Recovery

30. Summary
1
2
3
Don’t compromise between
speed and quality.
Support Continuous Delivery
Initiative with Continuous Testing
Cloud Sandboxes can be
orchestrate and automate
environments for continuous test
CONTINUOUS
TESTING
From “Missing Link” to “Glue”

31. 2017 DevOps and Cloud Survey
http://info.quali.com/2017-devops-and-cloud-survey

32. For More Information:
www.quali.com www.trace3.com www.devopsinstitute.com
Quali Trace3 DevOps Institute

33. 33
www.quali.com
THANK YOU