This document discusses continuous testing in an agile environment. It defines continuous testing as testing throughout the development process to identify bugs early. It explains that continuous testing helps control side effects, avoid defects, support multiple environments, get fast results, anticipate risks, and create reliable processes. The document provides an overview of how continuous testing works, including test environments, data management, automatic deployment, and test automation. It also discusses creating a continuous testing project, the agile test process, and how to implement effective continuous testing to improve quality and business value.

1. ! "
#
$
%
&
'
$./run
{
Created by
Fabrício Epaminondas ©2015-2020
Continuous
TESTING
Testing in an
agile environment

2. • Continuous Testing
• Agile principles
• Why Continuous Testing?
• Creating your first test project in minutes
• How it works
• Test process
• Effective Continuous Testing
15/10/2015 - 20:30
Location: Cinema
didático - Facisa
Agenda

3. The goal
AGILE
PRINCIPLES

4. The Big Picture of General SDLC 4
3
21
SDLC
Requirement
Initiation
Analysis & Specification
Risks
Delivery
User Acceptance (UAT)
Deployment
Maintenance and Support
Operations
4
4 }
( '
Planning
Scope, Time and Cost
Prioritization
Development
Design / Prototype
Implementation
Integration
Testing
Documentation
Release
Software-Development-Life-Cycle

5. 5Agile process: Scrum
SPRINT REVIEW /
RETROSPECTIVE
2-4
Weeks
1 2 3
4

6. 6
“Our highest priority is to satisfy the customer through
early and continuous delivery of valuable software.”
The Agile Manifesto, First Principle

7. WHY
CONTINUOUS
TESTING?
The metaphor

8. 8Software-Development-Life-Cycle Metaphor
1.Jack is a happy programmer, he has just done his
first commit
2. After a few days, he receives the test report
from QA Team reproving his changes and pointing
a new BUG X in production.
3.Jack is sad now, but he fixes the bug and
submit his changes again.
4. Now it is the week of a new release and Jack
receives other report confirming the BUG X is
fixed
5. But the report also contains new BUGS: A, B, C.

9. 9
6.Jack tries to figure out if his changes caused the
other bugs
7.But Jack is not alone, he tries to figure out if
someone else changes caused the other bugs
8.Jack tries a new solution to get done and re-
tested in all environments before the deadline
9.Poor, poor, Jack… it is too late…
the deadline has come.
Software-Development-Life-Cycle Metaphor

10. 10
What does Jack do?
Software-Development-Life-Cycle Metaphor

11. 11
Let’s think for a moment…
What are bugs?
What do they eat?
Where do they live?
How to avoid new bugs?
What if bugs were living entities…
What are they thinking?
Bugs….

12. 12
"I WILL BE BACK"
(The Bug)
)

13. 13
TRUST ME, I CAN DO IT..
( Developer in a good day)

14. 14
• To control side effects
• To avoid software defect snowball
• To efficiently support multiple platforms,
environments and release versions
• To get fast results in a short time cycle
• To anticipate risks
• To create an agile, reliable and repeatable process
• Release in hours not months
Use Continuous Testing…

15. CREATING YOUR
FIRST PROJECT
IN MINUTES
Hands-On

16. 16Hands-On
• Let’s create our first project for continuous testing
using an amazing and totally free web hosting
platform with continuous integration support:
Let’s start from the test project template:
https://gitlab.com/fabricioepa/gitlab-ci-quickstart
gitlab.com

17. 17Hands-On
•You can learn how to implement your own test platform
in the training:
https://fabricioepa.wordpress.com/2015/02/02/
quickstart-for-continuous-integration/
•For more advanced continuous integration practices
see the link:
https://fabricioepa.wordpress.com/2015/02/03/
continuous-integration-practices-to-improve-the-
software-quality/

18. HOW IT
WORKS
Overview

19. 19Overview
• Test Environment
• Test Data Management
• Automatic Deployment
• Multiple environment configuration profiles
•Virtualization (production-like)
•Dynamic Infra-structure
•Test Automation
• Orchestration
*Continuos
Testing

20. 20Related Topics
• Continuous Integration (CI)
• Continuous Delivery
• Continuous Deployment
• See 3 high levels of continuous testing adoption
*Continuos
Testing

21. Design development pipelines 21
Commit Stage
Unit Tests
Code Analysis
Code Review
'
Triggers
/branches/product-1.0
Development
Unit Tests
Code Analysis
Development pipeline
Build
Package
Sanity Tests
Verify Metrics
Publishing
Test Report
Unit Test Coverage
Project Metrics
Release Package
Fast Builds pattern ,
Artifact
Repository$
01

22. 22
Setup
Environment
Infrastructure & Data
QA automation pipeline
Deploy
Install packages
Automated Test Plan
Integration,
Functional, Regression,
Performance, etc.
Publishing
Test Report
Test Coverage
Performance Benchmark
○ Push-buttons
Scheduled builds
Repository changes
{
'
/branches/product-1.0
Triggers
Design QA pipelines
Self-service " +
Nightly-builds {
Artifact
Repository$
02

23. 23
Setup
Environment
Infrastructure
Data
Testing
Manual
Exploratory
○
Deploy
Install packages
Health check
○
. QA Team
UAT
Self-service " deployments
Design delivery and deployment pipelines
Delivery / Deployment pipelines
User Acceptance
Demonstrations
Consumer Preview
Beta testing
GA releases
03
Artifact
Repository$

24. AGILE TEST
PROCESS
The real deal

25. 25Main Testing Stages
Planning
Scope, Time, Cost
Process guideline
Specification
Test Scenarios
Test Cases
Prioritization and
Classification
Execution
Manual
Automatic
Closure
Test Report
Bug Report
Qualification
Documentation
1 2 3 4

26. 26Test plan
A good test plan should answer at least 3 main
questions:
• What?
• How?
• When?
01

27. s
27Test plan: What?
Guideline for macro activities
•Test execution
• Defect detection (Bugs)
• Enhancement areas
)
01

28. • Strategies and methodology
• Environment and tools
28Test plan: How?01

29. • Traceability
29Test plan: How?01

30. 30
Schedule
Test plan: When?
/" 0
Team and
Resources
Test Scope and
Risks
4
01

31. 31
•Black-box / White-box
•Unit / Component /
Integration
•Smoke / Sanity
•Acceptance / Functional
•Regression
•System / End-To-end
•Exploratory
•Load / Stress /
Performance
•Parallel / Concurrent
•Usability
•Alpha / Beta
•Compatibility
•Destructive / Recovery
•Internationalization /
Localization
Choose the right test techniques01

32. 32Test specification
• Requirements study
•Test Scenarios and Estimative of Test Cases
•Test Case Creation, Review and Maintenance
•Prioritization and Classification
•Automatic vs. Manual
•Others: Acceptance, Functional, Performance,
etc.
•This information will be used for creating
execution test plans.
02

33. 33Executing test plans
•Manual execution
•Automatic execution (by QA test pipeline)
•Whatever the way you take, you must choose a
strategy to compute all test results in the final
analysis
•To use a tool like TestLink may help you to
manage all tests.
03

34. 34Closure
•Reports
• Test Execution Status
• Defects by Priority
• Requirements Test Coverage
• Other quality aspects:
•Performance, accuracy, security issues, etc.
04

35. 35Closure
•Final report and Qualification:
•Approval Status of Deliverables
04
1
X
1
Feature A
Feature B
Feature C
L Partial release
approval

36. The real value
EFFECTIVE
CONTINUOUS
TESTING

37. • Let’s remember the agile principle:
37Continuous Quality Improvement
“…continuous delivery of valuable software.”
The Agile Manifesto, First Principle
• Testing != Software Quality Assurance
• You must understand how to use tests to enhance
the process, reduce rework, increases the quality
and business value of the software product.
• This is Continuous Improvement, one of the
most important QA practices.

38. 38Continuous Improvement in 4 Steps
Plan
Targets, Methodology,
Process, etc…
Do
Process execution
Verification / Study
Measure Results
Study evidences and deviations
from original plan
Action
Diagnostic
Corrective and Preventive Action Points
Improvement Areas

39. Notes: Continuous Improvement and Agile
• Retrospectives
• Learned Lessons
• Action Points
• Use the PDCA Cycle
2

40. 40Effective Continuous Testing and Agile
•Objective
•For each sprint the deliverable must potentially
be production-ready
•Quality metrics and daily feedback
•Build status and reports
•Statistics and Trends
•Automatic Tests report
•Code violations
•Code coverage
•Project metrics
•Dev&QA teams must understand and maintain
the quality metrics

41. 3 Displays + Dashboards
% Email notifications
Continuous Monitoring
Effective Continuous Testing and Agile

42. 42
•Balance test levels according to the project
needs on each sprint
Effective Continuous Testing and Agile
Unit Test
Integration Test
System Test
Acceptance
Test
Test individual
component
Test integrated
system
Test component
groups
Test final system
stack
V-model

43. 43
•Continuously improve your automated tests
Effective Continuous Testing and Agile
Unit
Integration
E2E
• Recommended test distribution pyramid
Cheaper
automation
Complex
Automation
UI / Manual /
Exploratory Tests
Reference: [1] http://googletesting.blogspot.com.br/2015/04/just-say-no-to-more-end-to-end-tests.html

44. 44
•Unrelated activities (to the backlog)
•Implementing automation scripts
•Status monitoring
•Multiple environment maintenance: Testing,
Staging, Production…
•Sprint Retrospective Tasks
•Use CI feedback to continuous improve the
process and the product quality
•Diagnosing pitfalls and registering sprint action
points
Continuous Testing and Agile Planning

45. 45Business Value
•The cost of adoption has a fast return on
investment
•Rework reduction
•Increase team productivity.
•Release in days not months
•Reduce at about 50% of development, testing
and deployment costs.

46. ! "
#
$
%
&
'
$./run
{
Criado por
Fabrício Epaminondas ©2015
Contact
FABRICIO EPAMINONDAS
4 linkedin.com/in/fabricioepa
5 fabricioepa.wordpress.com
6 fabricioepa
7 @fabricioepa