The document discusses the implementation of DevOps in test automation by Toan Le, an engineering manager with over 8 years of testing experience. It highlights the transformation of a testing process through the use of tools like Katalon Studio and Jenkins pipelines, resulting in significant improvements in efficiency, reduced flaky tests, and increased ROI on test automation. The presentation emphasizes the importance of automating testing environments and processes to achieve continuous testing and delivery.

1. Power of DevOps in Test Automation
My story to enable
One click saving your days

2. Engineering Manager at KMS Technology
TOAN LE
ABOUT ME
• A tester with 8+ years of experience in various
testing types and domains
• Managing a team of 30+ members doing
Software Development, Test Automation,
Performance Test and DevOps for a legacy
system of Scientific Data Management
• Our Test Automation Team has 20+ QA and
DevOps engineers, we are using Katalon Studio
to automate various testing types and execute
them on AWS streamlined by Jenkins pipelines

3. OBJECTIVES
Today, I’m going to share with you my story to enable DevOps solutions in Test
Automation that can help to save a lot of our time and effort as well as to maximize
the ROI of our test automation.

4. Continuous Testing
How DevOps helps?
My Lesson Learn in Automated Testing.
What If?
My Old Days
Lesson
Power of
DevOps in
Test
Automation
AGENDA
Past
Solution
Take
away

5. Test environment created and managed by client
Test servers and execution nodes had to run 24/7
Developers helped to create and deploy new builds
INFRASTRUCTURE
TEST EXECUTION
TEST REPORTING
We started with a team of 3 testers to develop
automated tests within Katalon Studio and delivered
to client for execution. The client then send back to
us the reports of failed test cases, our testers will
analyze those reports and tell them if any new big
encountered with those tests. The execution
frequency was not high and it took time to go back
and forth.
Tests were not executed frequently (weekly)
Tests takes long time to complete (5000 test
runs)
Unable to test against multiple versions at same
time
High flaky test rate of UI tests (5-10%)
Test results were not centralized, took time to
collect results from different nodes
Not easy to access test results of previous builds
Took too much time to analyze and categorize
failed test cases even with known issues
ROI OF TEST AUTOMATION IS LOW
MY OLD DAYS

6. INFRASTRUCTURE FREQUENCY FLAKY TEST KNOWN ISSUES
More
dependencies on
infrastructure will
limit the execution
frequency and
utilization
Less frequently
executing tests will
lead to low ROI of
test automation
since it does not
utilize much
repeated works
saving by
automated tests
The higher flaky
test rate, the less
reliability of
automated testing
as it requires more
time and effort to
analyze failures
Difficulty in
accessing to
historial testing
data will waste
more time and
effort in analyzing
test cases failed
because of known
issues
STREAMLINE
Deployment and
execution cycle
that is not
streamlined and
fully automated
will require weird
manual effort by
human
MY LESSON LEARNT IN AUTOMATED
TESTING

7. MY LESSON LEARN IN AUTOMATED
TESTING

8. WHAT IF?
The entire test environments including servers and
clients are completely managed by automated
pipelines and CLI/API
INFRASTRUCTURE AS A SERVICE
STREAMLINE
HISTORICAL DATA
TEST ANALYTICS
Test execution is fully streamlined and automated,
able to run dynamic requests of testing and test
against multiple product versions more frequently
Test results are centralized and historical testing
data is accessible at any time
Test failures and known issues are automatically
captured, quality trends and other test-related
analysis can be visualized and instantly reported to
stakeholders
The more tests are executed
The more time is saved
The more value returned on investment
into test automation

9. Our suggested DevOps solutions to accelerate Test Operation in automated testing
HOW DEVOPS HELPS?

10. Create builds
Run important API integration
tests within Docker containers
Publish build artifacts
MANAGE INFRASTRUCTURE WITH AWS
ScalingDeploymentBuilding
Create servers (Beanstalk)
Create database (RDS)
Deploy build to server (S3)
Create execution nodes
Dynamic number of servers
Dynamic data set
Dynamic number of execution
nodes
Dynamic system
configuration
Build is automatically
created and validated
Environments are ready
for testing within 20
mins
Environments are
scalable and
configurable

11. Tests are distributed to
different servers and nodes
with a balancer
Tests can be executed in
parallel
MANAGE TEST EXECUTION WITH
JENKINS PIPELINE
Re-testing Unknown
Failures
Re-trying Failed TCsTest Distribution
Re-try failed test cases on
same environment (supported
by Katalon)
Analyze and query test cases
failed with unknown reason
Re-deploy new environments
and re-test failed test cases
Reduce execution
time of entire build
Reduce failures due to
network issue
Reduce failures due to
environmental issues

12. All relevant attributes of
testing are captured and
stored in DB
Historical testing data is
accessible at any time
MANAGE TEST RESULTS WITH
CUSTOM REPORTING TOOL
Auto-AnalysisConsolidate ResultsTrack Relevant Data
Collect all results of one test
case for same build on
different servers and nodes
Consolidate into one test
report
Automatically analyze failed
test cases based on historical
data
Copy test analysis from
previous builds for same fail
reasons
Compare with common errors
Raw testing data is
tracked as much as
possible
Test results are in one
place
No need to analyze
known issues
Note: This tool has been developed by our testers within this project, you can also use a similar tool as an
alternative, access ReportPortal.io for more details.

13. COMMON TEST ATTRIBUTES

14. Re-test
Test
Deplo
y
Collect and analyze test reports
Filter test cases failed due to
unknown reasons
Verify environment health
Filter test cases to be executed
Notify test starting
Instant update
to QA and
stakeholders
Deploy and re-
test unknown
failures
Distribute and
execute tests
Re-analyze entire build with new test results
Notify QA to manually analyze new issues
Build and Deploy Report
ALL-IN-ONE SOLUTION

15. Full test execution cycle
Build and deployment cycle
ALL-IN-ONE SOLUTION

16. Group test execution Test execution on one server
ALL-IN-ONE SOLUTION

17. REAL-TIME TEST REPORT VISUALIZATION

18. OUR
RESULTS
By combining DevOps solutions into Test Automation project,
we have achieved significant improvement in our Automated
Testing. That make our client happier, our QAs have more time
to focus on other more important testing activities and our
Developers have more confidence to change the product in
order to achieve greater goals. This combination also enables
us in Continuous Testing and to be much closer to Continuous
Delivery model.
 5000 test runs
finished within 12
hours
 Support up to 4
different builds per day
 Reduce up to 95%
flaky tests
 Save days of analyzing
failed test cases
 One click to trigger the
whole testing process

19. Build
Deploy
Test
Report
In order to streamline and
automate your
deployment process, you
may need to choose a
cloud service to manage
your test environments
(e.g. AWS).
Cloud Services
A Test Automation tool
that supports various
types of testing such as
UI, API, mobile as well as
a good CLI will help you to
control the whole testing
process (e.g. Katalon
Studio)
Test Automation
Tools
The whole test operation
process include interactions
with many parties, you will
need a Continuous
Integration tool support that
process (e.g. Jenkins
pipelines)
CI Tools
A Test Automation project
should go with a good
Reporting Tool that will help
to track and manage
historical testing data and
free yourselves from
repeated works (e.g.
ReportPortal.io)
Reporting Tools
CONTINUOUS
TESTING
Investment on DevOps solutions to put all those tools into one place is the key to
establish Continuous Testing and to increase ROI of your Test Automation
CONTINUOUS TESTING

20. Send me a message
whenever you need more
sharing on this topic.
KMS Technology
28 Truong Son, W.12, D. Tan Binh,
HCMC
toanle@kms-technology.com
CONTACT ME

21. THANKS FOR COMING
Inspiring IT Movement - Sharing makes us stronger
ON BEHALF OF DEVDAY 2019
ORGANIZERS