Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Power of DevOps in Test Automation
My story to enable
One click saving your days
Engineering Manager at KMS Technology
TOAN LE
ABOUT ME
• A tester with 8+ years of experience in various
testing types and...
OBJECTIVES
Today, I’m going to share with you my story to enable DevOps solutions in Test
Automation that can help to save...
Continuous Testing
How DevOps helps?
My Lesson Learn in Automated Testing.
What If?
My Old Days
Lesson
Power of
DevOps in
...
Test environment created and managed by client
Test servers and execution nodes had to run 24/7
Developers helped to creat...
INFRASTRUCTURE FREQUENCY FLAKY TEST KNOWN ISSUES
More
dependencies on
infrastructure will
limit the execution
frequency an...
MY LESSON LEARN IN AUTOMATED
TESTING
WHAT IF?
The entire test environments including servers and
clients are completely managed by automated
pipelines and CLI/...
Our suggested DevOps solutions to accelerate Test Operation in automated testing
HOW DEVOPS HELPS?
Create builds
Run important API integration
tests within Docker containers
Publish build artifacts
MANAGE INFRASTRUCTURE W...
Tests are distributed to
different servers and nodes
with a balancer
Tests can be executed in
parallel
MANAGE TEST EXECUTI...
All relevant attributes of
testing are captured and
stored in DB
Historical testing data is
accessible at any time
MANAGE ...
COMMON TEST ATTRIBUTES
Re-test
Test
Deplo
y
Collect and analyze test reports
Filter test cases failed due to
unknown reasons
Verify environment h...
Full test execution cycle
Build and deployment cycle
ALL-IN-ONE SOLUTION
Group test execution Test execution on one server
ALL-IN-ONE SOLUTION
REAL-TIME TEST REPORT VISUALIZATION
OUR
RESULTS
By combining DevOps solutions into Test Automation project,
we have achieved significant improvement in our Au...
Build
Deploy
Test
Report
In order to streamline and
automate your
deployment process, you
may need to choose a
cloud servi...
Send me a message
whenever you need more
sharing on this topic.
KMS Technology
28 Truong Son, W.12, D. Tan Binh,
HCMC
toan...
THANKS FOR COMING
Inspiring IT Movement - Sharing makes us stronger
ON BEHALF OF DEVDAY 2019
ORGANIZERS
Upcoming SlideShare
Loading in …5
×

[DevDay2019] Power of Test Automation and DevOps combination - One click saving your days! - By Toan Le, Engineering Manager at KMS Technology Vietnam

63 views

Published on

Test Automation is becomming a MUST in software development life cycle now. DevOps has been an emerging trend, and it's no longer new. Remebering the old days, when you have to stand-up the test servers, get the builds from developers, deploy it, start-up agent machines, run your tests, collect reports, shutdown all resources you have just started, and spend days to analyze the failures. Now it's time to bring DevOps into this game and let it streamline all of these processes then you can save your days for other greater jobs of software testing.

Published in: Technology
  • Be the first to comment

[DevDay2019] Power of Test Automation and DevOps combination - One click saving your days! - By Toan Le, Engineering Manager at KMS Technology Vietnam

  1. 1. Power of DevOps in Test Automation My story to enable One click saving your days
  2. 2. Engineering Manager at KMS Technology TOAN LE ABOUT ME • A tester with 8+ years of experience in various testing types and domains • Managing a team of 30+ members doing Software Development, Test Automation, Performance Test and DevOps for a legacy system of Scientific Data Management • Our Test Automation Team has 20+ QA and DevOps engineers, we are using Katalon Studio to automate various testing types and execute them on AWS streamlined by Jenkins pipelines
  3. 3. OBJECTIVES Today, I’m going to share with you my story to enable DevOps solutions in Test Automation that can help to save a lot of our time and effort as well as to maximize the ROI of our test automation.
  4. 4. Continuous Testing How DevOps helps? My Lesson Learn in Automated Testing. What If? My Old Days Lesson Power of DevOps in Test Automation AGENDA Past Solution Take away
  5. 5. Test environment created and managed by client Test servers and execution nodes had to run 24/7 Developers helped to create and deploy new builds INFRASTRUCTURE TEST EXECUTION TEST REPORTING We started with a team of 3 testers to develop automated tests within Katalon Studio and delivered to client for execution. The client then send back to us the reports of failed test cases, our testers will analyze those reports and tell them if any new big encountered with those tests. The execution frequency was not high and it took time to go back and forth. Tests were not executed frequently (weekly) Tests takes long time to complete (5000 test runs) Unable to test against multiple versions at same time High flaky test rate of UI tests (5-10%) Test results were not centralized, took time to collect results from different nodes Not easy to access test results of previous builds Took too much time to analyze and categorize failed test cases even with known issues ROI OF TEST AUTOMATION IS LOW MY OLD DAYS
  6. 6. INFRASTRUCTURE FREQUENCY FLAKY TEST KNOWN ISSUES More dependencies on infrastructure will limit the execution frequency and utilization Less frequently executing tests will lead to low ROI of test automation since it does not utilize much repeated works saving by automated tests The higher flaky test rate, the less reliability of automated testing as it requires more time and effort to analyze failures Difficulty in accessing to historial testing data will waste more time and effort in analyzing test cases failed because of known issues STREAMLINE Deployment and execution cycle that is not streamlined and fully automated will require weird manual effort by human MY LESSON LEARNT IN AUTOMATED TESTING
  7. 7. MY LESSON LEARN IN AUTOMATED TESTING
  8. 8. WHAT IF? The entire test environments including servers and clients are completely managed by automated pipelines and CLI/API INFRASTRUCTURE AS A SERVICE STREAMLINE HISTORICAL DATA TEST ANALYTICS Test execution is fully streamlined and automated, able to run dynamic requests of testing and test against multiple product versions more frequently Test results are centralized and historical testing data is accessible at any time Test failures and known issues are automatically captured, quality trends and other test-related analysis can be visualized and instantly reported to stakeholders The more tests are executed The more time is saved The more value returned on investment into test automation
  9. 9. Our suggested DevOps solutions to accelerate Test Operation in automated testing HOW DEVOPS HELPS?
  10. 10. Create builds Run important API integration tests within Docker containers Publish build artifacts MANAGE INFRASTRUCTURE WITH AWS ScalingDeploymentBuilding Create servers (Beanstalk) Create database (RDS) Deploy build to server (S3) Create execution nodes Dynamic number of servers Dynamic data set Dynamic number of execution nodes Dynamic system configuration Build is automatically created and validated Environments are ready for testing within 20 mins Environments are scalable and configurable
  11. 11. Tests are distributed to different servers and nodes with a balancer Tests can be executed in parallel MANAGE TEST EXECUTION WITH JENKINS PIPELINE Re-testing Unknown Failures Re-trying Failed TCsTest Distribution Re-try failed test cases on same environment (supported by Katalon) Analyze and query test cases failed with unknown reason Re-deploy new environments and re-test failed test cases Reduce execution time of entire build Reduce failures due to network issue Reduce failures due to environmental issues
  12. 12. All relevant attributes of testing are captured and stored in DB Historical testing data is accessible at any time MANAGE TEST RESULTS WITH CUSTOM REPORTING TOOL Auto-AnalysisConsolidate ResultsTrack Relevant Data Collect all results of one test case for same build on different servers and nodes Consolidate into one test report Automatically analyze failed test cases based on historical data Copy test analysis from previous builds for same fail reasons Compare with common errors Raw testing data is tracked as much as possible Test results are in one place No need to analyze known issues Note: This tool has been developed by our testers within this project, you can also use a similar tool as an alternative, access ReportPortal.io for more details.
  13. 13. COMMON TEST ATTRIBUTES
  14. 14. Re-test Test Deplo y Collect and analyze test reports Filter test cases failed due to unknown reasons Verify environment health Filter test cases to be executed Notify test starting Instant update to QA and stakeholders Deploy and re- test unknown failures Distribute and execute tests Re-analyze entire build with new test results Notify QA to manually analyze new issues Build and Deploy Report ALL-IN-ONE SOLUTION
  15. 15. Full test execution cycle Build and deployment cycle ALL-IN-ONE SOLUTION
  16. 16. Group test execution Test execution on one server ALL-IN-ONE SOLUTION
  17. 17. REAL-TIME TEST REPORT VISUALIZATION
  18. 18. OUR RESULTS By combining DevOps solutions into Test Automation project, we have achieved significant improvement in our Automated Testing. That make our client happier, our QAs have more time to focus on other more important testing activities and our Developers have more confidence to change the product in order to achieve greater goals. This combination also enables us in Continuous Testing and to be much closer to Continuous Delivery model.  5000 test runs finished within 12 hours  Support up to 4 different builds per day  Reduce up to 95% flaky tests  Save days of analyzing failed test cases  One click to trigger the whole testing process
  19. 19. Build Deploy Test Report In order to streamline and automate your deployment process, you may need to choose a cloud service to manage your test environments (e.g. AWS). Cloud Services A Test Automation tool that supports various types of testing such as UI, API, mobile as well as a good CLI will help you to control the whole testing process (e.g. Katalon Studio) Test Automation Tools The whole test operation process include interactions with many parties, you will need a Continuous Integration tool support that process (e.g. Jenkins pipelines) CI Tools A Test Automation project should go with a good Reporting Tool that will help to track and manage historical testing data and free yourselves from repeated works (e.g. ReportPortal.io) Reporting Tools CONTINUOUS TESTING Investment on DevOps solutions to put all those tools into one place is the key to establish Continuous Testing and to increase ROI of your Test Automation CONTINUOUS TESTING
  20. 20. Send me a message whenever you need more sharing on this topic. KMS Technology 28 Truong Son, W.12, D. Tan Binh, HCMC toanle@kms-technology.com CONTACT ME
  21. 21. THANKS FOR COMING Inspiring IT Movement - Sharing makes us stronger ON BEHALF OF DEVDAY 2019 ORGANIZERS

×