Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Orchestrating Automated Tests in Different Systems - Please Download to watch the embedded videos ;)

9 views

Published on

This talk was presented in DevOpsDays Sao Paulo (June/2018) and in Daitan DevWeek (July/2018).

What should you do if your product must be tested on multiple operating systems before delivering? And what if it must be tested on different browsers? Now it is a lot of combinations. And how to do this in an agile way?

Here is a talk about how we can orchestrate automated tests on different systems, on different browsers. Assuring the coverage on all operating systems and browsers supported by a product is one of the toughest tasks commonly faced when transitioning to a DevOps setup.
Due to the high quantity of environment combinations, the complexity of organizing and executing tests increases and can lead to an ineffective pipeline. In this talk, we will evaluate an automated orchestration of complex test suites that delivers the effectiveness needed for proper DevOps.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Orchestrating Automated Tests in Different Systems - Please Download to watch the embedded videos ;)

  1. 1. Orchestrating a big automated testing pipeline in multiple environments
  2. 2. © Daitan Group – Private & Confidential – v1.1 2 About me • 9 years experience in Software Engineering, and 2 in Electronics • BSc in Computer Science – IC-Unicamp • Electronics Technician – Cotuca-Unicamp • Currently working on Data Engineering, Performance Engineering and Test Automation at Daitan Group • I love developing and testing! kelvsar@gmail.com https://www.linkedin.com/in/kelvin-ksps/ Kelvin Ronny Marques da Silva
  3. 3. CHALLENGE
  4. 4. © Daitan Group – Private & Confidential – v1.1 4 • Mac OS 10.11 • Mac OS 10.12 • Mac OS 10.13 • 13 Environments Problem 1: Ensure the product runs well on all supported operating systems • Windows 7 x64 • Windows 7 x86 • Windows 8 x64 • Windows 8 x86 • Windows 8.1 x64 • Windows 8.1 x64 • Windows 10 x64 • Windows 10 x86
  5. 5. © Daitan Group – Private & Confidential – v1.1 5 • Mac OS 10.11 • Mac OS 10.12 • Mac OS 10.13 • Chrome • Firefox • Internet Explorer • Safari 13 * 3 = 39 setups! Problem 2: Ensure the product runs well on all supported operating systems and browsers • Windows 7 x64 • Windows 7 x86 • Windows 8 x64 • Windows 8 x86 • Windows 8.1 x64 • Windows 8.1 x86 • Windows 10 x64 • Windows 10 x86
  6. 6. © Daitan Group – Private & Confidential – v1.1 6 Ex.: • 1 machine is the host and 2 machines are guests of a video conferencing meeting • 2 or more machines are playing a Game • 1 machine is the presenter and 2 are the participants on a webinar Problem 3: Product only runs one instance per machine Problem 4: Test Suites contain scenarios using more than one instance of the product
  7. 7. © Daitan Group – Private & Confidential – v1.1 7 • Parallel Plans: Installation Tests, Memory Tests, and Test Cases using different sessions. • Sequential Plans: Account Limitations, Database restriction, and Sessions Tested individually Problem 5: Some complex test suites cannot run in parallel in different clients
  8. 8. © Daitan Group – Private & Confidential – v1.1 8 • In order to solve all those problems, we need a mechanism to automate and orchestrate everything • Due to the high quantity of environment combinations, the complexity of organizing and executing tests increases and can lead to an ineffective pipeline. Problem 6: Orchestrate
  9. 9. © Daitan Group – Private & Confidential – v1.1 9 Challenge • How to coordinate and schedule tests in an easy way? • How to continue delivering the tests in an agile way, despite those problems? • How creating and working with a solution for those problems could be simple?
  10. 10. Constructing the Solution
  11. 11. © Daitan Group – Private & Confidential – v1.1 11 Jmeter • The most famous free tool for performance testing against API • It could be used for regression testing against API as well • We could create test plan, steps from a test and Threads to collect the result of the API Calls • We can also integrate Jmeter with external libs, since we can use Java or Javascript inside JMeter • It is possible to send requests to more than one machine or server
  12. 12. © Daitan Group – Private & Confidential – v1.1 12 Instead of using JUnit or TestNG… ... We just need to create a server with the libraries used for test/task automation (Selenium, Sikuli, White, etc) It does not require a specific language (we use Java for this server, but a server built with the libraries for automation could be created using Python, C#, etc)
  13. 13. © Daitan Group – Private & Confidential – v1.1 13 We created our QA-Server! • Java + Maven • Spring Framework (dependency injection) • Apache CXF (our base for the server, btw) • External Libraries
  14. 14. © Daitan Group – Private & Confidential – v1.1 14 How QA-Server works 0. HTTP Client send a HTTP request to QA- Server; 1. Controller receives the request HTTP to execute some action or to check something; 2. Controller send the data to the proper Service, based on the request received; 3. Service chooses the proper Project to initiate some action or verification; 4. Project does the action or verification and returns the result to the Service; 5. Service returns the Result to the Controller; 6. Controller returns the Result to HTTP Client Controllers QA-SERVER Services HTTP Client Controller Service Project A
  15. 15. © Daitan Group – Private & Confidential – v1.1 15 Example: JMeter Side Jmeter Plan HTTP Requests
  16. 16. © Daitan Group – Private & Confidential – v1.1 16 Example: QA-Server Controller
  17. 17. © Daitan Group – Private & Confidential – v1.1 17 Example: QA-Server Service
  18. 18. © Daitan Group – Private & Confidential – v1.1 18 Example: Devopsdays Project
  19. 19. © Daitan Group – Private & Confidential – v1.1 19 QA-Server + Jmeter: Demo
  20. 20. © Daitan Group – Private & Confidential – v1.1 20 QA-Server + Jmeter: Another Example
  21. 21. © Daitan Group – Private & Confidential – v1.1 21 Remaining Problems: Problem 5: Support Sequential Plans Problem 6: Orchestrate to create an effective pipeline Problem 3: Several Instances of the Client Problem 4: One Instance per Machine JMeter coordinates what QA-Server will do on each machine being used by the test plan. Problem 1: Support all OS. Problem 2: Support all browsers. QA-Server on each OS, pointing to the project, libraries and frameworks. Partial Solution
  22. 22. © Daitan Group – Private & Confidential – v1.1 22 Solving the 2 problems left • We have created an automated orchestration of our complex test suites that delivers the effectiveness needed in Devops culture • Now, we can coordinate and schedule tests in an easy way, implement changes in parameters, and cover the maximum number of machines
  23. 23. © Daitan Group – Private & Confidential – v1.1 23 QA-Planner We have developed a tool called QA-Planner. FrontEnd: AngularJS + Javascript BackEnd: QA-Server / Java DB: MongoDB
  24. 24. © Daitan Group – Private & Confidential – v1.1 24 What is QA-Planner? • Coordinate and schedule tests in an easy way, in different machines, respecting the dependency of each test set (Sequential Plans, for instance) • Verify if the automation tasks are being done properly on each machine • Return an automated effective pipeline of the Test Suite The idea came from the following concepts:
  25. 25. © Daitan Group – Private & Confidential – v1.1 25 QA-Planner: Scheduler Algorithm The Algorithm needs to deal with these restrictions: • Sequential Test Plans must not run in parallel • We need to be able to choose an order for each Test Plan • It must be accessible/runnable by external tools, such as Jenkins • Everyone on the Devops team must be able to create or access the Tool in realtime
  26. 26. © Daitan Group – Private & Confidential – v1.1 26 QA-Planner: JMeter Plans Orchestrator • QA-Planner controls which plans will run and when • Once that pipeline starts to run, the orchestrator will be processing all plans and it will keep monitoring JMeter processes • It also changes the variables inserted into JMeter on execution time
  27. 27. © Daitan Group – Private & Confidential – v1.1 27 How QA-Planner works 0. User selects the plans and machines 1. User requests a pipeline 2. BackEnd uses the Scheduler Algorithm to create the pipeline 3. User receives a pipeline candidate 4. • A) User accepts the pipeline and the tests begin. • B) User accepts the pipeline and take the id to put on other tool • C) User rejects the pipeline. User can change the plans or machines selected before to create a new pipeline. Node A Node B Node C Node E Node D Planner Page QA-Server Leader Node
  28. 28. © Daitan Group – Private & Confidential – v1.1 28 QA-Planner: Page
  29. 29. © Daitan Group – Private & Confidential – v1.1 29 QA-Planner: Page
  30. 30. © Daitan Group – Private & Confidential – v1.1 30 QA-Planner: Page
  31. 31. © Daitan Group – Private & Confidential – v1.1 31 QA-Planner: Page
  32. 32. © Daitan Group – Private & Confidential – v1.1 32 QA-Planner: Roles Leader: Machine under testing Aux: One or more machines used to make all scenarios “testable” by automated tests Ex.: Leader is the Game running on Windows 8, others 3 machines (don’t care about OS) will join the Game as well, but the focus here is testing all features on Windows 8
  33. 33. © Daitan Group – Private & Confidential – v1.1 33 QA-Planner: Demo – Full Test Suite Pipeline
  34. 34. © Daitan Group – Private & Confidential – v1.1 34 QA-Planner: Demo If each test plan takes 10 minutes to be done. 5 scenarios * 13 machines = Traditional pipeline: 650 minutes! QA-Planner: 650/5 = 130 minutes!
  35. 35. © Daitan Group – Private & Confidential – v1.1 35 QA-Planner: Demo – Demo Test Suite Pipeline
  36. 36. © Daitan Group – Private & Confidential – v1.1 36 Problem 5: Support Sequential Plans Problem 6: Orchestrate to create an effective pipeline QA-Planner coordinates and orchestrates everything! Problem 3: Several Instances of the Client Problem 4: One Instance per Machine JMeter coordinates what QA-Server will do on each machine being used by the test plan. Problem 1: Support all OS. Problem 2: Support all browsers. QA-Server on each OS, pointing to the project, libraries and frameworks. Solved!
  37. 37. © Daitan Group – Private & Confidential – v1.1 37 Conclusion • This solution saved a lot of time • It’s really easy to create new tests and reuse the steps by using HTTP calls on JMeter • It’s not just solving problems, it is a new pattern followed here
  38. 38. Thank You! kelvsar@gmail.com / ksilva@daitangroup.com

×