This talk was presented in DevOpsDays Sao Paulo (June/2018) and in Daitan DevWeek (July/2018). What should you do if your product must be tested on multiple operating systems before delivering? And what if it must be tested on different browsers? Now it is a lot of combinations. And how to do this in an agile way? Here is a talk about how we can orchestrate automated tests on different systems, on different browsers. Assuring the coverage on all operating systems and browsers supported by a product is one of the toughest tasks commonly faced when transitioning to a DevOps setup. Due to the high quantity of environment combinations, the complexity of organizing and executing tests increases and can lead to an ineffective pipeline. In this talk, we will evaluate an automated orchestration of complex test suites that delivers the effectiveness needed for proper DevOps.

1. Orchestrating a big automated testing pipeline in
multiple environments

2. © Daitan Group – Private & Confidential – v1.1 2
About me
• 9 years experience in Software
Engineering, and 2 in Electronics
• BSc in Computer Science – IC-Unicamp
• Electronics Technician – Cotuca-Unicamp
• Currently working on Data Engineering,
Performance Engineering and Test
Automation at Daitan Group
• I love developing and testing!
kelvsar@gmail.com
https://www.linkedin.com/in/kelvin-ksps/
Kelvin Ronny Marques da Silva

3. CHALLENGE

4. © Daitan Group – Private & Confidential – v1.1 4
• Mac OS 10.11
• Mac OS 10.12
• Mac OS 10.13
• 13 Environments
Problem 1: Ensure the product runs well on all
supported operating systems
• Windows 7 x64
• Windows 7 x86
• Windows 8 x64
• Windows 8 x86
• Windows 8.1 x64
• Windows 8.1 x64
• Windows 10 x64
• Windows 10 x86

5. © Daitan Group – Private & Confidential – v1.1 5
• Mac OS 10.11
• Mac OS 10.12
• Mac OS 10.13
• Chrome
• Firefox
• Internet Explorer
• Safari
13 * 3 = 39 setups!
Problem 2: Ensure the product runs well on all
supported operating systems and browsers
• Windows 7 x64
• Windows 7 x86
• Windows 8 x64
• Windows 8 x86
• Windows 8.1 x64
• Windows 8.1 x86
• Windows 10 x64
• Windows 10 x86

6. © Daitan Group – Private & Confidential – v1.1 6
Ex.:
• 1 machine is the host and 2 machines are guests of a
video conferencing meeting
• 2 or more machines are playing a Game
• 1 machine is the presenter and 2 are the participants
on a webinar
Problem 3: Product only runs one instance per
machine
Problem 4: Test Suites contain scenarios using more
than one instance of the product

7. © Daitan Group – Private & Confidential – v1.1 7
• Parallel Plans:
Installation Tests, Memory Tests, and Test Cases using
different sessions.
• Sequential Plans:
Account Limitations, Database restriction, and
Sessions Tested individually
Problem 5: Some complex test suites cannot run in
parallel in different clients

8. © Daitan Group – Private & Confidential – v1.1 8
• In order to solve all those problems, we need a
mechanism to automate and orchestrate everything
• Due to the high quantity of environment
combinations, the complexity of organizing and
executing tests increases and can lead to an
ineffective pipeline.
Problem 6: Orchestrate

9. © Daitan Group – Private & Confidential – v1.1 9
Challenge
• How to coordinate and schedule tests in an easy
way?
• How to continue delivering the tests in an agile way,
despite those problems?
• How creating and working with a solution for those
problems could be simple?

10. Constructing the Solution

11. © Daitan Group – Private & Confidential – v1.1 11
Jmeter
• The most famous free tool for performance testing
against API
• It could be used for regression testing against API as
well
• We could create test plan, steps from a test and
Threads to collect the result of the API Calls
• We can also integrate Jmeter with external libs, since
we can use Java or Javascript inside JMeter
• It is possible to send requests to more than one
machine or server

12. © Daitan Group – Private & Confidential – v1.1 12
Instead of using JUnit or TestNG…
... We just need to create a server with the libraries
used for test/task automation (Selenium, Sikuli, White,
etc)
It does not require a specific language (we use Java for
this server, but a server built with the libraries for
automation could be created using Python, C#, etc)

13. © Daitan Group – Private & Confidential – v1.1 13
We created our QA-Server!
• Java + Maven
• Spring Framework (dependency injection)
• Apache CXF (our base for the server, btw)
• External Libraries

14. © Daitan Group – Private & Confidential – v1.1 14
How QA-Server works
0. HTTP Client send a HTTP request to QA-
Server;
1. Controller receives the request HTTP to
execute some action or to check
something;
2. Controller send the data to the proper
Service, based on the request received;
3. Service chooses the proper Project to
initiate some action or verification;
4. Project does the action or verification
and returns the result to the Service;
5. Service returns the Result to the
Controller;
6. Controller returns the Result to HTTP
Client
Controllers
QA-SERVER
Services
HTTP
Client
Controller
Service
Project
A

15. © Daitan Group – Private & Confidential – v1.1 15
Example: JMeter Side
Jmeter Plan HTTP Requests

16. © Daitan Group – Private & Confidential – v1.1 16
Example: QA-Server
Controller

17. © Daitan Group – Private & Confidential – v1.1 17
Example: QA-Server
Service

18. © Daitan Group – Private & Confidential – v1.1 18
Example: Devopsdays Project

19. © Daitan Group – Private & Confidential – v1.1 19
QA-Server + Jmeter: Demo

20. © Daitan Group – Private & Confidential – v1.1 20
QA-Server + Jmeter: Another Example

21. © Daitan Group – Private & Confidential – v1.1 21
Remaining Problems:
Problem 5: Support Sequential Plans
Problem 6: Orchestrate to create an effective pipeline
Problem 3: Several Instances of the Client
Problem 4: One Instance per Machine
JMeter coordinates what QA-Server will do on each machine being
used by the test plan.
Problem 1: Support all OS.
Problem 2: Support all browsers.
QA-Server on each OS, pointing to the project, libraries and frameworks.
Partial Solution

22. © Daitan Group – Private & Confidential – v1.1 22
Solving the 2 problems left
• We have created an automated orchestration
of our complex test suites that delivers the
effectiveness needed in Devops culture
• Now, we can coordinate and schedule tests in
an easy way, implement changes in
parameters, and cover the maximum number
of machines

23. © Daitan Group – Private & Confidential – v1.1 23
QA-Planner
We have developed a tool called QA-Planner.
FrontEnd: AngularJS + Javascript
BackEnd: QA-Server / Java
DB: MongoDB

24. © Daitan Group – Private & Confidential – v1.1 24
What is QA-Planner?
• Coordinate and schedule tests in an easy
way, in different machines, respecting the
dependency of each test set (Sequential
Plans, for instance)
• Verify if the automation tasks are being
done properly on each machine
• Return an automated effective pipeline of
the Test Suite
The idea came from the following concepts:

25. © Daitan Group – Private & Confidential – v1.1 25
QA-Planner: Scheduler Algorithm
The Algorithm needs to deal with these restrictions:
• Sequential Test Plans must not run in parallel
• We need to be able to choose an order for each Test
Plan
• It must be accessible/runnable by external tools,
such as Jenkins
• Everyone on the Devops team must be able to create
or access the Tool in realtime

26. © Daitan Group – Private & Confidential – v1.1 26
QA-Planner: JMeter Plans Orchestrator
• QA-Planner controls which plans will run and when
• Once that pipeline starts to run, the orchestrator will
be processing all plans and it will keep monitoring
JMeter processes
• It also changes the variables inserted into JMeter on
execution time

27. © Daitan Group – Private & Confidential – v1.1 27
How QA-Planner works
0. User selects the plans and machines
1. User requests a pipeline
2. BackEnd uses the Scheduler Algorithm
to create the pipeline
3. User receives a pipeline candidate
4.
• A) User accepts the pipeline and the
tests begin.
• B) User accepts the pipeline and take the
id to put on other tool
• C) User rejects the pipeline. User can
change the plans or machines selected
before to create a new pipeline.
Node
A
Node
B
Node
C
Node
E
Node
D
Planner
Page
QA-Server
Leader
Node

28. © Daitan Group – Private & Confidential – v1.1 28
QA-Planner: Page

29. © Daitan Group – Private & Confidential – v1.1 29
QA-Planner: Page

30. © Daitan Group – Private & Confidential – v1.1 30
QA-Planner: Page

31. © Daitan Group – Private & Confidential – v1.1 31
QA-Planner: Page

32. © Daitan Group – Private & Confidential – v1.1 32
QA-Planner: Roles
Leader: Machine under testing
Aux: One or more machines used to make all
scenarios “testable” by automated tests
Ex.: Leader is the Game running on Windows 8,
others 3 machines (don’t care about OS) will join the
Game as well, but the focus here is testing all features
on Windows 8

33. © Daitan Group – Private & Confidential – v1.1 33
QA-Planner: Demo – Full Test Suite Pipeline

34. © Daitan Group – Private & Confidential – v1.1 34
QA-Planner: Demo
If each test plan takes 10 minutes to be done.
5 scenarios * 13 machines =
Traditional pipeline: 650 minutes!
QA-Planner: 650/5 = 130 minutes!

35. © Daitan Group – Private & Confidential – v1.1 35
QA-Planner: Demo – Demo Test Suite Pipeline

36. © Daitan Group – Private & Confidential – v1.1 36
Problem 5: Support Sequential Plans
Problem 6: Orchestrate to create an effective pipeline
QA-Planner coordinates and orchestrates everything!
Problem 3: Several Instances of the Client
Problem 4: One Instance per Machine
JMeter coordinates what QA-Server will do on each machine being
used by the test plan.
Problem 1: Support all OS.
Problem 2: Support all browsers.
QA-Server on each OS, pointing to the project, libraries and frameworks.
Solved!

37. © Daitan Group – Private & Confidential – v1.1 37
Conclusion
• This solution saved a lot of time
• It’s really easy to create new tests and reuse the
steps by using HTTP calls on JMeter
• It’s not just solving problems, it is a new pattern
followed here

38. Thank You!
kelvsar@gmail.com / ksilva@daitangroup.com