Slides for a paper published at AST 2019: Shahnaz M. Shariff, Heng Li, Cor-Paul Bezemer, Ahmed E. Hassan, Thanh H. D. Nguyen, and Parminder Flora. 2019. Improving the testing efficiency of selenium-based load tests. In Proceedings of the 14th International Workshop on Automation of Software Test (AST '19). IEEE Press, Piscataway, NJ, USA, 14-20. DOI: https://doi.org/10.1109/AST.2019.00008

1. Improving the Testing Efficiency of
Selenium-based Load Tests
Thanh H.D. Nguyen, Parminder Flora
Shahnaz Shariff Ahmed E. HassanHeng Li Cor-Paul Bezemer

2. Load-related problems can have catastrophic
consequences
2

3. Load testing is used to determine a system’s
behavior under various load conditions
3
Concurrent
requests
System
Under Test
System
behavior

4. Usually load tests are conducted with
protocol-level tools (e.g., JMeter)
4
RequestsProtocol
level
requests
generator Feedback
System under test

5. Advantages of browser-based testing frameworks
(e.g., Selenium) over protocol-level testing
frameworks
5
Simulate user interactions with browsersUses protocol-level HTTP requests
Difficult to abstract complex behavior
(e.g., a secure login)
Cannot execute JavaScript in web
applications
Require dedicated load testing scripts Reuse functional testing scripts
Easily handle complex behavior (e.g., a
secure login)
Can execute JavaScript in web applications
Protocol-
level testing
Browser-based
testing

6. Traditionally, a browser is used to
generate the load for one user
6
System under test
Requests
Load driver

7. Selenium-based load tests are resource-heavy
7
Selenium-based load tests are resource-heavy
System under test
Requests
A normal load driver
can only simulate a
few users
Load driver

8. Selenium users are concerned about the performance
issues in Selenium-based load tests
8

9. We propose an approach to improve the testing
efficiency of Selenium-based load tests
9Load driver
Simulate more users
on a load driver
Load driver
Sharing browsers between
users

10. We use a webmail client (Roundcube) as our
application under test
10

11. We create test cases based on the MMB3
benchmark
11
● Reply email
● Reply all
● Send email
● Forward email
● Delete email
● View emails
● Browse contacts
● Permanently delete
MMB3 is a popular benchmark
used to measure the performance
of mail servers
MMB3 simulates a typical user’s
email actions during a working
day

12. In the MMB3 benchmark, 295 tasks are
scheduled to run in a 8-hour working day
12
We run 19 tasks in a 30-minute period
(reduced execution time & same intensity of tasks)
Send Email Delete Email Browse Contacts … Task# 295
Reply All View Emails Delete Email … Task# 295
Browse Contacts Reply … Task# 295Forward Email
8 hours

13. We run Selenium tests with
three different browser settings
13
Regular browser
E.g.; Google Chrome
Headless browser
A simplified browser
without a Graphical
User Interface
Xvfb
A regular browser
with its display
transferred to an in-
memory display

14. Traditionally, a browser is launched for
each user instance
14
T1: Send Email T2: Delete Email T3: Browse Contacts … Task#19
T1: Reply All T2: View Emails T3: Delete Email … Task#19
T1: Browse Contacts T2: Reply … Task#19T3: Forward Email
30 mins

15. We merge the tasks of all users to
form a common list of tasks
15

16. We use fewer browsers to simulate
the workload of more users
16

17. A pool of browsers parallelly run the tasks
from the common list
17
…
Pool of browsers
Randomly select a free
browser from the pool
Common list of tasks
…
Select the next task from
the common list
Run the selected task on
the selected browser
Release the selected
browser back to the pool
The process is running in
multi-threads
(the number of threads =
the number of browses)

18. We use error ratio, delay ratio and resource
usage to measure test efficiency
18
We use error ratio (0%) and delay ratio (<5%) to find the
maximum workload (i.e., # of error-free users)
that a load driver can generate
Delay Ratio
The ratio of tasks that
miss the scheduled
starting time
Error Ratio
The ratio of failed
tasks in a test run
Resource Usage:
CPU & Memory of
the testing processes

19. Headless browsers are much more efficient than
regular browsers
19
49
92
5
8

20. CPU & memory is significant even in
idle browser instances
20
Comparing the resource usage of idle and busy browsers for a period of 10 mins
95th Percentile Memory

21. With headless browsers, our approach
significantly reduces the system’s memory usage
21

22. With Xvfb browsers, our approach reduces the
system’s memory usage significantly
22

23. Our approach increases the number of
error-free users by at least 20%
23

24. Selenium-based load tests are resource-heavy
Selenium-based load tests are resource-heavy
System under test
Requests
A normal load driver can
only simulate a few users
Load driver

25. We propose an approach to improve the testing
efficiency of Selenium-based load tests
25Load driver
Simulate more users
on a load driver
Load driver
Sharing browsers between
users

26. We use a webmail client (Roundcube) as our
application under test

27. Our approach increases the number of
error-free users by at least 20%
27

28. Shahnaz M. Shariff
s.shariff@queensu.ca