This document discusses patterns for improving the reliability of large-scale Selenium tests. It recommends using techniques such as multi-tier deployments with rollout criteria to minimize risk when modifying test code, implementing fallbacks and load balancing for single points of failure, using circuit breakers to prevent unnecessary requests, and distributing tests across a cluster of virtual machines with a cluster manager. It also demonstrates using a library called Ephemerals to programmatically launch short-lived testing environments across a container cluster.

1. Reliability Patterns
for Large-scale
Selenium Tests

2. Waseem Hamshawi
Senior Automation Engineer, LivePerson
waseemh@liveperson.com
github.com/waseemh
linkedin.com/in/waseem-hamshawi

3. Liveperson transforms the connection between
brands and consumers
200BN
API calls
per month
7
Data
Centers
(worldwide)
6000
Virtual
Servers
3BN
Visits
Per month

4. 25
Scenarios
350,000
Executions
per day
7
Data
Centers
(Geo-distributed)
70
Virtual
Machines
SYML
LO
AMS
VA
CA
Distributed Testing

5. Scalability vs Reliability

6. ● Scale goes up → More deployments
● Minimize risk when modifying tests codebase
● Apply software deployment principals to test automation
Reliable Deployments
Pre-Staging Staging Production
(SSOT)
Failure rate > 3 % Failure rate > 3 %
Failure rate > 3 %
Rollout Rollout
● Multi-tier phased deployment
● Rollout criteria: failure rate < x% for [0,t] time range
● Feature Flags

7. ● Rule of Thumb: External access points and APIs will fail
● Test code should be tolerance to external failures
● Provide Fallbacks to every access point
getRemoteWebDriver()
Primary
Grid
Secondary
Grid
getFallback()
If primary fail
Single Point of Failures
● Selenium Grid - Single point of failure
Secondary Hub/Grid failover

8. Single Point of Failures
getRemoteWebDriver()
Grid 1
Grid 2 new RemoteWebDriver()
getFallback()
If primary or secondary fail
Grid Load Balancing
Grid 3
Grid N
.
.
Load Balancer
● Breakdown large Grids into smaller ones
● Redundant, highly available Grid setup
● Monitor and load balance
● Open source LB: Ribbon (github.com/Netflix/ribbon)

9. Circuit Breakers
● Don’t beat a dead horse
● Prevent unnecessary timed-out requests
● Shed load on external access point
● Failure Handling Open source libraries:
○ Hystrix (github.com/Netflix/Hystrix)
○ Failsafe (github.com/jhalterman/failsafe)
Single Point of Failures
OPENCLOSED
HALF OPEN
fail [threshold reached]
Open Circuit timeout
reached
fail
success
success
Fallback

10. Hystrix Command - Example

11. Test
Test
Cluster-wide Distribution
VM
VM
VM
VM
Z1-us_east
VM
VM
VM
VM
Z2-eu_west
VM
VM
VM
VM
Z3-asia
VM
VM
VM
VM
Z4-us_west
VM
VM
Z5-eu_east
VM
VMTest
Test
Test
Test
Test
Test
Test

12. Cluster-wide Distribution
● Launch short-lived Selenium Standalone Server containers
Test
initRemoteWebdriverSession(caps)
Selenium
Container
VM
Selenium
Container
VM
Selenium
Container
VM
Selenium
Container
VM
new RemoteWebDriver(External Endpoint)
Cluster
Manager
● Deploy Selenium containers on-demand
● Let Cluster Manager do the distribution
● Scale is dependent only on cluster’s capacity
● Initialize RemoteWebDriver based on deployment endpoint (IP:Port)

13. A library for creating short-lived testing endpoints over container clusters
● Tests Integration - Programmatically launch test environment
● Plugable Cluster Management Systems
● Cluster-based Scaling
Ephemerals
Test
Container
VM
Cluster
Manager
Ephemerals
API
Container
VM
Container
VM
Container
VM
Deploy
External Endpointnew Object(External Endpoint)
LivePersonInc/ephem
erals

14. Container Orchestration using Google Container Engine (GKE) and Kubernetes
● Healthchecks (Liveness/Readiness Probes)
● Self-healing mechanisms
● Autoscaling
● Resource Monitoring
Use Case: GCP, Kubernetes and Docker
Test
Ephemerals
API
Deploy
External IP:4444RemoteWebDriver(External IP:4444)
getRemoteWebDriver
(capabilities)

15. @Rule
public EphemeralResource<RemoteWebDriver> seleniumResource =
new EphemeralResource(
new SeleniumEphemeral.Builder(deploymentContext)
.withDesiredCapabilities(FIREFOX_42)
.build());
@Test
public void test() {
RemoteWebDriver remoteWebDriver = seleniumResource.get();
.....
}
Ephemerals - JUnit Integration

16. DEMO

17. github.com/LivePersonInc/ephemerals

18. ● Deploy safely, Just like your software
● No Assumptions - Failure Awareness
● Avoid or protect SPOFs
● Containerization - Short lived, throwaway instances
● Cluster-based Orchestration
● Cloud is the Future - also for Testing
Takeaways