This document discusses distributed automation (DA) for running UI tests across scaled cloud infrastructure using Selenium Grid Scaler on AWS. It covers: setting up Selenium Grid hub and nodes on AWS, different grid topologies for parallel testing, making the grid stable through timeouts and resources tuning, cost savings through autoscaling and large instance types, and a DA dashboard for monitoring tests. The goal of DA is to run hundreds of tests within the time of the slowest test case through parallelization and scalability.

1. DISTRIBUTED AUTOMATION
SELENIUM-GRID-SCALER / AWS
Ragavan Ambighananthan
@ragsambi
Expedia
London Test Automation In Devops Meetup - 2018
1
AKA ‘RUNNING TESTS WITHIN THE TIME TAKEN BY THE SLOWEST TEST CASE’
using

2. AGENDA
• Before DA and after DA
• Setting up in AWS
• Making the Grid stable
• Grid topologies
• Cost saving
• DA Dashboard
2

3. WHAT DO I GET?
3
• DA = SeleniumGridScaler (Selenium Grid + EC2) + AWS
• How we were before we had DA
• How DA will help to
• Achieve faster feedback cycle
• Move away from real hardware/VM in data center to
cloud
• Cost effective, scalable and reliable
• Enables Continuous Integration / Continuous
Deployment

4. PROBLEM DESCRIPTION
4
TOO MANY UI TESTS

5. PROBLEM DESCRIPTION
5
• Hundreds of Jenkins jobs to run all the tests
• Not having a scalable system to run
hundreds of UI automation tests reliably, fast
and in a cost effective way
• No intelligent automation report to narrow
down failures quickly!

6. SOLUTION
• To be able to run all UI automation
scenarios within the time taken by the
slowest test case
• Cost effective, scalable and reliable
• Effective dashboard
6

7. BEFORE DA
7

8. 8
SETTING UP
BIG PICTURE - BEFORE DA

9. 9
SETTING UP
BIG PICTURE - AFTER DA
c5.largec5.9xlarge

10. AFTER DA
10

11. TECHNOLOGIES / TOOLS USED
11
SELENIUM GRIDSCALER
Expcuke / ScalaTest / Nightwatch / Java mongo writer

12. SETTING UP
• Cucumber allows to run a scenario with the following
syntax
• sample_featurefile.feature:12
• For Scenario Outline, the line number would be
that of the line from the example table
line no 12 Scenario: eat 5 out of 12
13 Given there are 12 cucumbers
14 When I eat 5 cucumbers
15 Then I should have 7 cucumbers
12
CUCUMBER SCENARIO GENERATION

13. SETTING UP
checkout/lx:
features/lx_fraud.feature:21:en_US
features/lx_fraud.feature:47:en_US
features/lx_responsive_design.feature:25:en_US
features/lx_responsive_design.feature:26:en_US
features/lx_responsive_design.feature:27:en_US
features/lx_responsive_design.feature:90:en_US
features/lx_responsive_design.feature:240:en_US
search_landing_pages/flights_tg:
features/tg_flights_revamp_hero_image.feature:120:en_US
features/tg_flights_revamp_social_sharing.feature:156:en_US
features/tg_flights_revamp_search_wizard.feature:202:en_US
features/tg_flights_revamp_search_wizard.feature:203:nl_NL
features/tg_flights_revamp_top_destinations.feature:159:en_US
features/tg_flights_revamp_top_destinations.feature:160:en_US
features/tg_flights_revamp_top_destinations.feature:161:en_US
features/tg_flights_revamp_top_destinations.feature:207:en_US
• Only scenarios that matches @stubbed | @live and @acceptance |
@regression will be included in the list to run
• All these tests will be executed concurrently
13
SAMPLE GENERATED SCENARIOS

14. SETTING UP
14
SAMPLE GENERATED SCENARIOS
• Parallelize your tests
• ScalaTest
• mvn test --projects tests/partner_template -
Denvironment=LIVE -Dbrowser=grid -
Dgrid_browser=chrome -Dparallel=true -
Dgrid_server=http://1.1.1.1:4444/wd/hub
•./gradlew -PnumBrowsers=150 :loyalty.ui:scalaAcceptance -i -
Denvironment=JENKINS_STUBBED -Dbrowser=Grid
•NightwatchJS
•To maximise speed, one test in a .js file

15. SETTING UP
15
SELENIUM GRID HUB SETUP
• c5.large (2 cpu / 4 GB RAM ) for fewer
hundreds of tests
• Hub should have enough network bandwidth
but low CPU / Memory is fine
• Hub is created from an AMI with bootstrap
SeleniumGridScaler jar, which will act as the
hub that can autoscale

16. SETTING UP
• Open Source
• Acts as an intelligent hub
• Auto scales grid nodes depending on the number of
tests
• Adhoc launch of new nodes is also possible
• Terminates nodes when not in use / forced
termination
• Talks to AWS using EC2
16
SELENIUMGRIDSCALER - HUB

17. • c5.xlarge
• Capable of running maximum 15
Firefox/Chrome
• Node created out of AMI has bootstrap code
to help attach to the hub
17
SETTING UP
SELENIUM GRID NODE SETUP

18. SELENIUM NODE BOOTSTRAP
CODE
[root@ip-10-2-12-167 ~]# more /home/grid/grid/grid_start_node.sh
#!/bin/sh
PATH=/sbin:/usr/sbin:/bin:/usr/bin
cd /home/grid/grid
export EC2_INSTANCE_ID="`wget -q -O - http://169.254.169.254/latest/meta-data/instance-id || die "wget instance-id has
failed: $?"`"
# Pull down the user data, which will be a zip file containing necessary information
export NODE_TEMPLATE="/home/grid/grid/nodeConfigTemplate.json"
curl http://169.254.169.254/latest/user-data -o /home/grid/grid/data.zip
# Now, unzip the data downloaded from the userdata
unzip -o /home/grid/grid/data.zip -d /home/grid/ubuntu/grid
# Replace the instance ID in the node config file
sed "s/<INSTANCE_ID>/$EC2_INSTANCE_ID/g" $NODE_TEMPLATE > /home/grid/grid/nodeConfig.json
# Finally, run the java process in a window so browsers can run
xvfb-run --auto-servernum --server-args='-screen 0, 1600x1200x24' java -jar /home/grid/grid/selenium-server-node.jar -role
node -nodeConfig /home/grid/grid/nodeConfig.json -Dwebdriver.chrome.driver="/home/grid/grid/chromedriver" -log
/home/grid/grid/grid.log &
18

19. SETTING UP
• Hub creates the node based on a config:
AMI ID, subnet, security group, node type,
etc.
19
SELENIUMGRIDSCALER - NODE

20. SETTING UP
20
GENERAL AUTOMATION PIPELINE
• check and start the hub
• run automation concurrently
• terminate nodes and
• shutdown hub
3mins 6mins 15secs

21. */3 * * * * killall --older-than 10m firefox
*/3 * * * * killall --older-than 10m chrome
*/3 * * * * killall --older-than 10m chromedriver
*/3 * * * * killall --older-than 10m geckodriver
• Add a cronjob to kill any browser instances that is
running for more than 10mins in the node
MAKING THE GRID STABLE
TIMEOUTS
22

22. • Grid setup should be in the same AWS subnet
• Using multiple subnets will result in lots of
FORWARDING_TO_NODE_FAILED errors
MAKING THE GRID STABLE
AWS - SUBNET
23

23. • Subnet you are using should have enough free
IP addresses
• It will be a blocker for autoscaling the grid nodes
MAKING THE GRID STABLE
AWS - IP ADDRESS
24

24. • The webDriver object creation consumes bandwidth
in the range of 6Gbits/5min in the Hub for 250+ tests
in parallel
MAKING THE GRID STABLE
AWS - HUB BANDWIDTH
c5.large
25

25. • Fine tune your
• -Xms
• -Xmx
• -DPOOL_MAX
MAKING THE GRID STABLE
AWS - HUB / NODE MEMORY
26

26. • HUB becomes unstable after running thousands
of tests
• Automate stop/restarting of Hub after every
2000+ tests
MAKING THE GRID STABLE
AWS - STOP/RESTARTING HUB
27
HUB
c5.4xlarge
restart
hub

27. • Jenkins executor which would be running hundreds of
tests in parallel, needs to have enough CPU power.
MAKING THE GRID STABLE
AWS - JENKINS EXECUTOR CPU
c3.9xlarge when running 250+ tests in parallel
28

28. • Update browsers in the node and create a new
node AMI
MAKING THE GRID STABLE
UPDATE BROWSERS
30

29. MAKING THE GRID STABLE
SCALE THE TEST INFRASTRUCTURE
31

30. GRID TOPOLOGIES
• Different ways to create Grid Topologies
32

31. GRID TOPOLOGY - 1
HUB
• parallel execution for small projects
• 1 jenkins executor - 1 hub - 20 nodes (= 300 tests)
• Test run would finish in ~5mins
c5.4xlarge
c5.large
c5.xlarge
33
….

32. GRID TOPOLOGY - 2
HUB
• Suitable for medium size projects (500+ tests)
• Adding one more executor (2 executors 1 hub
and 33 node),this could double your parallel
execution cases, still taking only ~5mins
c5.4xlarge
c5.4xlarge
c5.xlarge
34
….
….

33. GRID TOPOLOGY - 3
HUB
• Takes 2x times as previous topology-2, but half
the cost! (1 executor - 1 hub - 17 nodes)
• Suitable for medium size projects
• Test run would finish in ~10mins
c5.4xlarge
c5.xlargejob runs sequentially
35
….
c5.2xlarge

34. GRID TOPOLOGY
HUB
• One more job? Probably NOT as HUB network traffic would
make it unstable especially during webDriver creation
c3.8xlarge
c3.8xlarge
c3.xlarge
36
….
….

35. GRID TOPOLOGY - 4
HUB
HUB
• Use two hubs to double
the tests (1000+)
• But speed is same as
topology 2 (~5mins)
• Double the cost
c5.4xlarge
c5.xlarge
37
c5.4xlarge
c5.4xlarge

36. DA FOR MOBILE WEB
38
HUB
c5.large

37. COST SAVING
39

38. AUTOSCALING OF GRID NODES
• EC2 instances are billed per sec whether we
use them or idling
• AWS gives us the ability to pay for only what
we use
• Terminate / stop resources if you are not using
it
• You should pay only for what you use
40
COST SAVING

39. CLOUD PROVIDERS VS DA
41
COST SAVING
Third party cross
browser providers
AWS
Parallel Connections 100 no limit
Cost Expensive Cheap
License Type Fixed Based on usage

40. AUTOSCALING OF GRID NODES
• SeleniumGridScaler autoscales the grid nodes
• It creates AWS nodes on demand based on the
number of tests to run
• Auto termination / force termination of nodes
• Hub can shut itself down
42
COST SAVING
curl --connect-timeout 10 --max-time 20 -X "DELETE" http://${IP_OF_HUB}:4444/grid/admin/AutomationTestRunServlet --stderr -)

41. 43
AUTOSCALING OF GRID NODES
COST SAVING
• http://x.x.x.x:4444/grid/admin/AutomationTestRunServlet?uuid=testRun1
&threadCount=250&browser=firefox”
• For 250 test cases, it will create 250/15 ~ 17 nodes
• It returns status codes
• 202 - request can be fulfilled by current capacity
• 201 - request can be fulfilled but AMI must be started to meet capacity
(wait for ~3mins)
• 409,400 codes

42. • c5.xlarge = $0.17 per Hour (can run 15 Firefox instances)
• t2.small = $0.023 per Hour
• Using one c5.xlarge costs half the equivalent number of
t2.small instances
Conclusion:
• Use c5.xlarge as it is more value add
• Saving 14 IP addresses for every 15 tests
But always this depends on your observation of your own setup!
44
LARGE VS SMALLER NODE TYPES

43. • When AWS release new instance
family, check the price and move to the
latest
• Eg: c3.large is expensive compared to
c5.large but c5.large is bit more
powerful.
45
USE LATEST AWS INSTANCE FAMILY
TYPE

44. REDUCING UI TESTS
46
MONITOR UI TEST WITH STRICT REVIEW
PROCESS

45. REDUCING UI TESTS
47
BREAK DOWN BIGGER SCENARIOS

46. REDUCING UI TESTS
• Create more unit / integration tests
• Categorize test cases appropriately
48

47. 49
TREND CHARTS
DA DASHBOARD

48. 50
TREND CHARTS
DA DASHBOARD

49. 51
POINT OF SALE GRID
DA DASHBOARD

50. 52
UNIQUE ERROR REPORT
DA DASHBOARD

51. 53
FAILURE HISTORY / ONE PAGE
DA DASHBOARD

52. 54
ONE TAB STATUS FOR ALL PODS
DA DASHBOARD

53. 55
NOTIFICATION
• DA Dashboard deep link is can be sent
via
• Slack
• Email
• using AWZ SNS

54. FEW WORDS
• Original
• https://github.com/mhardin/SeleniumGridScaler
• Few differences in Expedia specific SeleniumGrid Scaler
• https://ewegithub.sb.karmalab.net/EWE/seleniumgridscalerexp
• Ability run multiple hubs
• Running 15 FF or Chrome in 1 box instead of 1 browser per instance
• Hub can terminate all nodes and shutdown itself
• Hub can also terminate a node based on idle time and per-second billing cycle
• Most of these changes will be in original repo soon!
• Hub Dockerised!
56
Trying DA on Kubernetes! - in the next Meetup :)

55. REFERENCE
• Medium Expedia Engineering post
• https://medium.com/expedia-engineering/distributed-automation-how-
to-run-1000-ui-automation-tests-in-5mins-cf9a84ca32d1
• Selenium Conference / Camp Videos
• https://www.youtube.com/watch?v=cbIfU1fvGeo
• https://www.youtube.com/watch?v=dI9PzmiQc3A
57