This document summarizes a presentation on secrets of successful test automation. It discusses scheduling automated tests to run every night for 10-20 minutes and limiting resources to 1GB RAM, no persistent storage, and 10MB network bandwidth. Standard reports on endpoints, response times, requests, and errors are shown. Measuring UI performance includes page load time with different networks, response traffic size, request counts, and centralized monitoring. Tools mentioned include Selenium Grid, Selenoid, and Grafana for scaling acceptance tests across browsers and versions.

1. Secrets of
Successful Test Automation
Narek Arushanov - Ashot Karapetyan
Barcamp 2018
Synergy International Systems

2. Agenda
Backend Performance
UI Performance
Scaling Acceptance Test

3. Requirements
1. Periodicity
2. Resource Allocation.
3. Summary Reports
4. Centralized Monitoring.

4. Scheduled Execution
Every night
100 rps
10-20 min

5. Limits
1 GB RAM
No Pers.
Storage
10Mb
network
1 core

6. Tools

7. Percentiles

8. Standard Reports
Service endpoint Scale rps
95p in
millis
Total requests
count
# of Errors
ml LoadAllLanguages 1 100 19 59547 0
LoadLanguagesById 1 100 29 59548 0
LoadMessagesByKeys 1 100 92 59547 0
LoadMessagesByLanguage
Id 1 100 13 59548 0

9. Fail

10. OK

11. Centralized Monitoring

12. Measuring UI Performance
1. Page load time with different network speed
2. Response traffic size
3. Requests count
4. Automated
5. Centralized monitoring

13. UI performance tools
1. sitespeed.io
2. Navigation Timing API
3. BrowserMob Proxy + Selenium

14. HAR file structure

15. {HAR} {HAR}{HAR}
Graphite Database
Grafana

16. Page Load Time

17. Request Count

18. Scaling Acceptance Test

19. Scaling Acceptance Tests
1. Parallel Test Execution
2. Multi-browser
3. Browser Versions

20. Ways to test in browsers
1. Cloud web testing platform
BrowserStack Sauce Labs Testing Bot

21. Ways to test in browsers
1. Cloud web testing platform
2. Selenium Grid

22. Problems
1. Hub is crushing.
2. Different browser versions.
3. Temp folders
4. Window focus
5. etc …

23. Ways to test in browsers
1. Cloud web testing platform
2. Selenium Grid
3. Selenoid

24. How to work solenoid

25. How to work solenoid

26. Selenium Grid UI

27. Selenoid UI

28. Advantages
1. Stability
2. Tests run in isolated container
3. Run tests in different browser versions
4. Tmpfs support
5. Hot configuration reload

29. Resources control
1. Limit total number of sessions per host
2. Queues
3. Limit CPUs per container
4. Limit RAM per container

30. Cluster Solution

31. Cheers!
www.synisys.com