This document discusses using serverless architecture for test automation. It begins with an overview of serverless computing and its benefits like instant scaling, low deployment complexity, and pay-per-use model. It then explains how serverless is well-suited for test automation as it allows tests to run independently, quickly, and be executed frequently for continuous testing. However, challenges like monitoring, debugging and vendor lock-in need to be addressed. Example test types like API, performance, UI and unit tests are discussed. Open-source tools for serverless test automation are also mentioned.

1. SERVERLESSI N T E S T A U T O M AT I O N
2 0 1 8 O l e k s a n d r K h o t e m s k y i h t t p : / / x o t a b u 4 . g i t h u b . i o

2. Independent Contractor,
Software Developer Engineer in Test
Hello!
I Am Oleksandr Khotemskyi
Contacts: xotabu4.github.io

3. Test automation meets serverless
What is serverless?
Serverless architecture highlights
New world order?

4. Evolution of software architecture principles and programming styles
What is serverless?
1

5. Evolution of servers: Bare Metal

6. Evolution of servers: Virtualization

7. Evolution of servers: Containers

8. Evolution of servers: Serverless

10. You will fall in love or hate forever
Serverless architecture highlights
2

11. In a nutshell
• Code snippets
• Deployed to cloud
• Executed in response to some event
• Scaled on demand
• Costs nothing when idle

12. Serverless is using servers!
Load
Balancer
Function
invocations
requests
Nodes
Controller

13. Your code runs
according to
your logic
You will get
execution result
in HTTP
response
Invoke. For example -
just by sending HTTP
request
Upload to your
FaaS provider
Usage flow
Wrap your code
to special
“handler”

14. You don’t have single point of
failure. In case errors - only one
invocation is affected
Stability
Automatically start thousands and
millions of your code copies
Instant scaling
Just zip your code and upload. No
deploying to thousands of servers
with complex update process
Low deploy complexity
You don’t need to think about
hardware. Provider will automatically
add/remove machines for you
No servers managing
You do not pay for your idle
computer
No underconsumption
No race conditions, no multithreading.
Function is small piece of code that easy
to read. Developers are focusing on
business requirements
Simple developing
Benefits

15. Old solutions does not work
anymore. And again - lock to
vendor
Monitoring
There might be delay in
starting fresh deployed code
Warming up
Requires new approaches in
setting up secure backend
Security
Harder to debug, since your
code runs quickly, and not
clear where exactly
Debug
All communication is done
thru network (HTTP), which
is not the fastest way
Latency
Usually migrating from one
FaaS provider to another is
hard
Vendor lock
Tricky parts

16. AWS Lambda PRICE
• You pay only when your code is running
• Pricing calculated for megabytes/second
• Huge memory consumption - more expensive
• Long running code - more expensive
• Services usually offer free amount (AWS - 1 million
calls)
Table 1
Memory
(MB)
Free tier
seconds per
month
Price per 100ms
($)
128 3,200,000 0.000000208
512 800,000 0.000000834
1024 400,000 0.000001667
2048 200,000 0.000003334
3008 136,170 0.000004897

17. Best friends forever
Test automation meets serverless
3

18. Tests should be independent
one from another
Serverless architecture is absolutely stateless by
its nature. As your tests also should be. Each your
test runs in separate isolated environment.
Potentially even pre/postconditions can be isolated

19. Tests should run as quickly
as possible
• Each your single test CPU/RAM consumption won’t affect others
because of isolation (noticeable on huge test numbers)
• You can potentially have thousands and millions of threads. Your
test execution will take time of longest running test

20. Tests should be stable
• Serverless allows to avoid running code locally
• Different code versions can be used
• Retrying test - is just recursive invoking function by itself
• Clean runtime environment for each test gives additional stability
• Executor issues does not affect whole test run

21. Tests should be executed as
often as possible
• No limits on tests execution frequency. Single execution is cheap
• With unlimited parallelization thats possible to start tests for even
smallest code changes. Or even for many changes simultaneously
• Possible to run tests for multiple environments in the same time
• Running tests while developing won’t block your execution

22. Bright future of test automation. But even sun has sunspots
New world order?
3

23. Reporting on huge scale - Report Portal

24. Allure

25. Test runners
• Existing test runners were not intended for this tricks
• Possible execution flow can be as follows
Sending link to
report as final step
Tests are running,
and reporting
results to “Report
Portal”
Multiple recursive
invoke with test
names that will be
used for GREP
Invoke function-starter
that parses all tests, and
detects tests that should
be run

26. API tests
Small size of project
Simple environment
Test runner should be updated
This is looks like best way to start
using serverless for tests
Quick start, fast execution
!
Should not require special config

27. Performance tests as a bonus
• With theoretically unlimited test
threads, we can do huge load to Site-
Under-Test
• This can be used to see application
performance issues
• Bonus - reusing UI or API tests (or
both) for performance measurements
FaaS
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test
Test Test Test
Test Test Test
Test Test Test
Test Test Test
Test
Test
Test
Test
Test
Test
Test Test Test

28. UI tests
Size of project can be big
Complicated environment
Test runner should be updated
Notice that some cloud providers might have function execution
timeout! Your test cannot run longer than it
Slow start, slow execution
!
Requires separate test farm cluster OR
start Chrome Headless inside function itself
!
!
!
!

29. Unit tests
Size of project usually big
Simple environment
Test runner should be updated
Potentially HTTP requests latency might be bigger
than test execution itself. Grouping by bigger chunks
help
Fast start, fast execution
!
Does not require some special config
!

30. More to read
Functional programming: https://habr.com/post/142351/
Code with lambda-selenium usage (Java, Node): https://github.com/blackboard/lambda-selenium
Framework for building serverless applications: https://serverless.com/framework/
WebdriverIO lambda runner: https://github.com/webdriverio/webdriverio/tree/master/packages/wdio-lambda-
runner
Old, good, wikipedia: https://en.wikipedia.org/wiki/Serverless_computing
Martin Fowler about serverless: https://martinfowler.com/articles/serverless.html

31. Thanks!
You can find me at: xotabu4.github.io
Any questions?