1.
Ways of tests parallelization in
JavaScript
2019, Oleksandr Khotemskyi
xotabu4.github.io

2.
Independent Contractor,
Software Developer Engineer in Test
Hello!
I Am Oleksandr Khotemskyi
Contacts: xotabu4.github.io

3.
https://smplday.com/
13 July
http://start-it.ua/
9 April sold out
11 June
http://start-it.ua/
14 May

4.
Ways of tests parallelization in JavaScript
• Concurrency vs Parallelism
• Concurrent execution
• Parallelism by NodeJS processes
• Parallelism by worker threads
• Functions as a Service (Serverless)
• Isolates
• Comparing approaches

5.
Concurrency vs
Parallelism
Short introduction into
differences

6.
Concurrency
• Tasks are sharing single CPU and have
shared memory
• Task that reaches the queue earlier -
executes earlier (FIFO)
• Queue allows to avoid race conditions for
memory, but managing execution order is
on you
• Best fits to different kinds of event handling
systems (WebServers, UI …) CPU
Memory
Task
Task
Task
Task

7.
Parallelism
• Each task runs on separate CPU
• Usually task distribution is controlled with some
“master” process
• Managing execution flow can be easier, but you have to
deal with race conditions on memory
• Best fits for heavy computational tasks - you are
processing static data, without external events
(graphics, video, etc) CPU
1
Memory
TaskTask Task Task
CPU
2
CPU
3
CPU
4

8.
Concurrent test
execution
Using power of async code to
run multiple tests at once

9.
Running tests concurrently
• Test runner starts your tests as usual - sequentially one by
one
• When test execution reaches some async operation - test
runner does not wait for it to complete
• Instead, test runner starts next test, and again runs it until
first async operation
• So your tests “compete” for CPU time, and code
execution flow is actually jumps between async
operations CPU
Memory
Test 4
Test 3
Test 2
Test 1

10.
Pros
• CPU / RAM consumption is lowest
• Overhead costs are minimal
• Tests use shared memory
• No any infrastructure/os level setup
needed

11.
Cons
• Shared variables wont be synchronized
• Potentially might crash nodes process
• Harder to implement test timeouts,
concurrency limit
• Using a lot of synchronous code might
slowdown your tests
• Limited test runners support

12.
Parallelism by
NodeJS processes
Tests are shared between
processes

13.
How this works?
• Test runner gets absolute file paths with tests
• Test runner starts Worker Processes and
passes file paths to them
• Each Worker processes loads one file, and runs
test runner for it
• Worker processes return results back to parent
process
• Master process gives another file path to idle
Worker process
Master Process
67891011
CPU
1
1 2 3 4 5
Memory
CPU
2
CPU
3
CPU
4
CPU
5
CPU
6
Memory Memory Memory Memory
Memory

14.
+
mocha-parallel-tests

15.
Pros
• Can use all your CPUs
• Simplest existing way to run JS tests in
parallel
• Most of frameworks already support this
• Each file isolated from others, errors in
Worker process not affect Master process
• Can be combined with other methods

16.
Cons
• Consumes CPU/RAM the most
• On start - huge spike of CPU usage. Probably
launching processes in chunks might help
• Hard to share data between running processes
• Reporting can be harder than usual
• Existing implementations cannot parallel individual
tests, only whole test files (because of pre/post
conditions)

17.
Parallelism by
worker threads
Multithreading finally comes to
JavaScript

18.
Worker Threads
• Starting from nodejs 10.5 - behind the flag
node —experimental-worker . Or starting from
nodejs 11.7 - without flag
• Allows to run piece of code with own context,
stack, event loop
• No racing conditions because memory is not
shared (but there are low-level API for memory
sharing)
• Communication is done via Messages (events)
• Main idea is to allow to run your CPU
consuming blocking calculations in separate
thread

19.
NodeJS Worker Threads vs Worker Process
Threads Processes
Memory
Can be shared (ArrayBuffer, SharedArrayBuffer,
Atomics)
Cannot be shared
Communication
Messages system. Serialized data doesn’t
leave process. Much faster
Inner Process Communication (with JSON
serialization/deserialization). Much slower
CPU One CPU core per worker One CPU core per process
Startup Fast, less than 40ms Slower, more than 200ms
Isolation
Each thread with own event loop, stack,
garbage collection
Separate V8 instance for each process. Nothing
shared.

20.
Potential future scaling using IPC
• NodeJS has ability to communicate to
another process via Inner Process
Communication
• Also HTTP server can be used
• Node-IPC package provides support to
access other nodejs processes via Unix/
Windows sockets, TCP/TLS/UDP Sockets
• This allows to scale tests parallelization
horizontally
• There are existing frameworks using similar
approach in other languages (C# - Meissa)
Master
Server

21.
Existing implementations
• Test runner https://github.com/
AleksanderPopov/toundra
• Add-on for Jest - jest-worker
• Also, there are some experiments with using
worker threads with MochaJS
• Worker threads just came out from
‘experiment’ stage in nodejs. Expecting
more frameworks to support this in future

22.
Functions as a
service
Maybe infrastructure can help
us?

23.
FaaS

24.
Your code runs
according to
your logic
You will get
execution result
in HTTP
response
Invoke. For example -
just by sending HTTP
request
Upload to your
FaaS provider
Wrap your code
to special
“handler”
FaaS

25.
• Does not require to manage own hosts
• Autoscaling included
• Predictable and clean environment for each
test
• Each test isolated to own container
• Google Chrome can be launched inside
function - potential cost saving for test farm
Pros

26.
• Need to deploy code to FaaS
• Cold function starts might be slow
• Collecting test results might require
additional infrastructure setup and
increased costs
• Running each test in separate container is a
huge resource overhead
• Functions has concurrency limit (1000
simultaneous running instances by default)
• No existing production-ready test runners
that support this
Cons

27.
Ongoing experiments

28.
Isolates as FaaS
platform
Probably this is future of
Functions as a Service

29.
What is Isolate?
• Low-level V8 engine feature to create
separate context+stack+event loop for your
code. To be simple - this is creating new V8
instance
• Can be thousands of running Isolates inside
one process (limited by your hardwareOS)
• Worker Threads are actually based on V8
Isolates
• Used in Chrome to provide isolation
between different JS context (tabs,
iframes…)

30.
Existing implementation - Cloudflare workers
• Similar to FaaS, but does not use
containers/virtual machines
• Cold start - less than 5ms
• Up to 1000 requests per second, and can
be increased in configuration
• 5$ per month - enough for 10 millions
requests
• 5-50ms CPU time per request. 128mb RAM
per request

31.
CPU/RAM usage
Simple measurements of
resources consumption

32.
Measurements
• Test - sending HTTP request every second
until 50
• Ava for measuring async execution, 50 tests
in one file
• Jest for measuring processes performance -
50 files each file with one test
• Toundra - 50 Worker Threads with same test
• Code available here: https://github.com/
Xotabu4/comparing-process-threads-async

33.
Ava (Async tests)
Jest (Processes)
Toundra (Threads)
0 1000 2000 3000 4000
Max RAM usage Min RAM usage
CPU/RAM consumption

34.
Ava (Async tests)
Jest (Processes)
Toundra (Threads)
0 6.5 13 19.5 26
CPU%
Average CPU%

35.
• Paralleling by processes is a simplest for now, because it is supported by
majority of frameworks, but has principal problems
• Most of API/Webdriver tests operations is I/O. Async approach
theoretically is the best for I/O, but frameworks support is limited
• Worker Threads are really good for tests parallelization, and potentially
can get even better with using shared memory and messaging. I expect a
lot of test runners switch to Worker Threads soon
• FaaS can be good on huge scale, but hard to setup all parts of tests
execution (deploy, launching, reporting... )

36.
Concurrency vs Parallelizm
• JS event loop visualized
• Libuv - core of NodeJS
• JS test runners benchmark
Isolates
• Docs
• Isolates in cloudflare
• Cloudflare docs
Worker Threads
• Worker Threads PR
• FAQ worker threads
• Details about messaging system
• Alibaba workers implementation
• Shared memory in JS
• High-level abstraction over Worker Threads
Oleksandr Khotemskyi (xotabu4.github.io) 2019