Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Ways of tests parallelization in
JavaScript
2019, Oleksandr Khotemskyi 

xotabu4.github.io
Independent Contractor,
Software Developer Engineer in Test
Hello!
I Am Oleksandr Khotemskyi
Contacts: xotabu4.github.io
https://smplday.com/
13 July
http://start-it.ua/
9 April sold out

11 June
http://start-it.ua/
14 May
Ways of tests parallelization in JavaScript
• Concurrency vs Parallelism

• Concurrent execution

• Parallelism by NodeJS ...
Concurrency vs
Parallelism
Short introduction into
differences
Concurrency
• Tasks are sharing single CPU and have
shared memory

• Task that reaches the queue earlier -
executes earlie...
Parallelism
• Each task runs on separate CPU

• Usually task distribution is controlled with some
“master” process

• Mana...
Concurrent test
execution
Using power of async code to
run multiple tests at once
Running tests concurrently
• Test runner starts your tests as usual - sequentially one by
one

• When test execution reach...
Pros
• CPU / RAM consumption is lowest

• Overhead costs are minimal

• Tests use shared memory

• No any infrastructure/o...
Cons
• Shared variables wont be synchronized

• Potentially might crash nodes process

• Harder to implement test timeouts...
Parallelism by
NodeJS processes
Tests are shared between
processes
How this works?
• Test runner gets absolute file paths with tests

• Test runner starts Worker Processes and
passes file pat...
+
mocha-parallel-tests
Pros
• Can use all your CPUs

• Simplest existing way to run JS tests in
parallel

• Most of frameworks already support th...
Cons
• Consumes CPU/RAM the most

• On start - huge spike of CPU usage. Probably
launching processes in chunks might help
...
Parallelism by
worker threads
Multithreading finally comes to
JavaScript
Worker Threads
• Starting from nodejs 10.5 - behind the flag
node —experimental-worker . Or starting from
nodejs 11.7 - wit...
NodeJS Worker Threads vs Worker Process
Threads Processes
Memory
Can be shared (ArrayBuffer, SharedArrayBuffer,
Atomics)
Can...
Potential future scaling using IPC
• NodeJS has ability to communicate to
another process via Inner Process
Communication
...
Existing implementations
• Test runner https://github.com/
AleksanderPopov/toundra

• Add-on for Jest - jest-worker

• Als...
Functions as a
service
Maybe infrastructure can help
us?
FaaS
Your code runs
according to
your logic
You will get
execution result
in HTTP
response
Invoke. For example -
just by sendin...
• Does not require to manage own hosts

• Autoscaling included

• Predictable and clean environment for each
test

• Each ...
• Need to deploy code to FaaS

• Cold function starts might be slow

• Collecting test results might require
additional in...
Ongoing experiments
Isolates as FaaS
platform
Probably this is future of
Functions as a Service
What is Isolate?
• Low-level V8 engine feature to create
separate context+stack+event loop for your
code. To be simple - t...
Existing implementation - Cloudflare workers
• Similar to FaaS, but does not use
containers/virtual machines

• Cold start ...
CPU/RAM usage
Simple measurements of
resources consumption
Measurements
• Test - sending HTTP request every second
until 50

• Ava for measuring async execution, 50 tests
in one file...
Ava (Async tests)
Jest (Processes)
Toundra (Threads)
0 1000 2000 3000 4000
Max RAM usage Min RAM usage
CPU/RAM consumption
Ava (Async tests)
Jest (Processes)
Toundra (Threads)
0 6.5 13 19.5 26
CPU%
Average CPU%
• Paralleling by processes is a simplest for now, because it is supported by
majority of frameworks, but has principal pro...
Concurrency vs Parallelizm

• JS event loop visualized

• Libuv - core of NodeJS

• JS test runners benchmark
Isolates

• ...
JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript
JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript
Upcoming SlideShare
Loading in …5
×

of

JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 1 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 2 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 3 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 4 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 5 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 6 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 7 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 8 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 9 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 10 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 11 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 12 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 13 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 14 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 15 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 16 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 17 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 18 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 19 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 20 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 21 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 22 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 23 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 24 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 25 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 26 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 27 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 28 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 29 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 30 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 31 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 32 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 33 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 34 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 35 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 36 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 37 JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript Slide 38
Upcoming SlideShare
What to Upload to SlideShare
Next

0 Likes

Share

JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript

Люди часто думают, что выполнять параллельные тесты в JavaScript сложно, и на самом деле это так. Мы рассмотрим возможные способы одновременного выполнения тестов написанных JavaScript (nodejs). Рассмотрим разные подходы - начиная от базового и общеизвестного разбиения по файлам и заканчивая некоторыми сумасшедшими экспериментальными вещами, такими как worker threads, functions as a service, isolates. Пояснения и метрики производительности включены!
- Конкурентность vs параллельность
- Конкурентная паралелизация
- Паралельность используя NodeJS процессы
- Паралельность используя Worker Threads
- IPC (коммуникация между процессами)
- Functions as a service
- Isolates

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript

  1. 1. Ways of tests parallelization in JavaScript 2019, Oleksandr Khotemskyi 
 xotabu4.github.io
  2. 2. Independent Contractor, Software Developer Engineer in Test Hello! I Am Oleksandr Khotemskyi Contacts: xotabu4.github.io
  3. 3. https://smplday.com/ 13 July http://start-it.ua/ 9 April sold out
 11 June http://start-it.ua/ 14 May
  4. 4. Ways of tests parallelization in JavaScript • Concurrency vs Parallelism • Concurrent execution • Parallelism by NodeJS processes • Parallelism by worker threads • Functions as a Service (Serverless) • Isolates • Comparing approaches
  5. 5. Concurrency vs Parallelism Short introduction into differences
  6. 6. Concurrency • Tasks are sharing single CPU and have shared memory • Task that reaches the queue earlier - executes earlier (FIFO) • Queue allows to avoid race conditions for memory, but managing execution order is on you • Best fits to different kinds of event handling systems (WebServers, UI …) CPU Memory Task Task Task Task
  7. 7. Parallelism • Each task runs on separate CPU • Usually task distribution is controlled with some “master” process • Managing execution flow can be easier, but you have to deal with race conditions on memory • Best fits for heavy computational tasks - you are processing static data, without external events (graphics, video, etc) CPU
 1 Memory TaskTask Task Task CPU
 2 CPU
 3 CPU
 4
  8. 8. Concurrent test execution Using power of async code to run multiple tests at once
  9. 9. Running tests concurrently • Test runner starts your tests as usual - sequentially one by one • When test execution reaches some async operation - test runner does not wait for it to complete • Instead, test runner starts next test, and again runs it until first async operation • So your tests “compete” for CPU time, and code execution flow is actually jumps between async operations CPU Memory Test 4 Test 3 Test 2 Test 1
  10. 10. Pros • CPU / RAM consumption is lowest • Overhead costs are minimal • Tests use shared memory • No any infrastructure/os level setup needed
  11. 11. Cons • Shared variables wont be synchronized • Potentially might crash nodes process • Harder to implement test timeouts, concurrency limit • Using a lot of synchronous code might slowdown your tests • Limited test runners support
  12. 12. Parallelism by NodeJS processes Tests are shared between processes
  13. 13. How this works? • Test runner gets absolute file paths with tests • Test runner starts Worker Processes and passes file paths to them • Each Worker processes loads one file, and runs test runner for it • Worker processes return results back to parent process • Master process gives another file path to idle Worker process Master Process 67891011 CPU
 1 1 2 3 4 5 Memory CPU
 2 CPU
 3 CPU
 4 CPU
 5 CPU
 6 Memory Memory Memory Memory Memory
  14. 14. + mocha-parallel-tests
  15. 15. Pros • Can use all your CPUs • Simplest existing way to run JS tests in parallel • Most of frameworks already support this • Each file isolated from others, errors in Worker process not affect Master process • Can be combined with other methods
  16. 16. Cons • Consumes CPU/RAM the most • On start - huge spike of CPU usage. Probably launching processes in chunks might help • Hard to share data between running processes • Reporting can be harder than usual • Existing implementations cannot parallel individual tests, only whole test files (because of pre/post conditions)
  17. 17. Parallelism by worker threads Multithreading finally comes to JavaScript
  18. 18. Worker Threads • Starting from nodejs 10.5 - behind the flag node —experimental-worker . Or starting from nodejs 11.7 - without flag • Allows to run piece of code with own context, stack, event loop • No racing conditions because memory is not shared (but there are low-level API for memory sharing) • Communication is done via Messages (events) • Main idea is to allow to run your CPU consuming blocking calculations in separate thread
  19. 19. NodeJS Worker Threads vs Worker Process Threads Processes Memory Can be shared (ArrayBuffer, SharedArrayBuffer, Atomics) Cannot be shared Communication Messages system. Serialized data doesn’t leave process. Much faster Inner Process Communication (with JSON serialization/deserialization). Much slower CPU One CPU core per worker One CPU core per process Startup Fast, less than 40ms Slower, more than 200ms Isolation Each thread with own event loop, stack, garbage collection Separate V8 instance for each process. Nothing shared.
  20. 20. Potential future scaling using IPC • NodeJS has ability to communicate to another process via Inner Process Communication • Also HTTP server can be used • Node-IPC package provides support to access other nodejs processes via Unix/ Windows sockets, TCP/TLS/UDP Sockets • This allows to scale tests parallelization horizontally • There are existing frameworks using similar approach in other languages (C# - Meissa) Master Server
  21. 21. Existing implementations • Test runner https://github.com/ AleksanderPopov/toundra • Add-on for Jest - jest-worker • Also, there are some experiments with using worker threads with MochaJS • Worker threads just came out from ‘experiment’ stage in nodejs. Expecting more frameworks to support this in future
  22. 22. Functions as a service Maybe infrastructure can help us?
  23. 23. FaaS
  24. 24. Your code runs according to your logic You will get execution result in HTTP response Invoke. For example - just by sending HTTP request Upload to your FaaS provider Wrap your code to special “handler” FaaS
  25. 25. • Does not require to manage own hosts • Autoscaling included • Predictable and clean environment for each test • Each test isolated to own container • Google Chrome can be launched inside function - potential cost saving for test farm Pros
  26. 26. • Need to deploy code to FaaS • Cold function starts might be slow • Collecting test results might require additional infrastructure setup and increased costs • Running each test in separate container is a huge resource overhead • Functions has concurrency limit (1000 simultaneous running instances by default) • No existing production-ready test runners that support this Cons
  27. 27. Ongoing experiments
  28. 28. Isolates as FaaS platform Probably this is future of Functions as a Service
  29. 29. What is Isolate? • Low-level V8 engine feature to create separate context+stack+event loop for your code. To be simple - this is creating new V8 instance • Can be thousands of running Isolates inside one process (limited by your hardwareOS) • Worker Threads are actually based on V8 Isolates • Used in Chrome to provide isolation between different JS context (tabs, iframes…)
  30. 30. Existing implementation - Cloudflare workers • Similar to FaaS, but does not use containers/virtual machines • Cold start - less than 5ms • Up to 1000 requests per second, and can be increased in configuration • 5$ per month - enough for 10 millions requests • 5-50ms CPU time per request. 128mb RAM per request
  31. 31. CPU/RAM usage Simple measurements of resources consumption
  32. 32. Measurements • Test - sending HTTP request every second until 50 • Ava for measuring async execution, 50 tests in one file • Jest for measuring processes performance - 50 files each file with one test • Toundra - 50 Worker Threads with same test • Code available here: https://github.com/ Xotabu4/comparing-process-threads-async
  33. 33. Ava (Async tests) Jest (Processes) Toundra (Threads) 0 1000 2000 3000 4000 Max RAM usage Min RAM usage CPU/RAM consumption
  34. 34. Ava (Async tests) Jest (Processes) Toundra (Threads) 0 6.5 13 19.5 26 CPU% Average CPU%
  35. 35. • Paralleling by processes is a simplest for now, because it is supported by majority of frameworks, but has principal problems • Most of API/Webdriver tests operations is I/O. Async approach theoretically is the best for I/O, but frameworks support is limited • Worker Threads are really good for tests parallelization, and potentially can get even better with using shared memory and messaging. I expect a lot of test runners switch to Worker Threads soon • FaaS can be good on huge scale, but hard to setup all parts of tests execution (deploy, launching, reporting... )
  36. 36. Concurrency vs Parallelizm • JS event loop visualized • Libuv - core of NodeJS • JS test runners benchmark Isolates • Docs • Isolates in cloudflare • Cloudflare docs Worker Threads • Worker Threads PR • FAQ worker threads • Details about messaging system • Alibaba workers implementation • Shared memory in JS • High-level abstraction over Worker Threads Oleksandr Khotemskyi (xotabu4.github.io) 2019

Люди часто думают, что выполнять параллельные тесты в JavaScript сложно, и на самом деле это так. Мы рассмотрим возможные способы одновременного выполнения тестов написанных JavaScript (nodejs). Рассмотрим разные подходы - начиная от базового и общеизвестного разбиения по файлам и заканчивая некоторыми сумасшедшими экспериментальными вещами, такими как worker threads, functions as a service, isolates. Пояснения и метрики производительности включены! - Конкурентность vs параллельность - Конкурентная паралелизация - Паралельность используя NodeJS процессы - Паралельность используя Worker Threads - IPC (коммуникация между процессами) - Functions as a service - Isolates

Views

Total views

527

On Slideshare

0

From embeds

0

Number of embeds

0

Actions

Downloads

0

Shares

0

Comments

0

Likes

0

×