Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript


Published on

Люди часто думают, что выполнять параллельные тесты в JavaScript сложно, и на самом деле это так. Мы рассмотрим возможные способы одновременного выполнения тестов написанных JavaScript (nodejs). Рассмотрим разные подходы - начиная от базового и общеизвестного разбиения по файлам и заканчивая некоторыми сумасшедшими экспериментальными вещами, такими как worker threads, functions as a service, isolates. Пояснения и метрики производительности включены!
- Конкурентность vs параллельность
- Конкурентная паралелизация
- Паралельность используя NodeJS процессы
- Паралельность используя Worker Threads
- IPC (коммуникация между процессами)
- Functions as a service
- Isolates

Published in: Education
  • Login to see the comments

  • Be the first to like this

JS Fest 2019. Александр Хотемский. Способы распараллеливания тестов в JavaScript

  1. 1. Ways of tests parallelization in JavaScript 2019, Oleksandr Khotemskyi
  2. 2. Independent Contractor, Software Developer Engineer in Test Hello! I Am Oleksandr Khotemskyi Contacts:
  3. 3. 13 July 9 April sold out
 11 June 14 May
  4. 4. Ways of tests parallelization in JavaScript • Concurrency vs Parallelism • Concurrent execution • Parallelism by NodeJS processes • Parallelism by worker threads • Functions as a Service (Serverless) • Isolates • Comparing approaches
  5. 5. Concurrency vs Parallelism Short introduction into differences
  6. 6. Concurrency • Tasks are sharing single CPU and have shared memory • Task that reaches the queue earlier - executes earlier (FIFO) • Queue allows to avoid race conditions for memory, but managing execution order is on you • Best fits to different kinds of event handling systems (WebServers, UI …) CPU Memory Task Task Task Task
  7. 7. Parallelism • Each task runs on separate CPU • Usually task distribution is controlled with some “master” process • Managing execution flow can be easier, but you have to deal with race conditions on memory • Best fits for heavy computational tasks - you are processing static data, without external events (graphics, video, etc) CPU
 1 Memory TaskTask Task Task CPU
 2 CPU
 3 CPU
  8. 8. Concurrent test execution Using power of async code to run multiple tests at once
  9. 9. Running tests concurrently • Test runner starts your tests as usual - sequentially one by one • When test execution reaches some async operation - test runner does not wait for it to complete • Instead, test runner starts next test, and again runs it until first async operation • So your tests “compete” for CPU time, and code execution flow is actually jumps between async operations CPU Memory Test 4 Test 3 Test 2 Test 1
  10. 10. Pros • CPU / RAM consumption is lowest • Overhead costs are minimal • Tests use shared memory • No any infrastructure/os level setup needed
  11. 11. Cons • Shared variables wont be synchronized • Potentially might crash nodes process • Harder to implement test timeouts, concurrency limit • Using a lot of synchronous code might slowdown your tests • Limited test runners support
  12. 12. Parallelism by NodeJS processes Tests are shared between processes
  13. 13. How this works? • Test runner gets absolute file paths with tests • Test runner starts Worker Processes and passes file paths to them • Each Worker processes loads one file, and runs test runner for it • Worker processes return results back to parent process • Master process gives another file path to idle Worker process Master Process 67891011 CPU
 1 1 2 3 4 5 Memory CPU
 2 CPU
 3 CPU
 4 CPU
 5 CPU
 6 Memory Memory Memory Memory Memory
  14. 14. + mocha-parallel-tests
  15. 15. Pros • Can use all your CPUs • Simplest existing way to run JS tests in parallel • Most of frameworks already support this • Each file isolated from others, errors in Worker process not affect Master process • Can be combined with other methods
  16. 16. Cons • Consumes CPU/RAM the most • On start - huge spike of CPU usage. Probably launching processes in chunks might help • Hard to share data between running processes • Reporting can be harder than usual • Existing implementations cannot parallel individual tests, only whole test files (because of pre/post conditions)
  17. 17. Parallelism by worker threads Multithreading finally comes to JavaScript
  18. 18. Worker Threads • Starting from nodejs 10.5 - behind the flag node —experimental-worker . Or starting from nodejs 11.7 - without flag • Allows to run piece of code with own context, stack, event loop • No racing conditions because memory is not shared (but there are low-level API for memory sharing) • Communication is done via Messages (events) • Main idea is to allow to run your CPU consuming blocking calculations in separate thread
  19. 19. NodeJS Worker Threads vs Worker Process Threads Processes Memory Can be shared (ArrayBuffer, SharedArrayBuffer, Atomics) Cannot be shared Communication Messages system. Serialized data doesn’t leave process. Much faster Inner Process Communication (with JSON serialization/deserialization). Much slower CPU One CPU core per worker One CPU core per process Startup Fast, less than 40ms Slower, more than 200ms Isolation Each thread with own event loop, stack, garbage collection Separate V8 instance for each process. Nothing shared.
  20. 20. Potential future scaling using IPC • NodeJS has ability to communicate to another process via Inner Process Communication • Also HTTP server can be used • Node-IPC package provides support to access other nodejs processes via Unix/ Windows sockets, TCP/TLS/UDP Sockets • This allows to scale tests parallelization horizontally • There are existing frameworks using similar approach in other languages (C# - Meissa) Master Server
  21. 21. Existing implementations • Test runner AleksanderPopov/toundra • Add-on for Jest - jest-worker • Also, there are some experiments with using worker threads with MochaJS • Worker threads just came out from ‘experiment’ stage in nodejs. Expecting more frameworks to support this in future
  22. 22. Functions as a service Maybe infrastructure can help us?
  23. 23. FaaS
  24. 24. Your code runs according to your logic You will get execution result in HTTP response Invoke. For example - just by sending HTTP request Upload to your FaaS provider Wrap your code to special “handler” FaaS
  25. 25. • Does not require to manage own hosts • Autoscaling included • Predictable and clean environment for each test • Each test isolated to own container • Google Chrome can be launched inside function - potential cost saving for test farm Pros
  26. 26. • Need to deploy code to FaaS • Cold function starts might be slow • Collecting test results might require additional infrastructure setup and increased costs • Running each test in separate container is a huge resource overhead • Functions has concurrency limit (1000 simultaneous running instances by default) • No existing production-ready test runners that support this Cons
  27. 27. Ongoing experiments
  28. 28. Isolates as FaaS platform Probably this is future of Functions as a Service
  29. 29. What is Isolate? • Low-level V8 engine feature to create separate context+stack+event loop for your code. To be simple - this is creating new V8 instance • Can be thousands of running Isolates inside one process (limited by your hardwareOS) • Worker Threads are actually based on V8 Isolates • Used in Chrome to provide isolation between different JS context (tabs, iframes…)
  30. 30. Existing implementation - Cloudflare workers • Similar to FaaS, but does not use containers/virtual machines • Cold start - less than 5ms • Up to 1000 requests per second, and can be increased in configuration • 5$ per month - enough for 10 millions requests • 5-50ms CPU time per request. 128mb RAM per request
  31. 31. CPU/RAM usage Simple measurements of resources consumption
  32. 32. Measurements • Test - sending HTTP request every second until 50 • Ava for measuring async execution, 50 tests in one file • Jest for measuring processes performance - 50 files each file with one test • Toundra - 50 Worker Threads with same test • Code available here: Xotabu4/comparing-process-threads-async
  33. 33. Ava (Async tests) Jest (Processes) Toundra (Threads) 0 1000 2000 3000 4000 Max RAM usage Min RAM usage CPU/RAM consumption
  34. 34. Ava (Async tests) Jest (Processes) Toundra (Threads) 0 6.5 13 19.5 26 CPU% Average CPU%
  35. 35. • Paralleling by processes is a simplest for now, because it is supported by majority of frameworks, but has principal problems • Most of API/Webdriver tests operations is I/O. Async approach theoretically is the best for I/O, but frameworks support is limited • Worker Threads are really good for tests parallelization, and potentially can get even better with using shared memory and messaging. I expect a lot of test runners switch to Worker Threads soon • FaaS can be good on huge scale, but hard to setup all parts of tests execution (deploy, launching, reporting... )
  36. 36. Concurrency vs Parallelizm • JS event loop visualized • Libuv - core of NodeJS • JS test runners benchmark Isolates • Docs • Isolates in cloudflare • Cloudflare docs Worker Threads • Worker Threads PR • FAQ worker threads • Details about messaging system • Alibaba workers implementation • Shared memory in JS • High-level abstraction over Worker Threads Oleksandr Khotemskyi ( 2019