Download as PDF, PPTX
Uploaded byElixir Club
Performance measurement methodology — Maksym Pugach | Elixir Evening Club 3
The document discusses performance measurement methodology comparing the Vert.x framework and Elixir for backend services, highlighting test results in terms of concurrency and requests per second. It reveals insights into Vert.x's performance, particularly its inability to fully utilize CPU resources, and questions regarding its community and documentation. Additionally, it shares practical experiences and challenges encountered while integrating existing code with Vert.x.
More Related Content
Similar to Performance measurement methodology — Maksym Pugach | Elixir Evening Club 3
Performance measurement methodology — Maksym Pugach | Elixir Evening Club 3
- 1.
- 2.
- 3. A couple ofyears ago in a company, somewhere in the city….
- 6. Eclipse Vert.x isa polyglot event-driven application framework that runs on the Java Virtual Machine
- 7. Vert.x promises forour case ● Just wrap your existing code into a Vert.x and it works (JS and Ruby in our case) ● More performant than other solutions
- 8. The Test Environment GoogleCloud Platform 1. K8S services - 1 vCPU 3.75GB Ram 2. MongoDB - 1 vCPU 3.75GB Ram 3. Target(elixir-cowboy, elixir-elli, vertx-ruby) - 2 vCPUs 7.5GB Ram 4. Tester(ab, wrk) - 4 vCPUs 15GB Ram vCPU implemented as a single hardware hyper-thread on Intel Xenon Skylake 2.7GHz
- 9. The Test Environment Vert.x3.5.4 (official Docker image) Elixir 1.7.4 (official Docker image) Cowboy 2.6.0 Elli 3.1.0 Jiffy 0.15.2 (NIF JSON parser) Plug … not used
- 10. Tools and Methodology ab- ApacheBench 2.3 Pick concurrency number which results better requests per second Iterate same test while last two tries are better than any other Pick the best
- 11. GET /static serve astatic JSON string 1000 000 times Concurr ency Req/sec Time (s) CPU load (%) RAM (Mb) Comment Elixir Cowboy 10 4889.53 204.519 200 490 4807.77 rps on concurrency 30 Elixir Elli 30 9198.34 108.715 200 490 Not stable, +/- 500 req/sec Vertx Ruby 30 9667.33 103.441 170 820 Not stable, +/- 500 req/sec; CPU cores are not fully loaded even if performance start to degrade on high concurrency. Vertx Ruby 30 11176.91 89.470 180 820 “/blocking_static” endpoint
- 12. GET /mongo get 20documents, serialize them and server JSON string 100 000 times Concurr ency Req/sec Time (s) CPU load (%) RAM (Mb) Comment Elixir Cowboy 20 1724.53 57.987 200 500 1656.11 rps on concurrency 30 Elixir Elli 20 2155.92 46.384 200 500 2058.08 rps on concurrency 30 Vertx Ruby 30 2135.98 46.817 150 920 Vertx Ruby 30 2200.38 45.447 150 920 “/blocking_mongo” endpoint
- 13. POST /sum post ‘{"a":10,"b":20,"iterations_count":1}’,calculate the sum and return serialized JSON string 100 000 times Concurr ency Req/sec Time (s) CPU load (%) RAM (Mb) Comment Elixir Cowboy 42 4521.85 22.115 200 480 Elixir Elli 42 8302.44 12.045 200 500 Vertx Ruby 36 6725.28 14.869 150 1320 6574.72 rps on concurrency 42 Vertx Ruby 42 10884.58 9.653 195 1075 “/blocking_sum” endpoint
- 14. I never heardof Vert.x before, but seeing that it’s based on non-blocking event-based approach and runs on Java, I’d guess that it suffers from issues such as cooperative scheduling, anonymous implicit activities, and callback hell. Here’s one way how we could compare Elixir (or any other BEAM language) against that. Make a simple server which has two requests: short and infinite. Make short do something trivial like return “Hello World!”. Make infinite run an infinite tight CPU bound loop. Saša Jurić https://elixirforum.com/t/elixir-synthetic-performance-test/3558/6
- 15. Vert.x POST /sum 1> time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1}' http://vertx/sum real 0m0.016s 2 > time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1000000000}' http://vertx/sum real 0m35.825s 1 > time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1}' http://vertx/sum real 0m34.992s
- 16. Vert.x POST /blocking_sum 1> time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1}' http://vertx/blocking_sum real 0m0.016s 2 > time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1000000000}' http://vertx/blocking_sum real 0m31.166s 1 > time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1}' http://vertx/blocking_sum real 0m30.396s
- 17. Elixir Elli POST/sum --erl “+S 1” so only one Erlang scheduler is launched 1 > time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1}' http://elixir-elli/sum real 0m0.013s 2 > time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1000000000}' http://elixir-elli/sum real 0m32.924s 1 > time curl -X POST -d '{"a": 1, "b": 20, "iterations_count": 1}' http://elixir-elli/sum real 0m0.013s
- 18. Scheduling types Preemptive: Apreemptive scheduler does context switching among running tasks and has the power to preempt (interrupt) tasks and resume them at a later time without the cooperation of the preempted tasks. This is done based on some factors like their priority, time slice or reductions. Cooperative: A Cooperative scheduler needs tasks’ cooperation for context switching. This way the scheduler simply lets running tasks to voluntarily release control periodically or when idle, then starts a new task and again waits for it to return the control back voluntarily. Now the question is what scheduling mechanism is suitable for real-time systems which must response within a specified time. Cooperative Scheduling system cannot satisfy a real-time system because a running task in such system might never return control back or returns late after a deadline. So real-time systems commonly use Preemptive Scheduling. https://hamidreza-s.github.io/erlang/scheduling/real-time/preemptive/migration/2016/02/09/erlang-sche duler-details.html
- 20. “Just wrap yourexisting code into a Vert.x” Real project feedback: ● “console.log” to debug ● Objects are not always JS objects, for example they do not always have “toString” method to print it to the log ● Not all Node.js libs are usable ● It took a week to rewrite JS promises to the Vert.x type of async execution ● Elasticsearch client for Vert.x is a piece of crap (same thing performs faster on Rails and has less error rate)
- 21. They do havesome good abstractions to reduce callback hell. I played with it for awhile but in the end it was a lonely affair, with very little activity on their Google group. Mandemus https://elixirforum.com/t/elixir-synthetic-performance-test/3558/11?u=mpugach
- 22. Questions 1. Vert.x doesnot utilize the CPU fully in our test, is it a bottleneck? 2. Does Vert.x language translation affects performance? 3. Do they have :observer.start ? 4. Does the community active? 5. What about documentation? 6. How about other ecosystem, can I reuse libraries?
- 24.
- 25.
- 26.