Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

EUC2015 - Load testing XMPP servers with Plain Old Erlang

138 views

Published on

Slides from Erlang User Conference 2015 that I gave about my experience with load testing Mongoose IM

Published in: Software
  • Be the first to comment

  • Be the first to like this

EUC2015 - Load testing XMPP servers with Plain Old Erlang

  1. 1. Tutorial: Load testing XMPP servers with Plain Old Erlang EUC 2015 tutorial, Stockholm Paweł Pikuła
  2. 2. Who am I? • Paweł Pikuła pawel.pikula@erlang-solutions.com • Developer @ Erlang Solutions Ltd. Cracow office • I work on MongooseIM XMPP server and some IoT related project. • github.com/ppikula • twitter: @pawelpikula
  3. 3. Agenda • What is load testing • Introducing AMOC - our simple load testing tool • Overview of our test infrastructure • Breaking a single node • Distributed scenario (scaling MIM and AMOC horizontally) • What to do next ?
  4. 4. Prepare the environment • Install VirtualBox and Vagrant • git clone https://github.com/ppikula/euc2015/ • Follow instructions from the README file (Setting up the environment from a usb stick)
  5. 5. What is load testing ? “In software engineering, performance testing is in general testing performed to determine how a system performs in terms of responsiveness and stability under a particular workload. It can also serve to investigate, measure, validate or verify other quality attributes of the system, such as scalability, reliability and resource usage.” wikipedia
  6. 6. Why do we do load test ? • Measure capacity of the whole system. • Measure latencies, time to deliver and other quality attributes. • Find the bottlenecks in our system. • Test a bad case scenarios under heavy load (ex. net splits, overloaded DBs). • Discover possible race conditions . • Stress 3rd party software.
  7. 7. Types of tests - endurance
  8. 8. Types of traffic - stress
  9. 9. Types of traffic - spike ! • Check if the system can recover after the spike • How long does it take to go back to the normal state
  10. 10. TSUNG • We‘ve been using it for a long time • It does the job • Tsung is dumb - doesn’t understand XMPP • XML DSL … • Manual setup • It is not designed for bidirectional protocols • HTTP semantics (request, transaction, page time) • some strange race conditions? (I was never able to login 10K users every time I tried I had ~9997 of them )
  11. 11. TSUNG DSL
  12. 12. OUR USE CASE • stream management enabled (XEP 198) auto respond to <r> with <a> • authentication process requires acquiring a token from http service • carbon copies - multiple jabber resources
  13. 13. ESCALUS • esl/ejabberd_tests use it for black box testing • supports many transports (ws, bosh, tcp, tls) • stanza generators • built-in predicates (is_message, is_iq_result)
  14. 14. Escalus story
  15. 15. Can we use escalus ?
  16. 16. AMOC design goals • Use the esl/escalus library • Don’t reinvent the wheel - cooperate with graphite, ansible, graphite notifications, graylog. • Simple and practical • No DSL - plain erlang • build everything on one host
  17. 17. AMOC features • Automatic deployment and environment setup via ansible • Exometer for collecting metrics - very easy to add a new metric • Uses modified erlang slave module • Dynamically increase/decrease the amount of users • Sends notifications about events mentioned above
  18. 18. Environment checklist • File descriptors limit • Check firewall rules • TCP buffers and other options • Increase local port range • Create more network interfaces if you want to generate more than 60K users
  19. 19. Why there is a limit of 60K per interface? • TCP port is a16 bit integer - we have only 65536 outgoing ports - reserved ones
  20. 20. AMOC Inside ! • AMOC scenario - define init and start function • AMOC controller - “user interface” start/stop scenarios. • The controller spawns users on AMOC slaves in round robin fashion
  21. 21. SUT - MongooseIM
  22. 22. MongooseIM • Fork of ejabberd 2.1.8 • Removed “non-scalable” extensions • Added unit & integration tests (black box testing) • Added many metrics • Want to hear more? go to “MongooseIM - The Right Tool for Scalable Messaging” by Michał Piotrowski
  23. 23. Task 1 break single box
  24. 24. Task 2 scale it !
  25. 25. Task3 Add custom metric to AMOC • Message rate (spiral) • Connected users count (counter) • Time to deliver (histogram)
  26. 26. Future of AMOC • automatic tests • allow to pass inter arrival per scenario • support for multiple scenarios • ansible recipes for ubuntu/debian family • dockerized version of AMOC? • integration with elixir - that will allow us to create DSLs
  27. 27. Thanks !
  28. 28. Questions ?

×