Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Performance testing with 100,000 concurrent users in AWS

1,848 views

Published on

M-Square build an easy scalable performance test solution on AWS, using open source tools & CI servers, to allow cost-effective testing at scale. The solution is suitable for any organisation type, from startup to enterprise.

The talk covers VPC, EC2, S3, ELB’s, AWS API scripting, automation and interesting performance issues when running massive workloads on AWS.

Published in: Technology
  • Be the first to comment

Performance testing with 100,000 concurrent users in AWS

  1. 1. Performance Testing @ With 100,000 concurrent users
  2. 2. WHO ARE WE • AWS for most client projects and our own • Enterprise/Solution Architecture + Development • Performance Engineering, Profiling, Optimization • "DevOps" • AWS for most client projects and our own
  3. 3. WHAT WE WANTED Performance test applications with 100,000 concurrent users (Websocket + HTTP) • Easy to use • Traceable history of tests • Good fit with modern DevOps processes • Low maintenance and cost
  4. 4. 99 Problems Quickly max out a single test node (or even 10 of them) Performance tests should be available to whole team (not just the guru) Shouldn't have to write another App (or pay A LOT for one)
  5. 5. REQUIREMNTS
  6. 6. SOLUTION (one of many)
  7. 7. STEP 1 Build App (Compile, Test, Package)
  8. 8. STEP 2 Build your environment (Puppet, Chef, etc) Deploy new application version
  9. 9. STEP 3 Create load test slave instances
  10. 10. STEP 4 Start running the test plan against new environment
  11. 11. STEP 5 • Test slaves return results to Jenkins. • Then clean up • Shut down instances • Snapshot new AMI
  12. 12. PIPELINE Build app -> Create test env + Deploy -> Test new environment
  13. 13. LESSONS LEARNED 1/2 • Performance problems found and fixed Garbage Collection, Serialization and encoding • Very helpful when sizing infrastructure Webservice tier network was bottleneck Different instance sizes picked • Scaling the test clients can be hard
  14. 14. LESSONS LEARNED 2/2 • Performance testing via ELB • Ramp up slowly • Make sure client see DNS changes • ELB's not great for Websocket • Linux tuning • Ephemeral port range • Buffers everywhere • File handles, flood protection, etc • It does a pretty good job by default
  15. 15. Results & Deep Insight Deep Insight with dynaTrace APM beyond simple JMeter results Clearly understand the difference between: • Load generation • Real User Monitoring • Always-on deep transaction tracing
  16. 16. TEST AUTOMATION / CI • Unit Tests are fully integrated as part of CI • Each build also measures performance for each testcase
  17. 17. BONUSES Easily re-used for different types of build/testing • BYO load test tool • Spot instances are cheaper • Not just for load testing • Test Chef/Puppet scripts • OS updates • Browser based (Webdriver, PhantomJS, etc) • Auto-scale testing • Snapshot AMI after successful tests
  18. 18. Questions ? Thanks ! Twitter: msquareau http://m-square.com.au

×