This document discusses performance testing an application with 100,000 concurrent users. It outlines some of the challenges with performance testing at scale on a single node. The proposed solution is to build and deploy the application, create load test slave instances, run a test plan against the new environment from the slaves, and then clean up by shutting down instances. Key lessons learned include issues found and fixed related to garbage collection and serialization, bottlenecks identified, and challenges scaling test clients. Performance testing is integrated into continuous integration to measure performance for each test case with each build.

1. Performance Testing
@
With 100,000 concurrent users

2. WHO ARE WE
• AWS for most client projects and our own
• Enterprise/Solution Architecture + Development
• Performance Engineering, Profiling,
Optimization
• "DevOps"
• AWS for most client projects and our own

3. WHAT WE WANTED
Performance test applications with
100,000 concurrent users
(Websocket + HTTP)
• Easy to use
• Traceable history of tests
• Good fit with modern DevOps processes
• Low maintenance and cost

4. 99 Problems
Quickly max out a single test node
(or even 10 of them)
Performance tests should be available to whole
team
(not just the guru)
Shouldn't have to write another App
(or pay A LOT for one)

5. REQUIREMNTS

6. SOLUTION
(one of many)

7. STEP 1
Build App
(Compile, Test,
Package)

8. STEP 2
Build your environment (Puppet, Chef, etc)
Deploy new application version

9. STEP 3
Create load test slave instances

10. STEP 4
Start running the test plan against new
environment

11. STEP 5
• Test slaves return results to Jenkins.
• Then clean up
• Shut down instances
• Snapshot new AMI

12. PIPELINE
Build app -> Create test env + Deploy -> Test new
environment

13. LESSONS LEARNED 1/2
• Performance problems found and fixed
Garbage Collection, Serialization and encoding
• Very helpful when sizing infrastructure
Webservice tier network was bottleneck
Different instance sizes picked
• Scaling the test clients can be hard

14. LESSONS LEARNED 2/2
• Performance testing via ELB
• Ramp up slowly
• Make sure client see DNS changes
• ELB's not great for Websocket
• Linux tuning
• Ephemeral port range
• Buffers everywhere
• File handles, flood protection, etc
• It does a pretty good job by default

15. Results & Deep Insight
Deep Insight with dynaTrace APM beyond simple JMeter results
Clearly understand the difference between:
• Load generation
• Real User Monitoring
• Always-on deep transaction tracing

16. TEST AUTOMATION / CI
• Unit Tests are fully integrated as part of CI
• Each build also measures performance for each testcase

17. BONUSES
Easily re-used for different types of build/testing
• BYO load test tool
• Spot instances are cheaper
• Not just for load testing
• Test Chef/Puppet scripts
• OS updates
• Browser based (Webdriver, PhantomJS, etc)
• Auto-scale testing
• Snapshot AMI after successful tests

18. Questions ?
Thanks !
Twitter: msquareau
http://m-square.com.au