Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PAC 2020 Santorin - Joerek Van Gaalen

Extreme Performance Testing

  • Login to see the comments

  • Be the first to like this

PAC 2020 Santorin - Joerek Van Gaalen

  1. 1. PERFORMANCE IS NOT A MYTH P E R F O R M A N C E A D V I S O R Y C O U N C I L SANTORINI GREECE FEBRUARY 26 - 27 2020 Extreme Performance Testing Joerek van Gaalen
  2. 2. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Introduction • Joerek van Gaalen • Performance specialist since 2005 • Experience with many large scaled performance tests
  3. 3. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L 2 Million Vuser test
  4. 4. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L How to simulate so many virtual users?
  5. 5. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Neotys Neoload 800 load generators Every instance local DNS server Different (cloud) providers
  6. 6. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Challenges • Not being the bottleneck yourself • Being able to simulate the users / load realistic
  7. 7. P E R F O R M A N C E A D V I S O R Y C O U N C I L Optimise your tests
  8. 8. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Tune your script • Goal is to minimise resources • Use strict regular expressions without many wildcards • Do not store previous response • Reconsider assertion when you already have a variable extractor
  9. 9. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Tune your controller • Consider creating your own variable manager outside the controller • Avoid using live status updates of virtual users • Consider more aggregation of data • Limit stored number of errors per second • Use linear increase of users rather then big steps
  10. 10. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Tune your agent • Run stress test with single agent and verify its limits • Approx 3000 Vusers per agent is usually good • Tune the kernel (open file descriptors, TCP settings) • Tune Xmx vs real memory
  11. 11. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Use a lot cloud instances • Ideally use many different cloud providers (AWS, DigitalOcean, Google, Azure) • Use vendor provided cloud load generators • AWS is ideal, you can start hundreds in a few clicks • c5 instance is usually a good choice, but know if the test is CPU, Memory or Bandwidth heavy
  12. 12. P E R F O R M A N C E A D V I S O R Y C O U N C I L Making your tests realistic
  13. 13. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Mimic production usage • Try to do it in perfection
  14. 14. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L DNS Load balancing • Lower DNS Cache timeout to 1 second • Install local DNS Server per agent and override TTL • Use open DNS Server with different geographical locations • Balance in hostnames in your scripts
  15. 15. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Agent spread • Geographical spread • Different providers and peering
  16. 16. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Scripts • Do not exclude your CDN objects. They can face quotas or limits
  17. 17. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Streaming services • For streaming services, balance between available bandwidth and used bitrates • Or maximise the size of your video segment size Range: bytes=0-102400
  18. 18. P E R F O R M A N C E A D V I S O R Y C O U N C I L Specific heavy load performance issues
  19. 19. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L CDN performance issues • Quotas & Limits reached • Too much load against your origins • Edge servers cannot handle the load
  20. 20. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L CDN performance issues Many egde nodes can stress the origin Many stream files can stress the origin or bad caching /index.m3u8?time=1582235622351
  21. 21. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L CDN performance issues • Not much geographical user spread • 3rd party services can stress a single edge node ? ? ? ? ?? ? ? ? ? ? ? ?
  22. 22. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Joerek van Gaalen jvangaalen@breakingit.nl

×