Extreme Performance Testing

1. PERFORMANCE
IS NOT A MYTH
P E R F O R M A N C E A D V I S O R Y C O U N C I L
SANTORINI GREECE
FEBRUARY 26 - 27 2020
Extreme Performance Testing
Joerek van Gaalen

2. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Introduction
• Joerek van Gaalen
• Performance specialist since 2005
• Experience with many large scaled performance tests

3. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
2 Million Vuser test

4. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
How to simulate so many
virtual users?

5. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Neotys
Neoload
800 load
generators
Every instance local
DNS server
Different (cloud)
providers

6. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Challenges
• Not being the bottleneck yourself
• Being able to simulate the users / load realistic

7. P E R F O R M A N C E A D V I S O R Y C O U N C I L
Optimise your tests

8. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Tune your script
• Goal is to minimise resources
• Use strict regular expressions without many wildcards
• Do not store previous response
• Reconsider assertion when you already have a variable extractor

9. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Tune your controller
• Consider creating your own variable manager outside the controller
• Avoid using live status updates of virtual users
• Consider more aggregation of data
• Limit stored number of errors per second
• Use linear increase of users rather then big steps

10. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Tune your agent
• Run stress test with single agent and verify its limits
• Approx 3000 Vusers per agent is usually good
• Tune the kernel (open file descriptors, TCP settings)
• Tune Xmx vs real memory

11. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Use a lot cloud instances
• Ideally use many different cloud providers (AWS, DigitalOcean, Google,
Azure)
• Use vendor provided cloud load generators
• AWS is ideal, you can start hundreds in a few clicks
• c5 instance is usually a good choice, but know if the test is CPU, Memory
or Bandwidth heavy

12. P E R F O R M A N C E A D V I S O R Y C O U N C I L
Making your tests realistic

13. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Mimic production usage
• Try to do it in perfection

14. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
DNS Load balancing
• Lower DNS Cache timeout to 1 second
• Install local DNS Server per agent and override TTL
• Use open DNS Server with different geographical locations
• Balance in hostnames in your scripts

15. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Agent spread
• Geographical spread
• Different providers and peering

16. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Scripts
• Do not exclude your CDN objects. They can face quotas or limits

17. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Streaming services
• For streaming services, balance between available bandwidth and used
bitrates
• Or maximise the size of your video segment size
Range: bytes=0-102400

18. P E R F O R M A N C E A D V I S O R Y C O U N C I L
Specific heavy load performance issues

19. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
CDN performance issues
• Quotas & Limits reached
• Too much load against your origins
• Edge servers cannot handle the load

20. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
CDN performance issues
Many egde nodes can
stress the origin Many stream files can stress the origin
or bad caching
/index.m3u8?time=1582235622351

21. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
CDN performance issues
• Not much geographical user spread
• 3rd party services can stress
a single edge node
?
?
?
? ??
?
?
?
?
?
?
?

22. P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Joerek van Gaalen
jvangaalen@breakingit.nl