SlideShare a Scribd company logo
1 of 22
PERFORMANCE
IS NOT A MYTH
P E R F O R M A N C E A D V I S O R Y C O U N C I L
SANTORINI GREECE
FEBRUARY 26 - 27 2020
Extreme Performance Testing
Joerek van Gaalen
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Introduction
• Joerek van Gaalen
• Performance specialist since 2005
• Experience with many large scaled performance tests
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
2 Million Vuser test
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
How to simulate so many
virtual users?
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Neotys
Neoload
800 load
generators
Every instance local
DNS server
Different (cloud)
providers
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Challenges
• Not being the bottleneck yourself
• Being able to simulate the users / load realistic
P E R F O R M A N C E A D V I S O R Y C O U N C I L
Optimise your tests
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Tune your script
• Goal is to minimise resources
• Use strict regular expressions without many wildcards
• Do not store previous response
• Reconsider assertion when you already have a variable extractor
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Tune your controller
• Consider creating your own variable manager outside the controller
• Avoid using live status updates of virtual users
• Consider more aggregation of data
• Limit stored number of errors per second
• Use linear increase of users rather then big steps
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Tune your agent
• Run stress test with single agent and verify its limits
• Approx 3000 Vusers per agent is usually good
• Tune the kernel (open file descriptors, TCP settings)
• Tune Xmx vs real memory
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Use a lot cloud instances
• Ideally use many different cloud providers (AWS, DigitalOcean, Google,
Azure)
• Use vendor provided cloud load generators
• AWS is ideal, you can start hundreds in a few clicks
• c5 instance is usually a good choice, but know if the test is CPU, Memory
or Bandwidth heavy
P E R F O R M A N C E A D V I S O R Y C O U N C I L
Making your tests realistic
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Mimic production usage
• Try to do it in perfection
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
DNS Load balancing
• Lower DNS Cache timeout to 1 second
• Install local DNS Server per agent and override TTL
• Use open DNS Server with different geographical locations
• Balance in hostnames in your scripts
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Agent spread
• Geographical spread
• Different providers and peering
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Scripts
• Do not exclude your CDN objects. They can face quotas or limits
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Streaming services
• For streaming services, balance between available bandwidth and used
bitrates
• Or maximise the size of your video segment size
Range: bytes=0-102400
P E R F O R M A N C E A D V I S O R Y C O U N C I L
Specific heavy load performance issues
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
CDN performance issues
• Quotas & Limits reached
• Too much load against your origins
• Edge servers cannot handle the load
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
CDN performance issues
Many egde nodes can
stress the origin Many stream files can stress the origin
or bad caching
/index.m3u8?time=1582235622351
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
CDN performance issues
• Not much geographical user spread
• 3rd party services can stress
a single edge node
?
?
?
? ??
?
?
?
?
?
?
?
P E R F O R M A N C E A D V I S O R Y C O U N C I L
byP E R F O R M A N C E A D V I S O R Y C O U N C I L
Joerek van Gaalen
jvangaalen@breakingit.nl

More Related Content

What's hot

What's hot (7)

DevOps Continuous Testing and LaaS – The Perfect Match for SDN
DevOps Continuous Testing and LaaS – The Perfect Match for SDNDevOps Continuous Testing and LaaS – The Perfect Match for SDN
DevOps Continuous Testing and LaaS – The Perfect Match for SDN
 
Channel 7
Channel 7Channel 7
Channel 7
 
IEEE Buenaventura cs Chapter March 9 2016 v4
IEEE Buenaventura cs Chapter March 9 2016  v4IEEE Buenaventura cs Chapter March 9 2016  v4
IEEE Buenaventura cs Chapter March 9 2016 v4
 
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...
Replicated Siamese LSTM in Ticketing System for Similarity Learning and Retri...
 
Automated SDTM Creation and Discrepancy Detection Jobs: The Numbers Tell The ...
Automated SDTM Creation and Discrepancy Detection Jobs: The Numbers Tell The ...Automated SDTM Creation and Discrepancy Detection Jobs: The Numbers Tell The ...
Automated SDTM Creation and Discrepancy Detection Jobs: The Numbers Tell The ...
 
JavaOne - Performance Focused DevOps to Improve Cont Delivery
JavaOne - Performance Focused DevOps to Improve Cont DeliveryJavaOne - Performance Focused DevOps to Improve Cont Delivery
JavaOne - Performance Focused DevOps to Improve Cont Delivery
 
Sydney Continuous Delivery Meetup May 2014
Sydney Continuous Delivery Meetup May 2014Sydney Continuous Delivery Meetup May 2014
Sydney Continuous Delivery Meetup May 2014
 

Similar to PAC 2020 Santorin - Joerek Van Gaalen

4Developers 2015: Measure to fail - Tomasz Kowalczewski
4Developers 2015: Measure to fail - Tomasz Kowalczewski4Developers 2015: Measure to fail - Tomasz Kowalczewski
4Developers 2015: Measure to fail - Tomasz Kowalczewski
PROIDEA
 

Similar to PAC 2020 Santorin - Joerek Van Gaalen (20)

The 12 Factor App
The 12 Factor AppThe 12 Factor App
The 12 Factor App
 
Monitoring and Logging in Wonderland
Monitoring and Logging in WonderlandMonitoring and Logging in Wonderland
Monitoring and Logging in Wonderland
 
PAC 2020 Santorin - Ankur Jain
PAC 2020 Santorin - Ankur JainPAC 2020 Santorin - Ankur Jain
PAC 2020 Santorin - Ankur Jain
 
Building a Data Ingestion & Processing Pipeline with Spark & Airflow
Building a Data Ingestion & Processing Pipeline with Spark & AirflowBuilding a Data Ingestion & Processing Pipeline with Spark & Airflow
Building a Data Ingestion & Processing Pipeline with Spark & Airflow
 
PAC 2020 Santorin - Vasilis Chatzinasios
PAC 2020 Santorin - Vasilis ChatzinasiosPAC 2020 Santorin - Vasilis Chatzinasios
PAC 2020 Santorin - Vasilis Chatzinasios
 
JavaFX Enterprise
JavaFX EnterpriseJavaFX Enterprise
JavaFX Enterprise
 
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
MongoDB Europe 2016 - Using MongoDB to Build a Fast and Scalable Content Repo...
 
4Developers 2015: Measure to fail - Tomasz Kowalczewski
4Developers 2015: Measure to fail - Tomasz Kowalczewski4Developers 2015: Measure to fail - Tomasz Kowalczewski
4Developers 2015: Measure to fail - Tomasz Kowalczewski
 
Measure to fail
Measure to failMeasure to fail
Measure to fail
 
Architecting for Sustainability - Community Summit v1.0.pptx
Architecting for Sustainability - Community Summit v1.0.pptxArchitecting for Sustainability - Community Summit v1.0.pptx
Architecting for Sustainability - Community Summit v1.0.pptx
 
React native first impression
React native first impressionReact native first impression
React native first impression
 
Everybody Lies
Everybody LiesEverybody Lies
Everybody Lies
 
Microservices Delivery Platform. Tips & Tricks
Microservices Delivery Platform. Tips & TricksMicroservices Delivery Platform. Tips & Tricks
Microservices Delivery Platform. Tips & Tricks
 
Using MongoDB to Build a Fast and Scalable Content Repository
Using MongoDB to Build a Fast and Scalable Content RepositoryUsing MongoDB to Build a Fast and Scalable Content Repository
Using MongoDB to Build a Fast and Scalable Content Repository
 
Nebula Cloud9 Public Cloud (IaaS)
Nebula Cloud9 Public Cloud (IaaS)Nebula Cloud9 Public Cloud (IaaS)
Nebula Cloud9 Public Cloud (IaaS)
 
Angular server side rendering with NodeJS - In Pursuit Of Speed
Angular server side rendering with NodeJS - In Pursuit Of SpeedAngular server side rendering with NodeJS - In Pursuit Of Speed
Angular server side rendering with NodeJS - In Pursuit Of Speed
 
SOA Latam 2015
SOA Latam 2015SOA Latam 2015
SOA Latam 2015
 
Synthetic and RUM: A Recipe for Web Performance Success
Synthetic and RUM: A Recipe for Web Performance SuccessSynthetic and RUM: A Recipe for Web Performance Success
Synthetic and RUM: A Recipe for Web Performance Success
 
Velocity NY 2014 - The Natives are Getting Restless
Velocity NY 2014 - The Natives are Getting RestlessVelocity NY 2014 - The Natives are Getting Restless
Velocity NY 2014 - The Natives are Getting Restless
 
PAC 2020 Santorin - Hari Krishnan Ramachandran
PAC 2020 Santorin - Hari Krishnan RamachandranPAC 2020 Santorin - Hari Krishnan Ramachandran
PAC 2020 Santorin - Hari Krishnan Ramachandran
 

More from Neotys

More from Neotys (20)

PAC 2020 Santorin - Andreas Grabner
PAC 2020 Santorin - Andreas Grabner PAC 2020 Santorin - Andreas Grabner
PAC 2020 Santorin - Andreas Grabner
 
PAC 2020 Santorin - Stephen Townshend
PAC 2020 Santorin - Stephen TownshendPAC 2020 Santorin - Stephen Townshend
PAC 2020 Santorin - Stephen Townshend
 
PAC 2020 Santorin - Leandro Melendez
PAC 2020 Santorin - Leandro MelendezPAC 2020 Santorin - Leandro Melendez
PAC 2020 Santorin - Leandro Melendez
 
PAC 2019 virtual Stephen Townshend
PAC 2019 virtual Stephen TownshendPAC 2019 virtual Stephen Townshend
PAC 2019 virtual Stephen Townshend
 
PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo   PAC 2019 virtual Federico Toledo
PAC 2019 virtual Federico Toledo
 
PAC 2019 virtual Leandro Melendez
PAC 2019 virtual Leandro Melendez PAC 2019 virtual Leandro Melendez
PAC 2019 virtual Leandro Melendez
 
PAC 2019 virtual Mark Tomlinson
PAC 2019 virtual Mark TomlinsonPAC 2019 virtual Mark Tomlinson
PAC 2019 virtual Mark Tomlinson
 
PAC 2019 virtual Srivalli Aparna
PAC 2019 virtual Srivalli AparnaPAC 2019 virtual Srivalli Aparna
PAC 2019 virtual Srivalli Aparna
 
PAC 2019 virtual Reuben Rajan George
PAC 2019 virtual Reuben Rajan GeorgePAC 2019 virtual Reuben Rajan George
PAC 2019 virtual Reuben Rajan George
 
PAC 2019 virtual Joerek Van Gaalen
PAC 2019 virtual Joerek Van GaalenPAC 2019 virtual Joerek Van Gaalen
PAC 2019 virtual Joerek Van Gaalen
 
PAC 2019 virtual Hemalatha Murugesan
PAC 2019 virtual Hemalatha Murugesan  PAC 2019 virtual Hemalatha Murugesan
PAC 2019 virtual Hemalatha Murugesan
 
PAC 2019 virtual Arjan Van Den Berg
PAC 2019 virtual Arjan Van Den Berg  PAC 2019 virtual Arjan Van Den Berg
PAC 2019 virtual Arjan Van Den Berg
 
PAC 2019 virtual Antoine Toulme
PAC 2019 virtual Antoine ToulmePAC 2019 virtual Antoine Toulme
PAC 2019 virtual Antoine Toulme
 
PAC 2019 virtual Scott Moore
PAC 2019  virtual   Scott Moore PAC 2019  virtual   Scott Moore
PAC 2019 virtual Scott Moore
 
PAC 2019 virtual Stefano Doni
PAC 2019 virtual Stefano Doni   PAC 2019 virtual Stefano Doni
PAC 2019 virtual Stefano Doni
 
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRAN
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRANPAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRAN
PAC 2019 virtual Uma Malini ; Hari Krishnan RAMACHANDRAN
 
PAC 2019 virtual Stijn Schepers
PAC 2019 virtual Stijn SchepersPAC 2019 virtual Stijn Schepers
PAC 2019 virtual Stijn Schepers
 
PAC 2019 virtual Philip Webb
PAC 2019 virtual Philip Webb PAC 2019 virtual Philip Webb
PAC 2019 virtual Philip Webb
 
PAC 2019 virtual Christoph NEUMÜLLER
PAC 2019 virtual Christoph NEUMÜLLERPAC 2019 virtual Christoph NEUMÜLLER
PAC 2019 virtual Christoph NEUMÜLLER
 
PAC 2019 virtual Alexander Podelko
PAC 2019 virtual Alexander Podelko PAC 2019 virtual Alexander Podelko
PAC 2019 virtual Alexander Podelko
 

Recently uploaded

Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...
Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...
Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...
Dr.Costas Sachpazis
 
UNIT-V FMM.HYDRAULIC TURBINE - Construction and working
UNIT-V FMM.HYDRAULIC TURBINE - Construction and workingUNIT-V FMM.HYDRAULIC TURBINE - Construction and working
UNIT-V FMM.HYDRAULIC TURBINE - Construction and working
rknatarajan
 
AKTU Computer Networks notes --- Unit 3.pdf
AKTU Computer Networks notes ---  Unit 3.pdfAKTU Computer Networks notes ---  Unit 3.pdf
AKTU Computer Networks notes --- Unit 3.pdf
ankushspencer015
 

Recently uploaded (20)

College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik
College Call Girls Nashik Nehal 7001305949 Independent Escort Service NashikCollege Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik
College Call Girls Nashik Nehal 7001305949 Independent Escort Service Nashik
 
Call Girls Service Nagpur Tanvi Call 7001035870 Meet With Nagpur Escorts
Call Girls Service Nagpur Tanvi Call 7001035870 Meet With Nagpur EscortsCall Girls Service Nagpur Tanvi Call 7001035870 Meet With Nagpur Escorts
Call Girls Service Nagpur Tanvi Call 7001035870 Meet With Nagpur Escorts
 
ONLINE FOOD ORDER SYSTEM PROJECT REPORT.pdf
ONLINE FOOD ORDER SYSTEM PROJECT REPORT.pdfONLINE FOOD ORDER SYSTEM PROJECT REPORT.pdf
ONLINE FOOD ORDER SYSTEM PROJECT REPORT.pdf
 
Booking open Available Pune Call Girls Koregaon Park 6297143586 Call Hot Ind...
Booking open Available Pune Call Girls Koregaon Park  6297143586 Call Hot Ind...Booking open Available Pune Call Girls Koregaon Park  6297143586 Call Hot Ind...
Booking open Available Pune Call Girls Koregaon Park 6297143586 Call Hot Ind...
 
Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
Top Rated  Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...Top Rated  Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
Top Rated Pune Call Girls Budhwar Peth ⟟ 6297143586 ⟟ Call Me For Genuine Se...
 
Processing & Properties of Floor and Wall Tiles.pptx
Processing & Properties of Floor and Wall Tiles.pptxProcessing & Properties of Floor and Wall Tiles.pptx
Processing & Properties of Floor and Wall Tiles.pptx
 
KubeKraft presentation @CloudNativeHooghly
KubeKraft presentation @CloudNativeHooghlyKubeKraft presentation @CloudNativeHooghly
KubeKraft presentation @CloudNativeHooghly
 
UNIT-III FMM. DIMENSIONAL ANALYSIS
UNIT-III FMM.        DIMENSIONAL ANALYSISUNIT-III FMM.        DIMENSIONAL ANALYSIS
UNIT-III FMM. DIMENSIONAL ANALYSIS
 
(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...
(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...
(SHREYA) Chakan Call Girls Just Call 7001035870 [ Cash on Delivery ] Pune Esc...
 
Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...
Booking open Available Pune Call Girls Pargaon  6297143586 Call Hot Indian Gi...Booking open Available Pune Call Girls Pargaon  6297143586 Call Hot Indian Gi...
Booking open Available Pune Call Girls Pargaon 6297143586 Call Hot Indian Gi...
 
Glass Ceramics: Processing and Properties
Glass Ceramics: Processing and PropertiesGlass Ceramics: Processing and Properties
Glass Ceramics: Processing and Properties
 
The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...
The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...
The Most Attractive Pune Call Girls Budhwar Peth 8250192130 Will You Miss Thi...
 
Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...
Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...
Sheet Pile Wall Design and Construction: A Practical Guide for Civil Engineer...
 
Introduction to IEEE STANDARDS and its different types.pptx
Introduction to IEEE STANDARDS and its different types.pptxIntroduction to IEEE STANDARDS and its different types.pptx
Introduction to IEEE STANDARDS and its different types.pptx
 
The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...
The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...
The Most Attractive Pune Call Girls Manchar 8250192130 Will You Miss This Cha...
 
Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...
Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...
Structural Analysis and Design of Foundations: A Comprehensive Handbook for S...
 
UNIT-V FMM.HYDRAULIC TURBINE - Construction and working
UNIT-V FMM.HYDRAULIC TURBINE - Construction and workingUNIT-V FMM.HYDRAULIC TURBINE - Construction and working
UNIT-V FMM.HYDRAULIC TURBINE - Construction and working
 
AKTU Computer Networks notes --- Unit 3.pdf
AKTU Computer Networks notes ---  Unit 3.pdfAKTU Computer Networks notes ---  Unit 3.pdf
AKTU Computer Networks notes --- Unit 3.pdf
 
High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts
High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur EscortsHigh Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts
High Profile Call Girls Nagpur Meera Call 7001035870 Meet With Nagpur Escorts
 
Online banking management system project.pdf
Online banking management system project.pdfOnline banking management system project.pdf
Online banking management system project.pdf
 

PAC 2020 Santorin - Joerek Van Gaalen

  • 1. PERFORMANCE IS NOT A MYTH P E R F O R M A N C E A D V I S O R Y C O U N C I L SANTORINI GREECE FEBRUARY 26 - 27 2020 Extreme Performance Testing Joerek van Gaalen
  • 2. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Introduction • Joerek van Gaalen • Performance specialist since 2005 • Experience with many large scaled performance tests
  • 3. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L 2 Million Vuser test
  • 4. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L How to simulate so many virtual users?
  • 5. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Neotys Neoload 800 load generators Every instance local DNS server Different (cloud) providers
  • 6. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Challenges • Not being the bottleneck yourself • Being able to simulate the users / load realistic
  • 7. P E R F O R M A N C E A D V I S O R Y C O U N C I L Optimise your tests
  • 8. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Tune your script • Goal is to minimise resources • Use strict regular expressions without many wildcards • Do not store previous response • Reconsider assertion when you already have a variable extractor
  • 9. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Tune your controller • Consider creating your own variable manager outside the controller • Avoid using live status updates of virtual users • Consider more aggregation of data • Limit stored number of errors per second • Use linear increase of users rather then big steps
  • 10. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Tune your agent • Run stress test with single agent and verify its limits • Approx 3000 Vusers per agent is usually good • Tune the kernel (open file descriptors, TCP settings) • Tune Xmx vs real memory
  • 11. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Use a lot cloud instances • Ideally use many different cloud providers (AWS, DigitalOcean, Google, Azure) • Use vendor provided cloud load generators • AWS is ideal, you can start hundreds in a few clicks • c5 instance is usually a good choice, but know if the test is CPU, Memory or Bandwidth heavy
  • 12. P E R F O R M A N C E A D V I S O R Y C O U N C I L Making your tests realistic
  • 13. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Mimic production usage • Try to do it in perfection
  • 14. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L DNS Load balancing • Lower DNS Cache timeout to 1 second • Install local DNS Server per agent and override TTL • Use open DNS Server with different geographical locations • Balance in hostnames in your scripts
  • 15. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Agent spread • Geographical spread • Different providers and peering
  • 16. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Scripts • Do not exclude your CDN objects. They can face quotas or limits
  • 17. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Streaming services • For streaming services, balance between available bandwidth and used bitrates • Or maximise the size of your video segment size Range: bytes=0-102400
  • 18. P E R F O R M A N C E A D V I S O R Y C O U N C I L Specific heavy load performance issues
  • 19. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L CDN performance issues • Quotas & Limits reached • Too much load against your origins • Edge servers cannot handle the load
  • 20. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L CDN performance issues Many egde nodes can stress the origin Many stream files can stress the origin or bad caching /index.m3u8?time=1582235622351
  • 21. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L CDN performance issues • Not much geographical user spread • 3rd party services can stress a single edge node ? ? ? ? ?? ? ? ? ? ? ? ?
  • 22. P E R F O R M A N C E A D V I S O R Y C O U N C I L byP E R F O R M A N C E A D V I S O R Y C O U N C I L Joerek van Gaalen jvangaalen@breakingit.nl