July webinar l How to Handle the Holiday Retail Rush with Agile Performance Testing
•Download as PPTX, PDF•
0 likes•164 views
In this Q&A-style webinar, you'll learn: 1. How and why to load test at least three months prior to the holidays 2. How to integrate CI/CD into your holiday load testing 3. How to determine and evaluate load curves
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to July webinar l How to Handle the Holiday Retail Rush with Agile Performance Testing
Similar to July webinar l How to Handle the Holiday Retail Rush with Agile Performance Testing (20)
More from Apica
More from Apica (12)
Recently uploaded
Recently uploaded (20)
July webinar l How to Handle the Holiday Retail Rush with Agile Performance Testing
- 1. How to Handle the Holiday Retail Rush with Agile Performance Testing Troy Presley & Carter Ransom
- 3. Performance Matters Organizations with degraded business apps (IDG) Problems identified for IT by end-users (EMA) - employees – 43% - business stakeholders – 13% - customers – 12% Problems resolved > 1 month, if ever (Forrester) Engineering time spent in triage (TechValidate) 75% 68% 31% 40%
- 4. Unit and Functional Tests Don’t Find Everything • Concurrency Bugs – Bugs that only present when same code is run multiple times at the same time • Compositional Bugs – Bugs caused by the interplay between separate pieces of your application • DB Indexes and Locks – Do the indexes and query locks on your database play nicely at load • Application or Web Server Configuration – Do your application or web servers exceed memory, socket, or other configuration constraints • Infrastructure Limitations – Bandwidth, Session Tables, Disk IO, Etc.
- 6. THEN
- 7. Application Complexity is Increasing NOW
- 8. Can You Answer These Basic Questions? • What are your application’s bottlenecks? • Performance at normal and peak usage levels? • Most common causes of performance issues or load based failures? • Is your hardware optimized to meet business objectives?
- 9. When to Load Test?
- 10. When to Load Test – Early is GOOD When performing load tests to prepare for holiday traffic, start early so you have time to fix problems • Allow enough time for several fix+test iterations • Take your code lock date into account • Understand the complexity of your application • Compensate for external teams
- 12. When to Load Test – Always is BETTER Consistently running load tests ensures that you already know your limits, and aren’t forced to rush development for peak events • Integrate load testing as part of your deployment strategy • Run smaller load tests before or during deployment, and schedule periodic larger load tests to understand your performance, bottlenecks, and breaking points under peak loads
- 14. Unit Testing Functional Testing User Acceptance DevOps Tools & Automation Application Development Lifecycle Performance Testing Automation
- 15. What Kind of Load Tests Should You Use?
- 16. Three types of load tests Automation Stress Test Concurrency Continuous Ramp
- 17. Stress Tests Continuously increase load to determine absolute theoretical bottlenecks and breaking points • Start with known good traffic levels • Run successive tests while doubling the traffic each time • When breaking point is found, you can run tests between the last successful test and this one to find your exact breaking point(s)
- 18. Concurrency Tests Realistic traffic to find practical performance and failure limits • Determine your actual traffic levels (average and peak), and the ratio of user flows/actions • Run a combination of load tests that simulate realistic traffic • Perform “what if” tests by either changing the mix of user flows/actions, or increasing the load of one or more of them • Concurrency Tests against auto-scaling environments can test scaling response times
- 19. Single test, slowly increasing traffic to find practical performance thresholds Continuous Ramp-up • Find the thresholds to trigger auto-scaling • Include server performance metrics • Perfmon/Sysstat agent on app server • APM integration
- 21. What Does It Mean? (Load Curves) USERS SESSIONDURATION USERS THROUGHPUT USERS FAILURES/ERRORS Ideal Results
- 22. What Does It Mean? (Load Curves) USERS SESSIONDURATION USERS THROUGHPUT USERS FAILURES/ERRORS Realistic Results Finding Your Peak Load SAFE PEAK UNSTABLE
- 23. Real Stress Test – Session time per loop increase
- 24. Real Stress Test – Transaction Rate Impact
- 25. Real Stress Test – Stability
- 26. Questions?
- 27. Additional Resources https://www.apicasystem.com/about-us/resources/ - Whitepapers, Webinars, Case Studies - Blog - Signup for Apica LoadTest Free Trial - Signup for Apica Monitoring Free Trial Contact: Troy Presley - troy.presley@apicasystem.com Carter Ransom - carter.ransom@apicasystem.com
- 28. Recording Your Load Test
- 29. Questions to Ask When Recording Scripts What do you want to know? • Max concurrent users before failure • Performance at expected peak • 3rd party content impact on performance • Current bottlenecks • Auto-scaling results at various loads • Etc.
- 30. Questions to Ask When Recording Scripts What requests matter? • Do you care about static/CDN served content? • Do you need to include 3rd party content? • Do supporting requests (e.g. auto-populate) matter?
- 31. Questions to Ask When Recording Scripts Do you need unique or dynamic data? • Usernames and passwords • Form data (e.g. dates) • Product or record ID’s • Etc.
- 32. Questions to Ask When Recording Scripts What constitutes a failure? • Content verification • HTTP Status checks • General or request specific time limits • Unusual response sizes
- 33. Steps to Recording Scripts 1. Determine the user flow 2. Record using a proxy recorder 3. Remove unnecessary requests 4. Handle variables 5. Add external data 6. Configure failure detection
- 35. Considerations When Running Load Tests •SaaS vs Private •Target Environment (Dev, DR, Prod) •Other performance metrics (APM, etc.) •Database preparation •Team communication methods •Criteria for aborting
- 37. Load Testing @ Deployment Stages • Dev Environments – Good for finding problems early, but unreliable for absolute performance metrics – Catch concurrency, composition, or DB problems • Staging/QA Environments – Catch problems caused by rolled up commits that weren’t visible when testing singular changes – Compare to previous runs to get early warnings for reduced performance or peak capacity – May or may not be reliable indicators of production performance depending on the stability of the environment, and similarity to the production environment • Pre-Production – Should be as close to the production environment as possible – Best indication of production performance and load capacity before live deployment • Production – Useful for absolute verification of readiness for expected traffic spikes (e.g. product launces, sales, holidays, marketing events) – Can be used for periodic validations (DR, seamless code deployment, performance validation)
Editor's Notes
- Integration – with continuous delivery - minimize effort insert performance testing in the release cycle Graph – Simple Page level detail of response times from regularly scheduled load test If you could Kick off a load test prior to release and automate the results that come back from the test…. You could gate deployments if the any of the issues from those 3 test profiles arises Apica Load Test – has Apis that allow for integration with the CI/CD tools here and many others, to kick off a tests, or (regularly schedule them from the portal) Return high level or low level error level resolution detail to gate deployments Not only execution but also Purpose for Performance Tests in your agile release cycle… What constitutes a failure? Content verification HTTP Status checks General or request specific time limits Unusual response sizes
- OK 3 types of load tests – so what – back to Release lifecycle
- What constitutes a failure? Content verification HTTP Status checks General or request specific time limits Unusual response sizes