Effective Use of Load Testing Tools to Tune Web Applications


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Effective Use of Load Testing Tools to Tune Web Applications

  1. 1. Effective Use of Load Testing Tools to Tune Web Applications Tom Igielski, CSTE, CPC, CI The Thomas Consulting Group, Inc. [email_address] January 8 th , 2009
  2. 2. Discussion Topics <ul><li>Web Terminology and Architectures </li></ul><ul><li>Common Problem Areas </li></ul><ul><li>Suggested Testing Approaches </li></ul><ul><li>How Does Architecture Affect Scalability </li></ul><ul><li>Tuning Your Application </li></ul>
  3. 3. Application Scalability Web applications are considered “scalable” if a graph depicting Web page response times compared to the number of concurrent users accessing the system increases in a linear fashion Users Response Time Expected Acceptable Crash Ideal
  4. 4. Bottleneck <ul><li>A limiting resource that affects the system under test </li></ul><ul><li>Typically the cause of slow or unacceptable performance </li></ul><ul><li>Affects performance and scalability of a system </li></ul><ul><li>Only one in existence at any point in time </li></ul>
  5. 5. Typical Web Problem Areas Application Server Hub Hub Router ISP1 Router Load Balancer ISP2 Network vs. Server issues External Datafeed Web Server Database Server Firewall Internet bottlenecks Application server issues End-user Connection issues Web Server Load balancer problems Faulty web server within a cluster Server HW, device and OS issues ISP performance and peering Content issues Errors
  6. 6. Candidate Tuning Areas
  7. 7. Why Do Bottlenecks Exist? <ul><li>Obvious </li></ul><ul><ul><li>Inadequate Hardware </li></ul></ul><ul><ul><li>Inadequate Network Capacity </li></ul></ul><ul><ul><li>Poor Implementation / Tuning of Databases </li></ul></ul><ul><li>Not So Obvious </li></ul><ul><ul><li>Poor Architecture and System Design </li></ul></ul><ul><ul><li>Limited Expertise in System Design / Development </li></ul></ul><ul><ul><li>Changing Requirements </li></ul></ul><ul><ul><li>Changing Project Plans </li></ul></ul><ul><ul><li>Not Thinking About “Load” or “Scalability” or “Performance” During Des & Dev </li></ul></ul>
  8. 8. Suggested Testing Approach <ul><li>Plan the Testing Activities </li></ul><ul><ul><li>Develop a Strategy </li></ul></ul><ul><ul><li>Clearly State Your Objectives </li></ul></ul><ul><ul><li>Define the Scope of the Performance Test </li></ul></ul><ul><ul><li>Identify the Performance Requirements </li></ul></ul><ul><ul><li>Develop Necessary Test Cases </li></ul></ul><ul><ul><li>Line Up the Necessary Testing Resources </li></ul></ul><ul><ul><li>Define Needed Test Data and Process to Maintain </li></ul></ul><ul><ul><li>Identify any Necessary Tools </li></ul></ul>
  9. 9. Suggested Testing Approach <ul><li>Build the Test Team </li></ul><ul><ul><li>QA / Testing Personnel </li></ul></ul><ul><ul><li>Database Administrators </li></ul></ul><ul><ul><li>Network Engineers </li></ul></ul><ul><ul><li>Application Architects </li></ul></ul><ul><ul><li>Developers </li></ul></ul><ul><ul><li>Analysts </li></ul></ul><ul><ul><li>SME’s </li></ul></ul>
  10. 10. Suggested Testing Approach <ul><li>Develop the Test Scripts </li></ul><ul><ul><li>Select Busiest Pages </li></ul></ul><ul><ul><ul><li>Always Include the Home Page </li></ul></ul></ul><ul><ul><li>Test Most Common Business Transactions </li></ul></ul><ul><ul><li>Test Most Critical Business Functions </li></ul></ul><ul><ul><li>Test Pages that are Suspect </li></ul></ul>
  11. 11. Suggested Testing Approach <ul><li>Establish the Test Environment </li></ul><ul><ul><li>Procure the Load Test Tool(s) </li></ul></ul><ul><ul><ul><li># Vusers Needed </li></ul></ul></ul><ul><ul><ul><li>Protocols to Support </li></ul></ul></ul><ul><ul><ul><li>Monitors Necessary </li></ul></ul></ul><ul><ul><li>Procure / Configure Test Servers </li></ul></ul><ul><ul><li>Establish Necessary Databases </li></ul></ul><ul><ul><li>Create / Maintain Test Data </li></ul></ul>
  12. 12. Running Load Tests <ul><li>Start load testing software and allow the system to stabilize before adding many users </li></ul><ul><li>Start ramping up users until you see some system slowdown appear </li></ul><ul><li>Check system components for problems </li></ul><ul><ul><li>Could be servers, application, network </li></ul></ul>
  13. 13. Ideal Business Scenario 12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11 12 Customer Search 250 users, 50 / hr / user Customer Referrals 100 users, 5 / hr / user Credit Checks 500 users, 5 / hr / user Recommend Product 300 users, 5 / hr / user Loan Approval 150 users, 2 / hr / user Business Process am pm Peak Load = BP Mix for Scenario
  14. 15. Load Test Cases <ul><li>Generally 3 Categories </li></ul><ul><ul><li>End-to-end </li></ul></ul><ul><ul><li>Modular </li></ul></ul><ul><ul><li>Single or low number of pages </li></ul></ul><ul><li>Modular tests of single / low number of pages work best for isolating bottlenecks </li></ul><ul><li>Create short tests that focus narrowly on problem pages or functions </li></ul><ul><li>Run scripts one at a time, with either fixed delays or no delays </li></ul>
  15. 16. Successful Load Tests <ul><li>Metrics </li></ul><ul><ul><li>Hits / Sec </li></ul></ul><ul><ul><li># Virtual Users </li></ul></ul><ul><ul><li>Transaction Response Time </li></ul></ul><ul><ul><li>Transactions / Second </li></ul></ul><ul><ul><li>Pages / Second </li></ul></ul><ul><ul><li>% CPU Utilization </li></ul></ul>
  16. 17. Desired Metric: Hits / Second <ul><li>The number of requests made to the web server per second </li></ul><ul><li>How many should the application support? </li></ul><ul><li>Determine peak based on the assumption that a % of activity in a day comes within a window of a few hours </li></ul>Let H = peak hits per second Let h = # hits received over a one month period Let a = % of activity that comes during peak time Let t = peak time in hours then H = h * a / (days * t * minutes * seconds) H = h * a / (108,000 * t)
  17. 18. Desired Metric: # Virtual Users <ul><li>How many users should the application support? </li></ul><ul><li>Determine the peak Virtual Users </li></ul><ul><ul><li>Peak hits/second + page view times </li></ul></ul>Let U = peak virtual users Let H = peak hits per second Let p = average number of hits / page Let v = average time a user views a page U = (H / p) * v
  18. 19. Metrics - Example h = 150,000,000 hits per month a = 10% of traffic occurs during peak time t = peak time is 2 hours p = a page consists of 6 hits v = the average view time is 30 seconds H = (h X a) / (108,000 * t) H = (150,000,000 * .1) / (108,000 X 2) H = 48 U = (H / p) * v U = (48 / 6) * 30 U = 8 * 30 U = 240 Desired Metric – 48 Hits / Sec or 240 Virtual Users
  19. 20. How # VUsers Affect Response Time
  20. 21. Performance Requirements <ul><li>Number of “registered” users </li></ul><ul><li>Number of simultaneous users </li></ul><ul><li>Data growth rate </li></ul><ul><li>Transaction Rates </li></ul><ul><ul><li>Pages / Sec </li></ul></ul><ul><ul><li>Hits / Sec </li></ul></ul><ul><ul><li>Business Functions / Unit of Time </li></ul></ul><ul><li>Response time limits </li></ul><ul><li>How users will use the system </li></ul><ul><ul><li>Business scenarios </li></ul></ul><ul><li>Impending events that will drive use </li></ul>
  21. 22. Architecture and Tuning <ul><li>Servers </li></ul><ul><ul><li>System Utilization (CPU) </li></ul></ul><ul><ul><li>Memory Utilization </li></ul></ul><ul><ul><li>Throughput </li></ul></ul><ul><ul><li>Disk I/O </li></ul></ul><ul><ul><li>Current Connections </li></ul></ul><ul><ul><li>Queue </li></ul></ul><ul><ul><li>Queue Wait Time </li></ul></ul><ul><li>Databases </li></ul><ul><ul><li>Table Scans </li></ul></ul><ul><ul><li>Table Locks </li></ul></ul><ul><ul><li>Cache Hits </li></ul></ul><ul><ul><li>Indexes </li></ul></ul>
  22. 23. Architecture and Tuning <ul><li>Load Balancers </li></ul><ul><ul><li>Total Concurrent Connections </li></ul></ul><ul><ul><li>System Utilization (CPU) </li></ul></ul><ul><ul><li>Throughput </li></ul></ul><ul><ul><li>Load Distribution </li></ul></ul><ul><ul><li>Load Balancing Mechanism </li></ul></ul><ul><li>Firewalls </li></ul><ul><ul><li>Total Concurrent Connections </li></ul></ul><ul><ul><li>Total Current SSL Connections </li></ul></ul><ul><ul><li>System Utilization (CPU) </li></ul></ul><ul><ul><li>Throughput </li></ul></ul>
  23. 24. Example Problems <ul><li>High server CPU usage but goals not met </li></ul><ul><ul><li>Have servers been tuned? </li></ul></ul><ul><ul><li>Has application code been optimized? </li></ul></ul><ul><ul><li>SSL overhead? </li></ul></ul><ul><ul><li>Additional Hardware </li></ul></ul><ul><li>Low server memory availability </li></ul><ul><ul><li>Sufficient memory for application </li></ul></ul><ul><ul><li>Memory leak? </li></ul></ul>
  24. 29. Example Problems <ul><li>Low activity on systems but increased transaction response time </li></ul><ul><ul><li>Look at requests queued </li></ul></ul><ul><ul><li>Too few connections available </li></ul></ul><ul><li>Application server activity ↓ but response time and timeouts ↑ as observed by client </li></ul><ul><ul><li>Check server configuration </li></ul></ul><ul><ul><li>Look at execute threads available </li></ul></ul><ul><ul><li>Network problems </li></ul></ul>
  25. 30. Example Problems <ul><li>Response times “spike” during the run </li></ul><ul><ul><li>Set up application servers in “cluster” mode </li></ul></ul><ul><ul><ul><li>Eliminates problems with garbage collection </li></ul></ul></ul><ul><ul><li>Garbage Collection </li></ul></ul><ul><ul><ul><li>Heap size large, incorrect garbage collection settings </li></ul></ul></ul><ul><li>Disk I/O and CPU high on Application Servers </li></ul><ul><ul><li>Application or server logging levels too high </li></ul></ul>
  26. 31. Affects on Response time
  27. 32. Example Problems <ul><li>Poor Java server performance </li></ul><ul><ul><li>Optimize the JVM </li></ul></ul><ul><ul><li>Check JVM switches for starting </li></ul></ul><ul><ul><ul><li>Memory consumption </li></ul></ul></ul><ul><ul><ul><li>Garbage collection </li></ul></ul></ul><ul><ul><ul><li>Thread counts </li></ul></ul></ul><ul><ul><li>Consider Using Multiple Instances </li></ul></ul>
  28. 35. Comparing Thread Counts
  29. 36. Comparing Thread Counts
  30. 37. Example Problems <ul><li>Web servers being used unequally with load balancer in place </li></ul><ul><ul><li>Problem with the load balancing algorithm being used </li></ul></ul><ul><ul><li>Servers sending “false” busy stats back to load balancer </li></ul></ul><ul><ul><li>“Run away” processes on select servers </li></ul></ul>
  31. 38. Load Balancing Problems
  32. 39. Example Problems <ul><li>Slow database query response </li></ul><ul><ul><li>Use profiling tool to identify slow queries </li></ul></ul><ul><ul><li>Optimize queries </li></ul></ul><ul><ul><ul><li>Eliminate unnecessary sorting </li></ul></ul></ul><ul><ul><ul><li>Eliminate unnecessary triggers </li></ul></ul></ul><ul><ul><ul><li>Minimize number of temp tables </li></ul></ul></ul><ul><ul><ul><li>Use stored procedures where possible </li></ul></ul></ul><ul><ul><li>Create necessary indexes </li></ul></ul>
  33. 40. Database Problems
  34. 41. Monitoring Activity
  35. 42. When Things Go Well
  36. 43. When Things Go Well
  37. 44. Summary <ul><li>Web architectures are complex and provide many areas that can be “tuned” </li></ul><ul><li>Develop a specific process for Load Testing </li></ul><ul><li>Take a surgical approach to address problems – one at a time </li></ul>
  38. 45. Questions?