Performance Testing Process SASQAG  March 2007 Emily Ren  T-Mobile
Why We Need Performance Testing? <ul><li>Before release, managers need to know: </li></ul><ul><ul><li>Do we have enough ha...
Nature of Performance Testing <ul><li>It is very different from functional testing. A very challenging job  </li></ul><ul>...
Why We Need Performance Testing?  <ul><li>The failure of an application can be costly </li></ul><ul><li>Assure performance...
When we do it <ul><li>During design and development   </li></ul><ul><ul><li>What is the best server to support target load...
What we are doing <ul><li>Performance testing before release : </li></ul><ul><ul><ul><li>Application response times   </li...
Plan Test Create  Scripts Scenario Creation Scenario Execution Result Analysis Load Testing Process Performance Tuning
Perf. Test Planning Documents <ul><li>Performance Testing Initial Assessment </li></ul><ul><li>- Pre-test plan document </...
What we are doing <ul><li>1. Test Planning - Before we run load testing </li></ul><ul><li>-  Setup goals </li></ul><ul><ul...
What we are doing – Cont. <ul><li>- Identify usage information -  Business Profile   </li></ul><ul><ul><ul><li>Which busin...
Sample : Business Profile 1 - HR App.   Business Processes 1 min 3 min 2-3 min Preferred  Response  Time (Total, including...
Sample : Business Profile 2 – eCommerce  Business Processes 3-5 sec 3-5 sec 3-5 sec Preferred  Response Time (Each transac...
What we are doing – Cont. <ul><li>-  Business Profile   is the basis for load testing </li></ul><ul><ul><ul><li>It is the ...
What we are doing – Cont. <ul><li>2. Create Scripts </li></ul><ul><li>-  Automate business processes  in LoadRunner VUGen ...
Sample Script <ul><li>web_submit_data(&quot;logon.sap&quot;,  </li></ul><ul><li>&quot;Action=http://watstwscrm02:50000/bd/...
What we are doing – Cont. <ul><li>3. Create Test Scenario  </li></ul><ul><li>-  Build test scenario according to usage inf...
What we are doing – Cont. <ul><li>4. Execute Load Testing  </li></ul><ul><li>Execute test scenarios with automated test sc...
Example Parameters to Monitor <ul><li>system  - % total processor time  </li></ul><ul><li>Memory - page faults/sec </li></...
What we are doing – Cont. <ul><li>5. Analyze Test Result - Analysis </li></ul><ul><li>- Collect statistics and graphs from...
What we are doing – Cont. <ul><li>6. Test Report  </li></ul><ul><li>-  Don’t send LoadRunner result and graphs directly </...
What we are doing – Cont. <ul><li>7.   Performance Tuning </li></ul><ul><li>- Help identify the bottlenecks and degradatio...
What we are doing – Cont. <ul><li>8.  Communication Plan   </li></ul><ul><li>-  Internal Support Team:   </li></ul><ul><li...
Timeline/Activities - Example <ul><li>Test Planning, Script Creation – 4 weeks </li></ul><ul><li>Test Execution – 4 weeks ...
Projects  <ul><li>Projects  : </li></ul><ul><li>All performance testing projects in  </li></ul><ul><li>T-Mobile’s IT dept ...
Automation Tools  - Mercury LoadRunner <ul><ul><li>Scripting : VUGen (Virtual User Generator) </li></ul></ul><ul><ul><li>P...
Automation Tools  – Performance Center   <ul><li>Web-enabled  global load testing tool  Performance Testing team can manag...
Automation Tools - Diagnostics <ul><li>- Pinpoint Root Cause </li></ul><ul><ul><li>Solve tough problems </li></ul></ul><ul...
Diagnostics Methodology in Pre-production <ul><li>Start with monitoring of business process </li></ul><ul><ul><li>Which tr...
Broad Heterogeneous Platform Support <ul><li>WebSphere J2EE/Portal Server </li></ul><ul><li>WebLogic J2EE/Portal Server </...
Performance Engineering  - Bridge the Gap <ul><li>80% of IT Organizations experience failures in apps that passed the test...
Performance Engineering  - HyPerformix Optimizer <ul><li>Configuration sizing, Capacity planning   </li></ul><ul><li>Creat...
What Performance Testing can do for business?  <ul><li>Performance testing is critical. Competition in market is high: cus...
Questions? <ul><li>[email_address] </li></ul><ul><li>[email_address] </li></ul><ul><li>Tel:   (425)748-6655 (desk) </li></...
Upcoming SlideShare
Loading in...5
×

PowerPoint Handouts

1,327

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,327
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
73
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • QA MANAGER—RYALND/RAJ
  • Performance Center is Web-enabled global load testing tool, which is specially designed to streamline the testing process and increase the test efficiency
  • Transcript of "PowerPoint Handouts"

    1. 1. Performance Testing Process SASQAG March 2007 Emily Ren T-Mobile
    2. 2. Why We Need Performance Testing? <ul><li>Before release, managers need to know: </li></ul><ul><ul><li>Do we have enough hardware? </li></ul></ul><ul><ul><li>Can we handle the target load? </li></ul></ul><ul><ul><li>How many users can we handle? </li></ul></ul><ul><ul><li>Is the system fast enough to make customers happy? </li></ul></ul>
    3. 3. Nature of Performance Testing <ul><li>It is very different from functional testing. A very challenging job </li></ul><ul><li>It requires stellar cooperation and coordination: it is a whole team effort! </li></ul><ul><li>Automation tools are very powerful, but expensive and complex, training is needed </li></ul><ul><li>It can be fun too! </li></ul>
    4. 4. Why We Need Performance Testing? <ul><li>The failure of an application can be costly </li></ul><ul><li>Assure performance and functionality under real-world conditions </li></ul><ul><li>Locate potential problems before our customers do </li></ul><ul><li>Reduce development time – multiple rounds of load testing </li></ul><ul><li>Reduce infrastructure cost </li></ul>
    5. 5. When we do it <ul><li>During design and development </li></ul><ul><ul><li>What is the best server to support target load? </li></ul></ul><ul><ul><li>Define system performance requirements </li></ul></ul><ul><li>Before release </li></ul><ul><ul><li>Is the system reliable enough to go into production? </li></ul></ul><ul><ul><li>After functional testing done </li></ul></ul><ul><li>Post-deployment </li></ul><ul><ul><li>What is the cause of performance degradation? </li></ul></ul>
    6. 6. What we are doing <ul><li>Performance testing before release : </li></ul><ul><ul><ul><li>Application response times </li></ul></ul></ul><ul><ul><ul><li>- How long does it take to complete a task? </li></ul></ul></ul><ul><ul><ul><li>Configuration sizing </li></ul></ul></ul><ul><ul><ul><li>- Which configuration provides the best performance level? </li></ul></ul></ul><ul><ul><ul><li>Capacity planning </li></ul></ul></ul><ul><ul><ul><li>- How many users can the system handle? </li></ul></ul></ul><ul><ul><ul><li>Regression </li></ul></ul></ul><ul><ul><ul><li>- Does the new version of the software adversely affect response time? </li></ul></ul></ul><ul><ul><ul><li>Reliability </li></ul></ul></ul><ul><ul><ul><li>- How stable is the system under heavy work load? </li></ul></ul></ul>
    7. 7. Plan Test Create Scripts Scenario Creation Scenario Execution Result Analysis Load Testing Process Performance Tuning
    8. 8. Perf. Test Planning Documents <ul><li>Performance Testing Initial Assessment </li></ul><ul><li>- Pre-test plan document </li></ul><ul><li>- Help project team to brainstorm their test scope </li></ul><ul><li>Performance Test Request Form </li></ul><ul><li>- Detail information related to whole performance testing process, including setup goals, environment, business process, performance requirement (e.g., response time), usage information, internal support team, etc. </li></ul>
    9. 9. What we are doing <ul><li>1. Test Planning - Before we run load testing </li></ul><ul><li>- Setup goals </li></ul><ul><ul><ul><li>Measure application response time </li></ul></ul></ul><ul><ul><ul><li>Configuration sizing </li></ul></ul></ul><ul><ul><ul><li>Capacity planning </li></ul></ul></ul><ul><ul><ul><li>Regression </li></ul></ul></ul><ul><ul><ul><li>Reliability </li></ul></ul></ul><ul><li>- Type of testing </li></ul><ul><ul><ul><li>Load Testing (System performance testing with SLA target load) </li></ul></ul></ul><ul><ul><ul><li>Stress Testing (Capacity testing to find out breaking point) </li></ul></ul></ul><ul><ul><ul><li>Duration Testing (Reliability testing to test the system under load) </li></ul></ul></ul>
    10. 10. What we are doing – Cont. <ul><li>- Identify usage information - Business Profile </li></ul><ul><ul><ul><li>Which business processes to use </li></ul></ul></ul><ul><ul><ul><ul><li>BA, Dev team responsible for definition </li></ul></ul></ul></ul><ul><ul><ul><li>Isolate peak load and peak time </li></ul></ul></ul><ul><ul><ul><ul><li>BA, Dev, application support responsible for definition </li></ul></ul></ul></ul><ul><ul><ul><li>Document user actions and input data for each business process </li></ul></ul></ul><ul><ul><ul><ul><li>SME/Functional Testing team responsible for creation of business process document </li></ul></ul></ul></ul>
    11. 11. Sample : Business Profile 1 - HR App. Business Processes 1 min 3 min 2-3 min Preferred Response Time (Total, including think time) 9000 350 100% Total > 3 min 1000 50 20% Update personal info. > 5 min 6000 200 60% Time Entry > 5 min 2000 100 20% Browse Unacceptable Response Time (Total , including think time) Peak number of concurrent users Avg Number of concurrent users Total Users (%) Business Process
    12. 12. Sample : Business Profile 2 – eCommerce Business Processes 3-5 sec 3-5 sec 3-5 sec Preferred Response Time (Each transaction) 8000 100% Total > 8 sec 1000 4-6 pm 20% Display order > 8 sec 6000 4-6 pm 60% Browse Catalog > 8 sec 1000 4-6 pm 20% Create Order Unacceptable Response Time (Each transaction) Peak Load (# of users) Peak Time Total Users (%) Business Process
    13. 13. What we are doing – Cont. <ul><li>- Business Profile is the basis for load testing </li></ul><ul><ul><ul><li>It is the traffic model of the application </li></ul></ul></ul><ul><ul><ul><li>The better the documentation of the business processes, the better the test scripts and scenarios. </li></ul></ul></ul><ul><ul><ul><li>Save time on script and scenario creation </li></ul></ul></ul><ul><ul><ul><li>Good business profile can make it possible to reuse existing load testing scripts and results later. </li></ul></ul></ul>
    14. 14. What we are doing – Cont. <ul><li>2. Create Scripts </li></ul><ul><li>- Automate business processes in LoadRunner VUGen (Virtual User Generator): </li></ul><ul><ul><li>Scripts are C, C++-like code </li></ul></ul><ul><ul><li>Scripts are different with different protocol/technology </li></ul></ul><ul><ul><li> LoadRunner has about 50 protocols, including WAP </li></ul></ul><ul><li>Record user actions </li></ul><ul><ul><li>Need assistance of SME/Functional Testing group </li></ul></ul><ul><li>Add programming and test data in the scripts </li></ul><ul><ul><li>E.g. add correlation to handle dynamic data, e.g. session id </li></ul></ul><ul><ul><li>Test data may need lot of work from project team </li></ul></ul>
    15. 15. Sample Script <ul><li>web_submit_data(&quot;logon.sap&quot;, </li></ul><ul><li>&quot;Action=http://watstwscrm02:50000/bd/logon.sap&quot;, </li></ul><ul><li>&quot;Method=POST&quot;, </li></ul><ul><li>&quot;RecContentType=text/html&quot;, </li></ul><ul><li>&quot;Referer=http://watstwscrm02:50000/bd/startEBPP.sap&quot;, </li></ul><ul><li>&quot;Snapshot=t3.inf&quot;, </li></ul><ul><li>&quot;Mode=HTML&quot;, </li></ul><ul><li>ITEMDATA, </li></ul><ul><li>&quot;Name=login_submit&quot;, &quot;Value=true&quot;, ENDITEM, </li></ul><ul><li>&quot;Name=j_authscheme&quot;, &quot;Value=default&quot;, ENDITEM, </li></ul><ul><li>&quot;Name=j_alias&quot;, &quot;Value={UserName}&quot;, ENDITEM, </li></ul><ul><li>&quot;Name=j_password&quot;, &quot;Value=coffee@2&quot;, ENDITEM, </li></ul><ul><li>&quot;Name=j_language&quot;, &quot;Value=EN&quot;, ENDITEM, </li></ul><ul><li>&quot;Name=AgreeTerms&quot;, &quot;Value=on&quot;, ENDITEM, </li></ul><ul><li>&quot;Name=Login&quot;, &quot;Value=Log on&quot;, ENDITEM, </li></ul><ul><li>LAST); </li></ul>
    16. 16. What we are doing – Cont. <ul><li>3. Create Test Scenario </li></ul><ul><li>- Build test scenario according to usage information in Business Profile </li></ul><ul><li>Load Calculation </li></ul><ul><li>Can use rendezvous point, IP Spoofing, etc. </li></ul><ul><li>- Run-Time setting </li></ul><ul><li>Think time </li></ul><ul><li>Pacing </li></ul><ul><li>Browser Emulation: simulate browser cache, new user each iteration </li></ul><ul><li>Browser version, bandwidth, etc. </li></ul>
    17. 17. What we are doing – Cont. <ul><li>4. Execute Load Testing </li></ul><ul><li>Execute test scenarios with automated test scripts in LoadRunner Controller </li></ul><ul><li>Isolate top time transactions with low load </li></ul><ul><li>Overdrive test (120% of full load) to isolate SW & HW limitations </li></ul><ul><li>- Work with Internal Support Team to monitor the whole system, e.g., web server, DB server, middleware, etc. </li></ul>
    18. 18. Example Parameters to Monitor <ul><li>system - % total processor time </li></ul><ul><li>Memory - page faults/sec </li></ul><ul><li>Server work queues - bytes transferred/sec </li></ul><ul><li>HTTP Response </li></ul><ul><li>Number of connections </li></ul><ul><li>Support team will have better ideas for what to monitor </li></ul><ul><li>Individual write-up is highly suggested as part of test report </li></ul><ul><li>---need to get csv files, then import to LoadRunner </li></ul>
    19. 19. What we are doing – Cont. <ul><li>5. Analyze Test Result - Analysis </li></ul><ul><li>- Collect statistics and graphs from LoadRunner </li></ul><ul><li>- Report results </li></ul><ul><ul><li>- Most commonly requested results: </li></ul></ul><ul><ul><li>Transaction Response time </li></ul></ul><ul><ul><li>Throughput </li></ul></ul><ul><ul><li>Hits per sec </li></ul></ul><ul><ul><li>HTTP response </li></ul></ul><ul><ul><li>Network Delay </li></ul></ul><ul><ul><li>*Server Performance </li></ul></ul><ul><ul><li>- Merge graphs to make it more meaningful </li></ul></ul><ul><ul><li>Transaction response time under load </li></ul></ul><ul><ul><li>Response time/Vuser vs CPU utilization </li></ul></ul><ul><ul><li>Cross scenario graphs </li></ul></ul>
    20. 20. What we are doing – Cont. <ul><li>6. Test Report </li></ul><ul><li>- Don’t send LoadRunner result and graphs directly </li></ul><ul><li>- Send summary to the whole team </li></ul><ul><li>- Report key performance data and back end performance data </li></ul><ul><li>- Add notes for each test run </li></ul><ul><li>- Keep test history: for team to compare test runs </li></ul>
    21. 21. What we are doing – Cont. <ul><li>7. Performance Tuning </li></ul><ul><li>- Help identify the bottlenecks and degradation points to build an optimal system </li></ul><ul><ul><li>- Hardware, Configuration, Database, Software, etc </li></ul></ul><ul><li>- Drill down on transaction details, </li></ul><ul><li>- e.g. webpage breakdown </li></ul><ul><li>- Diagnostics </li></ul><ul><li>- Show Extended Log to dev team </li></ul><ul><li>- Data returned by server </li></ul><ul><li>- Advanced Trace: Show logs of all VUser messages and function calls </li></ul>
    22. 22. What we are doing – Cont. <ul><li>8. Communication Plan </li></ul><ul><li>- Internal Support Team: </li></ul><ul><li>- PM, BA, environment / development / architect, network, DBA, functional test lead, etc. </li></ul><ul><li>- Resource plan </li></ul>
    23. 23. Timeline/Activities - Example <ul><li>Test Planning, Script Creation – 4 weeks </li></ul><ul><li>Test Execution – 4 weeks </li></ul><ul><li>Trail run - 2 days </li></ul><ul><li>Round 1 – Load Testing: Response time with SLA target load: 1 week </li></ul><ul><li>Round 2 – Stress Testing: find breaking point: 1 week </li></ul><ul><li>Round 3 – Duration (Reliability) test: 2 days </li></ul><ul><li>More performance tuning – 3 days </li></ul><ul><li>Document and deliver final report – 2-3 days </li></ul>
    24. 24. Projects <ul><li>Projects : </li></ul><ul><li>All performance testing projects in </li></ul><ul><li>T-Mobile’s IT dept </li></ul><ul><li>40+ projects in <3 years </li></ul><ul><li>The Standard Performance Testing Process has worked very well on all projects </li></ul>
    25. 25. Automation Tools - Mercury LoadRunner <ul><ul><li>Scripting : VUGen (Virtual User Generator) </li></ul></ul><ul><ul><li>Performance test execution: </li></ul></ul><ul><ul><ul><li>Controller – build test scenarios according to business profile and load calculation </li></ul></ul></ul><ul><ul><ul><li>Load Generator – run virtual users </li></ul></ul></ul><ul><ul><li>Performance test result analysis </li></ul></ul><ul><ul><ul><li>Analysis </li></ul></ul></ul><ul><ul><ul><ul><li>provides test reports and Graphs </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Summarize the system performance </li></ul></ul></ul></ul>
    26. 26. Automation Tools – Performance Center <ul><li>Web-enabled global load testing tool Performance Testing team can manage multiple, concurrent load testing projects across different geographic locations </li></ul><ul><ul><ul><li>User Site - conduct and monitor load tests. </li></ul></ul></ul><ul><ul><ul><li>Privilege Manager- manage user and project access rights </li></ul></ul></ul><ul><ul><ul><li>Administration Site - for overall resource management and technical supervision </li></ul></ul></ul>
    27. 27. Automation Tools - Diagnostics <ul><li>- Pinpoint Root Cause </li></ul><ul><ul><li>Solve tough problems </li></ul></ul><ul><ul><li>Memory leaks and trashing </li></ul></ul><ul><ul><li>Thread deadlock and synchronization </li></ul></ul><ul><ul><li>Instance tracing </li></ul></ul><ul><ul><li>Exceptions </li></ul></ul>
    28. 28. Diagnostics Methodology in Pre-production <ul><li>Start with monitoring of business process </li></ul><ul><ul><li>Which transactions are problematic </li></ul></ul><ul><li>Eliminate system and network components </li></ul><ul><ul><li>Infrastructure monitors and metrics </li></ul></ul><ul><li>Isolate application Tier and method </li></ul><ul><ul><li>Triage (using Transaction Breakdown) </li></ul></ul><ul><li>Correct behavior and re-test </li></ul>
    29. 29. Broad Heterogeneous Platform Support <ul><li>WebSphere J2EE/Portal Server </li></ul><ul><li>WebLogic J2EE/Portal Server </li></ul><ul><li>JBoss, Tomcat, JServ </li></ul><ul><li>Oracle Application Server J2EE </li></ul><ul><li>MS .NET </li></ul><ul><li>Generic/Custom JAVA </li></ul><ul><li>SAP Net/Weaver J2EE/Portal </li></ul><ul><li>Oracle 11i Applications </li></ul><ul><li>Siebel </li></ul>
    30. 30. Performance Engineering - Bridge the Gap <ul><li>80% of IT Organizations experience failures in apps that passed the test phases and rolled into production </li></ul><ul><li>HyPerformix – Performance Engineering </li></ul><ul><ul><li>Production line: Designer, Optimizer and Capacity Manager </li></ul></ul><ul><li>HyPerformix Optimizer (Capacity Planning): can bridge the gap between testing and production environments and leverage load test data to accurately show how the application will perform when in production. </li></ul>
    31. 31. Performance Engineering - HyPerformix Optimizer <ul><li>Configuration sizing, Capacity planning </li></ul><ul><li>Create production-scale models </li></ul><ul><ul><li>– Perf. Test team and Architect team work together </li></ul></ul><ul><li>Load test and production perf. data are seamlessly integrated with Optimizer </li></ul><ul><li>Ensure capacity is match to current and future business requirements </li></ul><ul><li>Reduce risk before application deployment </li></ul>
    32. 32. What Performance Testing can do for business? <ul><li>Performance testing is critical. Competition in market is high: customer switch cost is low, cost to keep customers is high </li></ul><ul><li>Performance Testing can protect revenue by helping to isolate and fix problems in the software infrastructure </li></ul><ul><li>Improve availability, functionality, and scalability of business critical applications </li></ul><ul><li>Ensure products are delivered to market with high confidence that system performance will be acceptable </li></ul><ul><li>Proactive performance testing can decrease costs of production support and help desk </li></ul><ul><li>A good Performance Testing Process is essential to get performance testing done right and on time! </li></ul>
    33. 33. Questions? <ul><li>[email_address] </li></ul><ul><li>[email_address] </li></ul><ul><li>Tel: (425)748-6655 (desk) </li></ul><ul><ul><ul><li>(425)922-7100 (cell) </li></ul></ul></ul>
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×