Performance TestingFundamentals for Managers              Presented by:               Jeff Roth        VP of Consulting Se...
Agenda Performance Testing 101 Components of Performance Testing Performance Testing Lifecycle                      © 2012...
What is Performance Testing?        Batch Jobs                         Stress                                             ...
Business Goals Will my system meet my goals?    User expectations (response time, accuracy)    Load capacity (user volume,...
Please excuse me while I shout .PERFORMANCE TESTING IS DEFINED BY WHAT YOU  WANT TO ACCOMPLISHNOT WHAT TOOLS YOU OWN      ...
What Can Performance Testing Do?Before we answer this question, let s start with a littleanatomy of IT systems            ...
User Interfaces Input Keyboard, Touch Screen, Voice, Camera, Hand Scanner, etc. Output    Screen, Printer, Sound, etc. Pro...
Performance Testing DevicesBusiness Goals  Configuration Optimization    Best hardware for the job    OS/Software configur...
Performance Testing Systems Include devices or not Real vs. simulated integration points Isolate risk areas Iterative proc...
COMPONENTS OF APERFORMANCE TEST         © 2012 BenchmarkQA
What Makes up a Performance Test    Load Generator                            Target of Test                              ...
What Makes up a Performance TestVirtual User Script  Home Page              Search Item        Select Item  Activity/Busin...
What Makes up a Performance TestTechnical Details of a Virtual User Script  Protocol - HTTP, AJAX, SOAP, SAP, CITRIX, etc....
Performance Test Scenario One or more test scripts Quantity of VUsers per script Velocity of activity per user Designate r...
Ramp Up And Failure                                           Number of       Percent of                          VUsers A...
PERFORMANCE TESTLIFECYCLE         © 2012 BenchmarkQA
PMBOK - PROJECT MANAGEMENT PHASES Software Development Lifecycle                                  SELECTION   INITIATION  ...
Performance Testing Lifecycle                                   Initial                                                   ...
4 Test Effort Scope                             4 Software Components                             4 Target System Architec...
4 Load Test Tools                              4 Test Script IDE                              4 Load Generation Farm      ...
4 Test Scripts                         4    Record Business Processes                         4    Correlate Data         ...
4 Schedule Execution                             4 No Users in Target System                             4 Anti-Virus Scan...
Typical User Response Graph               © 2012 BenchmarkQA   Slide 23
4 Test Goal Assessment                              4 Response Time Performance                              4 System Util...
Things to Remember Set clear objectives    To ensure all team members agree on the questions performance    testing will a...
Things to Remember Understand what happens   Every component of a system must be monitored and data collected   Subject ma...
Thank You!For additional information on Consulting Services with             BenchmarkQA, please contact:                 ...
Upcoming SlideShare
Loading in …5
×

BenchmarkQA Performance Testing Quality Forum March 2012

1,389 views
1,332 views

Published on

Understanding the fundamentals of performance and load testing will help you budget, plan, and manage performance testing to effectively find and resolve performance issues before they become a problem for your users and your company.

Jeff Roth delivered this presentation at BenchmarkQA’s Software Quality Forum on March 22, 2012. Jeff provided a non-technical explanation of the fundamentals of successful performance testing.

Topics include:
• Performance Testing 101
• Components of Performance Testing
• Performance Testing Lifecycle

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,389
On SlideShare
0
From Embeds
0
Number of Embeds
63
Actions
Shares
0
Downloads
84
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

BenchmarkQA Performance Testing Quality Forum March 2012

  1. 1. Performance TestingFundamentals for Managers Presented by: Jeff Roth VP of Consulting Services BenchmarkQA Date: March 22, 2012
  2. 2. Agenda Performance Testing 101 Components of Performance Testing Performance Testing Lifecycle © 2012 BenchmarkQA Slide 2
  3. 3. What is Performance Testing? Batch Jobs Stress Load Volume Scalability System Benchmark Failure Point Baseline Durability Failover System Optimization Infrastructure Validation System Tuning Desktop Benchmark © 2012 BenchmarkQA Slide 3
  4. 4. Business Goals Will my system meet my goals? User expectations (response time, accuracy) Load capacity (user volume, cost of configuration) Data integrity (at normal loads, peak loads, and failure) Stand up over time (durability) Failover capability (continuity through failure) System growth (scalability) When will my system degrade and when will it fail? What do I need (infrastructure) for effective performance? © 2012 BenchmarkQA Slide 4
  5. 5. Please excuse me while I shout .PERFORMANCE TESTING IS DEFINED BY WHAT YOU WANT TO ACCOMPLISHNOT WHAT TOOLS YOU OWN © 2012 BenchmarkQA Slide 5
  6. 6. What Can Performance Testing Do?Before we answer this question, let s start with a littleanatomy of IT systems © 2012 BenchmarkQA Slide 6
  7. 7. User Interfaces Input Keyboard, Touch Screen, Voice, Camera, Hand Scanner, etc. Output Screen, Printer, Sound, etc. Processing CPU, GPU Storage RAM, Disk, SDD, etc. Connectivity LAN, WAN, WiFi, 3G/4G © 2012 BenchmarkQA Slide 7
  8. 8. Performance Testing DevicesBusiness Goals Configuration Optimization Best hardware for the job OS/Software configuration Baseline Device Performance Embedded devices software dependent upon hardware With and without connectivity © 2012 BenchmarkQA Slide 8
  9. 9. Performance Testing Systems Include devices or not Real vs. simulated integration points Isolate risk areas Iterative process Load testing tools © 2012 BenchmarkQA Slide 9
  10. 10. COMPONENTS OF APERFORMANCE TEST © 2012 BenchmarkQA
  11. 11. What Makes up a Performance Test Load Generator Target of Test Web Farm Mainframe Controller Databases © 2012 BenchmarkQA Slide 11
  12. 12. What Makes up a Performance TestVirtual User Script Home Page Search Item Select Item Activity/Business process © 2012 BenchmarkQA Slide 12
  13. 13. What Makes up a Performance TestTechnical Details of a Virtual User Script Protocol - HTTP, AJAX, SOAP, SAP, CITRIX, etc. Uniqueness Session variables User IDs Test Data Files Correlate within the script Made up Think time © 2012 BenchmarkQA Slide 13
  14. 14. Performance Test Scenario One or more test scripts Quantity of VUsers per script Velocity of activity per user Designate ramp up & test schedule based on goal © 2012 BenchmarkQA Slide 14
  15. 15. Ramp Up And Failure Number of Percent of VUsers Axis Errors Axis Errors begin to occur @ 400 VUsers © 2012 BenchmarkQA Slide 15
  16. 16. PERFORMANCE TESTLIFECYCLE © 2012 BenchmarkQA
  17. 17. PMBOK - PROJECT MANAGEMENT PHASES Software Development Lifecycle SELECTION INITIATION PLANNING EXECUTION & CONTROL CLOSURE Project Definition Requirements Analysis & Design Technical Analysis & (SDLC) Design Build/Integrate & Test User Acceptance Test Deployment & Wrap-UpPerformance Testing Initial Planning Initial Design Planning Lifecycle Analyze Build Execute © 2012 BenchmarkQA Slide 17
  18. 18. Performance Testing Lifecycle Initial Design PlanningThe Performance Testlifecycle requires some initial Analyze Buildupfront planning work, andbecomes a cyclical effort toachieve your performancegoals. Execute Typical engagements are 7 to 9 weeks including initial planning. © 2012 BenchmarkQA Slide 18
  19. 19. 4 Test Effort Scope 4 Software Components 4 Target System Architecture 4 High Volume Business Processes and Resource Intensive System Activities 4 Workload Profiles 4 Performance Test Goals Participants 4 Resources 4 Target Test System HardwareØ Project StakeholdersØ Project Manager 4 Project Personnel and Test Effort RolesØ System Architects 4 Test Data Management PlanØ Key SMEsØ Performance Architect 4 Test Tools (load injection hardware and software) 4 Test Schedule Duration 4 Project Schedule 4 Project Dependencies 1 to 4 Weeks 4 Performance Test Work Breakdown Structure © 2012 BenchmarkQA Slide 19
  20. 20. 4 Load Test Tools 4 Test Script IDE 4 Load Generation Farm 4 Protocol Licensing (Web, Winsock, Citrix, etc.) 4 Business Process 4 Screen by Screen Walk Through 4 Usage Frequency and Pacing 4 Test Data & Test User IDs Participants 4 System Intensive ProcessesØ Business SMEsØ System SMEs 4 Batch JobsØ Performance Architect 4 Reporting SystemsØ Performance Engineers 4 Interfaces/Feeds 4 Test Monitoring Strategy 4 Servers/Databases/Network Duration 4 Integrate with Load Test Tool (Security Permissions) 1 to 4 Weeks 4 Data Center/ SME System Monitoring © 2012 BenchmarkQA Slide 20
  21. 21. 4 Test Scripts 4 Record Business Processes 4 Correlate Data 4 Parameterized Data 4 Pacing 4 End User Response Time Measurements 4 Typically 4 to 8 Scripts 4 Test Scenarios Participants 4 User Ramp Up ScenarioØ Business SMEs 4 Frequency of Business Process ActivitiesØ System SMEs 4 User Managed ActivitiesØ Performance Engineers 4 Results Storage 4 Verify Test Readiness Duration 4 User Concurrency 2 to 12 Weeks 4 Multi-Script Concurrency 4 Load Generators © 2012 BenchmarkQA Slide 21
  22. 22. 4 Schedule Execution 4 No Users in Target System 4 Anti-Virus Scans, Automated Backups 4 Off Hours Scheduling 4 Monitor Execution 4 Load Test Progress 4 Load Test System Participants 4 Real Time Response Time MetricsØ Project Stakeholders 4 Target System Utilization Under LoadØ Project Manager 4 General Execution ApproachØ System ArchitectØ Key SMEs 4 Small Load First To Confirm System Is StableØ Performance Engineers 4 Reasonable User Load Ramp Up 4 More Than One Test Execution To Confirm Goals Duration 1 to 8 Weeks © 2012 BenchmarkQA Slide 22
  23. 23. Typical User Response Graph © 2012 BenchmarkQA Slide 23
  24. 24. 4 Test Goal Assessment 4 Response Time Performance 4 System Utilization 4 Bottleneck Identification 4 Correlate Response Metrics with System Utilization Metrics 4 Isolate Causal System/User Activity Participants 4 Remediation StrategyØ Project Stakeholders 4 Configuration ChangesØ Project ManagerØ System Architect 4 System ChangesØ Business SMEs 4 Hardware ChangesØ System SMEsØ Performance Architect 4 Test Script Changes & Engineers 4 Load Scenario Change 4 Monitoring Changes Duration 1 to 4 Weeks Repeat Cycle Until Goals Achieved © 2012 BenchmarkQA Slide 24
  25. 25. Things to Remember Set clear objectives To ensure all team members agree on the questions performance testing will answer non-functional requirements solution requirements specification To understand the expectations of the business owners To know when testing is complete Strive for realism Subject the system under test to conditions that match production as closely as possible If possible use the actual production hardware for the performance test © 2012 BenchmarkQA Slide 25
  26. 26. Things to Remember Understand what happens Every component of a system must be monitored and data collected Subject matter experts analyze collected data Provide meaningful recommendations Must improve the end-user experience Must be based on a cost-benefit analysis © 2012 BenchmarkQA Slide 26
  27. 27. Thank You!For additional information on Consulting Services with BenchmarkQA, please contact: Jeff Roth 952.392.2384 jeff.roth@benchmarkqa.com © 2012 BenchmarkQA

×