Performance testing: respect the difference at swqd14

832 views

Published on

Performance testing: respect the difference, highlighting important points of performance testing, presented at Software Quality Days 2014 in Vienna

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
832
On SlideShare
0
From Embeds
0
Number of Embeds
29
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Performance testing: respect the difference at swqd14

  1. 1. Performance Testing: Respect the Difference Software Quality Days 2014 January 16, 2014 Alexander Podelko apodelko@yahoo.com http://alexanderpodelko.com/blog Performance Testing: Respect the Difference @apodelko
  2. 2. A bout Me  Have specialized in performance for the last 16 years  Currently performance testing and optimization of Hyperion products at Oracle  Board director at the Computer Measurement Group (CMG) http://cmg.org, organization of performance and capacity professionals  Next conference November 3-6, 2014 in Atlanta, GA Disclaimer: The views expressed here are my personal views only and do not necessarily represent those of my current or previous employers. All brands and trademarks mentioned are the property of their owners. Performance Testing: Respect the Difference Slide 2
  3. 3. A genda  Introduction  Points to ponder Performance Testing: Respect the Difference Slide 3
  4. 4. Ter minology Applying multi-user synthetic load to the system  Load testing  Concurrency testing  Performance testing  Burst testing  Stress testing  Endurance testing  Scalability testing  Longevity testing  Volume testing  Soak testing  Reliability testing  Stability testing Performance Testing: Respect the Difference Slide 4
  5. 5. Goal  To contrast load and functional testing  To highlight points to ponder in performance testing Performance Testing: Respect the Difference Slide 5
  6. 6. Load Testing Pr ocess Collect Requirements Define Load Create Test Assets Run Tests Modify System Analyze Results Goals are not met Goals are met Done Performance Testing: Respect the Difference Slide 6
  7. 7. What Load Testing A dds  Verification that the system handles the load  Verification of multi-user performance  Performance optimization  Exactly the same load  Debugging/verification of multi-user issues  Testing self-regulation functionality  Such as auto-scaling or changing the level of service depending on load Performance Testing: Respect the Difference Slide 7
  8. 8. Ster eotype  Load / Performance Testing is:        Last moment before deployment Protocol Level Record-and-Playback Large corporations Expensive tools requiring special skills Lab environment Scale-down environment … Performance Testing: Respect the Difference Only one option ! Slide 8
  9. 9. Perfor mance Testing  In no way limited to pre-production: • • • • • • • Technology evaluation Infrastructure evaluation Prototypes / POC Component / unit Continuous testing What/if scenarios Performance optimization Performance Testing: Respect the Difference • Performance troubleshooting • Benchmarking • Production Slide 9
  10. 10. A genda  Introduction  Points to ponder Performance Testing: Respect the Difference Slide 10
  11. 11. What to Test  Can’t be comprehensive  Typical scenarios  Classes of users  Risk  Component coverage Performance Testing: Respect the Difference Slide 11
  12. 12. Performance Requir ements  Goals vs. Requirements  Response times  Aggregate values  Throughput Performance Testing: Respect the Difference Slide 12
  13. 13. Performance Requir ements  Concurrency  Number of users and think time   Indirect way to define throughput Named, active, concurrent users  Resource utilization Performance Testing: Respect the Difference Slide 13
  14. 14. Wor kload Implementation  Manual  Load testing tool  Recording/playback, protocol level  Test harness  Recoding/playback, GUI level  Programming Load Generator Server Load Testing Tool Application Virtual Users Performance Testing: Respect the Difference Network Slide 14
  15. 15. Wor kload Ver ification  Tools default  Minimal (such as timeouts or HTTP error codes)  Direct  Analyzing server response  Indirect    Logs Data Monitoring Performance Testing: Respect the Difference Slide 15
  16. 16. The Effect of Data  Size and structure  Caching / concurrency  Adding new data Performance Testing: Respect the Difference Slide 16
  17. 17. Test Envir onment  Sizing / Configuration  Lab vs. Cloud vs. Service  Scope vs. Variability  Testing Limits vs.  Performance Optimization Performance Testing: Respect the Difference Slide 17
  18. 18. Explor ing the System  Explore a new system  Run all scenarios manually  Iterative approach  Run scripts as soon as they are ready  Monitor all components Performance Testing: Respect the Difference Slide 18
  19. 19. Implicit Tasks  Tuning  Diagnostics  Capacity planning Performance Testing: Respect the Difference Slide 19
  20. 20. Time Consider ations  Steady mode  Reliability  Tuning and diagnostics  Real / intensive users  Throughput / connections  Different tuning Performance Testing: Respect the Difference Slide 20
  21. 21. Pr ocess  Synchronous process may be needed     vs. Asynchronous Problem can block further testing Full setup can be required to reproduce Diagnosis is a collaborative process Performance Testing: Respect the Difference Slide 21
  22. 22. Systematic A ppr oach to Changes  Tuning / Diagnostics  Systematic approach  One factor at a time  Log all changes Performance Testing: Respect the Difference Slide 22
  23. 23. A nalyzing Results  Far more than passed/failed  Challenge for Continuous Integration (CI)  Response times, throughput, number of users  Resource utilization  Logs / diagnostics  Variability between runs Performance Testing: Respect the Difference Slide 23
  24. 24. Summar y  Performance testing is a complex activity involving testing, performance analysis, administration, development, and business processes  Skills and processes needed are quite different, don’t apply functional testing practices blindly  A load testing tool is one point on one slide  Excellence requires time Performance Testing: Respect the Difference Slide 24
  25. 25. Questions? Alexander Podelko apodelko@yahoo.com @apodelko To learn more check my collection of performance-related links and documents at www.alexanderpodelko.com Performance Testing: Respect the Difference Slide 25

×