Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Next-Generation Performance Testing with Lifecycle Monitoring


Published on

With the increasing market demand for “always on” high performance applications, many organizations find that their traditional load testing programs have failed to keep pace with expectations and competitive pressures. Agile development practices and DevOps concepts of continuous delivery cause old load testing approaches to become unacceptable bottlenecks in the delivery process. Although it remains true that the only way to know for certain how a system will respond to load is to subject it to load, much of what load testing has traditionally accomplished is rooting out performance issues that are detectable and resolvable without actually applying load. The trick is knowing when and how to look for these issues. With specific examples from recent client implementations, Scott Barber shares the T4APM™ approach, a simple and universal process to detect and manage performance issues—with or without applying load—throughout the lifecycle.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Next-Generation Performance Testing with Lifecycle Monitoring

  1. 1. T5 Performance Testing 5/8/2014 9:45:00 AM Next-Generation Performance Testing with Lifecycle Monitoring Presented by: Scott Barber SmartBear Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888-268-8770 ∙ 904-278-0524 ∙ ∙
  2. 2. Scott Barber SmartBear Chief performance evangelist for SmartBear Scott Barber is a respected leader in the advancement of software testing practices, an industry activist, and load testing celebrity of sorts. Scott authored several books―Performance Testing Guidance for Web Applications, Beautiful Testing, How to Reduce the Cost of Testing, and Web Load Testing for Dummies―and more than 100 articles and blog posts. Founder/president of PerfTestPlus, Scott co-founded the WOPR, served as director of the AST and CMG, and is a founding member of ISST. His industry writing, speaking, and activism focus on improving the effectiveness and business alignment of software development practices. Learn more about Scott Barber.
  3. 3. 4/26/2014 1 Next-Generation Performance Testing with Lifecycle Monitoring By: Scott Barber Product Owner – Load Testing Tools SmartBear Software, Inc. Revised for: Next-Generation Performance Testing with Lifecycle Monitoring
  4. 4. 4/26/2014 2 Product Owner – Load Testing Tools SmartBear Software, Inc. @sbarber Co-Founder: Workshop On Performance and Reliability Co-Author:Author: Contributing Author: Books: About me: System or application characteristics related to: Speed: - responsiveness - user experience Scalability: - capacity - load - volume Stability: - consistency - reliability - stress What is Performance?
  5. 5. 4/26/2014 3 What is a Performant System? A system optimized to balance: - Cost - Time to Market - Capacity …while remaining focused on: Quality of Service to System Users What is Performance Testing? Testing designed to: Inform software system optimization by balancing: - Cost - Time to Market - Capacity while remaining focused on the quality of service to system users.
  6. 6. 4/26/2014 4 Performance vs. Load Testing? Conception to Headstone Not Cradle to Grave The Performance LifeCycle is…
  7. 7. 4/26/2014 5 © PerfTestPlus, Inc. Prevent Poor Performance… … don’t just react when it happens. Who is Responsible?
  8. 8. 4/26/2014 6 Where does it fit in the SDLC? Delivering Performant Systems; Commonly, Y2K -> Today SPE (Software Performance Engineering) Math-centric Architect/Design method + Profiling Context-free code optimization + Load Testing Demonstrate (too late) poor performance characteristics + Capacity Planning Math-centric guessing to determine hardware needs + APM (Application Performance Management) Proactively react to production issues => Inefficient & Reactionary Delivery & Maintenance
  9. 9. 4/26/2014 7 Delivering Performant Systems; Moving Forward Minimize Need for Complex Mathematics Few teams have mathematicians w/ advanced degrees Disconnect Accuracy from Actual Production Data Because we can’t depend on what we don’t have yet Remove Socio-Political Barriers Collaboration improves information sharing & efficiency Keep/Adapt most Valuable Practices Many good concepts have been poorly implemented Distribute Valuable Practices Throughout LifeCycle Many valuable practices have suffered from stove-piping We need one model for the whole ‘Performance Journey’ © PerfTestPlus, Inc.
  10. 10. 4/26/2014 8 “DevOps” Perf Team Individual • Code Profiling • Requirements Revision • Design Validation • Budgets • Units & Components • Monitoring • Tuning • Load/Stress Testing • Environment Config • Capacity Planning • Concurrency Checks • Situation Analysis • Snapshots • Comparisons • Niche Concerns • Mini-Field Studies (UAT) • “What-if…” Key Points • Entire Lifecycle • Entire Team Focused • Individual, Role, & Team Accountability & Responsibility • Continuous & Collaborative © PerfTestPlus, Inc. © PerfTestPlus, Inc. Preventing Poor Performance with a little work… every day… from every one.
  11. 11. 4/26/2014 9 © PerfTestPlus, Inc. Units Stories Tiers Resources Goals Assess Compare Investigate Accept Answer Dev & Prod Times Resources Sizes Frequencies Dashboard! Proactive Micro & Macro Establish Goals Update Targets © PerfTestPlus, Inc.
  12. 12. 4/26/2014 10 © PerfTestPlus, Inc. An Ounce of Prevention…
  13. 13. 4/26/2014 11 © PerfTestPlus, Inc. © PerfTestPlus, Inc.
  14. 14. 4/26/2014 12 Load Tests Stress Tests Endurance Tests Infrastructure/Configuration Verification Load Variability Analysis Background Load for Acceptance Tests… What is it? Or any other deliberate approximation of Production Conditions © PerfTestPlus, Inc.
  15. 15. 4/26/2014 13 RPT is: Inspired by Rapid Software Testing Consistent with Rapid Software Testing themes Sanctioned by James Bach, Michael Bolton & the RST instructors to as a specific implementation of the Rapid Testing Methodology For more information about RST, visit: “What have we got? What do we want? How do we get there…?” --Bob Barber (Scott’s dad) … as quickly, simply, and cheaply as possible? Evolved from:
  16. 16. 4/26/2014 14 …a common man’s way of expressing the problem solving approach that classical engineers employ. • Given: “What have we got?” • Find: “What do we want?” • Solve: “How do we get there?” An approach to respond to a specific performance-related question after 4 or fewer hours of team effort with 1 or more of: A) The answer B) A partial answer • To determine the value of additional effort • The level of effort to provide the answer C) Better questions to address the underlying concern What is it?
  17. 17. 4/26/2014 15 1. Receive Question • Clarify the question • Understand the driver(s) behind the question 2. Generate Test Coverage Outline (TCO) (~20 min) • Simplest path to (partial) answer(s) • Comprehensive path to (partial) answer(s) 3. Transform TCO into Rapid Strategy (~20 min) • Only tasks that fit in time box • Stick to tasks requiring available resources 4. Execute Strategy (~2.5 hrs) • Snapshots are your friends • Anecdotal is sufficient 5. Consolidate/Analyze Data (~30 min) • Identify patterns • Confirm patterns (time permitting) 6. Report Results (~20 min) • Answer(s) / Time or Effort to get answer(s) • Follow-on questions of interest ~4 hr Conceptual Approach © PerfTestPlus, Inc. © PerfTestPlus, Inc.
  18. 18. 4/26/2014 16 © PerfTestPlus, Inc. The Bottom Line © PerfTestPlus, Inc.
  19. 19. 4/26/2014 17 Addendum Questions?
  20. 20. 4/26/2014 18 Contact Info Product Owner – Load Testing Tools SmartBear Softwear, Inc. E-mail: Blog: Web Site: Twitter: @sbarber