Your SlideShare is downloading. ×
0
Incorporating Performance Testing in Agile Process<br />Michael Vax<br />President, onmaas Software<br />www.onmaas.com<br />
Need Performance Numbers for the Web Site<br />
Types of Performance Testing<br />Performance Testing<br />Load Testing<br />Stress Testing<br />Focused Testing<br />Capa...
Performance Project Phases<br />Proactive<br />Reactive<br />Sizing and Capacity Planning<br />Performance Engineering<br ...
What do we need to test performance?<br />
Performance Testing & Waterfall<br />Sizing and Capacity Planning<br />Performance Engineering<br />Performance Testing<br...
Bugs prevent test execution
Last minute changes across the board
Test script development is delayed
 Testing is delayed to the end of the project
Too late to fix performance unfriendly design decisions
Developers may not be available by the time the performance testing starts</li></li></ul><li>Performance Testing in Agile ...
Tools selection
Performance requirements
Test data preparation
 Load tests
Stress tests
Capacity testing
Resilience testing</li></li></ul><li>Performance Testing is Iterative Process<br />Done in collaboration with Senior Devel...
Performance Testing is Team Activity<br />Performance analyst<br />Performance tester <br />Senior developer / architect<b...
Performance Testing and Iterations<br />Performance<br />Spike<br />Feature A<br /><ul><li> Test requirements
 Test development</li></ul>Feature A - Focus test<br />Feature B - Focus test<br />Feature B<br /><ul><li> Test requirements
 Test development</li></ul>Feature C<br /><ul><li> Test requirements
Upcoming SlideShare
Loading in...5
×

Incorporating Performance Testing in Agile Development Process

4,126

Published on

This presentations explains different aspects of software performance testing and give actionable recommendations on how to integrate it into the Agile Software development process

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,126
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
121
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • What is performance testing?Performance testing is a real-life investigation into a system’s responsiveness and scalability. This typically includes stability considerations as well.What is Load Testing?Load testingLoad testing takes performance testing and applies a “normal” amount of load, reflective of real-world numbersLoad testing shows us bottlenecks with different user levels and any immediate problems when typical load is appliedWhat is stress testing?Stress testingStress testing takes load testing even further and applies above normal or “peak” amounts of load and possibly for extended periods of timeAll of this testing is essential to truly understand the systems performance characteristicsWhat is Capacity Testing?To determine how many users and/or transactions a given system will support and still meet performance goals.Focused Testing?All of these are essential to truly understanding the performance characteristics of your system.
  • Sizing and Capacity PlanningSystem usage scenariosLoad during normal usage and peak hoursPerformance requirementsRight-sizing systems to meet needs without wasting money and creating unnecessary complexity.Performance EngineeringProactively engineering for performanceTechnical design with performance in mindTools SelectionTest environment &amp; data preparationTest script design and preparationPerformance TestingTests executionResult analysisRecommendationsFix problems Monitoring and TroubleshootingUnderstanding systems and solving problems in production
  • Test environmentYou need a separate hardware environment to test performance, including separate network It should be as close to production as possibleToolsTest scriptingMonitoring applications, DB, h/w (CPU, memory), network bandwidthWorking system- You cannot do performance testing on unstable systemSkills- See next slide
  • - Performance testing is delayed
  • Use performance spike to validate design decisions Do focus performance testing for completed features
  • NetworkingDNS, routers, load balancers configuration network monitoring toolsDatabases Monitoring Tuning Load balancing Backup Disaster recovery CachingApplication serversTuning Failover Load balancing Caching LogsDevelopment environments- Programming languages Virtual machines Debuggers Profilers Stack trace ExceptionsApplication domainWhat system does How it is used Integration points
  • automation is software development – use Agile process to manage itA fair amount of software/script/automation development needs to occur during performance testing and tuning. Therefore…
  • Performance testing is endlessCan be extremely time consuming with little resultsIn order to get through all the previously mentioned tests, it would take a full teama week, at which point they would need to start again because the next build would be out.It’s possible to measure a number of different performance characteristics in a single test. For exampleWe also have
  • What is performance testing?Performance testing is a real-life investigation into a system’s responsiveness and scalability. This typically includes stability considerations as well.What is Load Testing?Load testingLoad testing takes performance testing and applies a “normal” amount of load, reflective of real-world numbersLoad testing shows us bottlenecks with different user levels and any immediate problems when typical load is appliedWhat is stress testing?Stress testingStress testing takes load testing even further and applies above normal or “peak” amounts of load and possibly for extended periods of timeAll of this testing is essential to truly understand the systems performance characteristicsWhat is Capacity Testing?To determine how many users and/or transactions a given system will support and still meet performance goals.Focused Testing?All of these are essential to truly understanding the performance characteristics of your system.
  • Throughput testing – typically load generation tools will simply measure this with respect to data flow, such as bytes/second. This isn’t all that useful. I much more meaningful metric is how many user transactions per second can be completed. One user transaction can be thought of as a single click on the website, such as clicking on “view product”. The response time is then measured for the time it takes the request to come back in its full user-end response.Saturation Testing – involves pushing the system to its maximum throughput capacity where the throughput will level out to a max, and the average response times will simply increase with additional user loadEnd-user Response – this sort of testing is typically executed measuring end user response time and gives a good indication of the end-user experience. The Apdex performance statistic is useful for measuring this a meaningful manner.Server Response – this sort of testing gives an indication of how well the server is responding under the load an is usually measured by system resource statistics such as CPU, memory, hard disk and network use
  • Most commercial systems today employ a multi-tiered architecture with redundancy and failover methodologies applied at each tier in order to provide some level of protection against component failure, and therefore allowing their services to continue operating in a lower throughput state.  By avoiding a single point of failure in this way, system downtime can be reduced and High Availability (HA) can be achieved.  Maintaining High Availability is especially valuable for 24-hour sectors such as e-commerce, where problems cannot be fixed overnight and the failure of a retail website for even a few minutes may result in loss of sales.
  • Software is always workingPerformance testing is iterative processPerformance testing is team activity
  • Transcript of "Incorporating Performance Testing in Agile Development Process"

    1. 1. Incorporating Performance Testing in Agile Process<br />Michael Vax<br />President, onmaas Software<br />www.onmaas.com<br />
    2. 2. Need Performance Numbers for the Web Site<br />
    3. 3. Types of Performance Testing<br />Performance Testing<br />Load Testing<br />Stress Testing<br />Focused Testing<br />Capacity Testing<br />
    4. 4. Performance Project Phases<br />Proactive<br />Reactive<br />Sizing and Capacity Planning<br />Performance Engineering<br />Performance Testing<br />Monitoring & Troubleshooting<br />
    5. 5. What do we need to test performance?<br />
    6. 6. Performance Testing & Waterfall<br />Sizing and Capacity Planning<br />Performance Engineering<br />Performance Testing<br />Monitoring & Troubleshooting<br />Problems<br /><ul><li> No functional system until the very end
    7. 7. Bugs prevent test execution
    8. 8. Last minute changes across the board
    9. 9. Test script development is delayed
    10. 10. Testing is delayed to the end of the project
    11. 11. Too late to fix performance unfriendly design decisions
    12. 12. Developers may not be available by the time the performance testing starts</li></li></ul><li>Performance Testing in Agile Project<br />Benchmark Test<br />Benchmark Test<br />Benchmark Test<br />Focus Test<br />Focus Test<br />Focus Test<br />Focus Test<br /><ul><li> Test environment setup
    13. 13. Tools selection
    14. 14. Performance requirements
    15. 15. Test data preparation
    16. 16. Load tests
    17. 17. Stress tests
    18. 18. Capacity testing
    19. 19. Resilience testing</li></li></ul><li>Performance Testing is Iterative Process<br />Done in collaboration with Senior Developers<br />Multiple Test Cycles are Needed<br />Doing performance test right before the launch day is not helpful.<br />You need Calendar time! <br />
    20. 20. Performance Testing is Team Activity<br />Performance analyst<br />Performance tester <br />Senior developer / architect<br />DBA<br />Network engineer<br />
    21. 21. Performance Testing and Iterations<br />Performance<br />Spike<br />Feature A<br /><ul><li> Test requirements
    22. 22. Test development</li></ul>Feature A - Focus test<br />Feature B - Focus test<br />Feature B<br /><ul><li> Test requirements
    23. 23. Test development</li></ul>Feature C<br /><ul><li> Test requirements
    24. 24. Test development</li></li></ul><li>Performance Testing - Agile Best Practices<br /><ul><li>Put performance tasks on a SCRUM board
    25. 25. Make fixing functional bugs that block performance tests a high priority
    26. 26. Do not delay fixing performance issues identified by tests
    27. 27. Retest after fixes are applied to know if they worked
    28. 28. You cannot guess where the next bottleneck is
    29. 29. Don’t pre-optimize before testing </li></li></ul><li>Performance Testing-Agile Best Practices (2)<br /><ul><li>Understand the production environment
    30. 30. Analyze logs
    31. 31. Know when performance is good enough
    32. 32. You can always add more h/w 
    33. 33. Apply low level load to the system under test during manual testing
    34. 34. Use load test scripts:
    35. 35. to find functional and stability bugs
    36. 36. to look for memory leaks
    37. 37. to reproduce intermittent problems
    38. 38. as part of smoke test</li></li></ul><li>Skills<br />Networking<br />Databases<br />Application servers<br />Development environments<br />Application domain<br />
    39. 39. Performance Scripting is software development<br />It should follow the same rules and be treated the same way<br />Design, modularization, documentation, guidelines, source versioning, code reviews should all be carried out<br />Allows for long term use and quick adaptability to changes in the system under test<br />Allows for easy creation of new tests<br />Invest in automation of test setup, data population, monitoring, and analysis of results<br />
    40. 40. Break down your system-interaction model into a modular framework that can be re-used<br />Design Smart<br />Session X<br />Use Case A<br />This modularization allows quick test changes and creation<br />User Transaction A<br />User Transaction B<br />User Transaction C<br />Use Case B<br />User Transaction C<br />
    41. 41. This model allows Focused Tests or any sort of testing to be created quickly<br />Design Smart<br />“Long Running” Stress Test<br />“Check-out” Focused Test<br />Session Type Y<br />Session Type A<br />Session Type Z<br />Session Type X<br />
    42. 42. Types of Performance Testing<br />Performance Testing<br />Load Testing<br />Stress Testing<br />Focused Testing<br />Capacity Testing<br />
    43. 43. Load Testing<br /><ul><li>Load Testing can encompass a large number of tests specific to some area of performance, such as:</li></ul>Throughput testing<br />Saturation testing<br />End-user response <br />Server response<br />
    44. 44. Capacity Testing<br /><ul><li>Capacity Testing is typically more specific than general load testing
    45. 45. Involves measuring “sizing” statistics, such as:</li></ul>Peak Active Sessions<br />Sessions / Hour<br />Transaction / Hour<br />
    46. 46. Stress Testing<br /><ul><li>Stress Testing is typically based off load testing but pushes things even further until the break point
    47. 47. Resilience and Availability Testing
    48. 48. Disaster recovery</li></li></ul><li>Focused Testing<br /><ul><li>Not a ‘typical’ performance category
    49. 49. Allows us to put strain on a specific aspect or functionality of the system such as product views, check-out, or user registration.
    50. 50. Short duration
    51. 51. Heavy load
    52. 52. Usually run repeatedly with small tweaks</li></li></ul><li>In Conclusion<br />Agile Development Process enables performance testing throughout the entire development cycle and ensures the adequate performance by the end of the project<br />
    53. 53. Q & A<br />?<br />
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×