© Copyright GlobalLogic 2011 1Connect. Collaborate. Innovate.Performance testing in Agile development.Automation challenges.
© Copyright GlobalLogic 2011 2Connect. Collaborate. Innovate.Performance critical applicationPerformance critical java ent...
© Copyright GlobalLogic 2011 3Connect. Collaborate. Innovate.Time estimates5 core load testing scenariosOn average manual ...
© Copyright GlobalLogic 2011 4Connect. Collaborate. Innovate.Before any work is started1. Identify Test Environment2. Iden...
© Copyright GlobalLogic 2011 5Connect. Collaborate. Innovate.More performance engineersCons:• Noticeably bigger budget of ...
© Copyright GlobalLogic 2011 6Connect. Collaborate. Innovate.Limit the number of tests and configurationsPros:• There is n...
© Copyright GlobalLogic 2011 7Connect. Collaborate. Innovate.AutomationPros:• Automated performance tests 24X7• Continuous...
© Copyright GlobalLogic 2011 8Connect. Collaborate. Innovate.Principles of performance testing in AgileMake performance ex...
© Copyright GlobalLogic 2011 9Connect. Collaborate. Innovate.Ant, Jmeter, Scalable Vector Graphics case study
© Copyright GlobalLogic 2011 10Connect. Collaborate. Innovate.TeamCity
© Copyright GlobalLogic 2011 11Connect. Collaborate. Innovate.Sample SVG Report
© Copyright GlobalLogic 2011 12Connect. Collaborate. Innovate.Concurrent performance testing
© Copyright GlobalLogic 2011 13Connect. Collaborate. Innovate.ConclusionInput data• 9 priority-driven configurations• 2 pe...
© Copyright GlobalLogic 2011 14Connect. Collaborate. Innovate.Q&AThank youskype: andreymelnyk
Upcoming SlideShare
Loading in...5
×

Agile performance testing

491

Published on

Published in: Education, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
491
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Agile performance testing

  1. 1. © Copyright GlobalLogic 2011 1Connect. Collaborate. Innovate.Performance testing in Agile development.Automation challenges.
  2. 2. © Copyright GlobalLogic 2011 2Connect. Collaborate. Innovate.Performance critical applicationPerformance critical java enterpriseapplicationOperatingsystemsApplication servers Databaseservers9 priority-driven configurationstwo performance engineers
  3. 3. © Copyright GlobalLogic 2011 3Connect. Collaborate. Innovate.Time estimates5 core load testing scenariosOn average manual execution of a single test takes 4 hours(setup environment, execute a test, basic results analysis)Estimated time to complete load testing against 9 configurations:9 configurations * 5 test scenarios * 4 hours = 180 hours1+ man-monthstwo-week sprints?
  4. 4. © Copyright GlobalLogic 2011 4Connect. Collaborate. Innovate.Before any work is started1. Identify Test Environment2. Identify PerformanceAcceptance Criteria3. Plan and Design Tests4. Implement Test Design5. Configure Test Environment7. Analyze Results, Report,and RetestMost time-consumingactivitiesSolutions6. Execute TestsCore performance testingactivitiesMore performance engineers ?Limit the number of tests andconfigurations?Automation ?
  5. 5. © Copyright GlobalLogic 2011 5Connect. Collaborate. Innovate.More performance engineersCons:• Noticeably bigger budget of performance testing effort• Additional activities to manage a team, usually with dedicatedlead/manager• Staff trainings: methodology, tools• Time consuming and boring tasks are performed byperformance engineers• Test LAB investments• Performance engineer positions are hard to fillPros:• There is no timemoney investment in automation• More work done in less time• Can be used when automation is impossible due to productcomplexity and architecture limitations• Can be applicable to a short-term project
  6. 6. © Copyright GlobalLogic 2011 6Connect. Collaborate. Innovate.Limit the number of tests and configurationsPros:• There is no timemoney investment in automation• Focus on most commonly used configurations• There is no need to expand performance testing team• Can be applicable to short-term project• Can be used when risk to miss performance issue isconsidered and acceptableCons:• There is a high risk to miss environment specific performanceissue which could cause performance problems in production• Very risky when an application is performance critical• The approach doesn’t guarantee that a team will have enoughtime to investigate problems deeper• Time-consuming and boring tasks are performed byperformance engineers
  7. 7. © Copyright GlobalLogic 2011 7Connect. Collaborate. Innovate.AutomationPros:• Automated performance tests 24X7• Continuous integration process• Concurrent performance tests• Early warning of performance bottleneck or performanceissue• Performance testing with a small team• More time for results analysis and investigations• There is no need to expand a performance team• Profitable for long running, performance critical projects• Reusable by any person in a teamCons:• Timemoney investment to build automation• Ongoing efforts to keep automation in a good state• Usually not applicable for short-term projects• Test LAB investments• Commercial tools are expensive
  8. 8. © Copyright GlobalLogic 2011 8Connect. Collaborate. Innovate.Principles of performance testing in AgileMake performance explicit• Make it an explicit requirement upfront and throughout the process (non-functionalrequirements can be dotted onto the corner of the story card for a functionalrequirement)Performance testing is a part of continuos integration process:• Automated compilation• Automated packaging• Automated deployment• Automated configuration• Automated tests execution• Automated analysis• Automated result archiving - can check raw output if discover issues later• Automated scheduling. Overnight runs.Use concurrent performance testingIterate and incrementGet feedback
  9. 9. © Copyright GlobalLogic 2011 9Connect. Collaborate. Innovate.Ant, Jmeter, Scalable Vector Graphics case study
  10. 10. © Copyright GlobalLogic 2011 10Connect. Collaborate. Innovate.TeamCity
  11. 11. © Copyright GlobalLogic 2011 11Connect. Collaborate. Innovate.Sample SVG Report
  12. 12. © Copyright GlobalLogic 2011 12Connect. Collaborate. Innovate.Concurrent performance testing
  13. 13. © Copyright GlobalLogic 2011 13Connect. Collaborate. Innovate.ConclusionInput data• 9 priority-driven configurations• 2 performance engineers• Two-week sprintsCompleted activities:• Performance testing process was automated (automatic application deployment,tests execution, results analysis)• Test LAB upgraded to support 4 concurrent performance testing executions againstisolated servers (25 servers in the LAB)Statistics• 3 months spent on building Jmeter based automated performance testing• Automated deployment and configuration process is ~25% faster than manual• Performance testing cycle takes ~4 days (instead of 1+ man-months estimated)• 50% of time: results analysis, troubleshooting• 10% of time: keep automation in a good state• 40% of time: new tests development, exploratory testing, cluster tests• 17 performance issues were found• Performance testing reports are iteratively used by PM’s to assess quality of theproduct• PMs are happy 
  14. 14. © Copyright GlobalLogic 2011 14Connect. Collaborate. Innovate.Q&AThank youskype: andreymelnyk

×