Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
BUGDAY BANGKOK 2014 SPEAKER 
BugDay Bangkok 2014 
Nitisak 
Nitisak Mooltreesri 
DST Worldwide Services 
NMooltreesri@dstwo...
Agenda 
• Why automated Load testing is important ? 
• What are the risks we are testing for ? 
• “Realism” vs “Simulation...
Why automated Load testing is 
important ? 
• Most users click away after 8 seconds of delay 
• Business revenue loss due ...
Agenda 
• Why automated Load testing is important ? 
• What are the risks we are testing for ? 
• “Realism” vs “Simulation...
What are the risks we are testing 
for ? (1) 
Scalability : 
• Operation Problems ? Solve with hardware ? 
• Tall Stacks -...
What are the risks we are testing 
for ? (2) 
Capacity : System can’t support the expected load structurally/as 
engineere...
What are the risks we are testing 
for ? (3) 
Concurrency : ( Race Conditions, Database locks, 
contention points) 
• Infr...
What are the risks we are testing 
for ? (4) 
Reliability: Degradation over time, system becomes 
slower, less predictable...
Agenda 
• Why automated Load testing is important ? 
• What are the risks we are testing for ? 
• “Realism” vs “Simulation...
“Realism” vs “Simulation Tests” 
Approaches. 
Traditional testing we have to test realistic things!!! 
• Will the complete...
“Realism” vs “Simulation Tests” 
Approaches. 
Simulations Tests 
• Maybe the answer is more control 
• Horizontal Scalabil...
“Realism” vs “Simulation Tests” 
Approaches. 
Suggestions: Build easily repeatable, reliable rapid 
test. 
• Login ( Measu...
Agenda 
• Why automated Load testing is important ? 
• What are the risks we are testing for ? 
• “Realism” vs “Simulation...
Performance Testing Incomplete 
Systems. 
Showcase : Daily performance testing in BMPS business 
unit at DSTWS and DSTSyst...
Approach: Simulation Approach 
Suggestions: Build easily repeatable, reliable rapid test. 
• Login ( Measure session overh...
BugDay Bangkok 2014 
Automated Tools 
Tools: 
• HP Performance Center 11.52 
• HP Vugen IDE 
• Electric Cloud for CI 
• In...
Performance Testing Incomplete 
Systems. 
Process and Lifecycle: 
BugDay Bangkok 2014
BugDay Bangkok 2014 
HP Performance Scripting.
Create Performance Test Scenarios 
BugDay Bangkok 2014
Create a Performance Test Workflow. 
BugDay Bangkok 2014
Integrate to Continues Integration 
BugDay Bangkok 2014
Reporting Early Performance 
Results. 
BugDay Bangkok 2014
The benefits of daily performance 
testing 
• Reduce performance tickets by 70% compare with the previous 
BugDay Bangkok ...
Some Recommendation for Agile Load 
Testing 
• Establish performance goals 
• Include performance tests in the build proce...
BugDay Bangkok 2014 
Q/A
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
Gmail
Next
Download to read offline and view in fullscreen.

0

Share

Download to read offline

Bugday bkk-2014 nitisak-auto_perf

Download to read offline

This talk covers how to apply the Automated Load Testing for Continuous Delivery approach

Related Books

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

Bugday bkk-2014 nitisak-auto_perf

  1. 1. BUGDAY BANGKOK 2014 SPEAKER BugDay Bangkok 2014 Nitisak Nitisak Mooltreesri DST Worldwide Services NMooltreesri@dstworldwideservices.com Facebook: Nitisak Mooltreesri Twitter: @theimagine Automated Load Testing for Continuous Delivery. Consider how much time you spend finding and fixing problems in production. Even with the most comprehensive functional testing process, you still discover elusive bugs under simultaneous load of thousands of users after going live. Early testing and rapid feedback, as well as effective communication, are cornerstones of the Agile approach to software development. This talk covers how the BPMS performance team at DST have done the Automated Load Testing for Continuous Delivery approach. The Approach and techniques for early testing can also be applied to non Agile projects, so even if you haven’t jumped on the Agile bandwagon, you and your team can benefit from this session.
  2. 2. Agenda • Why automated Load testing is important ? • What are the risks we are testing for ? • “Realism” vs “Simulation Tests” Approaches. • Performance Testing Incomplete Systems. • Q/A BugDay Bangkok 2014
  3. 3. Why automated Load testing is important ? • Most users click away after 8 seconds of delay • Business revenue loss due to poor web applications performance • Failure of US Based Airlines computerized system to maintain flight BugDay Bangkok 2014 plans in Nov 2009 • Aberdeen found that inadequate performance could impact revenue by up to 9% • Business performance begins to suffer at 5.1 seconds of delay in response times of web applications and 3.9 for critical applications
  4. 4. Agenda • Why automated Load testing is important ? • What are the risks we are testing for ? • “Realism” vs “Simulation Tests” Approaches. • Performance Testing Incomplete Systems. • Q/A BugDay Bangkok 2014
  5. 5. What are the risks we are testing for ? (1) Scalability : • Operation Problems ? Solve with hardware ? • Tall Stacks --> Wirsth’s Law : “Software is getting slower more BugDay Bangkok 2014 rapidly than hardware get faster” • Subject to use patterns and user models. What Does a Problem Look Like ? • Longer response times is a clue • “High” CPU/Memory/Storing/Network Utilization
  6. 6. What are the risks we are testing for ? (2) Capacity : System can’t support the expected load structurally/as engineered. • Response time very sensitive to load • Growing Queues • Hard or Soft Resource Limitations BugDay Bangkok 2014 • High CPU Limitation • Increasing I/O Latency • Run out of database threads
  7. 7. What are the risks we are testing for ? (3) Concurrency : ( Race Conditions, Database locks, contention points) • Infrequent functional issues that seem to only occur under load. • Process crashes • Not easily reproducible. BugDay Bangkok 2014
  8. 8. What are the risks we are testing for ? (4) Reliability: Degradation over time, system becomes slower, less predictable, or eventually fails. BugDay Bangkok 2014 • Memory or Object Leaks • More frequent Garbage Collection • Decaying response time
  9. 9. Agenda • Why automated Load testing is important ? • What are the risks we are testing for ? • “Realism” vs “Simulation Tests” Approaches. • Performance Testing Incomplete Systems. • Q/A BugDay Bangkok 2014
  10. 10. “Realism” vs “Simulation Tests” Approaches. Traditional testing we have to test realistic things!!! • Will the completed, deployed system support : • [a,b…) users • Performing ( e,f…] activities • At (j,k…] rates • On mn… configuration Uder rs … external conditions, • Meeting x,y… response time goals ? BugDay Bangkok 2014
  11. 11. “Realism” vs “Simulation Tests” Approaches. Simulations Tests • Maybe the answer is more control • Horizontal Scalability makes assumptions let’s use them • Test Subsets: Single servers, single components, cheaply and BugDay Bangkok 2014 repeatedly • Calibration tests.
  12. 12. “Realism” vs “Simulation Tests” Approaches. Suggestions: Build easily repeatable, reliable rapid test. • Login ( Measure session overhead/footprint) • Simple Workflows, avoid data caching effects. • Control variables • Be ready to repeat/troubleshoot when you find anomalies. BugDay Bangkok 2014
  13. 13. Agenda • Why automated Load testing is important ? • What are the risks we are testing for ? • “Realism” vs “Simulation Tests” Approaches. • Performance Testing Incomplete Systems. • Q/A BugDay Bangkok 2014
  14. 14. Performance Testing Incomplete Systems. Showcase : Daily performance testing in BMPS business unit at DSTWS and DSTSystem. BugDay Bangkok 2014
  15. 15. Approach: Simulation Approach Suggestions: Build easily repeatable, reliable rapid test. • Login ( Measure session overhead/footprint) • Simple Workflows, avoid data caching effects. • Control variables • Be ready to repeat/troubleshoot when you find anomalies. Cheap and Fast Performance Tests : BugDay Bangkok 2014
  16. 16. BugDay Bangkok 2014 Automated Tools Tools: • HP Performance Center 11.52 • HP Vugen IDE • Electric Cloud for CI • In-house software for generating report.
  17. 17. Performance Testing Incomplete Systems. Process and Lifecycle: BugDay Bangkok 2014
  18. 18. BugDay Bangkok 2014 HP Performance Scripting.
  19. 19. Create Performance Test Scenarios BugDay Bangkok 2014
  20. 20. Create a Performance Test Workflow. BugDay Bangkok 2014
  21. 21. Integrate to Continues Integration BugDay Bangkok 2014
  22. 22. Reporting Early Performance Results. BugDay Bangkok 2014
  23. 23. The benefits of daily performance testing • Reduce performance tickets by 70% compare with the previous BugDay Bangkok 2014 version. • Increase performance test efficiency by 150%. • Reduce performance test effort by 80+% (Compared with the previous project) • Increase the performance test machines utilization by 450%
  24. 24. Some Recommendation for Agile Load Testing • Establish performance goals • Include performance tests in the build process • Test incrementally • Reuse functional tests BugDay Bangkok 2014
  25. 25. BugDay Bangkok 2014 Q/A

This talk covers how to apply the Automated Load Testing for Continuous Delivery approach

Views

Total views

347

On Slideshare

0

From embeds

0

Number of embeds

2

Actions

Downloads

5

Shares

0

Comments

0

Likes

0

×