Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management
Combining Performance Testing an...
Upcoming SlideShare
Loading in …5
×

Combining Performance Testing And Modelling For Easy Jet.Com V 1.0

1,490 views

Published on

easyJet is Europe’s fourth largest airline and has seen growth in passenger numbers of 37% per annum over the last 10 years. It operates 175 aircraft over a pan-European network of 20 bases and 400 routes.
Capacitas is a UK-based technology consulting and training company focussed on ICT performance, capacity and cost.
Over 98% of easyJet’s revenue is earned through its e-commerce platform; meaning software reliability, performance and scalability are critical to successful business operation. easyJet is a recognised innovator in a highly competitive market and this necessitates frequent software releases using an agile development methodology. All of this must be achieved within the context of a low-cost business model.
In many organisations performance testing and performance modelling have long been used to reduce the risk of poor performance impacting business services, but often in isolation. This presentation describes how performance testing and modelling can be used in tandem to compensate for the weaknesses of each other, within the context of an agile development process. The presentation demonstrates how Capacitas and easyJet have integrated performance testing and modelling processes to provide cost-effective risk mitigation for the easyJet.com service.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,490
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
9
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Combining Performance Testing And Modelling For Easy Jet.Com V 1.0

  1. 1. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Combining Performance Testing and Modelling for easyJet.com Danny Quilton, COO, Capacitas Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Agenda • Business Context • easyJet’s Goals • Challenges with Agile • How can we Reduce Risk? • Performance Testing • Performance Modelling • What you need • Costs • Benefits © Capacitas 2002-2009 2
  2. 2. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com About Capacitas • UK-based Capacity and Performance Management specialists • Training and professional services • Hardware-vendor independent • Tools independent © Capacitas 2002-2009 3 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Business Context © Capacitas 2002-2009 4
  3. 3. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Business Context • Capacity planning service www.easyJet.com • Business critical service • 98% of revenue earned through the e-commerce service • Highly competitive market • Frequent innovation by adopting agile development • Monthly software releases • Legislative changes • Consistent growth; typically 20% p.a. • Low-cost operating model • Requirement to manage risk © Capacitas 2002-2009 5 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com easyJet’s Goals © Capacitas 2002-2009 6
  4. 4. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com easyJet’s Goals © Capacitas 2002-2009 7 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Agile Development Process Scrum Product (Daily Meeting) Backlog OUT Sprint IN Pre-release Sprint Sprint Sprint Activities Planning Backlog (15-30 days) Review (e.g. Training, Meeting Documentation) LESSONS LEARNED Sprint Retrospective Release Meeting © Capacitas 2002-2009 8
  5. 5. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Challenges Associated with Agile • Frequent change introduces risk • Window available for performance testing is typically 1-2 weeks • Final production candidate code often not available at the start of testing • Multiple code releases within the testing window • Increased probability of functional and environmental defects • Frequent rework of code necessitates modifications to test scripts • Cumulative effect on capacity from 12 releases per annum • Narrow time window to resolve performance defects © Capacitas 2002-2009 9 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Challenges Associated with Agile © Capacitas 2002-2009 10
  6. 6. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Typical Causes of Degraded Service Performance © Capacitas 2002-2009 11 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com How Can We Reduce Risk? Cause of degraded performance Mitigated by performance Mitigated by modelling? testing? Changes in user patterns No Yes Poor performance of 3rd parties May be limited by Yes environment Physical capacity bottlenecks May be limited by size of Yes the environment Logical bottlenecks Yes If known Software configuration changes Yes Yes, if tested Increases in demand May be limited by size of Yes the environment © Capacitas 2002-2009 12
  7. 7. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Why Test? • Performance testing is used to understand the software’s performance characteristics • Do unknown time and growth related defects exist? • Memory leaks • Poor database design/indexing • Do unknown concurrency related defects exist? • Database locking issues • Memory locking issues • Race conditions • Obtain model inputs © Capacitas 2002-2009 13 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Why Model? • Modelling uses our understanding of the software to predict end to end service performance under different scenarios © Capacitas 2002-2009 14
  8. 8. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Scenario Modelling © Capacitas 2002-2009 15 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Coverage of Testing and Modelling Testing Modelling Business Scenarios Application Infrastructure Technical Scenarios © Capacitas 2002-2009 16
  9. 9. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com High Level Process © Capacitas 2002-2009 17 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Risk Assessment © Capacitas 2002-2009 18
  10. 10. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Risk Assessment • Assess the potential performance impact on different elements of infrastructure • Plan tests to help mitigate that impact • Focus on areas of highest risk due to limited timeframes Risks Description ID Web DB Other Risk Mitigation Project X --- Medium Medium Low Test 1 Project Y --- Low High Medium Tests 2 & 3 Project Z --- Low Low Low None © Capacitas 2002-2009 19 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Performance Tests Employed © Capacitas 2002-2009 20
  11. 11. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Baseline Tests • Used to determine the impact Example service time comparison between two fictional baseline tests of the planned release 1400 • Comparative performance 1200 between releases Service Time (ms) 1000 • May be Load, Stress or Soak 800 Tests 600 • Service times and response 400 times are compared to each 200 other 0 Database Web Server Server Service Service Time Time Test 1.1 Release X 752 1240 Test 1.1 Release Y 813 1192 % Difference 8% -4% © Capacitas 2002-2009 21 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Single Transaction Type Tests • To determine the impact on the Milliseconds Web CPU per page (fictional data) infrastructure of individual user transactions • Every business process which is to 132 be modelled will have a calculated 328 service time for each hardware 233 element 221 156 122 Home Search Select Options Login Purchase © Capacitas 2002-2009 22
  12. 12. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Modelling Performance Test Results © Capacitas 2002-2009 23 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Performance Modelling Business Service Component Virtual Infrastructure © Capacitas 2002-2009 24
  13. 13. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Performance Modelling Aircraft Business Home Page Search Booking Service HTTP1 APP1 APP2 DB1 Component Virtual Infrastructure © Capacitas 2002-2009 25 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Performance Modelling 600 Business 100 80 5 Service 160 70 10 80 Component Virtual Infrastructure 3.5 transactions/s © Capacitas 2002-2009 26
  14. 14. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Model Scope • What infrastructure to include? • Major contributor to the performance experienced by end-user • Highly utilised • Critical • Load growth • Frequently reviewed Virtual Infrastructure © Capacitas 2002-2009 27 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Model Scope • What transactions to include • Business critical • Importance to revenue • High-volume • High component cost Change Home Page Search Booking Password Service © Capacitas 2002-2009 28
  15. 15. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Demand Model Validation • The model should be validated on an ongoing basis against actual 600 values • The first stage is to validate the demand part of the model to ensure 100 80 5 that arrival rates are being calculated correctly • Following this the true demand can be used to validate the utilisation 610 calculations 80 80 6 © Capacitas 2002-2009 29 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Capacity and Performance Model Validation Comparison of Actual Database Load to Predicted Database Load 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Actual Database CPU Predicted Database CPU © Capacitas 2002-2009 30
  16. 16. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com What you need – People • Capacity and performance management skills: • Demand forecasting • Modelling • Load testing • A strong understanding of risk management • A good understanding of the business model • Flexibility • Project management skills © Capacitas 2002-2009 31 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com What you need – Process • Interface with the release team • Participation in the agile development ‘scrums’ • Access to developers to understand application functionality • Interface with the test environment management team for test planning • Interface to the business to access business forecasts • Interface to operations to access production measurements © Capacitas 2002-2009 32
  17. 17. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com What you need – Tools & Environment • An appropriately sized test environment • Production-like environment is not required • Consider the ratio of capacity between presentation, business and data tiers • Toolset • Performance monitoring tools (production and test) • Load generation tool • Modelling tool © Capacitas 2002-2009 33 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com What you need – Environment © Capacitas 2002-2009 34
  18. 18. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com What you need – Data • Software/hardware configuration (production and test) • Performance measurements from production and test environments • Benchmark data (Spec, TPC, etc.) • Service times derived from performance test results • SLAs • Internal • SLAs with third-parties • Historical and forecast business demand • Historical and forecast service demand • Measured error of previous forecasts © Capacitas 2002-2009 35 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Historical Business Demand Fleet Plan to April 2009 180 y = 0.0461x - 1673.1 R² = 0.9668 160 140 GB Airways 120 acquisition Number of Owned Aircraft 100 80 60 40 20 0 Sep-04 Sep-05 Sep-06 Sep-07 Sep-08 Mar-04 Jun-04 Dec-04 Mar-05 Jun-05 Dec-05 Mar-06 Jun-06 Dec-06 Mar-07 Jun-07 Dec-07 Mar-08 Jun-08 Dec-08 Mar-09 Delivery Date © Capacitas 2002-2009 36
  19. 19. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Historical Service Demand © Capacitas 2002-2009 37 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Costs © Capacitas 2002-2009 38
  20. 20. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Benefits “Capacitas give us the confidence in making the correct release decision” James Mackay, Test Manager, easyJet Test Phase Status Risk Assessment Complete Performance Test Plan Complete Targeted Performance and Stress Testing Complete Regression Performance and Stress Testing Complete All Defects: Closed Complete Capacity Models Updated Complete Capacity Plan updated and communicated Complete © Capacitas 2002-2009 39 Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Benefits • Testing and modelling complement one another • The combined approach provides effective risk-mitigation in the context of: • Frequent change • Narrow performance test window • Business growth • The approach enables us to rapidly answer business and technical ‘what if’ scenarios • Costs are limited through the use of a scaled-down performance test environment and low-cost modelling tools • Provides performance assurance in a highly competitive market place © Capacitas 2002-2009 40
  21. 21. Butler Group Briefing – Software Testing: A Fresh Approach to Testing and Test Management Combining Performance Testing and Modelling for easyJet.com Questions? • Presentation will be available for download at www.capacitas.co.uk • Please visit us at our stand for any further questions © Capacitas 2002-2009 41

×