Positionner le logo de
la société du présentateur
à la place
de ce bloc texte
Président de filière
Orateur
Sponsor
Adminis...
Performance Testing Driving Capacity Management at easyJet
easyJet.com
3
Performance Testing Driving Capacity Management at easyJet
easyJet continues to grow
easyJet 2009
£2.6bn revenue
181 aircr...
Performance Testing Driving Capacity Management at easyJet
easyJet 1996
3 Airports
2 Countries
2 Aircraft
2 Routes
5
Base ...
Performance Testing Driving Capacity Management at easyJet 6
easyJet 2011
150 Airports
34 Countries
210 Aircraft
560 Route...
Performance Testing Driving Capacity Management at easyJet
easyJet a growth story
In 15 years easyJet has become the UK’s ...
Performance Testing Driving Capacity Management at easyJet
IT Budgets
8
Performance Testing Driving Capacity Management at easyJet
easyJet Revenue
9
Flight revenue
(easyJet.com + B2B)
Flight rev...
Performance Testing Driving Capacity Management at easyJet
How important is IT?
10
Performance Testing Driving Capacity Management at easyJet 11
How important is IT?
Performance Testing Driving Capacity Management at easyJet
Agile Development Process
12
Performance Testing Driving Capacity Management at easyJet
Business Context
Capacity planning service www.easyJet.com
Busi...
Performance Testing Driving Capacity Management at easyJet
easyJet’s Goals
14
Performance Testing Driving Capacity Management at easyJet
easyJet’s Goals
15
Performance Testing Driving Capacity Management at easyJet
Challenges Associated with Agile
Frequent change introduces ris...
Performance Testing Driving Capacity Management at easyJet
Challenges Associated with Agile
17
Performance Testing Driving Capacity Management at easyJet
Typical Causes of Degraded Service Performance
18
Performance Testing Driving Capacity Management at easyJet
How Can We Reduce Risk?
19
Cause of degraded performance Mitiga...
Performance Testing Driving Capacity Management at easyJet
Why Test?
Performance testing is used to understand the softwar...
Performance Testing Driving Capacity Management at easyJet
Why Model?
Modelling uses our understanding of the software to ...
Performance Testing Driving Capacity Management at easyJet
Scenario Modelling
22
Performance Testing Driving Capacity Management at easyJet
Coverage of Testing and Modelling
23
ModellingTesting
Applicati...
Performance Testing Driving Capacity Management at easyJet
High Level Process
24
Performance Testing Driving Capacity Management at easyJet
Risk Assessment
25
Performance Testing Driving Capacity Management at easyJet
Risk Assessment
Assess the potential performance impact on diff...
Performance Testing Driving Capacity Management at easyJet
Performance Tests Employed
27
Performance Testing Driving Capacity Management at easyJet
Baseline Tests
Used to determine the
impact of the planned
rele...
Performance Testing Driving Capacity Management at easyJet
Single Transaction Type Tests
29
To determine the impact on
the...
Performance Testing Driving Capacity Management at easyJet
Modelling
30
Performance Test Results
Performance Testing Driving Capacity Management at easyJet
Performance Modelling
31
Business
Service
Component
Virtual Inf...
Performance Testing Driving Capacity Management at easyJet
Performance Modelling
32
Business
Service
Component
Virtual Inf...
Performance Testing Driving Capacity Management at easyJet
Performance Modelling
33
Business
Service
Component
Virtual Inf...
Performance Testing Driving Capacity Management at easyJet
Demand Model Validation
The model should be
validated on an ong...
Performance Testing Driving Capacity Management at easyJet
Capacity and Performance Model Validation
35
0%
10%
20%
30%
40%...
Performance Testing Driving Capacity Management at easyJet
What you need – People
Capacity and performance management skil...
Performance Testing Driving Capacity Management at easyJet
What you need – Process
37
Interface with the release team
Part...
Performance Testing Driving Capacity Management at easyJet
What you need – Tool & Environment
An appropriately sized test ...
Performance Testing Driving Capacity Management at easyJet
What you need – Data
Software/hardware configuration (productio...
Performance Testing Driving Capacity Management at easyJet
Historical Business Demand
40
y = 0.0461x - 1673.1
R² = 0.9668
...
Performance Testing Driving Capacity Management at easyJet
Historical Service Demand
41
Performance Testing Driving Capacity Management at easyJet
Costs
42
Performance Testing Driving Capacity Management at easyJet
Benefits
Testing and modelling complement one another
The combi...
Performance Testing Driving Capacity Management at easyJet
Questions
Presentation will be available for download at www.ca...
Performance Testing Driving Capacity Management at easyJet
Upcoming SlideShare
Loading in …5
×

Performance Testing Driving Capacity Management at easyJet

1,027 views

Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Performance Testing Driving Capacity Management at easyJet

  1. 1. Positionner le logo de la société du présentateur à la place de ce bloc texte Président de filière Orateur Sponsor Administrateur itSMF 2Titre de la présentation Performance Testing Driving Capacity Management at easyJet Andy Bolton Nathalie Grandvuillemin Andy Caddy Benoit de Savignac
  2. 2. Performance Testing Driving Capacity Management at easyJet easyJet.com 3
  3. 3. Performance Testing Driving Capacity Management at easyJet easyJet continues to grow easyJet 2009 £2.6bn revenue 181 aircraft 45m passengers 422 routes “Low cost with care and convenience” 4 easyJet 2011 £3bn+ revenue 204 aircraft 50m+ passengers 560routes “Passionate about making travel easy & affordable” easyJet 2014 £4bn+ revenue 230+ aircraft 60m+ passengers 650+ routes ““Europe’s preferred short haul airline”
  4. 4. Performance Testing Driving Capacity Management at easyJet easyJet 1996 3 Airports 2 Countries 2 Aircraft 2 Routes 5 Base Airport Other Airport
  5. 5. Performance Testing Driving Capacity Management at easyJet 6 easyJet 2011 150 Airports 34 Countries 210 Aircraft 560 Routes 1500+ Flights 60m+ Passengers Base Airport Other Airport
  6. 6. Performance Testing Driving Capacity Management at easyJet easyJet a growth story In 15 years easyJet has become the UK’s #1 airline 7
  7. 7. Performance Testing Driving Capacity Management at easyJet IT Budgets 8
  8. 8. Performance Testing Driving Capacity Management at easyJet easyJet Revenue 9 Flight revenue (easyJet.com + B2B) Flight revenue (easyJet.com + B2B) Non Flight Revenue (Ancilary Partners) Non Flight Revenue (Ancilary Partners) 100% IT Reliance 100% IT Reliance
  9. 9. Performance Testing Driving Capacity Management at easyJet How important is IT? 10
  10. 10. Performance Testing Driving Capacity Management at easyJet 11 How important is IT?
  11. 11. Performance Testing Driving Capacity Management at easyJet Agile Development Process 12
  12. 12. Performance Testing Driving Capacity Management at easyJet Business Context Capacity planning service www.easyJet.com Business critical service 98% of revenue earned through the e-commerce service Highly competitive market Frequent innovation by adopting agile development Monthly software releases Legislative changes Consistent growth; typically 20% p.a. Low-cost operating model Requirement to manage risk 13
  13. 13. Performance Testing Driving Capacity Management at easyJet easyJet’s Goals 14
  14. 14. Performance Testing Driving Capacity Management at easyJet easyJet’s Goals 15
  15. 15. Performance Testing Driving Capacity Management at easyJet Challenges Associated with Agile Frequent change introduces risk Window available for performance testing is typically 1-2 weeks Final production candidate code often not available at the start of testing Multiple code releases within the testing window Increased probability of functional and environmental defects Frequent rework of code necessitates modifications to test scripts Cumulative effect on capacity from 12 releases per annum Narrow time window to resolve performance defects 16
  16. 16. Performance Testing Driving Capacity Management at easyJet Challenges Associated with Agile 17
  17. 17. Performance Testing Driving Capacity Management at easyJet Typical Causes of Degraded Service Performance 18
  18. 18. Performance Testing Driving Capacity Management at easyJet How Can We Reduce Risk? 19 Cause of degraded performance Mitigated by performance testing? Mitigated by modelling? Changes in user patterns No Yes Poor performance of 3rd parties May be limited by environment Yes Physical capacity bottlenecks May be limited by size of the environment Yes Logical bottlenecks Yes If known Software configuration changes Yes Yes, if tested Increases in demand May be limited by size of the environment Yes
  19. 19. Performance Testing Driving Capacity Management at easyJet Why Test? Performance testing is used to understand the software’s performance characteristics Do unknown time and growth related defects exist? Memory leaks Poor database design/indexing Do unknown concurrency related defects exist? Database locking issues Memory locking issues Race conditions Obtain model inputs 20
  20. 20. Performance Testing Driving Capacity Management at easyJet Why Model? Modelling uses our understanding of the software to predict end to end service performance under different scenarios 21
  21. 21. Performance Testing Driving Capacity Management at easyJet Scenario Modelling 22
  22. 22. Performance Testing Driving Capacity Management at easyJet Coverage of Testing and Modelling 23 ModellingTesting Application Infrastructure Business Scenarios Technical Scenarios
  23. 23. Performance Testing Driving Capacity Management at easyJet High Level Process 24
  24. 24. Performance Testing Driving Capacity Management at easyJet Risk Assessment 25
  25. 25. Performance Testing Driving Capacity Management at easyJet Risk Assessment Assess the potential performance impact on different elements of infrastructure Plan tests to help mitigate that impact Focus on areas of highest risk due to limited timeframes 26 Web DB Other Project X --- Medium Medium Low Test 1 Project Y --- Low High Medium Tests 2 & 3 Project Z --- Low Low Low None Description ID Risks Risk Mitigation
  26. 26. Performance Testing Driving Capacity Management at easyJet Performance Tests Employed 27
  27. 27. Performance Testing Driving Capacity Management at easyJet Baseline Tests Used to determine the impact of the planned release Comparative performance between releases May be Load, Stress or Soak Tests Service times and response times are compared to each other 28 Database Server Service Time Web Server Service Time Test 1.1 Release X 752 1240 Test 1.1 Release Y 813 1192 % Difference 8% -4% 0 200 400 600 800 1000 1200 1400 ServiceTime(ms) Example service time comparison between two fictional baseline tests
  28. 28. Performance Testing Driving Capacity Management at easyJet Single Transaction Type Tests 29 To determine the impact on the infrastructure of individual user transactions Every business process which is to be modelled will have a calculated service time for each hardware element 132 233 156 122 221 328 Milliseconds Web CPU per page (fictional data) Home Search Select Options Login Purchase
  29. 29. Performance Testing Driving Capacity Management at easyJet Modelling 30 Performance Test Results
  30. 30. Performance Testing Driving Capacity Management at easyJet Performance Modelling 31 Business Service Component Virtual Infrastructure
  31. 31. Performance Testing Driving Capacity Management at easyJet Performance Modelling 32 Business Service Component Virtual Infrastructure Aircraft Home Page Search Booking HTTP1 APP1 APP2 DB1
  32. 32. Performance Testing Driving Capacity Management at easyJet Performance Modelling 33 Business Service Component Virtual Infrastructure 600 100 80 5 160 70 10 80 3.5 transactions/s
  33. 33. Performance Testing Driving Capacity Management at easyJet Demand Model Validation The model should be validated on an ongoing basis against actual values The first stage is to validate the demand part of the model to ensure that arrival rates are being calculated correctly Following this the true demand can be used to validate the utilisation calculations 34 600 100 80 5 610 80 80 6
  34. 34. Performance Testing Driving Capacity Management at easyJet Capacity and Performance Model Validation 35 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% Comparison of Actual Database Load to Predicted Database Load Actual Database CPU Predicted Database CPU
  35. 35. Performance Testing Driving Capacity Management at easyJet What you need – People Capacity and performance management skills: Demand forecasting Modelling Load testing A strong understanding of risk management A good understanding of the business model Flexibility Project management skills 36
  36. 36. Performance Testing Driving Capacity Management at easyJet What you need – Process 37 Interface with the release team Participation in the agile development `scrums` Access to developers to understand application functionality Interface with the test environment management team for test planning Interface to the business to access business forecasts Interface to operations to access production measurements
  37. 37. Performance Testing Driving Capacity Management at easyJet What you need – Tool & Environment An appropriately sized test environment Production-like environment is not required Consider the ratio of capacity between presentation, business and data tiers Toolset Performance monitoring tools (production and test) Load generation tool Modelling tool 38
  38. 38. Performance Testing Driving Capacity Management at easyJet What you need – Data Software/hardware configuration (production and test) Performance measurements from production and test environments Benchmark data (Spec, TPC, etc.) Service times derived from performance test results SLAs Internal SLAs with third-parties Historical and forecast business demand Historical and forecast service demand Measured error of previous forecasts 39
  39. 39. Performance Testing Driving Capacity Management at easyJet Historical Business Demand 40 y = 0.0461x - 1673.1 R² = 0.9668 0 20 40 60 80 100 120 140 160 180 Mar-04 Jun-04 Sep-04 Dec-04 Mar-05 Jun-05 Sep-05 Dec-05 Mar-06 Jun-06 Sep-06 Dec-06 Mar-07 Jun-07 Sep-07 Dec-07 Mar-08 Jun-08 Sep-08 Dec-08 Mar-09 NumberofOwnedAircraft Delivery Date Fleet Plan to April 2009 GB Airways acquisition
  40. 40. Performance Testing Driving Capacity Management at easyJet Historical Service Demand 41
  41. 41. Performance Testing Driving Capacity Management at easyJet Costs 42
  42. 42. Performance Testing Driving Capacity Management at easyJet Benefits Testing and modelling complement one another The combined approach provides effective risk-mitigation in the context of: Frequent change Narrow performance test window Business growth The approach enables us to rapidly answer business and technical ‘what if’ scenarios Costs are limited through the use of a scaled-down performance test environment and low-cost modelling tools Provides performance assurance in a highly competitive market place “Capacitas give us the confidence in making the correct release decision” James Mackay, Test Manager, easyJet 43
  43. 43. Performance Testing Driving Capacity Management at easyJet Questions Presentation will be available for download at www.capacitas.fr Please visit us at our stand for any further questions 44

×