Testing Sociotechnical Systems: Heathrow Terminal 5


Published on

Published in: Technology, Business
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Testing Sociotechnical Systems: Heathrow Terminal 5

  1. 1. Testing Socio-Technical Systems Part 1: Heathrow Terminal Five John Rooksby University of St Andrews
  2. 2. <ul><li>Systems Testing </li></ul><ul><ul><li>Testing easily exceed 50% of the total effort of a project </li></ul></ul><ul><ul><li>Testing is (in theory) pervasive throughout any systems project – before, during and after system development </li></ul></ul><ul><ul><li>The problem with doing testing at the end of a project is that this is too late to discover major problems (or even lots of minor ones). </li></ul></ul><ul><ul><li>Testing should not (in theory) be the sole means of ensuring quality, testing cannot show the absence of problems, only their presence. </li></ul></ul><ul><ul><li>Testing can involve many different people – developers, professional testers, managers, customers, users, consultants, etc. </li></ul></ul><ul><ul><li>Inadequate testing is given as a cause of many failures </li></ul></ul>
  3. 3. <ul><li>Systems Testing </li></ul><ul><li>Testing is saturated with socio-technical issues. </li></ul><ul><li>These are evolving from individualistic to systems/cooperative-work issues: </li></ul><ul><ul><li>a shift in focus from programs to systems </li></ul></ul><ul><ul><li>‘ usefulness’ becoming relevant in testing alongside or in place of correctness </li></ul></ul><ul><ul><li>iterative development and reduced time to market entailing issues being knowingly left until post-deployment </li></ul></ul><ul><ul><li>testing increasingly becoming a professional, team activity and made accountable to the wider organization </li></ul></ul><ul><ul><li>technology transfer from research entailing the reorganisation of current practices and the acquisition of new skills. </li></ul></ul>
  4. 4. <ul><li>Load / Performance Testing </li></ul><ul><li>A blanket term, generally referring to modeling the expected usage of a software program by simulating multiple users accessing the system’s services concurrently </li></ul><ul><li>Often used synonymously with ‘performance’, ‘reliability’ and ‘volume’ testing </li></ul><ul><li>Will the system work when lots of people are using it? </li></ul>
  5. 5. The opening of Heathrow Terminal 5
  6. 6. Heathrow Terminal 5 <ul><li>Opened March 27 th 2008 </li></ul><ul><li>Operated by BAA with BA as the sole tenant </li></ul><ul><li>A 4.3 Billion Pound Structure (Western Europe’s largest building project), 40 Meters high, 176m wide and 396 meters long. 7500 construction workers on site, from 60 contractors working on 16 major projects and 147 sub projects </li></ul><ul><li>To encourage the contractors to concentrate on problem solving, and thus stay on time and within budget, BAA accepted it carried all the risk for the project. A “lean” method was used – and the project was heralded as a good example (e.g. keynote at ICSE 2007) </li></ul><ul><li>Most of it was done successfully (including the boring of a 13.5km tunnel under a live airfield and close to existing tunnels, the diversion of two rivers and a new spur road for the M25) </li></ul><ul><li>None of the IT was radically different to the other terminals, and this time they had the luxury of space. </li></ul><ul><li>96 Check-in kiosks, 140 customer service desks including 90 fast bag drops </li></ul>
  7. 7. Heathrow Terminal 5 <ul><li>The opening date of March 27 th 2008 was fixed, and was seen as immovable (as is the norm for new airport terminals) </li></ul><ul><li>The building was delivered late, and internal work was still in progress during the opening </li></ul><ul><li>The majority of BA’s short haul operations moved there on the first day (400 flights a day and 34000 passengers) </li></ul>
  8. 8. <ul><li>There was a problem with the baggage system, the first flight left with no bags </li></ul><ul><li>More bags were checked in than were put on aircraft, and the baggage system clogged up </li></ul><ul><li>By 5pm on the 31 st March, BA could no longer accept checked luggage </li></ul><ul><li>By the end of the opening weekend 28000 bags were in ‘temporary storage’ (half that according to BA) – The T5 bag store has capacity for 4000 bags and can theoretically store/retrieve 3600 per hour. </li></ul><ul><li>Incoming passengers experienced delays of up to 4 hours </li></ul><ul><li>Massive queues at check in, long waits for flights </li></ul><ul><li>Cancellation of 500 flights, delays to many more </li></ul><ul><li>The full schedule could not be operated until April 8th </li></ul>The opening was, in the words of Willie Walsh (BA CEO), “a disaster” and “a national embarrassment” The Failure
  9. 9. <ul><li>BA drafted in 400 volunteers over the weekend </li></ul><ul><li>I imagine they got someone in to fix the baggage system </li></ul><ul><li>Bags eventually sent to Italy for sorting </li></ul><ul><li>Effective security was maintained (it is claimed) </li></ul><ul><li>Estimated cost to BA of 25 Million </li></ul><ul><li>A contributing factor to the break-up of BAA </li></ul>Response
  10. 10. <ul><li>The problems on the opening days were caused by problems cascading after the failure of the baggage handling system. </li></ul><ul><li>They failed to test the baggage system at high enough loads </li></ul><ul><li>Failure to remove a network message filter after testing (In place to stop the T5 baggage system sending specimen messages to other terminals) – effected bags transferred from non BA flights. </li></ul><ul><li>“ We did compromise on our testing programme” it was “a calculated risk” </li></ul><ul><li>However he defended against criticism that the dress rehearsals 15000 volunteers were insufficiently large or realistic, and defended against criticism that they did not implement lessons learned. </li></ul><ul><li>An incorrect configuration of a data feed between baggage handling and reconciliation not spotted until April 5 th </li></ul><ul><li>Training supervisors were absent from certain key areas on the opening day, in particular the baggage make-up hall </li></ul><ul><li>BA ground staff not trained on how to drive 'jetties' - which connect planes to the airport - leaving many stranded and without power. </li></ul><ul><li>BA ground staff not trained on new systems which led to delays in directing planes to designated parking slots. </li></ul>Contributing factors 1 Testing Training Willie Walsh was summed before Government Transport Select Committee to explain. He informed them staff were given insufficient training and corners were cut on testing .
  11. 11. <ul><li>Construction of T5 overran, leaving work still to be done after it opened. </li></ul><ul><li>28 out of 275 lifts not working </li></ul><ul><li>Communication between BA and BAA failed with no joint crisis plan in place. </li></ul><ul><li>Contingency plans, including those to respond to baggage system failures, were not given to the right people. </li></ul><ul><li>Staff delayed at security points because there was up to 50% more employees than expected, exacerbated by the breakdown of an X-ray security machine. </li></ul><ul><li>Sixty employees were late on opening day because of troubles in staff parking. </li></ul><ul><li>Eight (some reports imply all) staff delayed in logging onto baggage system because of faulty setup. They then had to reconcile bags manually (very slow). Bags affected by the message filter problem also manually sorted. </li></ul><ul><li>Data transmission errors between BAA and SITA (an It contractor) affected flight data, and caused further problems for bad reconciliation. </li></ul><ul><li>A lack of server capacity at the terminal exacerbated the problems </li></ul>Contributing factors 2 Staffing Planning Building IT
  12. 12. Information about The Baggage Handling system
  13. 13. Information about The Baggage Handling system <ul><li>Designed to handle inter and intra terminal baggage – 70,000 items dialy </li></ul><ul><li>The largest baggage handling system in a single terminal in Europe </li></ul><ul><li>BAA and British Airways worked together on T5. The baggage system was designed by BAA, Vanderlande and IBM </li></ul><ul><li>A 2D barcode is attached to each bag with information on where it needs to go, and handlers use hand-held scanners to update the bag's position on the system and to work out which route it should take. </li></ul><ul><li>BAA decided against RFID, which uses radio signals to track bags, because, IT director Richard Rundle said, the value of RFID would only become apparent once airports all over the world use it. </li></ul><ul><li>The new baggage system uses 2D barcodes and scanners “before, the system was crammed into buildings that were too small for it” (BAA executive) </li></ul>
  14. 14. <ul><li>Construction of the building and installation of utilities </li></ul><ul><li>Installation of physical system (conveyers, scanners, cranes etc) </li></ul><ul><li>Installation of electrics </li></ul><ul><li>Installation of equipment controls – computer hardware and software </li></ul><ul><li>Configuration and testing of controls </li></ul><ul><li>Installation of high level controls – computer hardware and software for logistical functions etc. Integration testing of those controls with low level controls </li></ul><ul><li>Integration and systems testing of baggage handling system, in isolation and in context of other airport systems </li></ul><ul><li>Testing operational use of the system </li></ul>“ Rough” Development Plan
  15. 15. <ul><li>Functionality </li></ul><ul><li>Automatically identify bag label </li></ul><ul><li>Screen for explosives </li></ul><ul><li>‘ early bag’ storage </li></ul><ul><li>Manual encoding for resolution of information errors </li></ul><ul><li>Sort to flight allocated loading position </li></ul><ul><li>Fast tracking for late bags </li></ul><ul><li>Manual handling for large bags </li></ul><ul><li>Reconciliation that bag is authorised to travel (ie passenger on plane) </li></ul>Architecture
  16. 16. Testing The Baggage Handling system <ul><li>Go live dates for airport terminals are very difficult to delay </li></ul><ul><li>It is well known from past experience that problems out of the control of the baggage handling system test team can interfere (ie building not completed) </li></ul><ul><li>Factory testing was done using simulators </li></ul><ul><li>Automod simulation software (which does ‘3D virtual reality simulations’ was used to test the controls software) </li></ul><ul><li>4800 hours spent building and testing with the low level emulation models, 6000 hours the high level </li></ul><ul><li>However a lot of the problems identified by the test team were caused by the emulation model itself </li></ul><ul><li>On site testing is also necessary as defects can occur in the physical system during installation </li></ul><ul><li>Testing is completed with a Component Site Assurance Test (CSAT) – ensuring the low level controls are of high enough quality to be integrated with the high level controls </li></ul>
  17. 17. T5 in October 2008 The move of long haul flights was delayed from late April, and a phased approach taken (June, September, October) Now handling 6 out of 7 BA flights from Heathrow 70000 passengers a day The final transfer of flights will be on October 22 nd 2008 Most passengers using the fast bag drop achieve check in within the target 10 minutes. Average plane to carousel of luggage is “only” 24 minutes &quot;Terminal 5 is working” A direct advertising campaign by BA including billboards and a website displaying recently-taken images and up-to-date statistics: www.terminal5.ba.com/yesterday/
  18. 18. <ul><li>Discussion </li></ul><ul><ul><li>What role does testing have when there are fixed deadlines? </li></ul></ul><ul><ul><li>Do the outcomes of these cases actually tell us anything about testing other than “you should have tested it more” – are the lessons about design? </li></ul></ul><ul><ul><li>Can decisions in testing / not-testing be coupled to wider understanding of socio-technical systems? </li></ul></ul><ul><ul><li>Should testing be used to answer “what do I need to worry about at go-live”? (given that testing is routinely compromised, and a disastrous go live does not mean the system is a long-term failure) </li></ul></ul>
  19. 19. <ul><li>Key Documents </li></ul><ul><ul><li>Roger Derksen, Huub van der Wouden, Paul Heath (2007) Testing The Heathrow Terminal 5 Baggage Handling System – Before it is Built. Paper Presented at Eurostar 2007, Manchester UK. </li></ul></ul><ul><ul><li>House of Commons Transport Select Committee (2008) The Opening of Heathrow Terminal 5. Published by The Stationary Office LTD. </li></ul></ul>