Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Boots & all Telstra Operations summer 2010/11


Published on

As we head into a new summer, I think it is always good to reflect on and learn from what has been. It is also good to remember how our people tried their best to keep communities connected across this wide land in summers past.

Last summer brought with it some of the most devastating and frightening weather events Australians had ever seen, from bushfires to cyclones, storms and flooding.

Published in: Technology, Business, Sports

Boots & all Telstra Operations summer 2010/11

  1. 1. BOOTS & ALL: SUMMER 2010/11Firsthand accounts of how the Telstra Operations teamprepared for and responded to the summer’s devastating events.
  2. 2. CONTENTSThe Australian summer 2010/11 brought with it bushfires,cyclones, storms and flooding – all of which were typicalevents for this time of year. However, combine these with anunprecedented La Niña event and you have an extraordinarysummer.Extraordinary times call for extraordinary measures and somewonderful stories were being told about how the TelstraOperations team were responding. It is for this reason that a‘call to action’ was delivered throughout this group, asking theteam to share their experiences.This book is therefore dedicated to recognising the hard workand tireless efforts of those within Telstra Operations.This team is responsible for all aspects of the design,engineering, architecture, construction and operation ofTelstra networks, technology and information technology, plusthe delivery of customer services across those networks.It is acknowledged that the company’s response to thesummer events was an enormous Telstra-wide effort andspecial thanks to colleagues within the Consumer & CountryWide, Enterprise & Government, Telstra Business and theStrategy & Corporate Services business units.These Telstra groups offered invaluable support to strickencommunities and businesses, both large and small, includingkeeping those impacted well-informed concerning progressduring Telstra’s restoration, reparation and reconstructionphases.The owner of ‘the boot’ on the front cover of this book is technician Rob White.Rob is pictured here replacing pillar strips at St Lucia (Qld), 19/01/11.Boots & All: Summer 2010/11 has been collated and edited by Sandra Mobbs,Strategy and Corporate Services.
  3. 3. CONTENTSMatt Totterdell and Roger Williams replacing a 400 pair copper cable, Carnarvon (WA). Their team manager at the time advised, “These guys had just got changed out of their wet clothes five minutes before this downpour.”
  4. 4. CONTENTS CONTENTSMessages from Telstra’s Chairman and Chief Executive Officer 6 PM offers heartfelt thanks to volunteers 192 Salvos emergency volunteers recognised 193Introduction 10 Rod and the Reject Shop 194 An invitation to an informal event with His Royal Highness Prince William 195Our approach to storytelling 14 Letter from the Federal Member for Wright 196Firsthand accounts from the Telstra Operations team 18 Our people’s community contribution recognised 197Hoping for the best, preparing for the worst 20 Last words from the Telstra Operations leadership team 200Central East Region (New South Wales, Australian Capital Territory) 30 Note from Strategy and Corporate Services editor, Sandra Mobbs:South East Region (Victoria, Tasmania) 50 ‘Boots & All’ has captured over a hundred stories, recording an unprecedentedNorth East Region (Queensland) 92 moment of this company’s history – the summer of 2010/11.The comprehensiveCentral West Region (Western Australia, South Australia & Northern Territory) 158 nature of this publication has only been made possible due to the generous spirit of the Telstra Operations team. Therefore, thank you to our storytellers and subjects(See lists of all stories, under each of the headings above, on the following pages.) for giving up their time to share their experiences and providing consent to use images. My thanks also to Phill Sporton, Executive Director Service Delivery, forLearning from experience 176 initially commissioning this employee engagement project in March 2011.Past ideas and innovations • Raised earth platforms 179 • Mobile Exchange on Wheels or the MEoW® 180 • Cell on Wheels (CoW) and Satellite Cell on Wheels (SatCoW) 180 • TECKs and TREKs 181Ideas and innovations due to summer 2010/11 • RIM to CMUX conversion plate 182 • New website keeps customers informed during outages and disasters 183Community, customer and industry recognition of ourpeople’s efforts 184CFA says thanks 186Floods disrupt National Relay Service 187Queensland department thanks Telstra for speedy response 188Commendations for quick action 189A ‘Big Thank You’ from Volunteering Queensland 190Anglicare warmed by quick and relevant contribution 191
  5. 5. CONTENTSHoping for the best, preparing for the worst 20 Working in waders John Pridgeon 66Prior planning and preparation leads to rapid An extraordinary effort Terry Scott 67network restoration Grant Nicholson 22 A tough couple of months Ian Baker 70On the ground in bushfire exercise Joe Camilleri 24 The hard yards Daryl Crosbie 72Unprecedented events tests capabilities Phil Astle 26 Charlton isolation Peter Craig 74How to prepare for network impact caused Volunteering with the CFA Roger Smith 76 Bridgewater underwater Tony Ryan 77by a Cat 5 cyclone Pat Rutter 28 Getting stuck in John Hossack 78Central East Region (New South Wales, Australian Capital Territory) 30 The fastest CoW in the country Ron Wilson 80 Saved from a real dirty job Daryl Beseler 81Walls of water Grant Carroll 32 Tack’ling the tough stuff John Hewett 82Riverina-Murray recognition Loretta Willaton 34 Fire and flood John Fixter 84The return of a slippery customer Con Parslow 35 Doing the nightshift Michael Ennor 85Lightning storms, floods and road closures Bill Lloyd 36 Just out of harm’s way Brad Crocker 86Back to business David Gillett 38 Kerang customer rapt Brad Shaw 87Washed out Terry Walsh 39 What goes around comes around Janet Mackey 88Call Central weathers the storm David Coxhell and Natalie O’Malley 40 Telstra blokes everywhere Craig Lawry 90Late night deliveries Terry Lines 42 Record rain and river crossings Matt Shaw 91Proudest moments Danny Freeman 43Left-field solutions go a long way Stephen Chadd 44 North East Region (Queensland) 92Many rivers to cross Peter Chapple 46 Ready for action Malcolm McKellar 94Find the fibre access point Greg Byrne 48 Flying in to fix fibre Peter Nash 95 Quick fix connects customers Peter Spence 96South East Region (Victoria, Tasmania) 50 The magnificent seven Shaun Walliss 97Wet or what? Warren Shean 52 Homeward bound Mark Graham 98Copping it early Ian Baker, Des Ryan and Trevor Goudie 54 The Army, Police and Dave toBit of bother Laurie Barber 56 the rescue David Webb and Craig Bartlett 99Incident Control Centre support Nick Marotta 58 Exchange wrapped in plastic Steve Strugnell and Shane Golding 100Sending in the reinforcements Ken Hodgson 59 Toowoomba – an insider’s view Sue Ikin and Dave Marshman 101A changed vacation Rob Dahllof 60 Flash flooding hits with no notice Peter Sticklen 102East coast focus for Tassie Glenn Turner 62 Meeting the challenge head-on John Parkin 104North-west coast focus for Tassie Ian Pickering 64 Keep calm and carry on Craig Bartlett 106Better safe than sorry Alistair Cowie 65 Staying the course Greg Anderson 108
  6. 6. CONTENTSBrisbane CBD shuts down Lynne Bell 110 Central West Region (Western Australia, South AustraliaAbove and beyond in Rocky Peter Spence 111 and Northern Territory) 158Capacity request met in difficult circumstances Neil Francis 112 The Stockport CoW Peter Andreopoulos 160When your road becomes a boat ramp Susan Kuppens 114 100-year floods Alan Brown 161Community comes together Shanne Wright 116 What did you do over Christmas 2010? Tim Leahy 164The Kholo/Mt Crosby ‘break and enter’ incident Peter Leonard 118 Home Sweet Home Mick Cooper 166Fault frog finds infamy Noel Hand 120 Fires at Lake Clifton Peter Old 167Queensland Flood Relief Hotline John Tarlinton 121 Great Northern Highway disappears Mick Cooper 168Coming to terms with Grantham Peter Scherer 122 Snakes, bugs and crocs Neil Cooke 170Solution-oriented tech takes charge Tym Browne 124 Public support outstanding Shane Caratti 171What a weekend! Steve Burke 126 The Kimberley cougar strikes again Joe Ganino 172A surreal experience Heidi Pfeffer 128 Just keep going – a poem Tony Hunter 174Snakes on joint go viral Pete Milward 129Who let the CoWs out? Michael Steele and Aaron Kong 130Pitching in Paul McCarthy 132Help from across the ditch Malcolm McKellar 134Oh geez – this is real! Paul Montiford 135Thunderbird operators are go! Graham Ford 136Roadside cabinet restoration work Michael Steele 138MacGyver has nothing on our guys Phillip Stringini 139My night in Townsville Exchange Wayne Watling 140Ground Control John Shepherd 142War Room approach hits the mark Dru Dingwall 143Ugly but interesting – about our exchanges John Dempster 144Bill the builder comes to the rescue Ross Auger 148View from Cardwell tower Chris Frost 150BAM and it’s gone Mick Young 151Rick to the rescue Joanne Flood 152Mark gets up close and personal Mark O’Connell 153SatCoW flies to Palm Island Clint Dickson 154Whipping up support Stephen Bowen 156Stepping up Mark Pettiford 157
  7. 7. CONTENTSIt’s been a hard day. Dennis Dregmans at Moggill (Qld), 25/01/11.
  9. 9. CONTENTS A message from Telstra’s Chairman Keeping communities connected during times of natural disaster demands a huge commitment, and often personal risk, from our technicians and support staff in the field. The stories you will read in this book are only a small fraction of what we as a company contributed to the disaster response and recovery, but they highlight the lengths our people will go to keep our customers and their communities connected. Catherine Livingstone: Telstra Chairman The dedication and commitment of Telstra’s people truly shines during difficult times. The Board was regularly briefed about the challenges Telstra crews faced and how, in collaboration with emergency service agencies, they supported local communities. On behalf of the Board, I acknowledge with deep gratitude the efforts of all those who responded to the natural disaster events of the summer of 2010/11. Catherine Livingstone AO Chairman Telstra’s Chairman attending a briefing concerning progress by the Northern Flood Recovery team in Brisbane (Qld), 24/01/11.8
  10. 10. CONTENTS A message from Telstra’s CEO Summer 2010/11 brought with it some of the most devastating and frightening weather events Australians had ever seen. We should never forget that it is our team, the Telstra team, which is one of the first to respond when a disaster strikes. Communication is essential during these times and Telstra people play a critical role ensuring communitiesDavid Thodey: Chief Executive Officer and emergency service organisations stay connected. This book contains firsthand accounts of how the Telstra Operations team took swift and decisive action and came up with some innovative solutions to meet the many challenges which presented themselves along the way. I would like to extend my sincere thanks to the people involved in preparing for possible network impacts, monitoring our networks in the aftermath, restoring services, helping customers and supporting their local communities during the summer 2010/11 peak load season. We can all be enormously proud of the team’s efforts during these times. David Thodey Chief Executive Officer Allen Brazier, John Parkin, David Thodey, Dave Liddell, Dave Kincaid, Greg Anderson and trainee Jason Lewis during the CEO’s visit to check in with the troops working at St Lucia (Qld), 25/01/11. 9
  11. 11. CONTENTSRoyston Bruce jointing a new section of 800 pair cable damaged in the floods at Rocklea (Qld).
  13. 13. CONTENTS Each year Australians deal with natural disasters Ready – Prepare our people, customers, infrastructure Often these reviews spark ideas, ideas which result where lives, homes and livelihoods are often on and business. in technological innovations and changes which the line. improve the way we do things. Respond – Determine the impact on our people, Keeping connected – something most of us take customers’ infrastructure and businesses. Within these pages we will refer to some of these for granted during the best of times – is of utmost innovations, which now form part of our Disaster Restore – Prioritise the restoration of services in co- importance during the worst of times. Planning (DISPLAN) communication support to ordination with emergency service organisations. emergency service organisations, including our In an emergency, one of the first things we all do Repair – Develop solutions which restore services Telstra Emergency Communications Kits (TECKs), is reach for the phone to call for help or to make as quickly as possible to isolated communities – this Cells on Wheels (CoWs) and Mobile Exchange on contact with loved ones. may involve temporary fixes. Wheels (MEoW®). For some facts and figures about This book concentrates on the events of summer these important pieces of equipment and other Reconstruct – Work to permanently repair or 2010/11 to provide the reader with firsthand, behind- innovations, under the ‘Learning from experience’ rebuild infrastructure damaged by the event. the-scenes accounts of how Telstra Operations section of this book (p.176). team members prepare for and respond to disasters (Note: See breakout box on the following page The last sections of this book are dedicated to which have the potential to impact communities and for more details of each phase of the 5R disaster recognition by the community, customers, industry their ability to keep these communities connected recovery framework.) and the Telstra Operations senior leadership team, during critical times. After managing through major infrastructure highlighting further how team members went above- The Telstra Operations team manage major, impacting events, the Telstra Operations team and-beyond what is expected in support of our customer-impacting incidents in accordance with conducts a Post Implementation Review to gain customers during these times. their ‘5R’ disaster recovery framework. The 5R knowledge and learn from experience. phases are as follows. Above: Road access was a huge issue. Opposite Top: John (Tack) Hewett rows a boat across a customer’s paddock to get to the job (Vic). Opposite Middle: Preparing for flooding in Victoria, Jim Rogers checks out this roadside12 cabinet which has been well sandbagged. Opposite Middle Lower: Warren Collie working at St. Lucia (Qld). Opposite Bottom: Old rail bridge at Dubbo (NSW).
  14. 14. CONTENTSBreaking down the 5Rs into actions (Depending on the event, these phases may include the following) • Field workforce prepared (resource planning – techs on the ground) • Communications plan established (Telstra internal and external, e.g. emergency service agencies, media, etc.) • Business Continuity: - Central Business District (CBD) sites evacuated during floods - Staff working from home and where necessary moving functions performed by staff at other locationsReady • Protecting the core network: - Network redundancy tested - Critical infrastructure protection - Power and access - Sandbagging of exchanges, emergency power plants checked and refuelled. See the ‘Hoping for the best, preparing for the worst’ section (p.20) for more details concerning how the team gets ready. When dealing with disasters of the kind we faced this summer, our focus is on the safety of our people and the restoration of our core network, which is vital to ensure telecommunications traffic into and out of affected areas, including 000 and to assure communications to emergency service organisations. The biggest challenge to keeping people connected during major storms, flooding, cyclones and bushfires is the loss of mains power. From the outset of this summer’s weather events the Telstra Operations team worked in direct alignment with power utilities and partner Silcar Energy Solutions to ensure critical telecommunication sites were repowered. Where critical mobile sites could not be restarted the team deployed Telstra’s Cells On Wheels (or CoWs), which are portable mobile base stations. Telstra’s Mobile Exchange on Wheels (orRespond, MEoW®) was used where critical exchanges went down.Restore and The respond, restore and repair phases can also include:Repair • Telecommunications support to evacuation centres, recovery centres and volunteer organisations • Prioritisation of work based on customer needs • Constant review of power to core network infrastructure to all key network sites • Constant review to ensure access to sites is safe • Temporary network restoration • Temporary repair of damaged infrastructure, deployment of temporary network elements • Focus on repair or replacement of roadside cabinets, getting customers and businesses back online. The final ‘R’ is ‘reconstruct’ and focuses on the team’s program of work to permanently repair orReconstruct rebuild damaged infrastructure.
  15. 15. CONTENTSByron Griffiths, one of a team of five, who manually dug extremely sticky soil in hot, humid, fly-friendly conditions to locate and fix fibre issues at FlorinaStation (NT). See related story, ‘What did you do over Christmas 2010?’ (p.164)
  17. 17. CONTENTS “The main audience for this book are Telstra Operations team members and their family and friends.” If one picture could tell the story about the summer Where possible the editor has chosen shots taken 2010/11’s weather events, then this one; sourced during the event which feature Telstra crews at work, from and used with the kind permission of the to fully capture the essence of the stories being told. OUR Australian Government Bureau of Meteorology; would be it. However, some stories do not have images supplied of the team members involved, as one construction APPROACH TO The story this image tells is a simple one. During October operative candidly put it when asked about photos, “We were all too bloody busy to pose for photos!” 2010 to end-March 2011 (the definition of the summer STORYTELLING period for the purposes of this book), approximately ninety percent of our country received above-average The stories have been listed by region* and are in date order as to when the storyteller’s event occurred. to unprecedented, ‘highest on record’ rainfall, with all The main focus of the stories supplied here is the states impacted. However, parts of Western Australia’s team’s response when communication services and southern regions remained extremely dry, also a recipe communities were significantly impacted. for disaster – bushfires. Parts of Victoria also suffered bushfire events during this period. Finally, while devastating events happened globally during this period and in some cases Telstra Operations Within these pages many tales have been collated with people were involved (for example the Australia-Japan three types of stories told: firsthand accounts ‘by’ the cable assessment following a major 8.9 offshore quake actual person involved, second-hand accounts ‘as told which impacted large areas of Japan’s northern Pacific by’ a team mate about another’s efforts and a question coast in March), this book has been limited to a national and answer style ‘interview with’ those involved. view of summer events. Concerning all the stories captured here, the authentic *As our country’s regions are often defined differently by different groups, voice of the storyteller shines through; therefore the telephone ‘area code’ regions have been used for the purposes of this book. Therefore the regions used within these pages are as follows: Central East language and tone of the stories vary greatly depending Region (New South Wales, Australian Capital Territory), South East Region on the storyteller’s style and the situation team members (Victoria, Tasmania), North East Region (Queensland) and Central West found themselves in at the time. Region (Western Australia, South Australia and Northern Territory). The main audience for this book are Telstra Operations team members and their family and friends. Therefore, for ease of reading, each story within this book is self- contained with acronyms and terms explained. All the images contained in this book have been sourced from team members involved. Many were captured via their mobile handsets and therefore the quality and resolution of photographs contained within the book varies.16
  18. 18. CONTENTSAbove image: Sourced from and used with the kind permission of the Australian Government Bureau of Meteorology. 17
  19. 19. CONTENTSAfter a break and enter incident, Peter Leonard and Keiron Smith revisited the RIM to capture the scene of the crime.Read how the pair put in a marathon effort to keep the Kholo/Mt Crosby (Qld) community connected (p.118)
  21. 21. CONTENTSDuring December’s bushfire exercise. See ‘On the ground in bushfire exercise’ story (p.24).
  23. 23. CONTENTS PROACTIVE... PROACTIVE... “Work continued non-stop throughout the night and into the next day ...” Interview with Grant Nicholson, to engage stakeholders early – advising them of Network & IT Operations network preparations and communicating potential PRIOR Date: October 2010 onwards network risks and priorities. The Network Assurance Operations teams meet PLANNING AND Telstra’s Network Assurance Operations team based at Telstra’s Global Operations Centre are regularly to share information concerning the up coming emergency and to ensure they have rostered PREPARATION responsible for the control, co-ordination and communication of high impacting customer their teams appropriately and have made any necessary preparations relating to their technology LEADS TO RAPID incidents and networks at risk. They work closely with the Telstra Operations groups on the ground (for example the Voice and Traffic team generally manually perform backups of the data for each of the in the affected areas, including determining priorities NETWORK for preparation and, afterwards, in the restoration exchanges in the area). In preparation for both the Queensland floods efforts. RESTORATION For this team, early engagement with key and Cyclone Yasi, exchanges were made as safe as possible, including removal of debris and stakeholders for preparation saves a lot of time and sandbagging. Power preparation and refuelling allows all groups to act effectively and efficiently strategies were put into place for exchanges most should network impact be experienced. likely to lose power. Preparations can only go so far, sometimes Upon becoming aware of an emergency incident, the unexpected takes place. In the case of the the team first makes contact with Telstra’s assigned Queensland floods, a third-party organised a major Emergency Services Liaison Officers (ESLOs) telethon to raise funds from the community, but to ensure all preparations are aligned with field did not advise Telstra. The Network Assurance operatives (including emergency agencies involved Operations team had to think quickly to co-ordinate and Telstra’s Service Delivery communication extra capacity. technicians and Network Construction constructor operatives). Annie Martin, one of the Major Incident team managers co-ordinating the incident, Following this, key network infrastructure within commented, “Technical staff were quickly the area at risk of impact is identified. ‘Go/no go’ gathered together and implemented some zones (red, amber, green) are constantly reviewed network changes to redirect call traffic using with the relevant emergency service authorities to alternate technologies. ensure safe passage for any field staff working in, or Whilst the changes enabled callers to get through travelling through, the zones. and make their donation, it created some other Early communication with internal stakeholders challenges for how the calls were handled, as the (groups within Telstra that may also be impacted or centre also takes calls for Queensland SES and can assist in preparation and/or restoration work) is Health. Work continued non-stop throughout also very important and ensures business continuity the night and into the next day to address the plans are put in place in preparation for the coming network capacity issues and return their service emergency. One of the key steps in preparation is to normal.”22
  24. 24. CONTENTSAt the onset of Severe Tropical Cyclone Yasi, theNetwork Assurance Operations team developed a About the role of Emergency Services Liaison Officerspre-impact priority restoration plan which allowed Telstra’s Emergency Services Liaison Officers (ESLOs) provide the interface between Telstra and thethem to become a practical part of preparations, emergency service organisations during an emergency such as flood, fire, cyclone, etc.monitoring the network closely and working alongsideemergency service organisations and district disaster ESLOs are field managers from Service Delivery and are supported by a state and deputy groups. During emergencies, ESLOs support the communications requirements of emergency agencies byOnce the cyclone had passed, restoration efforts co-ordinating requests including: restricting staff movement in the affected area by applying ‘no go’ zonesinitially focused on protecting the IP core sites, (for safety reasons), the identification and preparation of infrastructure at risk, organising additional productswhich was made very difficult due to power being and services as required, prioritising fault management for critical emergency and utility sites, and theunavailable at a number of exchanges. As sites temporary and long-term restoration of communication services.became safely accessible, Silcar Energy Solutions,Service Delivery and Network Construction teamsworked tirelessly to hook up generators andcommence regular refuelling runs to keep thegenerators topped up with fuel.To safely access key sites Telstra goes to greatlengths, for example working with the AustralianDefence Force to fly in via Black Hawk helicopter. Nick Kellett, one of the Network Operations Managers working at the time recalls, “We knew that the mains power would be disrupted for a number of days so we had to come up with a plan to keep all the network sites linking the IP core together working. This meant getting portable generators out to the sites that just had batteries and keeping the overall generator network refuelled. Given the large geographical area we had to cover and in some instances the lack of access due to the massive disruption that the cyclone left behind, we had to be creative. A couple of times we were within minutes of losing our IP core network which would have meant a loss of all fixed and mobile communications to Telstra customers in far North Queensland. We leveraged our great relationships with Silcar Energy Solutions, Service Delivery and the Australian Defence Force to come up with a robust refuelling plan which we implemented and monitored for two weeks to avoid any sites failing.” Top Left: Black Hawk at Theodore (Qld), 30/12/10. Top Right: Some of the team involved in the Major Incident room, at Telstra’s Global Operations Centre were (l-r) Grant Nicholson, Joe Camilleri, Soula Duval, David Cheeseman, Glen Taylor and James Wright (back to camera). Bottom: Telstra’s Global Operations Centre provides 24/7 monitoring across all Telstra networks, including working with emergency services 23 and field groups to establish restoration priorities during emergency incidents.
  25. 25. CONTENTS PROACTIVE... PROACTIVE... “For Telstra, the exercise was an important part of our seasonal preparation...” By Joe Camilleri, Network & IT Operations SMR is used by the Police, CFA and Victorian Date: 05/12/10 Department of Sustainability and Environment (DSE) ON THE In December, members of the Telstra Major in country Victoria as well as a number of other emergency organisations. It is a vital communication Incident Management and State Managed GROUND IN Radio teams participated in the largest test of Victoria’s bushfire preparation since Black system in an emergency. See related story (p.186) ‘CFA says thanks’. BUSHFIRE Saturday (7 February 2009). Alongside the Country Fire Authority (CFA) and SES, the exercise EXERCISE tested communication, collaboration and processes between the organisations involved in emergency situations. For Telstra, the exercise was an important part of our seasonal preparation for emergencies, providing a unique opportunity to test our disaster planning processes which includes the communications support Telstra provides to emergency service organisations to assist them in managing emergency situations. Co-ordinated from the CFA’s Kangaroo Ground Incident Control Centre with a staging area at Yarrambat Park, members of the Telstra Major Incident Management and State Managed Radio teams were there to see firsthand how communication systems operate on the ground during an emergency. The Telstra team were on hand to support the deployment of communication systems and were able to showcase our emergency communications capability by deploying a Telstra Emergency Communications Kit (TECK). The TECK provided incoming and outgoing PSTN and fax services for the SES and CFA staff on the ground. Telstra teams were also able to observe how the StateNet Mobile Radio Networks (SMR) operated during the exercise.24
  26. 26. CONTENTSTop: Comms play an extremely important role in assisting emergency service organisations to manage in a crisis. Here we see the TECK in use during the exercise. Bottom: During December’s bushfire exercise. 25
  27. 27. CONTENTS COLLABORATION... “Restoration activities were difficult due to the unsafe conditions and a record number of red zones...” By Phil Astle, Network & IT Operations state, including Charlotte, Edison and Ipswich, Date: December 2010 onwards preventing any further adverse impact. UNPRECEDENTED The Queensland floods and Cyclone Yasi Protecting the core network and ensuring continuity of service for the community, including emergency were two unprecedented events that put our EVENTS TESTS network and capabilities to the test. The other weather events across other states, whilst they were and essential services, were our key priorities. The restoration of local power and site access restrictions CAPABILITIES significant for the communities, had a comparatively minimal impact on Telstra infrastructure and being lifted saw a steady decline in the number of impacted services. customers. The emergency incident management process was invoked to manage all the natural disasters. Phil Astle: Onsite liaison The Incident Operations team engaged a number of teams including Service Delivery, Network When it became apparent the recovery effort Construction, Silcar Energy Solutions and Strategy & for the Queensland floods was unprecedented, Corporate Services to co-ordinate asset protection, Phil Astle, emergency and relationship manager restoration activities and communications. within the Network Assurance Operations Incident team, relocated from Telstra’s Global Restoration activities were difficult due to the unsafe Operations Centre (GOC) in Melbourne to conditions and a record number of red zones (‘no Queensland. Phil’s role was to attend twice daily go’ zones) declared by emergency services across Regional Emergency Council meetings and flood and cyclone impacted areas. Take Queensland assist with the management of the technical for instance, where we had a total of 719 red zones bridge to ensure any issues relating to damage declared by the end of the season. were quickly identified and escalated to make Despite the significant customer impact the network sure the recovery effort was not compromised. withstood the treacherous conditions quite well with The role of team members at Telstra’s GOC the major impact being caused by a loss of local includes 24/7 monitoring and first in fix of mains power and back-up batteries depleting due incidents across all Telstra networks, customer to site access issues. experience monitoring, incident management, It is highly probable the impact would have grown event management, emergency services exponentially if we had lost any of our IP core sites. call tracing and working with field groups to establish restoration priorities during emergency With the collaborative efforts of the Telstra incidents. Operations teams we were able to successfully protect a number of major exchanges across the26
  28. 28. CONTENTSTop Left: Joe Camilleri, Tom Farrell, Phil Astle, Will Visser and Paul Gerreyn at the December bushfire exercise. Top Right: Our field crews’ safety, due to road access issues, was monitored closely at the time.Bottom Left: The devastation caused by Cyclone Yasi and flooding was shocking. Bottom Right: Restoring power was important. Here a 4WD forklift is used for a genset deployment to a hill top exchange at Cardwell (Qld). 27 27
  29. 29. CONTENTS PROACTIVE... PROACTIVE... “The preparation prior to Tropical Cyclone Yasi provided benefits in reducing the impact to Telstra’s infrastructure in the affected areas...” By Pat Rutter, Service Delivery Post-Cyclone Date: 01/02/11 onwards HOW TO PREPARE Telstra’s key learning from Tropical Cyclone • The immediate three day focus was on power restoration. FOR NETWORK Larry (2006) was power would be a significant issue if Tropical Cyclone Yasi crossed the coast • A ‘4-Point Recovery Framework’ communications was established to provide for IMPACT CAUSED in a populated area in Far North Queensland. This proved correct as 680 major network sites were an easy flow of communications between the four main groups involved at Telstra’s Global BY A CAT 5 left without power. This required the deployment of Operations Centre, in Brisbane, Cairns and generators and the co-ordination of power restoration Townsville. Open phone bridges (technical and with Ergon Energy and Powerlink Queensland. CYCLONE Pre-Cyclone operations) were also in operation. • Power restoration meetings continued at a high • Core and priority network sites were identified level between Silcar Energy Solutions, Ergon by the Global Operations Centre / Major Incident Energy and Telstra. Management in Melbourne and the list reviewed • Restoration efforts including timing and access by the cross-Telstra business unit team to include were co-ordinated with Ergon Energy and SES. other known priorities based on local knowledge. • Daily Major Incident Management status updates • The Global Operations Centre / Major Incident provided visibility of Telstra zone status (ability to Management then completed scenario modelling access sites), summary of customer impact, and on the sites, with Telstra’s partnered contractor a power summary (including the number of sites for the provision of emergency power, Silcar without mains power, sites restored by Ergon Energy Solutions, in turn predicting the impact Energy and the number of sites being monitored and requirements for power at these sites post- and managed by Silcar Energy Solutions for cyclone. refuelling, batteries, generators, etc.). • A pre-disaster technical phone bridge was The preparation prior to Tropical Cyclone Yasi established to assist in planning for potential provided benefits in reducing the impact to Telstra’s issues identified. infrastructure in the affected areas by pre-planning the availability of generators for interim power to the key network sites. Relationships already established between Telstra, Telstra’s partnered contractor and Ergon Energy allowed all parties to have a co-ordinated approach to the restoration of power to these essential sites.28
  30. 30. CONTENTS Our top three network priorities: Power! Power! Power! Like many Telstra partners, Silcar Energy Solutions were only too happy to assist Telstra and their customers with restoration of services following the devastating and destructive effects of ongoing flooding, including the aftermath of Cyclone Yasi. Silcar Energy Solutions deployed resources and material from both Queensland and New South Wales to assist in around the clock efforts which initially focused on the restoration of the communications network. Silcar Energy Solutions commitment with the recovery saw personnel based in these regions for in excess of three weeks. During this time Silcar Energy Solutions’ employees worked 12 hour shifts alongside Silcar field technicians, Telstra field teams and other agencies also engaged in recovery efforts. Whilst many involved described the efforts as strenuous and demanding given the destruction caused, they also said they would be happy to do it all over again. Trevor Starcevich, Network Construction Contract ManagerAbove Top: Craig Bartlett, the Emergency Services Liaison Officer (ESLO) who undertook the massive task of co-ordinating efforts with emergency service organisations for Queensland and Northern New South Wales duringthe summer 10/11 period. Above Bottom: David Thodey visiting some of the Northern Region Operations team (l-r) Trudy Deighton, Jay Patel, Chris Hazelton, Tony Broadway, Tim Lostroh, Sym Puskaric, Simon McLean, 29David Thodey, Shellie Cave, Sandra Perry and Pat Rutter
  31. 31. CONTENTSGreg Cooke at a washout at Lue. See related story ‘Lightning, storms, floods and road closures’ (p.36).
  32. 32. CONTENTS CENTRAL EAST REGION(NEW SOUTH WALES, AUSTRALIAN CAPITAL TERRITORY) Note: Due to Service Delivery’s Southern Region extending into parts of New South Wales, to differentiate between stories from Service Delivery’s Southern and Central Regions we have noted when a story is from Southern teams in this section only.
  33. 33. CONTENTS COMMUNITY SPIRIT... “Traffic was cut for about 20 hours and there was extensive damage done to some heritage buildings...” As told by Grant Carroll, Countless SES meetings were attended by the local Service Delivery [Southern] team manager David Gillett who supplied updates WALLS OF Date: October 2010 onwards and assistance to customers. After waters receded to a safe level David and two field technicians were During the summer the Riverina Snowy ferried into North Wagga sitting in the bucket of a WATER Field Service Area (FSA) experienced major severe weather events. On each occasion, large front-end loader to assess damage. amounts of rain fell over very short periods, creating The last of these severe weather events for the walls of water which rushed down mountains, rivers summer period took place in March, when the and creeks, causing extensive damage to properties, largest amount of rain fell; 300-400 millimetres was bridges and Telstra infrastructure. recorded over a period of 48 hours in the south coast area around the Bega Valley. Natural disaster zones were declared in numerous local shires within the area, making it one of the Due to the mountainous terrain along the coast, most challenging years on record. the sheer volume of rain at this time caused more damage to infrastructure in the region than any of The first of these events in October saw 100- the other events throughout the year. 200 millimetres of rain fall in a 24-hour period in the Wagga and Albury areas. The small towns of Due to unsafe driving conditions, our team was not Culcairn, Adelong, Holbrook, The Rock, Lockhart, allowed to drive their vehicles into the area to affect Rand, Urana and Jingellic had to be evacuated due repairs. to widespread flooding and inundation of homes, Our local team managers did however continue properties and roads. Tragically, during this event to work closely with the SES and due to Peter we saw the loss of life just outside the township of Chapple’s involvement in the fire brigade, and the Lockhart. relationships built, Peter was able to arrange for two The Wagga and surrounding areas were struck field techs to join him on flights into isolated areas by by torrential rain again in December, causing the helicopter and commence restoration of damaged Murrumbidgee River to peak at its highest level since fibre crossings. 1974. Towns along the river including Gundagai, Peter was also able to set up a satellite phone in the Narrandera and Wagga saw residents forced small township of Rocky Hall for our customers to to evacuate from their properties and homes use until all services could be restored. including the evacuation of 1,800 people from North Wagga alone.32
  34. 34. CONTENTSMy sincere thanks and appreciation go out to thewhole team who worked long hours over manyweeks to restore services. They all worked tirelesslyand considered the needs of the people in thesecommunities and the importance for them to be ableto communicate with family and friends. About the Adelong area Chris Hargreaves, Service Delivery [Southern] said, “Telstra’s network was impacted by a 100-year flood event but we all pulled together as a team and got the work done, supplying telecommunications to the local people as fast as we possibly could. Traffic was cut for about 20 hours and there was extensive damage done to some heritage buildings, but the Adelong Telephone Exchange remained just out of reach, with the water level coming within millimetres of the floor level.”Top Left: What was left of East Grahamstown Bridge, East Grahamstown Road, Adelong after flash flooding.Top Right: Local team manager David Gillett took this aerial shot while surveying flooded North Wagga area with the local SES crew. Bottom Right: Travelling to Culcairn. 33
  35. 35. CONTENTS RECOGNITION... “Many of these towns were isolated geographically as floodwaters covered roads.” By Loretta Willaton, Telstra Country Wide Date: October 2010 onwards RIVERINA- Late last year and again early this year, With an extra 22 people in his team, Service Delivery floodwaters savaged the Riverina Murray area field manager Grant Carroll and his team managers MURRAY washing away trees, which piled up against tirelessly led restoration efforts working long hours bridges. Many bridges were destroyed. As the and weekends to fix faults as quickly as possible. RECOGNITION bridges often carry our fibre links, many were broken and we lost service to several exchanges. Emergency satellite phones and temporary services were delivered to several customers who were likely The excessive flooding and resulting road closures to have longer outages due to major damage at prevented Telstra Operations service teams from creek crossings. being able to get straight in there to begin repairs. Many of these towns were isolated geographically During the first major impact in November 2010, we as floodwaters covered roads. experienced problems with a number of our small Ten extra services were provided to the SES in rural exchanges due to fibre cables being washed Tumut, a small town just outside of Wagga, to away or broken, including one of the major back-up ensure emergency workers had access to essential fibres servicing the district. communication services during the rescue and relief Whilst all fibre-based services and related exchange operations. outages were restored relatively quickly, there were Then again, on 11 February, Wodonga was inundated also huge numbers of individual service faults with floodwaters and fibre was damaged with about where lead-in or distribution cables were broken or 600 customers impacted. Once again Grant’s team inundated with water. Our local service team had a swung into action and had all customer services up huge task repairing these. and running within 10 hours. Grant’s team were fantastic and, all in all, the network held up remarkably well considering the breadth and severity of the flooding.34
  36. 36. CONTENTSWILDLIFE... WILDLIFE... WILDLIFE...By Con Parslow, Service DeliveryDate: 12/10/10During the summer’s inclement weather When he returned, the snake had returned too and THE RETURNeven the wildlife had to find alternative was sunning himself wrapped around the pillar andaccommodation. the technician’s Loop-a-Line. OF A SLIPPERYTwo communication technicians, Mark O’Callaghanand Robert Hancock, opened a pillar in Austral The Wildlife Information, Rescue and Education Service were contacted and they relocated the CUSTOMERto test a cable length. A snake was inside the lid snake.and dropped down into the pillar while they were *The Loop-a-Line is a product developed specifically for telecommunica-working. tions technicians involved in fault location activities. It is the orange tool with the blue and white sticker pictured here.Once the snake removed itself, they were able tocontinue their job, locking the pillar down when theywere finished.The very next day Gavin Kaimoana, anothercommunication technician, came along and openedthe very same pillar, attached his Loop-a-Line* andleft to complete the job.Far Left: Tech Russell Thompson crossing Tumut River, Darbalara.Above: A slippery customer returns. 35
  37. 37. CONTENTS TEAM WORK... TEAM WORK... “Technicians spent several months working very long days and giving their very best...” As told by Bill Lloyd, Service Delivery Muswellbrook Date: October 2010 onwards The Muswellbrook team experienced extensive LIGHTNING During the summer at a Service Delivery Central North Field Service Area (FSA) level we floods over a period of three months in the Mudgee area. Roads were washed away and flooding creeks STORMS, FLOODS had 56 team members move out of our FSA. The majority of moves being for more than a month impacted our ability to serve our customers. Wollemi National Park customers were isolated for AND ROAD each and some being two-plus months to help out in other areas. days at a time. Lake Burrendong Dam went from having just 12 per cent capacity for many years, to CLOSURES over 130 per cent capacity. There were many other moves within the FSA, as we moved communication technicians (CTs) to The team supported each other with the serve our customers while maintaining our FSA’s Muswellbrook part of the team travelling the 400 appointment and commitment performance. kilometres round trip, for one week at a time, over three months. The Construct & Maintenance Greater Foster West team also assisted by responding promptly to Over summer the Foster team’s main customer volume holds* and cable outages. impacting concerns were caused by lightning in the mountain areas. The coastal areas had minimal Dubbo weather impacts which allowed the team to support The Dubbo team experienced flooding right across other areas. its vast area, with the Macquarie and the Castlereagh Rivers reaching record levels. Coonamble and Over the summer period, the team had on average numerous communities were isolated. The Northern three CTs away working in other regions. There was Queensland rains then started to move down the country a period of three weeks where the team pulled hard causing flooding in the Darling River isolating Kilpa. together to look after our customers to allow five technicians to be away working in other regions. Meeting the challenges Cessnock Controlled road closures and dirt roads being impassable were a continual challenge as the This summer the Cessnock team area was country absorbed the welcome rain. impacted by many lightning storms. The committed CTs spent many hours clearing multiple faults on our Technicians spent several months working very customers’ lightning damaged services. While this long days and giving their very best under imposing was happening the team supplied two technicians conditions. to go to Queensland for several months. *A Volume Hold Queue is created by Service Delivery’s Service Outage Cessnock team members also spent time in adjoining Management team when a number of fix line services are impacted by one fault (say a cable break or a roadside cabinet is inundated with water teams including the Muswellbrook, Newcastle and and stops functioning). Volume holds are put in place while the fault is Dubbo teams. The region was also supported by being repaired. Creating a Volume Hold Queue provides Telstra teams with a single point of reference to obtain updates on outages affecting our Cessnock sending a technician to Central South FSA. customers in one particular area.36
  38. 38. CONTENTS “I am proud of my team. We did all this work and still managed to support our colleagues who were in a worse situation. The team has performed excellently and always put the customer at the front of their minds concerning all decisions and actions made.” – Niall Carey, Service Delivery Area Manager, Central North.Top Right: Clouds building for another lightning storm across Cessnock. Bottom Left: Coonamble fields flooded. Bottom Right: Highway through Dubbo. 37
  39. 39. CONTENTS COLLABORATION... “Many technicians travelled from around the country to help...” As told by David Gillett, Service Delivery Many technicians travelled from around the [Southern] country to help out as part of Service Delivery’s BACK Date: 3 December 2010 peak load strategy. Whether it’s fires, floods or cyclones anywhere in the country, we have During flooding in Wagga Wagga the basement TO car park of the local main shopping mall was the flexibility to get our skilled people interstate quick smart to assist in restoring services. filled with water, up to 1.5 metres in places. BUSINESS The communications room, which is located in the car park, was inundated. The batteries, four access panels, the backplane, socket and plug assembly, including the DC power access, were partially submerged causing the loss of communications (fixed phone lines, internet, data, alarms, fax machines and EFTPOS services). With co-operation between the local Network Construction and Service Delivery teams, equipment was sourced and the recovery process commenced. It certainly was an around-the-clock affair, with technicians Phil Briggs and James Tibbetts working through Friday night 3 December in the flooded car park to get the local shopping mall back online. At around the same time, after receiving advice from the SES that two RIM cabinets were in danger of inundation (one on Hammond Avenue and the other on Schiller Street, Wagga Wagga), members of the Wagga team got to work sandbagging the roadside cabinets and potentially saved around 5,000 customer services from being impacted.38
  40. 40. CONTENTSTEAM WORK... TEAM WORK...By Terry Walsh, Service Delivery [Southern] John positively identified the fault location as aDate: December 2010 washed-out creek crossing and our techniciansDuring December I received a call from Service were onsite by 4:30pm. WASHED OUTDelivery Enhanced Services technician John The site was a disaster area, too wet and boggyCollins, informing us of a possible fibre outage to get any vehicle on site, including the Coolac and Jugiong affecting more than 170 The bridge was mostly washed away as well andPSTN (or fixed line) customers. this meant the gear had to be carried, by the field crew involved, across a plank on what was left ofWhile John continued his investigations, the bridge.communication technicians Geoff Sonneman andLawrie O’Callaghan organised an excavator and With all the challenges, and to the team’s credit, thetravelled from Wagga to Jugiong. fibre was up and running by 8:30pm.Colleague Craig Price hooked up the fibre trailer in Early communications between Service Delivery’sAlbury and also headed to Jugiong. Enhanced Services and field teams ensured a quick restoration time for our customers.Opposite: (l-r) Russell Thompson and Geoffrey Sonneman protecting our assets. Above Left: Bridge at Jugiong. Above Right: Washed-out optical fibre cable in Jugiong 39
  41. 41. CONTENTS CUSTOMER DRIVEN... “Our team spoke to people who lost a great deal and they handled themselves admirably in the face of adversity...” As told by David Coxhell and Natalie O’Malley, Service Delivery CALL Date: December 2010 onwards Call Central Newcastle, though based in While we did our best to relay potentially displeasing CENTRAL Newcastle, operates nationally and comprises several teams including Rescheduling, news to our customers, it’s understandable many calls had to be escalated due to customer WEATHERS Call Central Front of House, Service Delivery Customer Care, Recalls, Fee For Service and dissatisfaction. Our consultants are well versed in dealing with unhappy customers and did what they THE STORM Vetting and Validations, Outage Management and Interim Provision & Recovery. could to assist them within set guidelines (diversions, Personal Interim Phone Services (or PIPS), trouble- shooting, etc). Rescheduling Each day we managed escalations and liaised with With some of the fiercest weather conditions seen in the On-The-Day Schedulers in Service Delivery to get several years hitting multiple areas of Australia this the best possible results for our valued customers. summer, it was not surprising workloads skyrocketed This has been one of the busiest periods the team throughout the various areas of our business. has managed in over 10 years. A case in point was the rescheduling work done by Our team spoke to people who lost a great deal the dedicated Call Central staff in Newcastle. and they handled themselves admirably in the face Calls to customers to reschedule work the field have of adversity and always treated our customers with unfortunately had to delay, due mainly to unavoidable dignity, respect and understanding. weather impacts, rose significantly during December With many of our people cross-trained across 2010 to February 2011 (a total of more than 70,000 various functions, we pulled together to perform as reschedules actioned during this time). a true team should, putting our best foot forward to help provide the best possible levels of service The increase in workload meant Call Central had which our customers expect. to train several new classes of consultants in quick time given the need to keep our customers informed The Service Outage Management of the status of their services. With the fault volumes pouring in, and field crews The training was very hands on and very successful. dependent on being provided accurate and timely Call Central Front of House staff, many of whom information on where and how the network had been were newly trained themselves, had also been damaged, a huge effort was required from everyone affected by the increased workload brought on by involved in Service Outage Management. The team the unpredictable weather patterns. really stepped up and the effort put in by this team, to support the field crews and our customers, was truly impressive.40
  42. 42. CONTENTSInterim Provisioning and Recovery (IPR)With significant network damage across Queensland,it became increasingly vital for Telstra to be ableto provide our customers with temporary InterimTelephone Services.However, in a particularly cruel blow, the floods thathad done such damage to the network infrastructurealso inundated the main depot that holds our interimphones.So with demand for interims through the roof and alarge number of our units also unavailable, the teamat IPR really had to dig deep. And dig deep they did,working massive hours trying to get interim phonesback from where they were no longer needed, sothey could quickly be deployed to customers whoreally required them in the flood-ravaged areas.During January and February, the peak period ofthe emergency, they recovered and redeployedthousands of interims.The majority of IPR staff are cross-trained andhelped other workstreams where they could. Theentire IPR team’s attitude and work rate over theperiod of the emergency is a tribute to their passionand commitment for their customers.Top: Some of the members of our Interim Provision & Recovery team (front) Truli Balloch (l-r behind the partition) Clinton Yates and Kathy Cox.Bottom: (l-r) Some members of the Service Outage Management team (l-r) Brenton Crabb, Phil Ross and Chris Box. 41
  43. 43. CONTENTS DEDICATION... DEDICATION... “Reid’s Flat community was isolated by floodwaters for days with the optical fibre cable that feeds the telephone exchange being washed away...” As told by Terry Lines, Service Delivery Date: 16/12/10 LATE NIGHT This is a story about technician Richard Leary He drove around the back blocks of Reid’s Flat and shows the extraordinary lengths our people locating the customers and installing the satellite DELIVERIES go to, to serve our customers. phones. He found one customer had already been evacuated. Reid’s Flat community was isolated by floodwaters for days with the optical fibre cable that feeds the Even though in the end just two installs were telephone exchange being washed away at a river required, it was a very long haul with Richard arriving crossing. back home at 2:10am the next morning. On Thursday 16 December, Richard received a call from Dave Minney, Area Manager Central South, asking if he had any satellite phones available in Cowra. There were three medical customers in Reid’s Flat requiring satellite phones, but Richard only had one. However, he did some follow-up and soon located more satellite phones in Orange and arranged to meet a communication technician between Cowra and Orange to collect them. Richard then drove on to Reid’s Flat. Luckily, by late- afternoon that day, the floodwaters had receded enough for Richard to be able to drive through and he arrived just on dark.42