Loading…

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

Like this presentation? Why not share!

To Hell and Back in Five Days

on

  • 554 views

Between February 7 and March 14, 2009, more than 400 bush fires across the state of Victoria, Australia, scorched over a million acres of land, killing 173 and injuring 414. Engineers at Goulburn ...

Between February 7 and March 14, 2009, more than 400 bush fires across the state of Victoria, Australia, scorched over a million acres of land, killing 173 and injuring 414. Engineers at Goulburn Valley Water (GVW), provider of urban water and wastewater services to 54 towns and cities on the outskirts of Melbourne, watched as their telemetry system from the Kilmore Dissolved Air Filtration plant reported an ambient control room temperature of 142 °F before going silent on the afternoon of Saturday, February 7. A site visit on the following day revealed that while the treatment plant survived the fire, its control room was completely incinerated, destroying the electrical switchgear, plant HMI, laboratory, instrumentation and chemical dosing systems. With only five days worth of water stored, an emergency response plan to rebuild the control room and recommission the plant went into action.

Statistics

Views

Total Views
554
Views on SlideShare
541
Embed Views
13

Actions

Likes
0
Downloads
7
Comments
0

2 Embeds 13

http://www.jondipietro.com 12
http://leftbrainhandbook.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Role is storyteller.Many emergency response stories cautionary tales with bad endings.Aside from the tragic fires themselves, this is a success story.
  • Between February 7th and March 14th, 2009, more than 400 bush fires across the state of Victoria, Australia scorched over a million acres of land, killing 173 and injuring 414.
  • Goulburn Valley Water (GVW), provider of urban water and wastewater services to 54 towns and cities on the outskirts of Melbourne, Kilmore dissolved air filtration plant reported ambient control room temp 142 then went silent on Sat, Feb 7
  • A site visit on the following day revealed that while the treatment plant survived the fire, its control room was completely incinerated, destroying the electrical switchgear, plant HMI, laboratory, instrumentation, and chemical dosing systems. What followed over the course of the next five days is a case study in how proper backup and change management procedures, strong vendor relationships, and dedicated, cross-trained employees can pull off near miracles. Armed with a full set of accurate as-built drawings, up to date PLC programs, and HMI computer backups, GVW was able to assemble a portable control room in the parking lot of their operations center, deliver it to the site, connect thousands of control points, commission the plant, and resume delivery of 10ML per day of water to the towns of Kilmore, Wandong, and Heathcote Junction in five days. This presentation will discuss the challenges faced by management, employees, and vendors, while outlining the policies and procedures that allowed them to execute their emergency response plans so successfully during this catastrophe.
  • In the months and years leading up to the fire in Kilmore, GVW had built a foundation that would give it this chance at success. There were three key areas of preparation – documentation, human resources, and relationships – and lacking any one of them would have spelled failure.
  • Documentation is obvious and crucial requirements.GVW was very well preparedStrict change managementAccurate, recent (enough) as-built drawingsBetter to be lucky than good:Not all plants had accurate drawingsNo system for instrument settings, including flow meter but salvaged
  • Documents, plans & procedures only go so far in emergencyGVW had good internal team w/ wide range of skills (electrician, instrumentation tech)IT department embedded personnel in operations groupDiverse team with balance of skills and experienceAs GVW’s IT Manager Noel Squires describes, “A number of years back, we had a couple of near-misses with losing ladder logic, which was at a time when looking after PLCs and process controls was really considered operations and had nothing to do with IT. But we pretty quickly realized that it’s got a lot to do with IT and that disciplines we use in IT routinely like change management and backups applied equally to process control. So we formed the Operations IT group, and that was a little different in that we were taking these formerly operations people and putting them into the IT section.”
  • Teamwork = cliché (for a reason)Three relationships had to work for successful response:Horizontal (between disciplines – electrical, SCADA, PLC, construction, IT)Vertical (between management layers – response team and management)Management team fetched coffee, arranged meals, filled out paperworkExternal (vendors – occurred on weekend but vendors abandoned family functions to open warehouses and fetch parts w/o PO’sSolid relationships
  • Apollo 13 CO2 scrubbers – “We need to find a way to make this fit into that using only these…”Realized they had approx 5 days of water stored3 Operations IT people bunkered down in conference room on Sunday evening to plan
  • Assessment difficult due to crime sceneBureaucracy – county police had different radio freq than local police. Took several hours to gain access.
  • Using laptop, data projector, white board, OP IT team limited scope – treat water within time frame. “Nice to have” was put on the bus. Worked next five hours into Monday AM devising plan.Employee idea = build control room in shipping containerOther employee had family connection w/ insulated trailer close byCrucial = 1) save construction time 2) built near operations center instead of remote siteDrafted conceptual design & sketched layoutDevised shopping list (cabinets, cables, motor starters, PLC, computer, lumber, generator)Emailed spreadsheet to vendors in middle of nightDevised a timeline for construction, installation & commissioning
  • Core team focused on construction.Support teams focused on procurement, vendors picked up components.Management focused on fatigue – arranged campers, enforced sleep schedulesDuring construction, coordination was key due to restricted space (only 5 at a time)Demolition of existing site occurred parallel with construction.Field wiring cut and carefully labeled.IT establishing wireless connectivity.Corporate provided steady stream of food & refreshments.Boosted morale by organizing massive barbeque afternoon before shipping container.
  • Planning and construction complete, time for installation and commissioning.
  • Construction went more or less according to plan.Schedule was watched carefully & resources were added to bottlenecks.Container in place at 5PM on Wed.
  • Setbacks during installation included wiring challenges, incorrect parts, blocked generator fuel line.Overcame with brute force (manpower) or ingenuity
  • Power was restored in 3 days instead of 3 weeks.Instead of generator sequencing, installed a switchboard.Initial estimates were 5 days of water based on historical consumption.Turns out towns were consuming at much higher rates.Had to quickly arrange water tankers.
  • Commissioning began Thu afternoon.Two way radios were crucial – checking motor rotation & valve actuation.Friday morning = troubleshooting wiring issues & mechanical problems with pumps.Friday afternoon = water quality instruments interfaced to PLC & plant creating good quality water.Friday night = storage tank began filling9PM Fri plant started up & team supervised in early hours of Saturday.Final commissioning occurred throughout Saturday.Dealt with instrument, motor, valve & dosing issues.
  • When interviewing GVW IT Manager Noel Squires about the ordeal, he mentioned that, “Strangely enough, when we looked back, we actually rebuilt that control room for about $440,000. Now if you did that as a contracted job, it’s probably going to be like $1.5M to $2M. We didn’t need multiple quotes, didn’t need all the bureaucracy, and we just cut the chase and did the job.” So the obvious lesson learned is that the next time you need to rebuild a control room, instead of bidding it out, just set it on fire! But seriously…With a combination of solid emergency preparedness, cross-trained and motivated personnel, strong vendor relationships, solid planning, and – yes – a little bit of luck, GVW built a brand new water treatment plant control room in a box and had the plant running in five days. The experience of the individuals involved even allowed for the implementation of some on-the-fly improvements to the plant, such that it was actually running better after the ordeal than before.

To Hell and Back in Five Days To Hell and Back in Five Days Presentation Transcript

  • To Hell and Back in Five Days:
    Lessons in disaster recoveryand emergency response.
    Jon DiPietro
    Principal, Bridge-Soft
    Jon.dipietro@bridge-soft.com
  • 400 fires
    1,000,000+ acres
    173 dead
    414 injured
    “Burning Trees” courtesy of Sascha Grant via Creative Commons on Flickr
  • Preparation
    “If you want to make an apple pie from scratch,you must first create the universe.”- Carl Sagan
  • Documentation
  • Human Resources
  • Relationships
  • Planning
    “Strategy without tactics is the slowest route to victory. Tactics without strategy is the noise before defeat.”- Sun Tzu
  • Assessment
  • Strategy
  • Tactics
  • Execution
    “No battle plan ever survives contact with the enemy.”- Field Marshall Helmuth Carl
  • Assembly
  • Installation
  • Improvisation
  • Commissioning
  • Conclusion
    Emergency preparedness
    Cross-trained & motivated team
    Strong vendor relationships
    Solid planning
    LUCK
  • Thank You
    Jon DiPietro
    Principal, Bridge-Soft
    Jon.dipietro@bridge-soft.com