Thinking the unthinkable± some lessons from september 11 by Paul Bodart
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Thinking the unthinkable± some lessons from september 11 by Paul Bodart

on

  • 807 views

Seminar by Paul Bodart during Infosecurity.be 2011

Seminar by Paul Bodart during Infosecurity.be 2011

Statistics

Views

Total Views
807
Views on SlideShare
779
Embed Views
28

Actions

Likes
0
Downloads
22
Comments
0

1 Embed 28

http://www.infosecurity.be 28

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Thinking the unthinkable± some lessons from september 11 by Paul Bodart Presentation Transcript

  • 1. March 22, 2011The Bank of New York Mellon and the Impact ofSeptember 11, 2001 Presented by Paul BodartInformation Security Identification: Confidential
  • 2. Agenda I. Impact of WTC Attacks on New York City II. Implementing Disaster Recovery III. Steps taken IV. Where are we now V. Lessons learned2 Information Security Identification: Confidential
  • 3. Section I. Impact of WTC Attacks on New York City Information Security Identification: Confidential
  • 4. Impact of WTC Attacks on New York City Estimated loss of life = over 5,000 people Estimated total property losses = $60 billion, 15 million square feet More than 50,000 employees displaced Implosion of the towers knocked out more than 400 megawatts of electricity to lower Manhattan (enough to power 400,000 homes) Total amount of time needed for clean-up = approx. 1 year Estimated cost of clean-up = $40-$60 billion Estimated cost to replace the technology infrastructure, which was obliterated inside the towers and adjacent buildings = approx. $3.2 billion NYSE closed for almost 4 consecutive days, the longest period since the stock market crash of 1929 Sources: The Daily News and The New York Times4 Information Security Identification: Confidential
  • 5. All of Lower Manhattan Affected5 Information Security Identification: Confidential
  • 6. Aerial view of lower Manhattan after attacks6 Information Security Identification: Confidential
  • 7. Business Primary Locations in Lower Manhattan for The Bank of New York  One Wall Street Corporate Headquarters, Custody Staff, Broker Dealer Services staff, Relationship Management  32 Old Slip Global Foreign Exchange, Securities Lending and Treasury Operations  101 Barclay Street Stock Transfer, Funds Transfer systems, Corporate Trust, Broker/Dealer Services systems, Unit Investment Trusts and Depositary Receipts  (One of BNYM’s two Primary Data Centres, along with Teaneck, NJ)  100 Church Street Mutual Fund Custody Administration and Fund Accounting7 Information Security Identification: Confidential
  • 8. Prior to September 11th, BNY’s US-based processing and recoverycenters were concentrated in the NY/NJ area  Two primary production facilities  101 Barclay  Teaneck  Three recovery centers  Maywood, One Wall Street and Comdisco  Verizon connected all centers through a campus-type network  Customers and external agencies connected through 101B, TNK and OWS One Wall Street Recovery Center 101 Barclay Primary Production Maywood, NJ Recovery Center Teaneck Primary Production Comdisco/N. Bergen, NJ Recovery Center8 Information Security Identification: Confidential
  • 9. Section I. Implementing Disaster Recovery Information Security Identification: Confidential
  • 10. Implementing Disaster Recovery What Happened to BNY on September 11? Evacuation of 8,300 BNY staff, including: 1 Wall Street -- Broker Dealer Services Operations Center 101 Barclay -- Broker Dealer Services Data Center Corporate Headquarters moved from One Wall Street to 1290 Sixth Avenue Contingency plans immediately implemented 101 Barclay Street and 100 Church Street most impacted due to proximity to World Trade Centre10 Information Security Identification: Confidential
  • 11. Primary and Backup Business Locations (Staff) Primary Back-up Business Site Business Site Broker/Dealer Services One Wall Street Maywood, NJ Custody One Wall Street Pleasantville, NY Mutual Fund Custody & Acctg 100 Church Street Pleasantville, NY Funds Transfer Utica Utica (no relocation necessary) Depositary Receipts 101 Barclay Street West Paterson, NJ Corporate Trust 101 Barclay Street West Paterson, NJ Stock Transfer 101 Barclay Street West Paterson, NJ Securities Lending 32 Old Slip Greenwich, CT Trading 32 Old Slip Greenwich, CT Treasury 32 Old Slip Greenwich, CT Foreign Exchange 32 Old Slip Greenwich, CT Additional warm sites for operations/business staff were at West Paterson, NJ; Teaneck, NJ; Lodi, NJ and Secaucus, NJ.11 Information Security Identification: Confidential
  • 12. After September 11th, and the collapse of WTC 1, 2 and 7,101 Barclay was unusable  Processing from Barclay was distributed to recovery centers and Teaneck  Maywood, One Wall Street, and Comdisco/SunGard (vendor)  Rolling telecommunications outages delayed recovery  Personnel had difficulty reaching recovery centers  BNY was at risk until new recovery centers were established One Wall Street X Recovery Center 101 Barclay Primary Production Maywood, NJ Recovery Center Teaneck Primary Production Comdisco/N. Bergen, NJ Recovery Center12 Information Security Identification: Confidential
  • 13. Telecommunications Infrastructure Damage Loss of switching stations crippled service to downtown Manhattan. As a result:  Client connectivity with several critical Bank’s systems like Government Securities Clearing System (GSCS) and collateral management system (RepoEdge) interrupted  Communications lines between the Bank and key providers (like Federal Reserve or DTC) interrupted  Connectivity among different Bank’s systems (located in different locations) interrupted  Voice communications in Metropolitan area interrupted13 Information Security Identification: Confidential
  • 14. An Example The Bank of New York US Government Clearing Services Situation before September 11. Client/Broker Dealers The Bank of New York Providers US Government Securities Clearing Federal Reserve Morgan Stanley Downtown Manhattan Bank of New York Operations Systems (“The Fed”) Downtown Downtown Goldman Sachs Manhattan Downtown Manhattan Manhattan Downtown Manhattan US Money Transfer Depositary Trust Lehman Brothers and Downtown Manhattan Clearance Cy Operations Systems (“DTCC”) Utica (NY) Downtown Manhattan New Jersey14 Information Security Identification: Confidential
  • 15. An Example On or after September 11, a majority of the players located in Down town Manhattan migrated to back-up Centers. Operational Back-up are not always in the same building as System back-up Client/Broker Dealers The Bank of New York Providers US Government Securities Clearing Federal Reserve Morgan Stanley Bank of New York Downtown Manhattan Operations Systems (“The Fed”) Downtown Downtown Downtown Goldman Sachs Manhattan Manhattan Manhattan Downtown Manhattan Depositary Trust US Money Transfer and Lehman Brothers Clearance Cy Downtown Manhattan Operations Systems (“DTCC”) Utica (NY) Downtown New Jersey Manhattan Have migrated to a Back-up Center15 Information Security Identification: Confidential
  • 16. An Example On the telecommunications front, two things happened : 1. Loss of switching stations crippled services to/from Downtown Manhattan 2. Communications between Back-up Centers was not sufficiently robust or tested. Client/Broker Dealers The Bank of New York Providers US Government Securities Clearing Federal Reserve Morgan Stanley Bank of New York Downtown Manhattan Operations Systems (“The Fed”) Downtown Downtown Downtown Goldman Sachs Manhattan Manhattan Manhattan Downtown Manhattan Depositary Trust US Money Transfer and Lehman Brothers Clearance Cy Downtown Manhattan Operations Systems (“DTCC”) Utica (NY) Downtown New Jersey Manhattan Have migrated to a Back-up Center Communication Lines impacted16 Information Security Identification: Confidential
  • 17. Section III. Steps taken Information Security Identification: Confidential
  • 18. Steps Taken 1. Operations Centers Relocation from One Wall Street  Operations personnel began establishing operations in Back-up Centers, NJ 9-11  Immediately began client contact 9-11  Facility expansion initiated 9-12 - 150 workstations operational by 9-13 2. Computer Centers Relocation  Personnel relocated 9-11  Systems successfully switched to back-up for processing  All data integrity maintained - 100% capture of all transactions 3. Restoration of Communications Lines Between Banks and Providers  Lines monitored on continuous basis  Replaced ISDN with dedicated dual high speed lines18 Information Security Identification: Confidential
  • 19. Steps Taken 4. Restoration of Client Connectivity  Switched to client back-up communications where available  Worked with clients to establish back-up communications where necessary  Implemented manual solutions for clients where no communications back-up was available  Clients relocated to BNY back-up site  Established special disaster recovery e-mail location for bulk file to be received for transaction processing  Established e-mail delivery process for extracts and client reports  E-mail fax service established 5. Re-establishing Voice Communications with Clients/Business Partners19 Information Security Identification: Confidential
  • 20. Section IV. Where are we today Information Security Identification: Confidential
  • 21. Where are we today  Two of the four buildings in downtown Manhattan were re-occupied within two weeks of September 11.  The two other buildings that were closed to WTC have started to be reoccupied and will be fully occupied during the summer.  Interim measures:  Leased 100.000 square meters in 9 new Manhattan locations;  most staff were moved to these 9 buildings by the end of Q3 2001  Data Centres  Established a new primary data center in Sterling Forest  moved out of contingency sites by November 30.  Established geographical diversity :  400 staff permanently relocated to New Jersey  accelerating occupancy of existing “growth center” in Orkado and Syracuse The organization has been battle-tested and emerged stronger21 Information Security Identification: Confidential
  • 22. Section V. Lessons learned Information Security Identification: Confidential
  • 23. Lessons learned  Be flexible. The disaster planned for is not the disaster that occurs.  Planning is critical. Have a strong plan, and practice it.  Telecommunications is a resource which is not entirely under your control. Route diversity planning via multiple central offices may be undone by the plans or actions of your service provider. Know the complete end-to-end routes of your mission critical circuits.  At the precise time that accurate and timely communication with clients is most essential, it becomes the most difficult.  Geographic diversity of operating units enhances recovery. The most successful relocations occurred when there was a geographically separate group with the same functions as the group being moved, permitting uninterrupted processing.23 Information Security Identification: Confidential
  • 24. Lessons learned  Systems integration is a changing landscape. Every critical system has supporting applications without which the base system cannot function. The support systems change over time, and periodic reevaluation is important.  Right size back up capabilities. Recovery may involve processing volumes that exceed normal. When in doubt, err on the side of too much backup capacity.  Processing the day’s work is the key objective, but do not overlook accounting, financial, and proof and control systems. Getting the work accomplished is important, but is only part of the job – you still have to be able to reconcile internally and with external systems and vendors.24 Information Security Identification: Confidential
  • 25. Lessons learned  Individual client CPU links become a liability in a disaster recovery scenario, as each must be individually recovered. In a major disaster, clients may also be operating out of contingency sites, which further complicates recovery. It is far easier to restore a single link to an industry utility such as SWIFT than to 50 or 100 clients.  Going forward, we all need to create a state-of-the-art data centre, telecommunications, and business environment with full redundancy, real- time backup, and adequate dispersal of staff and systems to ensure continued operation through any disaster event, regardless of severity or breadth.25 Information Security Identification: Confidential
  • 26. Site Selection Considerations  Geographic diversity. Adequate separation to insure survivability, close enough to facilitate communications.  Exposure to terrorist attack.  Exposure to natural disasters - flood, hurricane, tornado, volcanic activity.26 Information Security Identification: Confidential
  • 27. Cost  The price of safety and assurance of business continuation is going to rise. Fully redundant data centres with multiple dedicated links from multiple processing centres to multiple client sites and utilities will add significant cost.  These cost will have to be recovered by all of us – which means higher prices in all industries, especially those heavily dependent upon data processing, such as the financial services industry.27 Information Security Identification: Confidential
  • 28. For more information  Paul BODART : Executive Vice President Head of EMEA Global Operations for Asset Servicing Phone : (32-2) 545.82.37 E-mail : paul.bodart@bnymellon.com28 Information Security Identification: Confidential
  • 29. Information Security Identification: Confidential