INOC - Art Meierdirk Presentation from UTC April 15, 2014

335 views

Published on

Art Meierdirk of INOC presents at the UTC conference.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
335
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
8
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

INOC - Art Meierdirk Presentation from UTC April 15, 2014

  1. 1. © 2013 Utilities Telecom Council Transforming Critical Infrastructure The Cost of Modernization: Perspectives on Effectively Managing Today's Advanced Technology Assets without Breaking the Bank •  Art Meierdirk, Senior Director, Business Services – INOC •  Dan Belmont, Director, Energy & Utilities - West Monroe Partners •  Gary Sanders, Telecom Manager – Sun Prairie Utilities
  2. 2. Utility & Advanced Technology Overview Dan Belmont, Director, Energy & Utilities
  3. 3. Utility & Advanced Technology Overview 3 •  Utilities – Drivers for Advanced Technologies •  The Telecom “Tipping Point” •  Advanced Technologies – Applications vs. Telecom Capabilities •  Creating a Roadmap •  Cyber Security and Telecom Synergy •  Lesson’s Learned and Path to Success
  4. 4. Utilities – Drivers for Advanced Technologies 4 •  Mission? •  Over the next 10-15 years, we will transform our delivery infrastructure into a model we call the “Utility of the Future” which will: –  Empower customers to control electricity usage and costs –  Improve service, enhance reliability and cost controls –  Improve the customer experience –  Increase efficiency of our operations –  Enable Distributed Generation –  Networks and Microgrids
  5. 5. Utilities – Drivers for Advanced Technologies 5 GRID Modernization End Use Programs Enabling Infrastructure Investments Information Technology Core Telecommunications Service and Data Center, Substation, and SCADA Communications Transmission, Distribution, and Substation Automation (DA/SA) Advanced Metering Infrastructure (AMI) Mission Critical Voice and Data Systems Backhaul
  6. 6. The Telecom Tipping Point 6 AT&T, Verizon, and LECsannounce discontinued analogcircuit support in 2013 andservice in 2018 NERC CIP version 5 removes the non- routable protocol exclusion for critical assets NISTiR 7628 Smart Grid cyber security standards must address attacks from disgruntled employees, industrial espionage, and terrorists …. 95% of Utility companies aregoing to experience coppertheft this year. Cooper theft caused 456,000outage minutes, 52 injuriesand 35 deaths in one year. Utilities with AMI have more managed devices per square mile than LECs, Cable, or Cellular Carriers
  7. 7. Applications vs. Telecom Networks Capabilities 7 Smart Metering & Home Area Network Distribution Automation Substation Protection and Automation Network Backhaul Enterprise Communications Mobile Voice/ Data Communication Applications New challenges for processes and people
  8. 8. Applications vs. Telecom Networks Capabilities 8 Distribution Automation Substation Protection and Automation §  Coverage of >99% of Meters (preferably Electric, Gas and Water) §  Higher Latency, Excellent Coverage §  Typically Need Mesh Capability between Meters and HAN Devices Network Backhaul Smart Metering & Home Area Network Enterprise Communications Mid-Tier Backhaul Mobile Voice/ Data Telecom Backbone AMI and HAN Communications Mobile Voice/ Data Applications Communication Systems Communication Systems Characteristics 8 §  Mission Critical Voice/Data §  Wide Band Mobile Data §  Coverage across Service Territory §  Power Generation Back-up §  Low Latency Broadband §  Good Uplink Capacity for Video §  Coverage across Service Territory §  Power Generation Back-up §  Mission Critical Voice/Data §  Wide Band Mobile Data §  Relay Teleprotection Services §  Coverage across Service Territory §  Power Generation Back-up
  9. 9. Applications vs. Telecom Networks Capabilities 9 Tier  One Core  MPLS Tier  Three   Access  Collectors Tier  Two Backhaul Explore combinations of Tier 2 solutions with Tier 3 networks to minimize net cost while maintaining Cyber Security and IP design standards
  10. 10. Creating a Telecom Roadmap 10 Microwave/Fiber  Backhaul     • Backbone  IP   Communica0ons   • Connec0vity  to  AMI  comm.   • Connec0vity  to  substa0ons   • Connec0vity  to  2-­‐Way  Radio   IT  “Hand-­‐Off”   ENTERPRISE  BUS   • Customer  Billing/CRM     • Load  Control  Management  System   • Meter  Data  Management  System   • Outage  Management  System   • Distribu0on  Management  System   • Energy  Management  System   Service Center AMI     •   AMI  Communica0ons   •   Smart  Meters   •   Distribu0on  Automa0on   • Power  Quality  Meters     • Vehicular  Radios     • Portable  Radios   2-­‐Way  Voice  &  Mobile  Data   •   Microwave  or  Fiber  Based     •   SCADA  to  Substa0ons   •   Communica0ons  to  AMI     •   Relay  Teleprotec0on  services     •   Distribu0on  Automa0on   SubstaLon  &  DistribuLon     CommunicaLons   Home  Area  Network  (HAN)     •   Home  Energy  Display   •   Programmable  Thermostats   •   Load  Control  Relays   •   Renewable  Energy  Genera0on   •   PHEV/EV  Smart  Charging  
  11. 11. Cyber Security and Telecommunications Synergy 11
  12. 12. Cyber Security and Telecommunications Synergy 12  MPLS service with four forwarding classes §  Reserve 2 Mb/s for voice and 4 Mb/s for video §  Enforce 14 Mb/s PIR for overall service §  Allow critical and best-effort traffic to burst up to 14 Mb/s if bandwidth is available  Less overall bandwidth required §  lower overall cost  Priority and best-effort traffic are equally well served §  Voice and video do not always consume all the reserved bandwidth CIR=0 PIR=4 Mb/s PIR=max PIR=max PIR=2 Mb/s CIR=2 Mb/s CIR=4 Mb/s CIR=4 Mb/s Overall SLA=14 Mb/s Voice Video SCADA FE Normal Data
  13. 13. Change Management and Training 13 = Aware = Knowledgeable = Expert Telecom/Cyber Security will Require Training across the Utility
  14. 14. •  Create a holistic business case and roadmap •  Design cyber security in from the ground up •  Tight Quality Control, Quality Assurance, and Audit •  Carefully Orchestrated Cutover/Migration •  Business Process Design •  Organizational Change Management Lesson’s Learned and Path to Success REB REBMEB Ping  MEB  to  verify   power  (From  ALU   gear) Verify  in  GS  which   MEB(s)/REB(s)  are   communicating  with Ping  7705  to  verify  power    (Can  be  done  within  SW  or   with  a  device  on  the  same  subnet/network).    Also   check  SW  for  downed  equipment  alert Ping  6855  from  7705  to  verify  power.  (Can  be  done   within  SW  or  a  device  on  the  same  subnet/network).       Also  check  SW  for  downed  equipment  alert Ping  7705  from  6855  to   test  connectivity  and   power   Check  SAM  to   ensure  MEB  port  is   up  and  active  on   either  6855  or  7705 Check  success  rates   in  G.S  than  PI  to   identify  other   possible  issues Still  up? No Refer  to  site  master  to  verify  what   equipment  is  on  site If  a  7705  is  o n-­‐s ite If  6 855  is  o n-­‐s ite ME B Fail  ping Then Pass  ping Pass  Ping If  all  else  fails: On-­‐site  visit  by  qualified  and  identified  group  through  process  of   elimination.    Worst  case  scenario.    (SSN,  BIG,  ALU,  Standby,  MJ,  CED) Pass  ping Pass  ping Alert/Remedy  Ticket   from  SSN  that  MEB  is   down Find  out  alerting  tool  for  AC  power  loss,   DC  backup  activated,  Backhaul  issues.     That  way  if  other  steps  “fail”  we  can   start  a  new  process  flow Pass  Ping Fail  Ping Fail  Ping Fail  Ping Yes
  15. 15. Gary Sanders Telecom Manager The Sun Prairie Utilities Experience
  16. 16. Sun Prairie Utilities 16 •  Who and what we are –  Municipality –  100+ years old –  Population of 30,000 –  Electric / Water / Internet / WAN
  17. 17. Municipal Entry Into Telecom 17 •  Approached by Sun Prairie Area School District in 1998 •  They had been quoted a price from the local ILEC for $1.2 million to furnish WAN services to School District •  Cable company was not interested in furnishing services at the time •  City of Sun Prairie got involved •  Formed study committee between the three entities
  18. 18. •  They used a map of the city to lay out 10 school sites, 5 city sites and 12 Sun Prairie Utilities sites. •  The result was a natural overhead loop on our electric distribution poles. •  At that point, we did an engineering study which was funded by the three Committee members for $5,000 each. •  Also, at this point our Utility Commission became involved and was very supportive. •  Obtained CLEC license •  Saving to the SPASD & City exceeds $1M The Committee Met… 18
  19. 19. Launching Fiber WAN 19 •  Simple P2P roll out for School District , City & Utility sites for SCADA •  Began offering business solutions •  Evolving expectations / scope •  Transition from ATM to Ethernet •  Engineering, design & support needs •  Partner with NOC provider –  24x7x365 Support –  Reduced staff & tools (monitoring / ticketing) costs
  20. 20. •  At the time customers were not happy with services & availability offered by Incumbent providers •  Temporary solution – FTTH technology had not matured enough at the time. We wanted to offer the community something until we found a FTTH product that we were comfortable with •  Wireless solution would ultimately complement FTTH technology Why Wireless? 20
  21. 21. Muni Wireless Hype Cycle 21
  22. 22. Enhanced Fiber WAN 22 •  Growing services and support •  Required outside resource / expertise •  Higher level of expectations from subscribers Partner with NOC provider –  Network engineering, design & deployment –  24x7 Support –  Reduced staff & tools (monitoring / ticketing) costs •  Multi-node 10Gb ring architecture •  Serving MDU’s as way to expand infrastructure and grow network cost effectively
  23. 23. Lessons Learned 23 Fixed Wireless •  Line of sight / unlicensed spectrum interference •  Technology / hardware life cycles •  Complimentary to fiber infrastructure – Not end-all solution Fiber •  High demand •  Symmetrical bandwidth •  Reliability •  Scalability
  24. 24. •  GPON •  High demand •  Economic Development •  Smart Grid •  Quality of Life •  Customer Service •  Public Private Partnerships •  Political challenges at local & state levels 2014 FTTX Initiative 2.0 24
  25. 25. The Network Operations Center On-going Support Art Meierdirk Senior Director, Business Services
  26. 26. On-going Support – Service Level Agreement (SLA) Based 26 •  Define the “product” •  Facilities / power / environmental •  Transport / switching •  Connectivity •  Set Service Level Requirements •  Metrics (alarm / call handling / accuracy) •  Service availability (uptime) •  Time to entry •  Restoration response time (time to action) •  Quality Assurance metrics & reports Value Proposition Elements Value Converted to SLAs
  27. 27. Meeting Business / Communication Expectations 27 •  Build a solution for 24x7 support to meet SLA requirements: •  Action on Monitored Infrastructure Alarms •  Answer Incoming Calls / Messages •  Notifications of Clients / Customers / Users •  Escalation and/or Dispatch for Restoration •  Time to Restoration •  Documentation of all Actions for Reporting SLA Performance Tools Process People Integration
  28. 28. Solution Elements 28 •  Tools – Monitoring, Workflow, Documentation & Reporting •  Process – Best-in-class (ITIL) / Focus on SLA •  People – Skill Set to Match Functional Requirements •  Professional (Consultation) Services – Integration of Tools and Process-based on SLA Requirements ”Structured NOC”
  29. 29. Tools 29 •  Monitoring, Workflow Documentation & Reporting •  Device / EMS / NMS / Dry Contacts – Alarm Aggregation from all Infrastructure Systems •  Workflow Management / Ticketing System with Time- stamp as Work Progresses or Escalation of Work is Delayed •  Portal / Web Access for Customer Access to their Information
  30. 30. Comprehensive (Single) View 30 NOC Monitoring System (Alarm Aggregation) Secure Connection Security Systems HVAC Systems EMS / NMS Devices Ideal: Monitoring System to Aggregate ALL Alarms into “Single Pane of Glass”
  31. 31. Process 31 •  Process (ITIL) – Documented, Standards-based and Focus on SLA Fulfillment •  Process Flow Mapping •  Work Instructions for Every Process •  Expected Results •  Escalation when Required •  Closure and Documentation •  Reporting •  Real Need for Best-in-class Process Development
  32. 32. People 32 Skill Set to Match Functional Requirements •  Service Desk •  Tier 1 Support •  Advanced Support •  Subject Matter Expertise Quantity Time Staffing W orkLoad Staffing Limitations: •  Budget Constraints •  Staff Additions Often Lag Growth •  Multiple Responsibilities - Staff for Highest Job Requirements
  33. 33. Structured NOC 33 ACTIVITY TYPE BY TIER: Tier 1 Tier 2/3 PERIODIC REVIEW 3% 10% DOCUMENTATION 2% 3% INCIDENT MANAGEMENT 7% 18% 24x7 EVENT MONITORING 39% CALLS/E-MAILS 14% 4% Tier  1   Activities 65% Tier  2/3   Activities 35% •  Tiered Jobs •  Resource Pool •  Maintain Focus •  Meet SLAs •  Control Costs •  Right Sized” Solution
  34. 34. Knowledge Base / Runbook 34 •  Documentation for all Aspects of the Business •  Architecture •  Organization •  Process Flow •  Work Instructions •  Contingencies •  Business Continuity Plan
  35. 35. Other Considerations 35 Service Turn-Up Team •  Dedicated service turn-up team •  Initial set-up and changes – process and procedure, ticketing configuration, equipment monitoring, etc. Systems & Development •  Continued development of monitoring tools •  Integration with third-party/client systems •  Custom monitoring, reporting Project Management & NOC Run Book Development •  Development and documentation of operational support processes for clients •  Advanced Runbook including comprehensive client-specific process flow diagrams and detailed work instructions for Incident Management, Problem Management and Change Management Customer Advocacy •  Represents customer’s interests •  Reports, QA, QC programs •  Ongoing reviews with client team
  36. 36. Transforming Critical Infrastructure Questions / Open Discussion •  Art Meierdirk, Senior Director, Business Services – INOC •  Dan Belmont, Director, Energy & Utilities - West Monroe Partners •  Gary Sanders, Telecom Manager – Sun Prairie Utilities

×