Download as PDF, PPTX
Uploaded byIlissa Miller
INOC - Art Meierdirk Presentation from UTC April 15, 2014
This document discusses utilities modernizing their infrastructure through advanced technologies and telecommunications networks. It covers: 1) Drivers for utilities to adopt advanced technologies like smart metering, distribution automation, and enterprise communications to empower customers, improve reliability and operations, and enable new technologies. 2) How utilities' telecommunications needs have outgrown traditional networks due to new applications and increased cybersecurity requirements, requiring them to create roadmaps balancing different network tiers. 3) Lessons learned from Sun Prairie Utilities' experience launching a fiber WAN and fixed wireless network initially before expanding to a fiber network, including the importance of partnerships, scalability, and customer service.
More Related Content
Similar to INOC - Art Meierdirk Presentation from UTC April 15, 2014
INOC - Art Meierdirk Presentation from UTC April 15, 2014
- 1. © 2013 UtilitiesTelecom Council Transforming Critical Infrastructure The Cost of Modernization: Perspectives on Effectively Managing Today's Advanced Technology Assets without Breaking the Bank • Art Meierdirk, Senior Director, Business Services – INOC • Dan Belmont, Director, Energy & Utilities - West Monroe Partners • Gary Sanders, Telecom Manager – Sun Prairie Utilities
- 2. Utility & AdvancedTechnology Overview Dan Belmont, Director, Energy & Utilities
- 3. Utility & AdvancedTechnology Overview 3 • Utilities – Drivers for Advanced Technologies • The Telecom “Tipping Point” • Advanced Technologies – Applications vs. Telecom Capabilities • Creating a Roadmap • Cyber Security and Telecom Synergy • Lesson’s Learned and Path to Success
- 4. Utilities – Driversfor Advanced Technologies 4 • Mission? • Over the next 10-15 years, we will transform our delivery infrastructure into a model we call the “Utility of the Future” which will: – Empower customers to control electricity usage and costs – Improve service, enhance reliability and cost controls – Improve the customer experience – Increase efficiency of our operations – Enable Distributed Generation – Networks and Microgrids
- 5. Utilities – Driversfor Advanced Technologies 5 GRID Modernization End Use Programs Enabling Infrastructure Investments Information Technology Core Telecommunications Service and Data Center, Substation, and SCADA Communications Transmission, Distribution, and Substation Automation (DA/SA) Advanced Metering Infrastructure (AMI) Mission Critical Voice and Data Systems Backhaul
- 6. The Telecom TippingPoint 6 AT&T, Verizon, and LECsannounce discontinued analogcircuit support in 2013 andservice in 2018 NERC CIP version 5 removes the non- routable protocol exclusion for critical assets NISTiR 7628 Smart Grid cyber security standards must address attacks from disgruntled employees, industrial espionage, and terrorists …. 95% of Utility companies aregoing to experience coppertheft this year. Cooper theft caused 456,000outage minutes, 52 injuriesand 35 deaths in one year. Utilities with AMI have more managed devices per square mile than LECs, Cable, or Cellular Carriers
- 7. Applications vs. TelecomNetworks Capabilities 7 Smart Metering & Home Area Network Distribution Automation Substation Protection and Automation Network Backhaul Enterprise Communications Mobile Voice/ Data Communication Applications New challenges for processes and people
- 8. Applications vs. TelecomNetworks Capabilities 8 Distribution Automation Substation Protection and Automation § Coverage of >99% of Meters (preferably Electric, Gas and Water) § Higher Latency, Excellent Coverage § Typically Need Mesh Capability between Meters and HAN Devices Network Backhaul Smart Metering & Home Area Network Enterprise Communications Mid-Tier Backhaul Mobile Voice/ Data Telecom Backbone AMI and HAN Communications Mobile Voice/ Data Applications Communication Systems Communication Systems Characteristics 8 § Mission Critical Voice/Data § Wide Band Mobile Data § Coverage across Service Territory § Power Generation Back-up § Low Latency Broadband § Good Uplink Capacity for Video § Coverage across Service Territory § Power Generation Back-up § Mission Critical Voice/Data § Wide Band Mobile Data § Relay Teleprotection Services § Coverage across Service Territory § Power Generation Back-up
- 9. Applications vs. TelecomNetworks Capabilities 9 Tier One Core MPLS Tier Three Access Collectors Tier Two Backhaul Explore combinations of Tier 2 solutions with Tier 3 networks to minimize net cost while maintaining Cyber Security and IP design standards
- 10. Creating a TelecomRoadmap 10 Microwave/Fiber Backhaul • Backbone IP Communica0ons • Connec0vity to AMI comm. • Connec0vity to substa0ons • Connec0vity to 2-‐Way Radio IT “Hand-‐Off” ENTERPRISE BUS • Customer Billing/CRM • Load Control Management System • Meter Data Management System • Outage Management System • Distribu0on Management System • Energy Management System Service Center AMI • AMI Communica0ons • Smart Meters • Distribu0on Automa0on • Power Quality Meters • Vehicular Radios • Portable Radios 2-‐Way Voice & Mobile Data • Microwave or Fiber Based • SCADA to Substa0ons • Communica0ons to AMI • Relay Teleprotec0on services • Distribu0on Automa0on SubstaLon & DistribuLon CommunicaLons Home Area Network (HAN) • Home Energy Display • Programmable Thermostats • Load Control Relays • Renewable Energy Genera0on • PHEV/EV Smart Charging
- 11. Cyber Security andTelecommunications Synergy 11
- 12. Cyber Security andTelecommunications Synergy 12 MPLS service with four forwarding classes § Reserve 2 Mb/s for voice and 4 Mb/s for video § Enforce 14 Mb/s PIR for overall service § Allow critical and best-effort traffic to burst up to 14 Mb/s if bandwidth is available Less overall bandwidth required § lower overall cost Priority and best-effort traffic are equally well served § Voice and video do not always consume all the reserved bandwidth CIR=0 PIR=4 Mb/s PIR=max PIR=max PIR=2 Mb/s CIR=2 Mb/s CIR=4 Mb/s CIR=4 Mb/s Overall SLA=14 Mb/s Voice Video SCADA FE Normal Data
- 13. Change Management andTraining 13 = Aware = Knowledgeable = Expert Telecom/Cyber Security will Require Training across the Utility
- 14. • Create aholistic business case and roadmap • Design cyber security in from the ground up • Tight Quality Control, Quality Assurance, and Audit • Carefully Orchestrated Cutover/Migration • Business Process Design • Organizational Change Management Lesson’s Learned and Path to Success REB REBMEB Ping MEB to verify power (From ALU gear) Verify in GS which MEB(s)/REB(s) are communicating with Ping 7705 to verify power (Can be done within SW or with a device on the same subnet/network). Also check SW for downed equipment alert Ping 6855 from 7705 to verify power. (Can be done within SW or a device on the same subnet/network). Also check SW for downed equipment alert Ping 7705 from 6855 to test connectivity and power Check SAM to ensure MEB port is up and active on either 6855 or 7705 Check success rates in G.S than PI to identify other possible issues Still up? No Refer to site master to verify what equipment is on site If a 7705 is o n-‐s ite If 6 855 is o n-‐s ite ME B Fail ping Then Pass ping Pass Ping If all else fails: On-‐site visit by qualified and identified group through process of elimination. Worst case scenario. (SSN, BIG, ALU, Standby, MJ, CED) Pass ping Pass ping Alert/Remedy Ticket from SSN that MEB is down Find out alerting tool for AC power loss, DC backup activated, Backhaul issues. That way if other steps “fail” we can start a new process flow Pass Ping Fail Ping Fail Ping Fail Ping Yes
- 15. Gary Sanders Telecom Manager TheSun Prairie Utilities Experience
- 16. Sun Prairie Utilities 16 • Who and what we are – Municipality – 100+ years old – Population of 30,000 – Electric / Water / Internet / WAN
- 17. Municipal Entry IntoTelecom 17 • Approached by Sun Prairie Area School District in 1998 • They had been quoted a price from the local ILEC for $1.2 million to furnish WAN services to School District • Cable company was not interested in furnishing services at the time • City of Sun Prairie got involved • Formed study committee between the three entities
- 18. • They useda map of the city to lay out 10 school sites, 5 city sites and 12 Sun Prairie Utilities sites. • The result was a natural overhead loop on our electric distribution poles. • At that point, we did an engineering study which was funded by the three Committee members for $5,000 each. • Also, at this point our Utility Commission became involved and was very supportive. • Obtained CLEC license • Saving to the SPASD & City exceeds $1M The Committee Met… 18
- 19. Launching Fiber WAN 19 • Simple P2P roll out for School District , City & Utility sites for SCADA • Began offering business solutions • Evolving expectations / scope • Transition from ATM to Ethernet • Engineering, design & support needs • Partner with NOC provider – 24x7x365 Support – Reduced staff & tools (monitoring / ticketing) costs
- 20. • At thetime customers were not happy with services & availability offered by Incumbent providers • Temporary solution – FTTH technology had not matured enough at the time. We wanted to offer the community something until we found a FTTH product that we were comfortable with • Wireless solution would ultimately complement FTTH technology Why Wireless? 20
- 21. Muni Wireless HypeCycle 21
- 22. Enhanced Fiber WAN 22 • Growing services and support • Required outside resource / expertise • Higher level of expectations from subscribers Partner with NOC provider – Network engineering, design & deployment – 24x7 Support – Reduced staff & tools (monitoring / ticketing) costs • Multi-node 10Gb ring architecture • Serving MDU’s as way to expand infrastructure and grow network cost effectively
- 23. Lessons Learned 23 Fixed Wireless • Line of sight / unlicensed spectrum interference • Technology / hardware life cycles • Complimentary to fiber infrastructure – Not end-all solution Fiber • High demand • Symmetrical bandwidth • Reliability • Scalability
- 24. • GPON • Highdemand • Economic Development • Smart Grid • Quality of Life • Customer Service • Public Private Partnerships • Political challenges at local & state levels 2014 FTTX Initiative 2.0 24
- 25. The Network OperationsCenter On-going Support Art Meierdirk Senior Director, Business Services
- 26. On-going Support –Service Level Agreement (SLA) Based 26 • Define the “product” • Facilities / power / environmental • Transport / switching • Connectivity • Set Service Level Requirements • Metrics (alarm / call handling / accuracy) • Service availability (uptime) • Time to entry • Restoration response time (time to action) • Quality Assurance metrics & reports Value Proposition Elements Value Converted to SLAs
- 27. Meeting Business /Communication Expectations 27 • Build a solution for 24x7 support to meet SLA requirements: • Action on Monitored Infrastructure Alarms • Answer Incoming Calls / Messages • Notifications of Clients / Customers / Users • Escalation and/or Dispatch for Restoration • Time to Restoration • Documentation of all Actions for Reporting SLA Performance Tools Process People Integration
- 28. Solution Elements 28 • Tools– Monitoring, Workflow, Documentation & Reporting • Process – Best-in-class (ITIL) / Focus on SLA • People – Skill Set to Match Functional Requirements • Professional (Consultation) Services – Integration of Tools and Process-based on SLA Requirements ”Structured NOC”
- 29. Tools 29 • Monitoring, WorkflowDocumentation & Reporting • Device / EMS / NMS / Dry Contacts – Alarm Aggregation from all Infrastructure Systems • Workflow Management / Ticketing System with Time- stamp as Work Progresses or Escalation of Work is Delayed • Portal / Web Access for Customer Access to their Information
- 30. Comprehensive (Single) View 30 NOC Monitoring System(Alarm Aggregation) Secure Connection Security Systems HVAC Systems EMS / NMS Devices Ideal: Monitoring System to Aggregate ALL Alarms into “Single Pane of Glass”
- 31. Process 31 • Process (ITIL)– Documented, Standards-based and Focus on SLA Fulfillment • Process Flow Mapping • Work Instructions for Every Process • Expected Results • Escalation when Required • Closure and Documentation • Reporting • Real Need for Best-in-class Process Development
- 32. People 32 Skill Set toMatch Functional Requirements • Service Desk • Tier 1 Support • Advanced Support • Subject Matter Expertise Quantity Time Staffing W orkLoad Staffing Limitations: • Budget Constraints • Staff Additions Often Lag Growth • Multiple Responsibilities - Staff for Highest Job Requirements
- 33. Structured NOC 33 ACTIVITY TYPEBY TIER: Tier 1 Tier 2/3 PERIODIC REVIEW 3% 10% DOCUMENTATION 2% 3% INCIDENT MANAGEMENT 7% 18% 24x7 EVENT MONITORING 39% CALLS/E-MAILS 14% 4% Tier 1 Activities 65% Tier 2/3 Activities 35% • Tiered Jobs • Resource Pool • Maintain Focus • Meet SLAs • Control Costs • Right Sized” Solution
- 34. Knowledge Base /Runbook 34 • Documentation for all Aspects of the Business • Architecture • Organization • Process Flow • Work Instructions • Contingencies • Business Continuity Plan
- 35. Other Considerations 35 Service Turn-UpTeam • Dedicated service turn-up team • Initial set-up and changes – process and procedure, ticketing configuration, equipment monitoring, etc. Systems & Development • Continued development of monitoring tools • Integration with third-party/client systems • Custom monitoring, reporting Project Management & NOC Run Book Development • Development and documentation of operational support processes for clients • Advanced Runbook including comprehensive client-specific process flow diagrams and detailed work instructions for Incident Management, Problem Management and Change Management Customer Advocacy • Represents customer’s interests • Reports, QA, QC programs • Ongoing reviews with client team
- 36. Transforming Critical Infrastructure Questions/ Open Discussion • Art Meierdirk, Senior Director, Business Services – INOC • Dan Belmont, Director, Energy & Utilities - West Monroe Partners • Gary Sanders, Telecom Manager – Sun Prairie Utilities