2. Utility & Advanced Technology Overview
Dan Belmont,
Director, Energy & Utilities
3. Utility & Advanced Technology Overview
3
• Utilities – Drivers for Advanced Technologies
• The Telecom “Tipping Point”
• Advanced Technologies – Applications vs. Telecom Capabilities
• Creating a Roadmap
• Cyber Security and Telecom Synergy
• Lesson’s Learned and Path to Success
4. Utilities – Drivers for Advanced Technologies
4
• Mission?
• Over the next 10-15 years, we will transform our delivery
infrastructure into a model we call the “Utility of the
Future” which will:
– Empower customers to control electricity usage and costs
– Improve service, enhance reliability and cost controls
– Improve the customer experience
– Increase efficiency of our operations
– Enable Distributed Generation
– Networks and Microgrids
5. Utilities – Drivers for Advanced Technologies
5
GRID Modernization
End Use Programs
Enabling Infrastructure Investments
Information Technology
Core Telecommunications
Service and Data
Center,
Substation, and
SCADA
Communications
Transmission,
Distribution, and
Substation
Automation
(DA/SA)
Advanced
Metering
Infrastructure
(AMI)
Mission Critical
Voice and Data
Systems
Backhaul
6. The Telecom Tipping Point
6
AT&T, Verizon, and LECsannounce discontinued analogcircuit support in 2013 andservice in 2018
NERC CIP version 5
removes the non-
routable protocol
exclusion for critical
assets
NISTiR 7628 Smart Grid cyber
security standards must
address attacks from
disgruntled employees,
industrial espionage, and
terrorists ….
95% of Utility companies aregoing to experience coppertheft this year.
Cooper theft caused 456,000outage minutes, 52 injuriesand 35 deaths in one year.
Utilities with AMI have more managed
devices per square mile than LECs,
Cable, or Cellular Carriers
7. Applications vs. Telecom Networks Capabilities
7
Smart Metering &
Home Area
Network
Distribution
Automation
Substation
Protection and
Automation
Network
Backhaul
Enterprise
Communications
Mobile Voice/
Data
Communication Applications New challenges for processes and people
8. Applications vs. Telecom Networks Capabilities
8
Distribution
Automation
Substation
Protection and
Automation
§ Coverage of >99% of Meters
(preferably Electric, Gas and Water)
§ Higher Latency, Excellent Coverage
§ Typically Need Mesh Capability
between Meters and HAN Devices
Network
Backhaul
Smart Metering
& Home Area
Network
Enterprise
Communications
Mid-Tier
Backhaul
Mobile Voice/
Data
Telecom
Backbone
AMI and HAN
Communications
Mobile Voice/
Data
Applications
Communication
Systems
Communication Systems Characteristics
8
§ Mission Critical Voice/Data
§ Wide Band Mobile Data
§ Coverage across Service Territory
§ Power Generation Back-up
§ Low Latency Broadband
§ Good Uplink Capacity for Video
§ Coverage across Service Territory
§ Power Generation Back-up
§ Mission Critical Voice/Data
§ Wide Band Mobile Data
§ Relay Teleprotection Services
§ Coverage across Service Territory
§ Power Generation Back-up
9. Applications vs. Telecom Networks Capabilities
9
Tier
One
Core
MPLS
Tier
Three
Access
Collectors
Tier
Two
Backhaul
Explore combinations of Tier 2 solutions with Tier 3 networks to
minimize net cost while maintaining Cyber Security and IP design
standards
10. Creating a Telecom Roadmap
10
Microwave/Fiber
Backhaul
• Backbone
IP
Communica0ons
• Connec0vity
to
AMI
comm.
• Connec0vity
to
substa0ons
• Connec0vity
to
2-‐Way
Radio
IT
“Hand-‐Off”
ENTERPRISE
BUS
• Customer
Billing/CRM
• Load
Control
Management
System
• Meter
Data
Management
System
• Outage
Management
System
• Distribu0on
Management
System
• Energy
Management
System
Service
Center
AMI
•
AMI
Communica0ons
•
Smart
Meters
•
Distribu0on
Automa0on
• Power
Quality
Meters
• Vehicular
Radios
• Portable
Radios
2-‐Way
Voice
&
Mobile
Data
•
Microwave
or
Fiber
Based
•
SCADA
to
Substa0ons
•
Communica0ons
to
AMI
•
Relay
Teleprotec0on
services
•
Distribu0on
Automa0on
SubstaLon
&
DistribuLon
CommunicaLons
Home
Area
Network
(HAN)
•
Home
Energy
Display
•
Programmable
Thermostats
•
Load
Control
Relays
•
Renewable
Energy
Genera0on
•
PHEV/EV
Smart
Charging
12. Cyber Security and Telecommunications Synergy
12
MPLS service with four forwarding classes
§ Reserve 2 Mb/s for voice and 4 Mb/s for video
§ Enforce 14 Mb/s PIR for overall service
§ Allow critical and best-effort traffic to burst up to 14 Mb/s if bandwidth is available
Less overall bandwidth
required
§ lower overall cost
Priority and best-effort traffic
are equally well served
§ Voice and video do not always consume
all the reserved bandwidth
CIR=0
PIR=4 Mb/s
PIR=max
PIR=max
PIR=2 Mb/s
CIR=2 Mb/s
CIR=4 Mb/s
CIR=4 Mb/s
Overall
SLA=14 Mb/s
Voice
Video
SCADA
FE
Normal Data
13. Change Management and Training
13
= Aware = Knowledgeable = Expert
Telecom/Cyber
Security will
Require Training
across the Utility
14. • Create a holistic business case and roadmap
• Design cyber security in from the ground up
• Tight Quality Control, Quality Assurance, and Audit
• Carefully Orchestrated Cutover/Migration
• Business Process Design
• Organizational Change Management
Lesson’s Learned and Path to Success
REB REBMEB
Ping
MEB
to
verify
power
(From
ALU
gear)
Verify
in
GS
which
MEB(s)/REB(s)
are
communicating
with
Ping
7705
to
verify
power
(Can
be
done
within
SW
or
with
a
device
on
the
same
subnet/network).
Also
check
SW
for
downed
equipment
alert
Ping
6855
from
7705
to
verify
power.
(Can
be
done
within
SW
or
a
device
on
the
same
subnet/network).
Also
check
SW
for
downed
equipment
alert
Ping
7705
from
6855
to
test
connectivity
and
power
Check
SAM
to
ensure
MEB
port
is
up
and
active
on
either
6855
or
7705
Check
success
rates
in
G.S
than
PI
to
identify
other
possible
issues
Still
up?
No
Refer
to
site
master
to
verify
what
equipment
is
on
site
If
a
7705
is
o
n-‐s
ite
If
6
855
is
o
n-‐s
ite
ME
B
Fail
ping
Then
Pass
ping
Pass
Ping
If
all
else
fails:
On-‐site
visit
by
qualified
and
identified
group
through
process
of
elimination.
Worst
case
scenario.
(SSN,
BIG,
ALU,
Standby,
MJ,
CED)
Pass
ping
Pass
ping
Alert/Remedy
Ticket
from
SSN
that
MEB
is
down
Find
out
alerting
tool
for
AC
power
loss,
DC
backup
activated,
Backhaul
issues.
That
way
if
other
steps
“fail”
we
can
start
a
new
process
flow
Pass
Ping
Fail
Ping
Fail
Ping
Fail
Ping
Yes
16. Sun Prairie Utilities
16
• Who and what we are
– Municipality
– 100+ years old
– Population of 30,000
– Electric / Water / Internet / WAN
17. Municipal Entry Into Telecom
17
• Approached by Sun Prairie Area School District in
1998
• They had been quoted a price from the local ILEC for
$1.2 million to furnish WAN services to School
District
• Cable company was not interested in furnishing
services at the time
• City of Sun Prairie got involved
• Formed study committee between the three entities
18. • They used a map of the city to lay out 10 school sites, 5
city sites and 12 Sun Prairie Utilities sites.
• The result was a natural overhead loop on our electric
distribution poles.
• At that point, we did an engineering study which was
funded by the three Committee members for $5,000
each.
• Also, at this point our Utility Commission became
involved and was very supportive.
• Obtained CLEC license
• Saving to the SPASD & City exceeds $1M
The Committee Met…
18
19. Launching Fiber WAN
19
• Simple P2P roll out for School District , City & Utility
sites for SCADA
• Began offering business solutions
• Evolving expectations / scope
• Transition from ATM to Ethernet
• Engineering, design & support needs
• Partner with NOC provider
– 24x7x365 Support
– Reduced staff & tools (monitoring / ticketing)
costs
20. • At the time customers were not happy with services &
availability offered by Incumbent providers
• Temporary solution – FTTH technology had not matured
enough at the time. We wanted to offer the community
something until we found a FTTH product that we were
comfortable with
• Wireless solution would ultimately complement FTTH
technology
Why Wireless?
20
22. Enhanced Fiber WAN
22
• Growing services and support
• Required outside resource / expertise
• Higher level of expectations from subscribers
Partner with NOC provider
– Network engineering, design &
deployment
– 24x7 Support
– Reduced staff & tools (monitoring /
ticketing) costs
• Multi-node 10Gb ring architecture
• Serving MDU’s as way to expand
infrastructure and grow network cost
effectively
23. Lessons Learned
23
Fixed Wireless
• Line of sight / unlicensed spectrum interference
• Technology / hardware life cycles
• Complimentary to fiber infrastructure – Not end-all
solution
Fiber
• High demand
• Symmetrical bandwidth
• Reliability
• Scalability
24. • GPON
• High demand
• Economic Development
• Smart Grid
• Quality of Life
• Customer Service
• Public Private Partnerships
• Political challenges at local & state levels
2014 FTTX Initiative 2.0
24
25. The Network Operations Center
On-going Support
Art Meierdirk
Senior Director, Business Services
26. On-going Support – Service Level Agreement
(SLA) Based
26
• Define the “product”
• Facilities / power / environmental
• Transport / switching
• Connectivity
• Set Service Level Requirements
• Metrics (alarm / call handling / accuracy)
• Service availability (uptime)
• Time to entry
• Restoration response time (time to action)
• Quality Assurance metrics & reports
Value
Proposition
Elements
Value
Converted
to SLAs
27. Meeting Business / Communication Expectations
27
• Build a solution for 24x7 support to meet
SLA requirements:
• Action on Monitored Infrastructure Alarms
• Answer Incoming Calls / Messages
• Notifications of Clients / Customers / Users
• Escalation and/or Dispatch for Restoration
• Time to Restoration
• Documentation of all Actions for Reporting SLA
Performance
Tools
Process
People
Integration
28. Solution Elements
28
• Tools – Monitoring, Workflow, Documentation &
Reporting
• Process – Best-in-class (ITIL) / Focus on SLA
• People – Skill Set to Match Functional Requirements
• Professional (Consultation) Services – Integration of
Tools and Process-based on SLA Requirements
”Structured NOC”
29. Tools
29
• Monitoring, Workflow Documentation & Reporting
• Device / EMS / NMS / Dry Contacts – Alarm
Aggregation from all Infrastructure Systems
• Workflow Management / Ticketing System with Time-
stamp as Work Progresses or Escalation of Work is
Delayed
• Portal / Web Access for Customer Access to their
Information
30. Comprehensive (Single) View
30
NOC
Monitoring
System (Alarm
Aggregation)
Secure
Connection
Security
Systems
HVAC Systems
EMS / NMS
Devices
Ideal: Monitoring System to
Aggregate ALL Alarms into “Single
Pane of Glass”
31. Process
31
• Process (ITIL) – Documented, Standards-based
and Focus on SLA Fulfillment
• Process Flow Mapping
• Work Instructions for Every Process
• Expected Results
• Escalation when Required
• Closure and Documentation
• Reporting
• Real Need for Best-in-class Process
Development
32. People
32
Skill Set to Match Functional Requirements
• Service Desk
• Tier 1 Support
• Advanced Support
• Subject Matter Expertise
Quantity
Time
Staffing
W
orkLoad
Staffing Limitations:
• Budget Constraints
• Staff Additions Often Lag Growth
• Multiple Responsibilities - Staff for Highest Job
Requirements
33. Structured NOC
33
ACTIVITY TYPE BY TIER: Tier 1 Tier 2/3
PERIODIC REVIEW 3% 10%
DOCUMENTATION 2% 3%
INCIDENT MANAGEMENT 7% 18%
24x7 EVENT MONITORING 39%
CALLS/E-MAILS 14% 4%
Tier
1
Activities
65%
Tier
2/3
Activities
35%
• Tiered Jobs
• Resource Pool
• Maintain Focus
• Meet SLAs
• Control Costs
• Right Sized” Solution
34. Knowledge Base / Runbook
34
• Documentation for all Aspects of the Business
• Architecture
• Organization
• Process Flow
• Work Instructions
• Contingencies
• Business Continuity Plan
35. Other Considerations
35
Service Turn-Up Team • Dedicated service turn-up team
• Initial set-up and changes – process and
procedure, ticketing configuration, equipment
monitoring, etc.
Systems
& Development
• Continued development of monitoring tools
• Integration with third-party/client systems
• Custom monitoring, reporting
Project Management
& NOC Run Book
Development
• Development and documentation of operational
support processes for clients
• Advanced Runbook including comprehensive
client-specific process flow diagrams and detailed
work instructions for Incident Management,
Problem Management and Change Management
Customer Advocacy • Represents customer’s interests
• Reports, QA, QC programs
• Ongoing reviews with client team
36. Transforming Critical Infrastructure
Questions / Open Discussion
• Art Meierdirk, Senior Director, Business Services – INOC
• Dan Belmont, Director, Energy & Utilities - West Monroe Partners
• Gary Sanders, Telecom Manager – Sun Prairie Utilities