Presentation from the EPRI-Sandia Symposium on Secure and Resilient Microgrids: Cyber Security R&D for Microgrids, presented by Jason Stamp, Sandia National Laboratories, Baltimore, MD, August 29-31, 2016.

1. Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin
Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.
Cyber
Security
R&D
for
Microgrids
Panel
Session:
Emerging
System
Design
Requirements
–
Security,
Resiliency,
and
Reliability
Jason
Stamp,
Ph.D.
Sandia
NaDonal
Laboratories
1

2. Sandia’s
Control
System
Security
Research
Mission: To reduce the risk of critical infrastructure
disruptions due to cyber attacks on control systems.
Provide decision makers with actionable information
• Red Team Assessments
• Field Device Analysis
• PLC monitoring and forensics
• PLC firmware forensics
• ICS network detection for ICS traffic
• Emulytics (SCEPTRE)
• Exercise/Test Bed support
Design resilient systems to withstand cyber-attacks
• Research next generation security solutions
• Partner with industry to “push” solutions to market
2

3. Control
System
Architecture
Human-Machine Interface (HMI) software
Status displays
Switches and dials
User Interfaces
Field Devices
Programmable Logic Controllers (PLC)
Remote Telemetry Units (RTU)
Intelligent Electronic Devices
Sensors
Thermocouples
Accelerometers
Photoresistors
Physical Process
Oil & Gas Refining
Electrical Distribution and Transmission
Manufacturing
Actuators
Breakers/Switches
Motors
Valves
Supervisory Control and Data Acquisition (SCADA)
Distributed Control Systems (EMS/DCS)
Data Historians
Control System Apps
3

4. RepresentaDve
ICS
TesDng
Environments
Emuly&cs™/SCEPTRE
4

5. SCEPTRE
OperaDonal
Overview
§ SCEPTRE
provides
a
cyber-­‐physical
environment
to
show
interacDon
between
cyber-­‐iniDated
events
and
the
physical
world
§ Balances
need
for
M&S
accuracy
against
tesDng
resources
§ Live
system
tesDng:
potenDal
damage
to
the
real
system
and
dangers
to
human
life
§ Test
bed
systems:
Expensive
to
build,
maintain,
configure,
and
operate
§ Labscale
hardware
tesDng
setups:
May
require
the
context
of
a
larger,
networked
system
§ Devices
(simulated,
emulated,
real)
communicate/interact
via
ICS
protocols
§ All
ICS
devices
are
able
to
interact
with
the
process
simulaDon,
providing
both
updates
and
subscribing
to
the
current
state
of
the
simulaDon
§ Overall
simulaDon
is
able
to
bridge
mulDple
infrastructures
into
the
same
experiment
to
show
interdependencies
§ Use
cases:
§ Test
and
evaluaDon
§ Mission
rehearsal
§ Other
analysis:
understand
vulnerabiliDes
and
exploitable
avenues,
idenDfy
criDcal
components
on
the
control
network,
model
infrastructure
interdependencies,
etc.
5

6. SCEPTRE
Cyber
Security
Analysis
for
ICS
§ Control
systems
devices:
simulated
RTUs,
PLCs,
relays;
emulated
PLCs,
FEPs,
HMI
services;
real
HITL
relays,
PLCs,
RTUs
§ High
fidelity
SCADA
protocols:
ModbusTCP,
DNP3,
IEC61850
§ Process
simulaDon:
industry
standard
so_ware
where
possible,
PowerWorld,
PyPower,
PSSE
for
electricity,
water
treatment,
refining,
oil/gas
pipelines
6

7. Cyber
Security
Architecture
§ Microgrid
cyber
security
reference
architecture
§ In
addiDon
to
DoD
IA
controls,
addiDonal
rigor
will
be
applied
to
protecDng
data-­‐in-­‐moDon
and
data-­‐
at-­‐rest,
along
with
ensuring
such
addiDonal
rigor
does
not
impede
the
operaDonal
data
exchange
requirements
of
the
SPIDERS
microgrid
§ Defense-­‐in-­‐depth
using:
§ Enclaves
§ FuncDonal
Domains
7
4
V. DESIGN APPROACH AND DEFENSE-IN-DEPTH
Best practices for securing ICSs leverage network segmen-
tation; for example, see [3], [6], and [7]. In most cases,
however, network segmentation is focused on separation of
the control system network from other less-trusted networks,
such as the enterprise network and the Internet. The concept of
network segmentation within the control system network itself
is addressed to a minimal degree in a recommended practices
document [3] published by the DHS Control System Security
Program (CSSP), but the additional complexities of configur-
ing and managing such a network often result in this level of
defense-in-depth being dismissed. In geographically dispersed
control systems and field devices, physical segmentation often
inherently exists within ICS command and control networks
due to the employment of third-party providers for communi-
cation services. This segmentation is not leveraged to enhance
security, however, as neither physical nor logical segmentation
is currently used as a basis for providing additional defense-
in-depth within modern ICS networks.
The SNL approach to designing a secure microgrid control
system network leverages segmentation to reinforce defense-
in-depth practices. The microgrid control system network is
segmented into enclaves defined by system functions, physical
locations, and security concerns. Enclaves are then grouped to-
gether into functional domains that allow actors to collaborate
in operational system functions that crosscut enclaves. Data
exchange worksheets describe communication between actors
within enclaves and functional domains.
A. Enclaves
An enclave is a collection of computing environments that
only by system function, rather than by physical location. For
example, consider that all of the actors at Site II are grouped
into a single enclave (Enclave 3) based on physical location,
whereas the actors at Site I are segregated into two enclaves
(Enclave 1 and Enclave 2), which may be based on physical
location, system function, security concerns, or a combination
of features.
Fig. 2. Example segmentation of network into enclaves and functional
domains.
B. Functional Domains
Although some enclaves are defined based on actors that
participate in a particular system function, some actors neces-
sarily crosscut enclaves that are defined by physical location,
functional characteristics, or security concerns. For example,
the EMS could interact with external actors at the electrical
points of common coupling (PCCs), which could belong to

8. Cyber
Security
Data
Exchange
§ Process:
§ Designate
actors
§ Describe
data
flows
using
tables
§ Assign
enclaves
§ Develop
funcDonal
domains
§ Design
cyber
security
controls
8
TABLE IV
DATA EXCHANGE ATTRIBUTES AND EXAMPLE VALUES.
Attribute Description Example Values
Exchange
Type Type of data exchange to occur monitor, control, report, write
Interval How often data exchange occurs e.g. milliseconds, seconds
Method How data will be exchanged unicast, multicast, broadcast
Priority Relative importance of exchanging the data high, medium, low
Latency Tolerance Tolerance to delayed control or delayed data exchange high (delays do not affect operation), medium, low
Data
Type Type of data to be exchanged voltage, setpoint, status
Accuracy Necessary precision/timeliness of data significant digits, time units
Volume Amount of data to transferred per exchange e.g. bytes, kilobytes, etc.
Reliability Necessity of access to control processes and data critical, important, informative
InformationAssurance
Confidentiality Importance of preserving restrictions to control
processes and information access (based on risk to
system operations and/or system security)
high, medium, low
Integrity Importance of preventing unauthorized changes to
control processes or data, including authenticity (based
on reliability with respect to operations)
high, medium, low
Availability Importance of timely and reliable access to control
processes and data (based on priority and latency
tolerance with respect to operations)
high, medium, low
influence of actors to a particular enclave, the consequences of
both local failures and vulnerabilities are isolated within that
enclave.
VIII. FIRST EXAMPLE FOR THE REFERENCE
ARCHITECTURE
The approach to segmenting the microgrid control system
network is to first identify system functions with a granularity
B. System Functions
Consider a basic microgrid function: Connect/Disconnect
Microgrid as applied to this system. Islanding of the microgrid
when the installation’s distribution system loses power and is
one of the key functions of the system’s operation. The power
actors typically involved in this system function include:
• IEDs at the utility (PCC) used to monitor voltage/current
sensors and to control breakers and disconnect switches,
EMS may also receive manual control messages from an
operator of an HMI system. These control messages are sent
from the HMI server via the EMS to the appropriate IEDs via
a FEP.
TABLE V
EXAMPLE FOR DATA EXCHANGE (AGMC OPERATIONS)
FROM A FEP TO A GENERATOR IED
Data Exchange Attributes for
Automated Grid Management and Control (AGMC) Operations
Source FEP FEP
Destination Generator controller Generator controller
Exchange
Type monitor control
Interval seconds seconds or minutes
Method unicast unicast
Priority medium medium
Latency
Tolerance
medium low
Data
Type run/stop/ATS status, fuel
level, active & reactive
output, frequency
start/stop/mode/breaker
control, voltage settings,
governor droop settings
Accuracy 1 decimal, second 1 decimal, second
Volume bytes bytes
Reliability important critical
Information Assurance
Confidentiality medium medium
Integrity medium high
Availability high high
TABLE VI
EXAMPLE FOR DATA EXCHANGE (AGMC OPERATIONS)
BETWEEN AN EMS AND A HMI SERVER
Data Exchange Attributes for
Automated Grid Management and Control (AGMC) Operations
Source EMS HMI Server
Destination HMI Server EMS
Exchange
network
concerns
because
or carry
Server
that auto
and req
the EM
the broa
sheer vo
of its o
through
microgri
relevant
The enc
• Dis
sys
• Ren
ren
• Ge
ing
Data
Exchange
Table
Format
Data
Exchange
Example
Example
Flat
Control
System
8

9. Defense-­‐in-­‐depth:
ApplicaDon
of
Enclaves
and
FuncDonal
Domains
9
Flat
Segmented

10. Cyber
Security
QuanDtaDve
Analysis
10
and “report” can be considered as “reading” (from the field to
the control center) and likewise all control traffic outward to
the field devices can be labeled “write.” Furthermore, “high,”
“medium,” and “low” are mapped to the numerical values 1,
2, and 3 respectively (although any could be used, the simplest
approach is simple incrementing values). Summarizing the
data exchange characteristics for each functional domain with
the read/write strategy yields the data shown in Table VII.
TABLE VII
SUMMARIZED DATA ATTRIBUTES FOR EXAMPLE MICROGRID CONTROL
SYSTEM.
Functional
Domain
Read/Write
Confidentiality
Integrity
Availability
Subtotal
Total
HMI- Read 2 3 2 7
13
Server Write 2 2 2 6
Server- Read 2 3 2 7
13
FEP Write 2 2 2 6
FEP- Read 1 3 3 7
15
RTU Write 2 3 3 8
Totals Both 11 16 14 41 41
The testing against this example system was performed by
cyber security Red Teams, modeling relevant threats (Section
III). The tests were scored by carefully monitoring the data
flows that form the functional domains during the exercise.
If any flow in a functional domain was impacted according
to confidentiality, integrity, or availability, then the affected
security attribute was scored as a zero; otherwise, if unaffected
it was scored according to the value in Table VII. Obviously,
if any security attribute was impacted, then test score was less
than perfect (100% of raw value 41). During testing, both read
and write flows were impacted, sometimes in different ways.
(a) Flat network
(b) Enclaved network
Fig. 7. Red Team access locations for the quantitative testing.
C. Experiment Results
Per the previous discussion, a total of eight versions of the
notional microgrid control system network were deployed and
tested in a laboratory setting at SNL. The Red Teams were
Fig. 6. Reference architecture test network (enclaved configuration).
the the “Type” attribute of the “Exchange” section for the ap-
plicable data exchange worksheets (Table IV). Here, “monitor”
and “report” can be considered as “reading” (from the field to
the control center) and likewise all control traffic outward to
the field devices can be labeled “write.” Furthermore, “high,”
“medium,” and “low” are mapped to the numerical values 1,
2, and 3 respectively (although any could be used, the simplest
approach is simple incrementing values). Summarizing the
data exchange characteristics for each functional domain with
the read/write strategy yields the data shown in Table VII.
TABLE VII
SUMMARIZED DATA ATTRIBUTES FOR EXAMPLE MICROGRID CONTROL
SYSTEM.
Functional
Domain
Read/Write
Confidentiality
Integrity
Availability
Subtotal
Total
HMI- Read 2 3 2 7
13
Server Write 2 2 2 6
Server- Read 2 3 2 7
13
FEP Write 2 2 2 6
FEP- Read 1 3 3 7
15
RTU Write 2 3 3 8
Totals Both 11 16 14 41 41
The testing against this example system was performed by
cyber security Red Teams, modeling relevant threats (Section
III). The tests were scored by carefully monitoring the data
flows that form the functional domains during the exercise.
If any flow in a functional domain was impacted according
• Access: where in the network the modeled adversary has
access (three choices, shown in Figure 7)
• Compliance: a binary variable representing the cyber
security of the platforms in the system, with “hardened”
representing systems that are fully patched and secured
according to current best practices, and “insecure” mean-
ing they are not; due to the operational reliability neces-
sary from energy control systems, hardware and software
patches are not always applied in a timely manner
(a) Flat network
(b) Enclaved network
Fig. 7. Red Team access locations for the quantitative testing.
constrained to reasonable threat parameters (specifically, the
“Mid” range shown in Table I). The results are in Table VIII.
TABLE VIII
MICROGRID CYBER SECURITY TEST RESULTS.
Architecture
Access
Compliance
Confidentiality
Integrity
Availability
Total
Flat High
Insecure 0 0 8 8
Hardened 9 0 14 23
Enclaved
High
Insecure 0 0 8 8
Hardened 9 0 14 23
Med- Insecure 7 6 11 24
ium Hardened 9 6 14 29
Low
Insecure 11 6 16 33
Hardened 11 6 16 33
Maximum Possible Score ! 11 16 14 41
The results indicate that each progressive variation to the
reference implementation led to an increase in system security.
More interesting is the fact that adding hardened systems to
the enclaved versions of the reference implementation only
increased the security by a small amount, and the small
The authors w
tricity Delivery
this work, as w
Idaho National L
Technology Linc
mand (USPACO
Warfare Center
participation in t
[1] Systems and N
Assessing and
Systems (Versi
(NSA), August
[2] Brian Van Leeu
Sandia Report
Albuquerque, N
[3] Control System
Improving Indu
Depth Strategie
(NCSD), Depa
[4] CSSP, Catalo
Standards Deve
[5] CSSP, Comm
Systems, techn
[6] Smart Grid In
Group (CSWG
Interagency Re
Standards and
H/M/L
SensiDvity
Scores
for
FuncDonal
Domains
Red
Team
Scoring
Results

11. Advanced
Field
Device
Monitoring
Network monitoring alone is not sufficient to adequately defend
against a sophisticated adversary
PLCs are vulnerable to targeted
attacks that cost millions in
equipment damage, lost
operation, or injured personnel.
PLCs are not monitored for
security compromise.
It is not enough to build “secure”
products. The ability to inspect
and detect is necessary for
systems that will be in place for
decades.
A backplane analysis system
examines the communication
between PLC modules
Cyber attacks on the control
systems will result in anomalies
visible on the PLC backplane.
New Capabilities for PLCs:
• Forensics: After compromises, detect
modifications to hardware, firmware, or
logic
• Detection: Actively detect anomalies
11

12. Advanced
Field
Device
Monitoring
§ WeaselBoard
plugs
into
the
backplane
and
listens
to
the
conversaDons
between
control
system
modules
§ There
is
a
lot
of
granularity
in
these
conversaDons,
which
allows
WeaselBoard
to
uniquely
observe
behavior
of
the
control
system
independent
of
the
processor
and
alert
when
the
system
is
not
operaDng
within
a
specifically
defined
manner
§ Because
it
alerts
on
effects
of
an
adack
in
progress,
and
not
on
signatures
of
prior
adacks,
WeaselBoard
can
detect
zero-­‐day
exploits
Processor
Module
Runs Process Logic
PLC Backplane
Comms
Module
Connects the PLC to
the Network
I/O
Module
Connects the PLC
to the Process
Isolation
WeaselBoard
Detects Intruders
12

13. Other
ICS
Cyber
Security
RecommendaDons
§ InvesDgate
all
miDgaDon
opDons,
covering
defend,
detect,
react,
and
recover
(including
incident
management/recovery
plans)
§ Develop
and
install
detecDon
capabiliDes
for
adack/anomaly
indicators
§ Complementary
opDons
include
network
traffic
monitoring
and
advanced
hardware
monitoring
§ Reduce
troubleshooDng
duraDon
§ Develop
effecDve
environments/procedures
for
tesDng
§ Minimize
adacker
opportuniDes
for
device
configuraDon
or
firmware
access
(possibly
disallowing
such
network
traffic)
§ Develop
logic-­‐
and
tamper-­‐checking
tools
for
devices
and
systems
§ Focus
on
cyber
security
assessment
for
field
devices
13

14. Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin
Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000.
Cyber
Security
R&D
for
Microgrids
Panel
Session:
Emerging
System
Design
Requirements
–
Security,
Resiliency,
and
Reliability
Jason
Stamp,
Ph.D.
Sandia
NaDonal
Laboratories
14