Nathan	Wallace,	PhD,	CSSA	
	
	
	
n.wallace.us@ieee.org	
Twi1er:	@NathanSWallace	
1	
Staff	Engineer	 Dir.	Cyber	Opera;ons
Personal	Background	
Volunteering:		
EE	Intern	 EE	Intern	 Associate	Engineer	 Research	Assistant	 Visi;ng	Lecturer	
Staff	Engineer	 Dir.	Cyber	Opera;ons	
Drafting Relay Settings T&D Protection Cybersecurity Researcher
Digital Forensics Examiner
Math & Engineering
Relay & RTU
Design & Commissioning
Risk Assessments
Cybersecurity Design & Integration
2
3	
Disclaimer	
•  Statements and opinions are my own which may or may not reflect
that of my current employer.
•  Statements are based on generalized observations of the industry
and do not represent any particular entity or asset owner.
•  Seek professional engineering assistance and vendor support prior to
implementing or developing any of the capabilities discussed.
4	RISK	
Two	Infrastructures		
ResidenAal		 Industrial	Commercial	
Genera;on	 Transmission	
Distribu;on	
•  Physical	
•  Cyber	
Control	Center	
DistribuAon	
Control	Center	
RTOs/ISO
5	Assets	
What	devices	need	to	be	protected?	
CommunicaAon	ComputaAonal	What	has	 and/or	 ability	and	
is	used	to	ensure	the	safe,	reliable,	and	con,nuous		
genera;on,	transmission	and	delivery	of	electricity?				
Cybersecurity Implementation Challenge:
How to maintain the
Confidentiality, Integrity, and Availability
of power system cyber assets?
IEEE Std C37.240TM-2014, IEEE Standard Cybersecurity Requirements for Substation
Automation, Protection, and Control Systems.
•  Must not impede access and operation during legitimate activities.
•  Must be technically, economically, and operationally feasible.
Each protective measure
6	Assets	
Cyber	Assets		
Programmable	electronic	devices,	including	the		
hardware,		
soOware,	and		
data,	in	those	devices.		
I/O	cards	
Network	cards	
CPU,	RAM,	Physical	Ports	
User	ApplicaAons	(EMS,	HMI,	FTP,	Apache,	etc.)	
Device	ApplicaAons	(Protocol	Parsers/Converters)		
ApplicaAon	Programming	Interface	(API)	
SoXware	Libraries	and	Tools	
SeZngs	and	ConfiguraAon	Files	
Stored	Binary	and	Analog	Values	
Sequence	of	Events	&	DFR	Logs	
Usernames,	Password	Hashes,	IP	Tables
7	Assets	
•  What	hardware	is	common	across	
all	devices?	
Engr,	Tech,	
Operator		
Imported	
Applica;ons	Library	
Vendor	
Applica;ons	
PAM	 Access/Login	
HMI	Apache	
iproute2	 Firewall	
pyPAM	
HTTP	
•  Do	vendors	write	their	own	code?	
•  What	3rd	Party	soOware	libraries	&	
applica;ons	are	being	used?	
Linux	Kernel	
19.5	million	
lines	of	code	
FUN-FACT
8	
Power	System	Cybersecurity	Implementa;on	
Who’s	Responsibility	is	it?	
IT	Dept.	 OT	Dept.	
t	
-  SoXware	to	determine	how	power	flows	and	when	breakers	open/closes	
-  Apache,	Telnet,	SSH,	MySQL,	FTP,	LDAP,	Embedded	Linux,	Windows,	etc.		
-  Virtual	power	plants	and	protecAon	relays,	soXware	defined	networking	
-  SCADA	in	the	cloud	
Present
9	
OT	Dept.	
VS
=> Adversarial Relationship
Example 1: Annual Funding
IT	Dept.	
IT
Manager
Engineering
ManagerNew Cyber Compliance
Manager & Dept.
Legal Team, Training,
Audit Specialists
No Change
Power	System	Cybersecurity	Implementa;on	
Who’s	Responsibility	is	it?	
More Personnel
and Resources
Present
Present	 10	
Cri;cal	Cyber	Assets	
Compliance	
		HIGH	 		MEDIUM	 		LOW	
15%	
85%	
Protected	Grid	
Cyber	
Security	
PROTECTED	
GRID?	
Cyber	
Assets	
(For US 80-90% grid’s cyber assets are out of scope for NERC-CIP) Source: Cybersecurity and the Evolving Role of State
Regulation: How it Impacts the California Public Utilities Commission, California’s PUC Policy Paper
Cybersecurity
Implementation
11	Present	
VS
Mostly
Policy Standards
Mostly
Technical Standards
and Best Practices
US: NERC-CIP
12	
IT	Dept.	 OT	Dept.	
VS
=> Adversarial Relationship
Example 2: Implementation
a) Securing laptops used by field personnel.
Power	System	Cybersecurity	Implementa;on	
Who’s	Responsibility	is	it?	
Engineer
“My company’s IT department has no idea I use this laptop…
I wouldn’t be able to do my job if they did.”
b) Securing edge devices (RTUs, relays, reclosers, etc.)
Engineer
Settings/Configurations
Power P&C logic
Cyber P&C logicIT Dept.?
Present
13	Present	
Cybersecurity
Design Implementation
Entity’s
1st Audit
v3 Audit
revealed over half
of the system’s
firewalls were
misconfigured.
•  Typically the current approach is to use network firewalls and call it a day.
•  Cybersecurity is an afterthought that ends up being “bolted on” only for compliance.
3	General	Types	of	Firewall	
Packet	Filtering		|		ApplicaAon-Proxy	Gateway		|		Stateful	InspecAon	
•  Some	can	be	bypassed	by	spoofing	network	layer	data	
•  All	are	based	on	soXware	
CVE-2016-**** The password-sync feature on [firewall vendor’s]
switches sets an SNMP community to the same string as the
administrator password, which allows remote attackers to obtain
sensitive information by sniffing network.
14	
What drives cybersecurity in the industry today?
Compliance
What will drive cybersecurity in the industry tomorrow?
Hopefully, engineering and best practices.
Goal
Cyber	
Security	
Present
Future	 15	
Cyber Infrastructure
(Computation & Communication)
Protection and Control
of the Modernized Grid
Physical Infrastructure
(Flow of Power)
Inputs: Currents, Voltages, Impedance,
Status (open,close, lockout)
Output: Open/Close Bkr, +/- Vars,
Inputs: Topology, traffic flows,
deep packet inspection, communication
state, state of physical power system
Output: NOTHING!
Future	 16	
What will drive cybersecurity in the industry tomorrow?
Hopefully, engineering and best practices.
American Engineers' Council for Professional Development defines Engineering as:
"The creative application of scientific principles to design or develop …."
Major
Hurdles
Power System
Cybersecurity &
Cyber Resiliency
Hurdle 1: Labeling of everything as Restricted, Classified, or Sensitive
Requires	Verifiable	Evidence	&	Repeatable	Tests	
Administrator
1.  Joe
2.  Alice
Example
Compliance/Legal
depts. stops
engineer from
discussing what
works and what
doesn’t at
technical industry
conference.
Negative Side Effects
•  Industry slow to advance and therefore slow to defend.
•  Engineers are not aware of solutions/approaches
resulting in the assumption that security is not feasible.
•  Approach is really security through obscurity.
Future	 17	
What will drive cybersecurity in the industry tomorrow?
Hopefully, engineering and best practices. Major
Hurdles
Hurdle 2: Viewing cybersecurity as only defending against malicious actors.
Power System
Cybersecurity &
Cyber Resiliency
Recall	Cybersecurity:			“The	facet	of	reliability	that	relates	to	the	degree	of	certainty	that	a	
				cyber	device	or	system	will	not	operate	incorrectly.”		
***	***	***	***	-2015	Firmware	Update	
Summary:	Corrected	an	issue	where	the	
meter	restarted	or	stopped	opera;ng	
during	file	transfers	in	the	presence	of	a	
saturated	network	
CVE-2013-****																																								
DNP3	vulnerability	causes a denial of
service (driver crash and process
restart) via a oddly crafted DNP3 TCP
packet.
	
State
Machines
Testing all states (known and unknown)
QA Challenge
Inputs, process, memory
Future	 18	
What will drive cybersecurity in the industry tomorrow?
Hopefully, engineering and best practices. Major
Hurdles
Hurdle 3: Viewing power system cybersecurity as only an IT issue for the IT dept.
Power System
Cybersecurity &
Cyber Resiliency
Operations
Design Implementation Maintenance Implementation
Cyber Risk Assessment
Requires input from power system
engineers and an understanding of
how each device is fundamentally
being used to control and/or monitor.
Examples
•  Testing changes prior to field
installations
•  Applying patches/updates in
energized systems
•  Does device support cyber feature?
•  Same logic and vendor software
used for relay P&C is used for cyber
•  Cybersecurity checkout &
commissioning
•  What is considered normal in the
control system application?
•  Real-time cyber-physical system
event modeling and contingency
analysis
Future	 19	
RISK	=	(Threat)	x	(Probability)	x	(Impact)	
Example:	Alter	soXware	on	Smart	Inverter		
Probability		
Impact	
Cyber	
Source:	h1p://acesolar.co.uk/services/solar-pv-panels/		
•  Cloud	based	management	
•  Code	on	device	to	detect	faults	
•  Regulates	energy	flows		
Commercial
or
Residential
•  Installed	in	untrusted	network	
•  Who	is	responsible	for	cybersecurity	
•  Security	issues	associated	with		
communicaAng	back	up	to	
connected	grid.
Future	 20	
RISK	=	(Threat)	x	(Probability)	x	(Impact)	
Example:	Spoofing	of	GPS	data	to	PMU	
~	
~	
Probability		
Impact	
Cyber	
?	
PMU		
A	
PMU		
A	θ1	 θ2	
θi
Future	 21	
RISK	=	(Threat)	x	(Probability)	x	(Impact)	
Example:	State	EsAmaAon	A1ack,	Spoofing	State	Variables	
Probability		
Impact	
Cyber	
?	
?	
Remote SCADA or Local Automation
Injected
Readings
Future	 22	
What will drive cybersecurity in the industry tomorrow?
Hopefully, engineering and best practices. Major
Hurdles
Hurdle 4: Documenting IED & System level cybersecurity capabilities
Power System
Cybersecurity &
Cyber Resiliency
Relay Design Engineer
Relay Settings Engineer
Cyber Design Engineer
Cyber Settings Engineer
•  Design protective relaying
functions based on operational
requirements and equipment ratings
•  Design protective cyber
functions based on operational
requirements and capabilities of devices
•  Programming of relaying and other
devices based on relay design specification
•  Programming of relaying and other
devices based on cyber design specification
Future	 23	
What will drive cybersecurity in the industry tomorrow?
Hopefully, engineering and best practices. Major
Hurdles
Hurdle 4: Documenting IED & System level cybersecurity capabilities
Power System
Cybersecurity &
Cyber Resiliency
Relay Design Engineer Cyber Design Engineer
Relay Operational One-Line Cyber Operational One-Line
ANSI /IEEE Standard C37.2 Standard for
Electrical Power System Device Function
Numbers, Acronyms, and Contact Designations
IEEE Standard ***** for Electrical Power System Cyber Device
Function Numbers, Acronyms, and Contact Designations
21 - Distance Relay
27 – Undervoltage Relay
32 – Directional Relay
50 – Instantaneous Relay
51 – AC Time Overcurrent Relay
52 – AC Circuit Breaker
59 – Overvoltage Relay
64 – Ground Detector Relay
87 – Differential Protective Relay
c48 – Firewall Type: Application Gateway
c49 – AAA Server
c50 – Role based access control
c51 – Report cyber events to master
c52 – Cyber-event concentrator (RTU)
c53 – Cyber-event converter, (e.g. DNP -> Syslog) (RTU)
c54 – Log cyber-events locally
c56 - Antivirus
c57 – Enable host firewall
c58 – Intrusion detection
c59 – Intrusion prevention
c60 – Web access
c61 – Application whitelisting
c62 – Email alerts
c63 – Network DoS detection
c90 – CPU & RAM Differential over Δt w/o protection event
c100 - Cyber lockout, revoke all remote control
Common
c64 – Network encryption
c65 – HD encryption
c66 – HTTPS
c67 – SSH
c68 – Telent
c69 – Active port detection
c70 – NSM/IDS with Protocol DPI
EXAMPLE
Maybe
one day…?
Future	 24	
What will drive cybersecurity in the industry tomorrow?
Hopefully, engineering and best practices. Major
Hurdles
Hurdle 4: Documenting IED & System level cybersecurity capabilities
Power System
Cybersecurity &
Cyber Resiliency
Cyber Design Engineer
Cyber Operational One-Line
c50 – Role based access control
c51 – Report cyber events to master
c52 – Cyber-event concentrator (RTU)
c53 – Cyber-event converter, (e.g. DNP -> Syslog) (RTU)
c54 – Log cyber-events locally
c56 - Antivirus
c57 – Enable host firewall
c58 – Intrusion detection
c59 – Intrusion prevention
c60 – Web access
c61 – Application whitelisting
c62 – Email alerts
c63 – Network DoS detection
c90 – CPU & RAM spike over Δt w/o protection event
c100 - Cyber lockout, revoke all remote control
Potential Benefits
•  Universally understood
•  Procurement: can the device do c**
•  (Scoping, Designing, Commissioning)
•  Multiple vendors, contractors, integrators
•  Maintenance
•  What devices require signature updates
•  Identify failed cyber component
•  Incident Response
•  What devices saw the event
•  What devices recorded the event
•  What devices were impacted
•  What device failed to alarm or take action
•  Saves time and money
•  Prevents extended operational downtime
c64 – Network encryption
c65 – HD encryption
c66 – HTTPS
c67 – SSH
c68 – Telent
c69 – Active port detection
c70 – NSM/IDS with Protocol DPI
25	
Safety	
Nathan	Wallace,	PhD,	CSSA	
n.wallace.us@ieee.org	
Twi1er:	@NathanSWallace	
Questions?
Thank You
Reliability

IEEE PES GM 2017 Cybersecurity Panel Talk

  • 1.
  • 2.
    Personal Background Volunteering: EE Intern EE Intern Associate Engineer Research Assistant Visi;ng Lecturer Staff Engineer Dir. Cyber Opera;ons Drafting Relay Settings T&D Protection Cybersecurity Researcher Digital Forensics Examiner Math & Engineering Relay & RTU Design & Commissioning Risk Assessments Cybersecurity Design & Integration 2
  • 3.
    3 Disclaimer •  Statements andopinions are my own which may or may not reflect that of my current employer. •  Statements are based on generalized observations of the industry and do not represent any particular entity or asset owner. •  Seek professional engineering assistance and vendor support prior to implementing or developing any of the capabilities discussed.
  • 4.
    4 RISK Two Infrastructures ResidenAal Industrial Commercial Genera;on Transmission Distribu;on • Physical •  Cyber Control Center DistribuAon Control Center RTOs/ISO
  • 5.
    5 Assets What devices need to be protected? CommunicaAon ComputaAonal What has and/or ability and is used to ensure the safe, reliable, and con,nuous genera;on, transmission and delivery of electricity? CybersecurityImplementation Challenge: How to maintain the Confidentiality, Integrity, and Availability of power system cyber assets? IEEE Std C37.240TM-2014, IEEE Standard Cybersecurity Requirements for Substation Automation, Protection, and Control Systems. •  Must not impede access and operation during legitimate activities. •  Must be technically, economically, and operationally feasible. Each protective measure
  • 6.
  • 7.
    7 Assets •  What hardware is common across all devices? Engr, Tech, Operator Imported Applica;ons Library Vendor Applica;ons PAM Access/Login HMI Apache iproute2 Firewall pyPAM HTTP •  Do vendors write their own code? •  What 3rd Party soOware libraries & applica;ons are being used? Linux Kernel 19.5 million lines of code FUN-FACT
  • 8.
    8 Power System Cybersecurity Implementa;on Who’s Responsibility is it? IT Dept. OT Dept. t -  SoXware to determine how power flows and when breakers open/closes - Apache, Telnet, SSH, MySQL, FTP, LDAP, Embedded Linux, Windows, etc. -  Virtual power plants and protecAon relays, soXware defined networking -  SCADA in the cloud Present
  • 9.
    9 OT Dept. VS => Adversarial Relationship Example1: Annual Funding IT Dept. IT Manager Engineering ManagerNew Cyber Compliance Manager & Dept. Legal Team, Training, Audit Specialists No Change Power System Cybersecurity Implementa;on Who’s Responsibility is it? More Personnel and Resources Present
  • 10.
    Present 10 Cri;cal Cyber Assets Compliance HIGH MEDIUM LOW 15% 85% Protected Grid Cyber Security PROTECTED GRID? Cyber Assets (For US 80-90% grid’s cyber assets are out of scope for NERC-CIP) Source: Cybersecurity and the Evolving Role of State Regulation: How it Impacts the California Public Utilities Commission, California’s PUC Policy Paper Cybersecurity Implementation
  • 11.
  • 12.
    12 IT Dept. OT Dept. VS => AdversarialRelationship Example 2: Implementation a) Securing laptops used by field personnel. Power System Cybersecurity Implementa;on Who’s Responsibility is it? Engineer “My company’s IT department has no idea I use this laptop… I wouldn’t be able to do my job if they did.” b) Securing edge devices (RTUs, relays, reclosers, etc.) Engineer Settings/Configurations Power P&C logic Cyber P&C logicIT Dept.? Present
  • 13.
    13 Present Cybersecurity Design Implementation Entity’s 1st Audit v3Audit revealed over half of the system’s firewalls were misconfigured. •  Typically the current approach is to use network firewalls and call it a day. •  Cybersecurity is an afterthought that ends up being “bolted on” only for compliance. 3 General Types of Firewall Packet Filtering | ApplicaAon-Proxy Gateway | Stateful InspecAon •  Some can be bypassed by spoofing network layer data •  All are based on soXware CVE-2016-**** The password-sync feature on [firewall vendor’s] switches sets an SNMP community to the same string as the administrator password, which allows remote attackers to obtain sensitive information by sniffing network.
  • 14.
    14 What drives cybersecurityin the industry today? Compliance What will drive cybersecurity in the industry tomorrow? Hopefully, engineering and best practices. Goal Cyber Security Present
  • 15.
    Future 15 Cyber Infrastructure (Computation& Communication) Protection and Control of the Modernized Grid Physical Infrastructure (Flow of Power) Inputs: Currents, Voltages, Impedance, Status (open,close, lockout) Output: Open/Close Bkr, +/- Vars, Inputs: Topology, traffic flows, deep packet inspection, communication state, state of physical power system Output: NOTHING!
  • 16.
    Future 16 What willdrive cybersecurity in the industry tomorrow? Hopefully, engineering and best practices. American Engineers' Council for Professional Development defines Engineering as: "The creative application of scientific principles to design or develop …." Major Hurdles Power System Cybersecurity & Cyber Resiliency Hurdle 1: Labeling of everything as Restricted, Classified, or Sensitive Requires Verifiable Evidence & Repeatable Tests Administrator 1.  Joe 2.  Alice Example Compliance/Legal depts. stops engineer from discussing what works and what doesn’t at technical industry conference. Negative Side Effects •  Industry slow to advance and therefore slow to defend. •  Engineers are not aware of solutions/approaches resulting in the assumption that security is not feasible. •  Approach is really security through obscurity.
  • 17.
    Future 17 What willdrive cybersecurity in the industry tomorrow? Hopefully, engineering and best practices. Major Hurdles Hurdle 2: Viewing cybersecurity as only defending against malicious actors. Power System Cybersecurity & Cyber Resiliency Recall Cybersecurity: “The facet of reliability that relates to the degree of certainty that a cyber device or system will not operate incorrectly.” *** *** *** *** -2015 Firmware Update Summary: Corrected an issue where the meter restarted or stopped opera;ng during file transfers in the presence of a saturated network CVE-2013-**** DNP3 vulnerability causes a denial of service (driver crash and process restart) via a oddly crafted DNP3 TCP packet. State Machines Testing all states (known and unknown) QA Challenge Inputs, process, memory
  • 18.
    Future 18 What willdrive cybersecurity in the industry tomorrow? Hopefully, engineering and best practices. Major Hurdles Hurdle 3: Viewing power system cybersecurity as only an IT issue for the IT dept. Power System Cybersecurity & Cyber Resiliency Operations Design Implementation Maintenance Implementation Cyber Risk Assessment Requires input from power system engineers and an understanding of how each device is fundamentally being used to control and/or monitor. Examples •  Testing changes prior to field installations •  Applying patches/updates in energized systems •  Does device support cyber feature? •  Same logic and vendor software used for relay P&C is used for cyber •  Cybersecurity checkout & commissioning •  What is considered normal in the control system application? •  Real-time cyber-physical system event modeling and contingency analysis
  • 19.
    Future 19 RISK = (Threat) x (Probability) x (Impact) Example: Alter soXware on Smart Inverter Probability Impact Cyber Source: h1p://acesolar.co.uk/services/solar-pv-panels/ •  Cloud based management • Code on device to detect faults •  Regulates energy flows Commercial or Residential •  Installed in untrusted network •  Who is responsible for cybersecurity •  Security issues associated with communicaAng back up to connected grid.
  • 20.
  • 21.
  • 22.
    Future 22 What willdrive cybersecurity in the industry tomorrow? Hopefully, engineering and best practices. Major Hurdles Hurdle 4: Documenting IED & System level cybersecurity capabilities Power System Cybersecurity & Cyber Resiliency Relay Design Engineer Relay Settings Engineer Cyber Design Engineer Cyber Settings Engineer •  Design protective relaying functions based on operational requirements and equipment ratings •  Design protective cyber functions based on operational requirements and capabilities of devices •  Programming of relaying and other devices based on relay design specification •  Programming of relaying and other devices based on cyber design specification
  • 23.
    Future 23 What willdrive cybersecurity in the industry tomorrow? Hopefully, engineering and best practices. Major Hurdles Hurdle 4: Documenting IED & System level cybersecurity capabilities Power System Cybersecurity & Cyber Resiliency Relay Design Engineer Cyber Design Engineer Relay Operational One-Line Cyber Operational One-Line ANSI /IEEE Standard C37.2 Standard for Electrical Power System Device Function Numbers, Acronyms, and Contact Designations IEEE Standard ***** for Electrical Power System Cyber Device Function Numbers, Acronyms, and Contact Designations 21 - Distance Relay 27 – Undervoltage Relay 32 – Directional Relay 50 – Instantaneous Relay 51 – AC Time Overcurrent Relay 52 – AC Circuit Breaker 59 – Overvoltage Relay 64 – Ground Detector Relay 87 – Differential Protective Relay c48 – Firewall Type: Application Gateway c49 – AAA Server c50 – Role based access control c51 – Report cyber events to master c52 – Cyber-event concentrator (RTU) c53 – Cyber-event converter, (e.g. DNP -> Syslog) (RTU) c54 – Log cyber-events locally c56 - Antivirus c57 – Enable host firewall c58 – Intrusion detection c59 – Intrusion prevention c60 – Web access c61 – Application whitelisting c62 – Email alerts c63 – Network DoS detection c90 – CPU & RAM Differential over Δt w/o protection event c100 - Cyber lockout, revoke all remote control Common c64 – Network encryption c65 – HD encryption c66 – HTTPS c67 – SSH c68 – Telent c69 – Active port detection c70 – NSM/IDS with Protocol DPI EXAMPLE Maybe one day…?
  • 24.
    Future 24 What willdrive cybersecurity in the industry tomorrow? Hopefully, engineering and best practices. Major Hurdles Hurdle 4: Documenting IED & System level cybersecurity capabilities Power System Cybersecurity & Cyber Resiliency Cyber Design Engineer Cyber Operational One-Line c50 – Role based access control c51 – Report cyber events to master c52 – Cyber-event concentrator (RTU) c53 – Cyber-event converter, (e.g. DNP -> Syslog) (RTU) c54 – Log cyber-events locally c56 - Antivirus c57 – Enable host firewall c58 – Intrusion detection c59 – Intrusion prevention c60 – Web access c61 – Application whitelisting c62 – Email alerts c63 – Network DoS detection c90 – CPU & RAM spike over Δt w/o protection event c100 - Cyber lockout, revoke all remote control Potential Benefits •  Universally understood •  Procurement: can the device do c** •  (Scoping, Designing, Commissioning) •  Multiple vendors, contractors, integrators •  Maintenance •  What devices require signature updates •  Identify failed cyber component •  Incident Response •  What devices saw the event •  What devices recorded the event •  What devices were impacted •  What device failed to alarm or take action •  Saves time and money •  Prevents extended operational downtime c64 – Network encryption c65 – HD encryption c66 – HTTPS c67 – SSH c68 – Telent c69 – Active port detection c70 – NSM/IDS with Protocol DPI
  • 25.