Tools and Techniques to Identify Threats to ICS Communications

1. Standards
Certification
Education & Training
Publishing
Conferences & Exhibits
Looking for Trouble on
OT Networks
Tools and Techniques to Identify
Threats to ICS Communications
2015 ISA Water / Wastewater and Automatic Controls Symposium
August 4-6, 2015 – Orlando, Florida, USA
Speakers:
Bryan Singer, CISSP, CAP
Kenexis

2. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA 2
Presenter
• Bryan L Singer, CISSP, CAP
– Principal Investigator, Kenexis
– Previous Chairman ISA-99 Industrial Automation and
Control Systems Security
– Experienced in cyber forensic investigations, network
architecture and design, security assessments, software
coding, penetration testing, vulnerability research,
malware research, and system commissioning
– Co-Author: Cybersecurity for Industrial Control
Systems: SCADA, DCS, PLC, HMI, and SIS ISBN:
1439801967
• Email: bryan.singer@kenexis.com
• Twitter: @BryanLSinger

3. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA 3
Presentation Outline
• Defining “Trouble” - The challenges to Understanding OT
Networks
• Looking for Trouble - Methods of Capture
• Removing Trouble - Analysis of OT Networks
– Performance Analysis
– Network Security Monitoring (NSM)

4. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
DEFINING TROUBLE
Challenges to Understanding OT Networks
4

5. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Greatest Hits
• In Nearly 20 Years of Industrial Networking:
5
Problem Network Cause
18 Month Delayed Startup and
$10k/week
Traced to >50% of fiber terminated
improperly and poor switch design
Catastrophic failure of batch process
in pharma
Excessive latency and jitter due to
consumer grade network switches
susceptible to electric noise
$8 Million product loss in engine plant Active virus on an open network
“Air Gapped” network in
entertainment (ride) industrial critical
failure resulted in system rebuild
Maintenance laptop introduced virus
to network
Safety Incident on Engine Production
Line
Saturated network caused
communication error
Network Problems Often Cause Unseen Process
Failures

6. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
• Intermittent Failures
• Nuisance Trips
• Stale HMI Data
• Missing HMI Data
• Errant behavior on HMI
• Reports from IT of “errors”
• Loss of remote communications
Symptoms of OT Network Problems

7. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Understanding IT versus OT Network
Performance
• IT Networks: Bandwidth, Port
Density, and Performance of
Core and Distribution
Switches
• OT Networks: All about
Latency and Jitter across
Distribution and Access
Switches
• IT Networks: Short TCP
Session times
• OT Networks: Long TCP
sessions and as many as 19
different TCP sessions for a
single process instruction
7

8. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
OT Network Performance
Considerations
8
Network Layer Consideration
Physical Layer Media Integrity, electrical or RF integrity,
collision prevention
Network Performance Layer Bandwidth, Network Protocol
Distribution, proper switch capacity and
design
Communications Layer Performance of the industrial protocols
themselves
Security Layer Ability to detect, report, and respond to
security events
Every Plant Should Have Appliances, Software, and other Tools to Manage Network
performance at each layer!

9. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
LOOKING FOR TROUBLE
9
I Have a Particular Set of Skills…
I will hunt you down, and I will find
you
I Have a Particular Set of Skills…
I will hunt you down, and I will find
you

10. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Successful Management of OT
Networks
• What data to capture – online analysis with COTS tools
and manual analysis with PCAP data
• How to analyze and interpret the data – Expert Info
tools, knowing where to capture
• What to do with the data – Compare to Process
Historians, maintenance records, or manual observations
10

11. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Types of Monitoring
• Passive Monitoring
– PCAP
– Wireshark, TCPDump
– Network Miner
• Active Monitoring
– IDS/HIDS/NIDS such as Snort, Fireeye, etc
– NSM
– Netflow, Syslog, SolarWinds, etc
11

12. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Physical Layer
• Validate Media
• Protect physical media and cable runs
• Test physical media (cable testers, Fluke meters, etc.)
12

13. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Network Layer Resources
• Syslog
• Commercial Tools
– IntraVue
– SolarWinds
• Open Source
– Wireshark and PCAP Analysis
13

14. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Network Layer: Protocol Distribution
May
18,
14
• From Wireshark Expert Info toosl

15. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Network Layer: Conversations
May
18,
15
• From Wireshark Conversations Tab
• Show top talkers and helps identify
problem devices

16. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Network Layer: IO Graph
• Wireshark IO Graphs
• Show performance of communications
over time
• Helps show dropouts
May
18,
16

17. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Tools for the Network Layer
• Wireshark PCAP
– Conversations Tab and Protocol dissectors
• Vendor specific tools
• Manual Analysis of PCAP Data
• Emerging tools for OT protocol performance
17

18. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Protocol Layer
• This is the hard one, and rarely is it managed well
• Many networks have been seen to be sound from an IT
perspective, yet don’t work for industrial protocols
• Typical failures come from not following vendor
recommended network design or excessive network
latency / jitter due to design limitations
• Successful analysis of this layer requires protocol
specific knowledge (EthernetIP, MODBUS, etc)
• SolarWinds or similar solutions indicate network stack up
or down via IGMP, but to analyze protocol performance,
need to look at protocol itself
18

19. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Protocol Layer
• Quick succession of command/response packets
• Minimal delay in command/response sequence
• Apparently large delay in a single packet
• Example: Rockwell tag reads
19
Quick Succession Read Commands
Delay Until Next Time Sequence

20. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Protocol Layer
20
Test Time (s)
Measured Packet Interval (ms)
~62 sec test
Mean MPI = 2ms
Min ~ 1.2
Max ~ 2.9

21. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Security Layer
• Tools for Watching the Security Layer
– Intrusion Detection Systems
– Snort, Bro-IDS, Suricata, FireEye
– Get familiar with Security Onion
– Manual Analysis using Wireshark (no magic at the packet layer)
– Network Security Monitoring (NSM) such as FireEye
• Each facility should have at least IDS at the OT
border/DMZ
21

22. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Security Layer: Intrusion Detection

23. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Security Layer: IDS

24. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
REMOVING TROUBLE
24

25. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
What to Do with the Data
• Correlate network events to process events where
possible
• Replace and reroute physical media as necessary
• Review ladder logic, test media, and further analyze
network traffic where protocol issues exist
• Utilize forensic tools to remove security threats
25

26. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Key OT Network Design Considerations
• Utilize Layer 3 Management on Core and Distribution
layers as much as possible
• Establish recovery point objectives to ensure that
network availability is in line with process availability –
never have a 5 hour network support a 5 minute
process!
• Utilize IDS/HIDS/NIDS at core and distribution layers
where possible
26

27. 2015 ISA WWAC Symposium
Aug 4-6, 2015 – Orlando, Florida, USA
Key Takeaways / Summary
• Every plant should have tools to:
– Diagnose Physical Network Media
– Gather network PCAP and monitor network health (Wireshark,
SolarWinds, etc)
– Ability to monitor industrial protocol specific performance
– IDS/HIDS/NIDS or other NSM to monitor for security events
• Bryan L Singer, CISSP, CAP
– Bryan.singer@Kenexis.com
– Twitter: @BryanLSinger
27