1.
Master Serial Killer
Chris Sistrunk PE, Mandiant
Adam Crain, Automatak

2.
About Us
Chris Sistrunk, PE
• Electrical Engineer
• SCADA Expert
• Loves Security
• DNP3 Member
• Button Pusher
Adam Crain
• Software Engineer
• OSS Advocate
• openDNP3 Author
• DNP3 Member
• Code Monkey

3.
How I Audit SCADA systems
http://securityreactions.tumblr.com/post/30866100673/how-i-audit-scada-systems

4.
ICS/SCADA Security
• ICS/SCADA lags IT by 10-15 years
• 708 SCADA-related vulns on OSVDB.org
since 2011. “Like kicking a puppy”
• Positive vs. Negative Testing: The front
yard is mowed, but the back yard is
overgrown.

5.
Software Testing

6.
When you scan ICS with nmap

7.
SCADA Protocol Vuln Research
We chose to focus on popular SCADA protocols
Fuzzers did exist, but only tested server side
Serial had not been fuzzed before (that we know of)
We chose to use Responsible Disclosure
• Inform the vendor, then ICS-CERT, DNP3 UG
• Worked with the vendor to help them replicate and
begin further negative testing

8.
Project Robus
• Latin for “bulwark”
• Started in April 2013
• 24 advisories / 30 tickets
• 22 DNP3, 1 Modbus,
1 Telegyr 8979
www.automatak.com/robus
www.automatak.com/aegis

9.
Fuzzing Master Stations
• Referenced in Nat’l SCADA Test Bed
reports but no data available
• Wurldtech & Spirent (Mu Dynamics) don’t
fuzz the master side of ICS
protocols…………..yet
Master Slave

10.
Fuzzing Master Stations
DNP3 Application Function Code 0x82
• If the Master Station has Unsol enabled, it must accept
messages from its RTUs at any time
• Design of System must be fine tuned...or else
DNP3 Outstation Unsolicited Response Storm
• If the Master parser has problem with one message,
you can imagine the problems with many many
messages

11.
Serial Fuzzing
All the security focus has been on ethernet networks, but
many ICS, especially SCADA, still utilize serial networks.
• DNP3 is same! (unlike Modbus)
• Impact to NERC/CIP v3 & v5
Physical Security (discuss later)
• Pole-mounted RTUs
• PQ Meters, etc

12.
DNP3 (IEEE 1815-2012) Primer
DNP3 is a SCADA protocol used by almost all of the
electric utilities and some water in North America,
Australia, and the UK.
Created in 1990s and turned over to DNP3 UG in 1993.
One of the few ICS protocols that has secure auth.
SCADA
Master
RTU with I/O

13.
Breaking Down DNP3
TCP 20000
TCP 19999 (TLS)
UDP 20000
Ref from IEEE Std 1815-2012

14.
Courtesy of

15.
Vendor Response Matrix
ICS-CERT Adv Company Protocol Bug Fix Days Advisory
ICSA-13-161-01 IOServer DNP3 4/24 5/24 30 6/10/2013
ICSA-13-213-03 IOServer DNP3 5/1 7/20 80 8/1/2013
ICSA-13-219-01 SEL DNP3 5/1 5/30 29 8/7/2013
ICSA-13-226-01 Kepware DNP3 4/24 6/18 55 8/14/2013
ICSA-13-234-02 TOP Server DNP3 4/24 6/18 55 8/22/2013
ICSA-13-240-01 TMW DNP3 4/24 6/17 54 8/28/2013
ICSA-13-213-04A Matrikon DNP3 4/24 6/17 54 8/29/2013
ICSA-13-252-01 Subnet DNP3 4/24 8/30 128 9/9/2013
ICSA-13-282-01 Alstom DNP3 4/24 6/4 41 10/21/2013
ICSA-13-297-01 Catapult DNP3 4/24 10/1 160 11/22/2013
ICSA-13-297-02 GE IP DNP3 Self Report 10/1 n/a 11/22/2013
ICSA-13-337-01 Elecsys DNP3 9/12 11/4 53 12/3/2013
ICSA-13-346-02 Cooper OPC DNP3 7/31 None ∞day™ 12/12/2013
ICSA-13-346-01 Cooper/Cybectec DNP3 5/1 12/12 225 12/12/2013
ICSA-13-352-01 Novatech DNP3 5/1 9/5 127 12/18/2013
ICSA-14-014-01 Schneider DNP3 8/6 8/23 17 1/14/2014
ICSA-14-100-01 IOServer Modbus 2/6 3/4 26 4/10/2014
ICSA-14-154-01 COPA-DATA DNP3 Self Report using Aegis! n/a 6/3/2014
ICSA-14-196-01 Subnet TG8979 4/18 6/18 61 7/31/2014

16.
Vendor Response
• Most of the vendors were very pleased
• A few were not >> head in the sand
• Some had never done negative testing
• Nearly all devices and hosts with DNP3
were affected, so it was an industry-wide
wakeup call.

17.
White Noise Fuzzing
#1 random == really “dumb”

18.
Template (mutational) Fuzzing

19.
Generational “Smart” Fuzzing

20.
Multi-field Anomalies

21.
Generational == most vulns!

22.
The Aegis ICS Fuzzing Framework
• We decided that we needed to release our
fuzzing framework tool as open source.
• Open source security tools have a proven
track record of raising security (hello MSF!)
• We do encourage people to join our efforts
to add more protocols to Aegis

23.
Aegis Specifics
• Version 0.1.x in Scala www.scala-lang.org
• Current version (private release) in C#
• Protocol boundary conditions
• Abstracts physical layer
• Combines aspects of generation and mutation
• Repeatable random seeds
• ~500,000 test cases with one seed

24.
Test DNP3 Message (DL, TL, or AL)
Request Link States
Link Status
x Num Test Cases
Request
Response
x Num Retry (10)
Fuzzer Test Flow

25.
I 0x0564 U...
Y U NO 0x0564
ME BAK ?!

26.
Combinatorics
val nums = List(1, 3)
val colors = List(“red”,”green”)
// repeat the reversed string num times
def combine(i: Int, s: String) = List.fill(i)(s.reverse).mkString
val result = Cartesian.Transform(colors,nums)(combine)
What is result?

27.
Lazy Generator
// val nums = List(1, 3)
// val colors = List(“red”,”green”)
> result.foreach(println)
der
derderder
neerg
neergneergneerg

28.
{ frames } = f (byte,Type)
{byte} = f (bool, bool, int) {Type} = f (.....)
{ true, false } { true, false } { 0, 1, 63 }
...........................
Fuzzing is O(2n)

29.
Generators can get large!
{ test cases } ● Many function codes
● Many objects
● Header types
● Many field values

30.
Types of Vulnerabilities

31.
FA 82 00 00 01 00 02 00 00 00 00 FF FF FF FF
Unsolicited
Response
Group 1
Variation 0
Sizeless?!
4 byte
start/stop
0 4294967295
● infinite loop
● missing data
● integer overflow?
● accepts broadcast
Vuln #1

32.
DD 82 00 00 0A 02 01 00 00 FF FF
UNSOL
Group 10
Variation 2
Binary
Output
Status
2 byte
start/stop
0 65535
● infinite loop
● missing data
● unexpected data
● integer overflow?
Vuln #2

33.
05 64 06 44 64 00 64 00 FF F2 C0 1D 0A
1 byte
payload
100 100
● transport header only
● unhandled exception
unconfirmed
user data
CRC CRC
FIR / FIN
SEQ = 0
Vuln #3

34.
Recorded Demos
Video 1: a DNP3 outstation
-application layer object fuzzing
Video 2: a DNP3 master
-unsolicited application layer fuzzing

35.
Vuln #4 (TMW integration)
DD 82 00 00 0C 01 00 00 01 rnd(11) rnd(11)
Unsolicited
Response
Control
Relay
Output Block
1 byte
start/stop
CROB #1 CROB #2
● buffer overrun
● not malformed!
● unexpected objects
● accepts broadcast

36.
Vuln #5 (TMW integration)
FA 82 00 00 02 02 01 01 00 FF FF
Unsolicited
Response
Group 2
Var 2
(event)
2 byte
start/stop
1 65535
● stable infinite loop
● max range - 1 and no data
● accepts broadcast

37.
Using Aegis

38.
So easy…Twitter can do it

39.
Examples
Run 10 link layer test cases starting at #123
$ aegis-console -mid dnp3 -pid lfuzz -start 123 -count 10
Unsolicited response fuzzing of a master listening on default port 20000 with master address of 0
and an outstation address of 1
$ aegis-console -mid dnp3 -pid aufuzz -dest 0 -src 1 -master -listen
Outstation link layer fuzzing test case #100 only
$ aegis-console -mid dnp3 -pid lfuzz -start 100 -count 1
Outstation application object fuzzing against 192.168.1.55:20001 with default addressing
$ aegis-console -mid dnp3 -id aofuzz -host 192.168.1.55 -port 20001

40.
Further Aegis Development
• In addition to DNP3 protocol, we’ve added
Modbus and Telegyr 8979 (serial only)
protocol modules to the framework.
• Migrated from scala to C#.
• Added a GUI
• Working with vendors and other trusted
researchers.

41.
New Aegis Demo
--- module: dnp3 - Test routines for the DNP3 protocol ---
Procedure ids:
link Fuzzing of the link layer (masters or outstations)
transport Fuzzing of the transport function (masters or outstations)
requests Fuzzes the application layer with malformed and unexpected requests (outstation)
unsol Fuzzes the application layer with malformed and unexpected unsolicited responses (master)
octetunsol Reports large numbers of 0-length octet string headers via unsolicited mode (master)
octetwrite Writes large numbers of 0-length octet string headers (outstation)
randrequest Fuzzes the application layer with semi-random requests (outstation)
randunsol Fuzzes the application layer with semi-random unsolicited responses (master)
-dest <arg>(1024)[0, 65535] link layer address of the target
-src <arg>(1)[0, 65535] link layer address of the fuzzer
-master <arg>(False) set the link-layer master bit for master fuzzing
-retries <arg>(10)[1, none] Number of link status retries
-timeout <arg>(1000)[10, none] Read timeout in milliseconds
-health <arg>(LinkStatus) Type of health check to use [linkstatus, resetlink]
--- module: modbus - Test routines for the Modbus protocol ---
Procedure ids:
request Sends malformed or unexpected requests at a Modbus slave

42.
ICS/SCADA Defense

43.
ICS/SCADA Defense
Network Security Monitoring (do it now!)
• Bro, SNORT, Wireshark can parse DNP3 & Modbus!
• Deep packet inspection firewalls
• Full packet capture (even serial) – 1TB y’all
• Use Security Onion to monitor ICS networks
http://www.liquidmatrix.org/blog/2014/07/01/is-there-a-cuckoo-in-your-control-system/
tl;dr ≥1 person, Security Onion, and an ICS Honeypot

44.
ICS/SCADA Defense
• Install patches – not quite like IT
• Robust device & network configuration
• Disable unused ports, protocol function codes
• Whitelist apps and even traffic
• DNP3 Secure Authentication v5 & TLS
• Signed software/firmware

45.
Physical Security
3/8” Mesh
ASTM Grade 6
Buys extra time

46.
What’s different about Robus?
SCADA Vulns reported for a while now
Adam and I aren’t security researchers
• He’s a software geek…I’m an engineer
• Our skills complemented each other
• Both experts in DNP3 protocol, but from
different angles

47.
Some theories
Why did the industry move instead of ignore?
• I was an end user and we really cared!
• Not just a wham-bam researcher
• Respectful, tactful, responsible
• We released our tool
…………we weren’t going away

48.
I’m still more worried about…

49.
SHODAN
Probably default configs
• Many similar responses
• Same DNP Addresses
python shell
>>> " ".join("%02x" % ord(i) for
i in "DNP3 paste from shodan”)
Unsolicited Response with
Binary and Analog Data
Class 1/2/3/0 Poll!!!
https://ics-radar.shodan.io/
https://maps.shodan.io/

50.
Conclusions
• DNP3 is not a special case, other protocols same fate
Modbus, IEC 60870, IEC 61850, ICCP, EtherNet/IP…
• Early testing both slave/server AND master/client sides
of protocols are important!
• Compliance != Security, but the culture is important
• Don’t have to be a nation/state or large firm to do this
• A few good folks can make a difference in the industry

51.
Questions?
@jadamcrain
@chrissistrunk