Downloaded 74 times
![Contains records with file description
struct PCK_file_record
{
char Name[252];
DWORD CRC;
DWORD Size;
}
PCK file format](https://image.slidesharecdn.com/hopelessrelayprotectionforsubstationautomationfin-171015042634/75/Recon-Hopeless-relay-protection-for-substation-automation-14-2048.jpg)
![KON file is set of tagged records with different types. Structure of the record header:
struct KON_section_header
{
char SectionTypeName[4];
DWORD size;
}
struct KON_file_header
{
char Signature[4];
KON_section_header Sections[];
}
In the present case we had the following types of records:
• “HEAD” (char code_type[4]; DWORD minaddr; DWORD maxaddr; DWORD entry_point ; DWORD xorcks)
• “INFO” (char unit[8]; char device[8]; char version[15]; BYTE number)
• "TITL“ (char title[] )
• "DATC“ (DWORD datc_start_addr; DWORD datc_size; DWORD crc)
• "DATA“ (DWORD start_addr; DWORD datca_size; DWORD crc)
• "ENDE“
• https://github.com/rigmar/Recon2017/tree/master/SIPROTEC
KON file format](https://image.slidesharecdn.com/hopelessrelayprotectionforsubstationautomationfin-171015042634/75/Recon-Hopeless-relay-protection-for-substation-automation-16-2048.jpg)
More Related Content
Similar to Recon: Hopeless relay protection for substation automation
![[CB20] Reverse Engineering archeology : Reverse engineering multiple devices ...](https://cdn.slidesharecdn.com/ss_thumbnails/shlomiobermanandmoshekolcb20-210112133843-thumbnail.jpg?width=640&height=640&fit=bounds)
![Eloi Sanfélix y Javier Moreno - Hardware hacking on your couch [RootedCON 2012]](https://cdn.slidesharecdn.com/ss_thumbnails/esanfelix-jmoreno-120324094426-phpapp02-thumbnail.jpg?width=640&height=640&fit=bounds)
![Too soft[ware defined] networks SD-Wan vulnerability assessment](https://cdn.slidesharecdn.com/ss_thumbnails/us-18-gordeychik-too-software-defined-networks-sd-wan-vulnerability-assessment-180808175551-thumbnail.jpg?width=640&height=640&fit=bounds)
![Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]](https://cdn.slidesharecdn.com/ss_thumbnails/codedagentsdeck-251215155422-5497c599-thumbnail.jpg?width=640&height=640&fit=bounds)
Recon: Hopeless relay protection for substation automation
- 1. Hopeless Relay Protectionfor Substation Automation Kirill Nesterov @k_v_nesterov Alexander Tlyapov
- 2. @scadasl Opinions are myown and not the views of my employer
- 3.
- 4.
- 5. Substation in anutshell • Primary system devices • Circuit breakers, Disconnect and grounding switches, Power transformers, Instrument transformers, Generators • Secondary system devices • Protection, Reclosers, Annunciators, Meters, sensors, Fault recorders, Control switches and interfaces • Computers are here! https://en.wikipedia.org/wiki/Electrical_substation
- 6. Substation in anutshell • Primary system devices • Circuit breakers, Disconnect and grounding switches, Power transformers, Instrument transformers, Generators • Secondary system devices • Protection, Reclosers, Annunciators, Meters, sensors, Fault recorders, Control switches and interfaces • Computers are here! https://en.wikipedia.org/wiki/Electrical_substation
- 7.
- 8. Substation in anutshell • Everything is in IEC 61850 • Set of protocols (GOOSE, MMS, SV, etc.) • Dafaq Substation Configuration Language (SCL) • Digital Protective Relay (also IEDs) • Network bacchanalia • Interconnections with substation • System operator, Billing, Transmission support • Ethernet, Power Line Communications (PLC)
- 9. Security of substations •IEC 61850 • tldr; No security • Exploiting the GOOSE Protocol: A Practical Attack on Cyber-infrastructure by Juan Hoyos, Mark Dehus, Timthy X Brown • Poisoned GOOSE: Exploiting the GOOSE Protocol http://crpit.com/confpapers/CRPITV149Kush.pdf • IEC 62351 • tldr; use No security via SSH tunnel • Set of words to encapsulate everything from IEC 61850 in encryption • Haha, you know, distribution owners update and vendors provide updates
- 10. Antiviolence reminder: transformers andgeoshmalitics • Santa Barbabararaba is in another universe • We are not electrical engineers and that is not the point of the talk • Yes, we heard that transformer is not like Optimus Prime • They just didn’t saw them transforming • While colors don’t match, Eleron gas source is planet Cybertron! • If you want bash us for electrical misanything - just call your therapist ?
- 11. Generic Relay TerminalInternals • PowerPC (MPC860) • RTOS • Protocols • IEC61850 (MMSLite) • Proprietary protocol for updates • Optional Web • Poor debug facilities • Today’s menu • En salada la Switzerland, Germany, France, USA
- 12.
- 13. The software isdivided into two main parts: • Common firmware (bootloader, RTOS pSOS+ code, …) • Modules that implement additional protocols (IEC61850, DNP3, Modbus, …) The firmware is available as a file with the extension ".PCK" included with the application for the installation - FIRMWAREUPDATE.EXE PCK File is a container with .KON files, xml with update options and soon Target device – SIPROTEC 7SJ64x
- 14. Contains records withfile description struct PCK_file_record { char Name[252]; DWORD CRC; DWORD Size; } PCK file format
- 15. Files in 7SJ64X_04.93.01.PCK Codestored in KON files. One PCK file may contain KON files for different CPU. In this example we have bootloader variants for CCPU and 384K. PCK file format
- 16. KON file isset of tagged records with different types. Structure of the record header: struct KON_section_header { char SectionTypeName[4]; DWORD size; } struct KON_file_header { char Signature[4]; KON_section_header Sections[]; } In the present case we had the following types of records: • “HEAD” (char code_type[4]; DWORD minaddr; DWORD maxaddr; DWORD entry_point ; DWORD xorcks) • “INFO” (char unit[8]; char device[8]; char version[15]; BYTE number) • "TITL“ (char title[] ) • "DATC“ (DWORD datc_start_addr; DWORD datc_size; DWORD crc) • "DATA“ (DWORD start_addr; DWORD datca_size; DWORD crc) • "ENDE“ • https://github.com/rigmar/Recon2017/tree/master/SIPROTEC KON file format
- 17. Section type name Sectionsize File type signature TITL section body INFO section body HEAD section body datc_start_addr Entry point addr File header of “bootld_C_V2.kon” from firmware: So, we know bootloader start address (0x40000) and entry point (0x40004). Trying to load in IDA PRO and see some problem: • Part of code has different than 0x40000 base address • ANOTHER part of code has base address that is different from the previous two KON file format
- 18. bootloader code 0x40000 0x66700 Firstpart 0x40750 0xFFF00000 Second part 0x53078 0xFFF1292C 0xFFF3361C0xFFF20000 Third part 0xFFF02C30 0x340000 0x34FCA0 Fourth part Hardcoded KON file0x2000 0xFFF25294 0x20948 Finally bootloader code map memory view 0x2000 - 0x20948 0x40000 - 0x66700 0x340000 – 034FCA0 0xFFF00000 – 0xFFF1292C 0xFFF20000 – 0xFFF3361C Nested Doll - Matreshka from Germany
- 19. Architecture and OSof device • pSOS+/PPC V2.0.7 • Upgrade with Ethernet module EN100 • Same CPU • Same OS • TCP/IP communication • Port forwarding
- 20. • HTTP (80/tcp) •Diagnostics and bonus features! • DIGSI (5000x/tcp) • Proprietary engineering protocol • Java Applet Remote Managing protocol (56797/udp) • Diagnostics • IEC61850 MMS (102/tcp) and GOOSE • Industrial process Services
- 21. Web Server • It’salways a good idea to wright your own
- 22. How to secureyour web? • Password of course! • CVE-2016-7112
- 23. Is your websecure? • strstr “taskname” • scanf “%d” into stack • No canary • What could go wrong?
- 24. Complicated auth • /fehler– error log URL -> • Very convenient • Looks promising/pwnable • PC = 0x41414140 • Network buffers looks RWX
- 25. Complicated CVE • CVE-2016-7113 •CVSS v3.0 Base Score 5.3
- 26.
- 27. sscanf problem • Nearlyevery call is vulnerable
- 28. Java applet High-voltage BayController Unit
- 29. Java applet • CPUservice • Some proprietary 56797/udp protocol • Some diagnostics • Some password check on user-side • But it’s not hardcoded • It’s confirmation code 311299 • Some read out of bounds => crash
- 30. Crash >= RCE •Defective mode • Can be fixed only with manual reboot • No protection • Terminal runs in “Monitor mode” • Tested with RETOM device • True for core CPU bugs Relay protection and automation testing system
- 31. D/F60 Feeder ProtectionSystem КЗХ
- 32. The firmware isavailable as a file with the extension “.bin”. For example – “A09ma604.000.bin”. At offset 0x100 starts loader code: First instruction – “ba 0xFFB10104”, therefore, the base address of firmware is a 0xFFB10104. Target device - F60
- 33. Loader copies 0x1D4F8bytes from 0xFFB10270 to 0x1F80000. F60 Firmware unpacking (1)
- 34. This bytes containszlib uncompress code that unpack main firmware code from 0xFFB2D768 to 0x8000. F60 Firmware unpacking (2)
- 35. If uncompressing isfinished successfully, code at offset 0x1F80000 is cleared F60 Firmware unpacking (3)
- 36. Finally, control ispassed at offset 0x8100 in the uncompressed code. F60 Firmware unpacking (4)
- 37. Global Device Objects •Thousands of them • Backed up by EEPROM • Inheritance level ~ 3 • Strongly typed => Unified access
- 38. This values hasdb based view, that initialized using hardcoded value descriptions. Example of such object
- 39. At start wewere analyzing firmware version 6.04. So, on vendor’s website has newer one. New in firmware v. 7.31: • VxWorks 6.8 • And that has VxWorks symbols! Well, knowledge of the names of functions and global variables really doing life better Sometimes, the new version is really better
- 40.
- 41. Simple web service •Very simple • No user interaction ЭЙЦЖЫЫ
- 42. Modbus • Authorization • Differentmodes • Password is a 32 bit number or username with password • Bruteforce protection • R/W Access control • Old Enervista protocol
- 43. New Modbus • NewEnervista protocol • SSH tunnel • MocanaSSH
- 44.
- 45. Secure CyberSecurity • Noresponse • Reported 26 Jul 2016 • Got 4 potential RCE
- 46. No demo • Nodebugger • No crash dump • No JTAG • No UART • Nothing at all
- 47. REF630 “DB based” FTP – full access to flash HTTP IEC 61850 ODBC ЪЮЙ
- 48. Comfortable terminal VxWorks PowerPC FS access VxWorks img is ELF Symbols Traceback with PC and LR And something more...
- 49. Comfortable terminal debugsrv 7755/tcp – stdout with additionals headers 7766/tcp – stdin Can be switched in boot • VxWorks console • Internal debugger • Arbitrary calls by name and by address • Many more
- 50. Ref630 DB based •All data in DB that is stored on file system • Database files are divided into three types: • Basic – plain data, no encryption and compression • Sequential – compressed data blocks • Secure sequential – compressed and encrypted data blocks
- 51. Ref630 Encrypted DBfiles • Blowfish algorithm • Encryption key depended on interfaces IP addresses
- 52. Ref630 Encrypted DBfiles • Two interfaces • Loopback with IP address 127.0.0.1 • Common with external IP address • Hardcoded string
- 53. Ref630 ODBC protocol •Releases! • Parser • https://github.com/rigmar/Recon2017/tree/master/DBS • Client • https://github.com/rigmar/Recon2017/tree/master/ODBC
- 54. IEC 61850 MMSLite from SISCO Cares about security Some info about secpatches But “SISCO does not provide detailed technical information of any kind (security related or otherwise) on our products to anonymous or unknown persons”
- 55. MMS Lite Nosources Some dumb fuzzing => No results Again some bug in user-hooks
- 56. Path traversal u_mvl_fopen_ind Used to read COMTRADE files But allows to read any file on flash Reported 26 Jul 2016 Device credentials stored in DB file Hashed with MD5
- 57. Pimp my term! Schneider Electric Fancy TV for your terminal 220 Service ready on KAOS system Magelis xbtgt5330 Only one default port
- 58. Firmware as OS Firmware consists of several .dlm files .dml – is ordinary PE x86 based Some kind of KAOS system But KAOS looks like Windows App PTC Perc “Real-Time” Java machine
- 59. Device management Vijeomanagement tool Works through FTP FTP has some proprietary extensions TGID WRDI ...
- 60. Smart TV Canbe integrated with bunch of terminals Some vendors even recommend it A lot of SW extensions
- 61. Augmented Smart TV A lot of HW extensions USB biometric switches
- 62. Third-party party • Almostevery IED (with IEC61850) uses SISCO MMSLite • Mocana SSH • Allegro ROM Pager • Third-party soft is Good • Update problems
- 63. Substation-ng • Remove embeddeddevices • Goodbye, VxWorks! • Goodbye, PowerPC! • Signal acquiring from power lines still required • Put all protection processing in virtual machines • Application running on Windows box • Only HI-TECH countries
- 64. In the end Still just an embedded device Real-Time requirements No encryption No exploit mitigations Updates are slow/manual/hard A lot of people still writing their own HTTP Servers
- 65. @scadasl kudos @atimorin AlexanderTimorin @_Rigmar_ Alexander Tlyapov @arbitrarycode Alexander Zaitsev @GiftsUngiven Alexey Osipov Anatoly Katushin @repdet Gleb Gritsai Sergey Gordeychik Sergey Sidorov iGrids Lab Maksim Nikandrov Viktor Nikitin And others http://scadastrangelove.blogspot.com
- 66. iGrids Lab • Cheboksary,home of ‘Bouquet of Chuvashia’ beer • https://en.wikipedia.org/wiki/Chuvashia • Substation (“releyka”) capital of RF • Certification laboratory • (ad) Access to numerous substation devices by subscription • (russian) http://igrids.ru/ • Open challenges on conferences
- 67. Thanks for Yourattention
Editor's Notes
- #2 Hello everyone, I’m Kirill Nesterov and my friend Alexander could come for some reasons. So I’ll tell you a story about our journey into world of relay protections which are the part of a digital substation. So let’s start. First of all I’ll describe what is relay protection and what role does it play in electric power transmission.
- #3 Small disclaimer. All opinions are my own and not the views of my employer. So let’s start.
- #4 Here is an awesome NIST picture with Conceptual Model shows overall lifecycle of energy management from generation sources, how the flow of electricity is controlled and how it gets to our homes, factories and other destinations. From now on we will talk only about Transmission part of this process, but interesting fact is that generation itself depends on transmission as it needs electricity to operate. For example, Nuclear Power Plant will have up to 3 substations powering it, cause in case of electricity faults reactors could be stopped to prevent overheating/melting – say hello to cascade effect.
- #5 Transmission is done by a set of substations and power lines. Basically, substations are routers for power lines coming from generation facilities to customers distribution. Those routers are controlled by various parties who manage, bill and whatsoever the economics of the energy distribution. Our point of interest is a substation and the equipment inside it. Though the same equipment for example Protection Relay Terminals, is used much wider: in distribution part, in railroad traction substations, small diesel/gas powered facilities – substations are the most important of them.
- #6 Substation differ a lot by voltage level, their role in energy network, and most interestingly by their customers. As we already mentioned Nuclear Power Plant will have 3 substations powering it, less critical destinations will not have such redundancy. Substation can have a number of input and output lines. Electricity from them is passed through a set of primary equipment (primary system devices), for example transformer, in order to either lower voltage or make it higher, or various switchgear to route electricity in required destination.
- #7 Though, the transformer does looks like a nice spot to conduct an attack as its explosion would sweep the substation, they are actually hard to attack as most of the time only electronics they have is diagnostics, they are insulated with some dielectric or have other safety features, and in the end, many things will melt or brake before transformer, but not him. There are obvious ways to get to transformer – rtfm sabotage manuals. There are a lot of them in the internet. Simple Sabotage Field Manual - http://svn.cacert.org/CAcert/CAcert_Inc/Board/oss/OSS_Simple_Sabotage_Manual.pdf
- #8 Here I’ll show you some demo. At first you can see two examples of triggering the circuit breaker while it’s line is powered. So It looks charming but real damage is minimal. As I have mentioned before It’ hard to get the transformer, but sometimes it’s still possible. So here you can see a fail of protection system and overheating of the transformer. But still it’s a petty damage. Because real one will look like blackout of the city or even stop of some kind of a factory. And restart of industrial objects after an accidental stop can be very expensive.
- #9 Secondary system devices are our goal as those are in charge of protection which includes overcurrent protection, differential protection, distance and other. This includes data acquisition from substation and sometimes joint substations, calculations and if required action to prevent damage due to some fault, for example trigger of circuit breakers. IEC 61850 is how substation work. And modern substation works almost exclusively via MMS and GOOSE protocols. Which are Ethernet, but GOOSE have another Ether type. The data inside those protocols is described in substation configuration language during substation configuration. SCL is represented as XML with description of configuration of protecting relays and supporting devices, like network switches. It can hold network configuration and electrical configuration for protection processes. Of course, industrial vendors included a pack of proprietary protocols to relays which allow to reconfigure, update and monitor devices. You can imagine that substations are part of some closed network, but this is actually the other way around. Inside substations you will find a ton of various systems from video surveillance, physical access system, various switchgear monitoring, SCADA system, network equipment, billing systems to utility communication (telephony, DECT, etc.). All these systems are not only managed by different teams by its role, but also represent up to 3 or 4 different companies.
- #10 Security of substations is a very dull thing. Current standard IEC 61850 has no security in it and this has been discussed one thousand hundred million billion times. The point is there are MMS and GOOSE protocols (and more others), they are open and non-authenticated, and provide means for reconfiguration or distribute measurements between Intelligent Electronic Devices. It’s a matter of replay or spoofing attack to start influence normal processes at substation. This could or could not cause the news headlines with words blackout, balalaika or baboon – not interesting. The point is that if your are in a subnet or have network access to protection relay on substations you’d only need ARP spoofing, packet replaying or open source MMS, GOOSE tools for attack. The fact that all protection relays are supported by a typical ICS out of this millennia software only emphasizes the legendary complexity of potential attack. New standard IEC 62351 is meant to add security to all this madness. And while the theory is good for ideologists and marketing, the madness is not easily fought with implementation phase. But this is an answer to a very good question: What is the purpose of the talk when things are so bad? We wanted to look at substation of future that would be more or less secure at least typical office network grade and analyze the security of embedded devices in charge of safe electric power distribution.
- #11 We are kindly asking you to leave your interesting question and thoughts about real world buzz in a trash bin. We most probably made and will make mistakes related to electrical engineering terms – just skip them. The point of the talk is reverse engineering of embedded devices found on susbstations. Элерон – это один из газов используемых в оболочках для трансформаторов. Оболочки с газами не проводящими электричество используют, чтобы когда ебнет дуги успокоить.
- #12 We have met nearly ten different relay protection terminals and here You can see the most generic things about them. So on such device you can expect to see some Real Time OS, the most popular was VxWorks. Also every device that we saw implements two protocols. It’s MMS and HTTP, which in some cases can be turned off. Another generic problem with them it’s debugging, without access to hardware it’s very hard to get even some crash info. And in the end there for some reasons vendors like to build their devices on top of PowerPC’s CPU MPC860.
- #13 Here You can see our first target which is SIPROTEC from Siemens. It’s the forth generation of such devices, we have heard that the fifth one exists but we have not met it anywhere. So it’s typical for relay protection box with small display and number of buttons. Let’s take a closer look at firmware format, because It’s complicated comparably to others.
- #14 Final firmware for the device consists of two parts – OS with bootloader and number of additional modules with different industrial protocols. But everything is packed inside of proprietary PCK format. Such PCK file is a container for different number of KON files and xml with their description.
- #15 As You can see PCK header is very simple. It have three fields: name, check sum and a size of current element.
- #16 Here is an example of enumeration of such PCK file. As you can see it contains several KON files with appropriate names. From this names you can understand that they contains firmware for different kind of devices with different CPU types.
- #17 And KON format is a set of tagged records with short name and size. This short name informs the updater about what kind of data it has. For example tag DATA means that it’s a raw data and tag DATC is a compressed record. Each type has it’s own fields. And because of code lays inside of DATA or DATC record it has appropriate field for loading address. Also you can notice that some of the records have additional check sums for some reason. Maybe because it can be a standalone file format. Also we released parser library for SIPROTEC’s firmware format.
- #18 So here is a colorful representation of a KON format based on bootloader. One of the presented sections here is DATC section which is compressed data with code. When you load it into IDA you can clearly see that whole binary can be divided into several parts depending of their offset.
- #19 If you dive deeper into the code you will see that bootloader’s startup code copy parts of itself into different memory locations. And here is some attempt to describe it’s memory map. So in the end we are able to parse and unpack firmware so let’s take a closer look into system.
- #20 So when we look inside the code of the firmare we will find that it’s based on some rare pSOS. It’s a RTOS which was born in early 80s and than after some business stuff was bought by Wind River – developer of VxWorks, they were rivals. So you can understand It’s fate. Also main CPU module can be upgraded with different kinds of communication modules. One of then is Ethernet module. Funny thing about it is that it has the same CPU, same OS and communicate with main module with some sort of TCP/IP protocol.
- #21 Here is services which we have discovered on nearly default configuration. Some of this services work on main CPU and forwarded their by EN100 module, other work on EN100 by itself. Here you can see two proprietary protocol. First one is DIGSI – it’s a protocol for the main configuration tool and on the Java’s applet protocol we will soon take a closer look. But let’s start from my favorite HTTP service. I think for some reason developers which need to implement HTTP service always have a desire to implement it from scratch as it was with SCADA system WinCC OA from same company.
- #22 And as we see developers of EN100 firmware couldn’t resist this desire. May be because HTTP always looks very simple. But one of the problems is how to hide some development or debug info inside of your new HTTP server?
- #23 The password is a good answer! So with this three ‘x’ symbols you can get inside of debug interface which allows to read whole memory and so on. So let’s take a closer look at some of “protected” options.
- #24 After the password check there are two input forms. They are rather similar. So first one tries to find a taskname substring in user request and if it finds it will try to read several integers into stack buffer. So we could get in some trouble in case of s specifier for scanf function. Because all the code relies in addresses with leading zero and because it’s a big endian system it’s impossible to overwrite return address without corrupting this zero byte. But here you just drop your payload and pivot the stack. Like in ninety's. So let’s try to write some values.
- #25 Of course the EN100 module will crash and web error log will show us crash info, which is very convenient for devices of this kind because you can see not only where it have crashed but also the state of all it’s registers. As you can see program counter is equals to ascii chars of A and this crash looks like it can lead to code execution. Moreover there are read write and execute memory rights on some of the network buffers which make the exploitation even more easier. But a little notice, the code section have only read and execute which is right. So I don’t know why execution prevention wasn’t implemented fully.
- #26 Also after we submitted bugs to Siemens and they patched everything we’ve got little stuck with CVSS scores which are 5.3 for remote code execution. It’s on some kind of a smart network card, but it’s still code execution.
- #27 So now I’ll show you some demo.
- #28 So we have found simple overflow with scanf function. But searching for vulnerabilities can be pretty simple if you taint network controlled data to the scanf calls. Because a lot of them can be vulnerable.
- #29 Another interesting service is Java applet which can be downloaded through web. But when loaded it connects to device through UDP port 56797. It allows to see device state and send some remote commands to it.
- #30 So it’s one of the services which forwarded to main CPU module. It communicates with custom protocol. Show small amount of diagnostics after you enter right password witch is checked on client side. At first we thought it’s hardcoded, but then we found some kind of document with it. This document describes actions in case of problems with the device. So it’s not really a password, but a “confirmation” code. Simple protocol fuzzing revealed some read out of bounds. In case of CPU service we can’t get any debug info through web error handler as before.
- #31 Why this read out of bound can be more serious threat than RCE in EN100 module? There is special mode inside of the device which is called Monitor mode. This mode is activated after three fails in short period of time. In this mode device turns into expensive brick and if any electrical accidents happens it will just ignore them and they will be escalated to the next level of protection dealing more damage. This case was tested with this device called RETOM, which simulates emergency situations and checks protection.
- #32 Next device is F60 from General Electrics.
- #33 This firmware format is much simpler than the previous one. It have some header and after one hundred bytes in hex it have code.
- #34 Loading process contains relocation of self part into new address.
- #35 And then it just uncompresses main part of it.
- #36 If uncompressing is finished successfully, code at this offset is cleared.
- #37 Then control is passed to uncompressed code. Nothing strange.
- #38 It’s more interesting how it works inside. There are thousands of global objects. Their initialization function even doesn’t shows in IDA graph view on default settings. They are strongly typed and every object have proper getters and setters, so you don’t have to worry about wrong usage of strcpy. Also they are build in heavy Object Oriented style with average inheritance level of 3. State of every such object is backed up by EEPROM. And when we saw this code at first there was no decompiler for PowerPC. I’m not a big fun of reversing large cplusplus code inside of just an disassembler, because of difficulties with structure offsets. Then HexRays choose PowerPC between PowerPC and MIPS and It made me on of the happiest man in the world, at least for some time.
- #39 Here is an example of initializer for such global object. I hope you can see that it has a type, a name, default value and also a Modbus number, because on this devices Modbus is one of the main protocols. And nearly every such object can be called for reading or writhing through Modbus. You can even virtually push buttons on the device through Modbus.
- #40 At first we took 6th version of firmware and wasted a lot of time on understanding it. Just because it was firmware version of lab device. But then we found that in 7th version vxworks symbols were not stripped. So sometimes it’s rather useful to analyze several firmware version at once.
- #41 Here is number of services we have found on newer firmware. As you see my favorite HTTP protocol still here.
- #42 It looks like selfmade HTTP server. But when I’ve played a little with it turned out that it’s extremely simple and have no user input at all. It’s so lazy that it ignores everything except GET and URL.
- #43 Next was Modbus. It looked promising. But it turned out to be an authorization on top of Modbus with bruteforce protection and proper access control. After some google we could found that different people had some research on security of this device and even made parts of their masters degree. Main configuration tool Enervista worked through Modbus and even update firmware through some extensions. The only bad thing about it is that it doesn’t have encryption. But even this was fixed soon.
- #44 In newer versions of firmware we have found MocanaSSH service. Mocana is an SSH implementation for embedded devices. It’s a new Enervista configuration protocol and It looks rather secure. Let’s take a glimpse at it.
- #45 I was staring on this function names for a long time. And after that I definitely see some correlation in main callback functions. Now I clearly can say that It has example substring inside of them. What can it mean?
- #46 But I won’t say what it mean. Communication is the sad part about this vendor. Before this presentation we contacted all vendors and asked about coordinated disclosure. So everyone responded with ok and provided detailed information about patches except this one. We had no response at all. I have found four potential remote code execution, some of them a really funny in my opinion. But in light of recent events I will not disclose any of found bugs because I heard that there is a Recon in Montreal and It’s hard to visit such conference when you are in jail or under sanctions.
- #47 But the main problem with multilin devices that without physical access they don’t have any any debug facilities. So after the devices crashes you will see a lonely string inside of log. Just a system exception. No debugger, no crash info, no traceback, nothing at all. And I’m pretty lazy to guess what have gone wrong in static.
- #48 And the next relay terminal with database architecture is REF630 from ABB. This DB architecture is similar to global objects in GE terminal, but it’s backed up by custom db format on an normal filesystem. Also to get a firmware you only have to get an access to an ftp service. Because this FTP gives full access to flash. As other devices it has http and iec61850 and two additional protocols SPA and ODBC. Which allows to work with different objects inside of database.
- #49 And it was the most comfortable terminal that I saw. It has Vxworks with symbols. Like the others it’s based on powerpc cpu. You can have full access to filesystem through FTP. If something gone wrong you will find a file with a traceback. And something more.
- #50 It has a debug server which can be turned on with special option in configuration file. So when it’s running it opens two ports. One for stdout with some custom headers and the second one is raw stdin. Every string received from stdin port is executed inside of vxworks shell. it’s differs a from telnet but just a bit. So if you will find some bug which can provide to write-what-where primitive you just need to turn this service on. And you don’t have to upload any code to the device at all. So this service allows to interact with native vxworks console which in most cases contains internal debugger and many other useful stuff. So we will walk through some of the services but before that I will describe format of the database files.
- #51 There are three types of database files. First one is a simple plain data format. Second one is compressed. And the last one is compressed and encrypted. So let’s take a look at that encryption.
- #52 It uses blowfish algorithm. And here is a key generation procedure. The resulting key depends on ip addresses of the device’s network interfaces.
- #53 So the ordinary story is two interfaces. First one is a loopback and the second one is Ethernet. So we concatenate VMX substring with ip addresses, then we xor the result with some hardcoded string. And this whole key generation procedure can be described with this six lines of python code.
- #54 If you need to parse this format for some reason you can get released parser and a client to communicate with devices of this kind.
- #55 Another interesting service is of course MMS. So like others it uses MMSlite from sisco. And let’s focus on this library. If you google something like sisco mmslite security you will find info about security patches. But details or any other kind of information is closed to ordinary people. Only clients have access to it.
- #56 As you can guess it’s a proprietary library. But as it often happens you can find some sources inside of Chinese internet. Of course they are not the latest one, but it still something. Also we have tried some simple fuzzing with zero results. And in the end it turned out that easy bugs can be found in user hooks code of this library.
- #57 There is an fopen implementation which allows to read comtrade files. It’s special files which contain information before and after the fault. And it doesn’t sanitize network data at all. So we can easily read any file on flash with simple path traversal. And because MMS have no authorization for such calls you can read database file with all credentials which are hashed with md5. After that you get full access to device with legitimate configuration software.
- #58 As you can notice before relay terminals have very simple display with small number of buttons. But there are a lot of different augmentation for them. One of them is HMI system from schneider electric. It have only one default port – it’s ftp with fingerprint that says that is has some kind of KAOS system on board.
- #59 But when we look through it’s firmware it seemed that KAOS is just a library for some embedded windows system. Because all imports lead to standard windows libraries. Also significant part of the system is written in java and runs under some kind of real-time java machine for internet of things. Whatever that meant.
- #60 It’s a little bit unusual for me, but device management works through ftp protocol. But with some proprietary extensions.
- #61 So this smart tv allows integration with different kind of devices and vendors. Also it has a lot of supported protocols.
- #62 Also it has some hardware extensions like this biometric switch which allows to authorize through fingerprints. I think I saw biometric switches with such design in some part of Deus Ex series. If you know what it is.
- #63 So modern relay terminal contains a lot of third party software. And it’s very good but it can have problems with one days because of update problems.
- #64 So we described pretty new technologies for substation automation build on industrial embedded devices. But do we have any real reasons to build all this equipment? It looks like ”no”. New advances bring several hardware servers where all protection relays are virtualized and run as standalone application inside Windows 7 box for example. Will it help security or not? At least, even Windows box provide more flexibility for security reconfiguration. In reality, this technology is probably only used in China nowadays and nobody had a chance to research it. Little notice here. Most of researched devices built on top of real time OS, and it’s seems that their tasks need to be real-time to get the job done. But some devices and this virtualized approach in my opinion does not provide hard real time capabilities. But still they are working somehow.
- #65 I my opinion such industrial devices still just an ordinary embedded device with some additional peripherals. So security of such device have the same weaknesses as other embedded systems. So because of real-time requirements there are no exploit mitigation techniques like canary words or address space randomization. All you can find is partial data execution prevention. But as you understand it doesn’t work when it’s partial. Most of the protocol have no encryption so they are vulnerable to man-in-the-middle attacks or even replay attacks. Updates for such devices are slow, hard or need a help of engineer. So network security of such systems relays on that fact that this network have to be isolated, but for some administrative or business reasons they still need to be accessed from the outside. And still a lot of people try to implement their own http servers AGAIN.
- #66 Here is our team, those who stay behind of this research.
- #67 And big thanks goes to our friends from certification laboratory iGrids, who build strong security community around their lab by providing access to a big number of substation equipment for security researches, create hacking challenges on various conferences, provide introduction workshops in electrical engineering for it engineers and speak on hot security topics regarding substations.
- #68 Thanks for your attention. Thank you Recon team. It’s all I’ve got. Now I can answer some questions if we’ve got some.