Virtualizing Industrial Controllers (PLCs/DCS Controllers) represents a fundamental shift in the industrial automation industry. Most industries have fully embraced virtualization as a means to support reliability, scalability and resource optimization. However, the industrial control system industry has been slow to adopt virtualization into automation controllers fully. These slides are from Austin Scott's S4 2019 presentation and outlines the benefits of industrial controller virtualization and why automation vendors see this as a threat to their business model. The slides describe a virtualized PLC deployment at a large refinery in North America that allowed them to scale to support the massive size of the plant and includes:
- What is PLC virtualization?
- A brief history of PLC virtualization
- Challenges with PLC virtualization
- The benefits of PLC/Controller virtualization
- Commodity controllers
4. 01
02
03
!!! Riskof cybersecurity vulnerabilities in
commodityhardware.
Consolidation,resiliency, and
manageability.
Hardened / cyberresilient Linux
machinesto hostVMs.
Virtual containeraround criticalICS
components.
ARE THERE CYBERSECURITY BENEFITS TO PLC
VIRTUALIZATION?
5. WHAT ATTRIBUTES HAVE MADE THE PLC A
SUCCESS?
01 Reliable
Will runautonomouslyforyearswithoutfail
02 Industrial
Withstandsin anindustrialenvironment
03 Modular
Expandabletomeet processrequirements
04 Maintainable
Is easytoprogramandsupport
6. Level 5-Enterprise
Level 4-Plant
Level 3– Operations Support
Level 2– Supervisory Control
Level 1– Control Devices
Level 0-Instrumentation
VIRTUALIZATION IN TODAY’S OT ENVIRONMENTS
FREQUENTLY
VIRTUALIZED
RARELY VIRTUALIZED
7. WHY ARE SOME SITES VIRTUALIZED TODAY?
Level 2– Supervisory Control
Level 1– Control Devices
CONTROLLERS REMOTE IO RACKS
CPU
UTILIZATION
#ERROR! #ERROR! #ERROR! #ERROR! #ERROR! #ERROR! #ERROR! #ERROR!
8. WHY ARE SOME SITES VIRTUALIZED TODAY?
Level 2– Supervisory Control
Level 1– Control Devices
CONTROLLERS REMOTE IO RACKS
CPU
UTILIZATION
VIRTUAL CONTROLLERS
9. OTHER USE CASES FOR CPU VIRTUALIZATION
TODAY
PLC SIMULATORS
Simulated PLC for
logic development
and simulation.
RESEARCH
Product
development and
vulnerability
research.
DCS CONTROLLERS
DCS Testing and
development
environments.
10. BUT … CPU VIRTUALIZATION IS ONLY PART OF THE
SOLUTION
CARDS
BACKPLANE
CPU
11. LATENCY TOLERANCE
IT virtualization (x86) is
designed for general-
purpose workloads where
throughput takes priority
over latency.
In OT,anydeviationfromthe required
latencywill causea processtrip.
250 µs 280 µs
800 µs
Motion Control
[1]
Electrical Control
[2]
Process Control
[3]
ESTIMATED PLC BACKPLANE
LATENCY TOLERANCE
[1] C. E. Pereira and P. Neumann, Industrial Communication Protocols, S. Y. Nof, Ed. Heidelberg, Germany: Springer-Verlag, 2009.
[2] L. Kean, “Microcontroller to Intel architecture conversion: PLC using Intel atom processor,” Intel Corp., Santa Clara, CA, USA, White
Paper, 2010.
[3] S. Balacco and C. Lanfear, “The embedded software strategic market intelligence program 2002/2003 vol. I: Embedded systems market
statistics,” Venture Develop. Corp., Mill Valley, CA, USA, Tech. Rep., 2003.
12. CAN WE ACHIEVE THE REQUIRED LATENCY IN A
REALTIME VM?
[1] Tiago Cruz, Paulo Simões, and Edmundo Monteiro “Virtualizing Programmable Logic Controllers: Toward a Convergent Approach” - IEEE EMBEDDED SYSTEMS LETTERS, VOL. 8,
NO. 4, DECEMBER 2016
(Gigahertz)
13. WHAT WOULD THE HARDWARE LOOK LIKE?
CARDS
RACK
VIRTUALIZED
ETHERNET
BACKPLANE
CPU
VIRTUALIZED
CARDS / RACKS
VIRTUALIZED
CPU
IN Dale Peterson’s Keynote he challenged us to ask better question.
In todays presentation, I will be using the socratic method of asking challenging questions to explore the possibility of a FULLY Virtualized PLC
In my presentation today I will be talking
1. The potential benefits of PLC Virtualization
2. the challenges to truly virtualizing a PLC
3. the impact will be for vendors and customers
Quite simply put: “Virtualization makes software look like hardware.”
The implications of virtualization within IT (and to a certain extent with in OT) have been massive.
Cost
Flexibility
Scalability
Reliability
Performance
No other advance in the past six decades of IT has offered more quantifiable benefits than virtualization.
More recently we have seen the benefits extend into OT environments as well within Windows and Linux based assets.
We have even seen a few different SoftPLCs try and fail to move into the market.
Steeplechase Software Inc – Visual Logic Controller (VLC)
Rockwell Softlogix
Today we are going to explore possibility of the FULLY Virtualized PLC –
How do I define fully virtualized?
If you think of how we can run Windows, Linux or BSD in VMWare. Imagine doing the same with GE, Schneider and Rockwell all on the same hardware.
PLC Software Container that behaves like a SPECIFIC vendors CPU, Backplane, Rack, IO Cards and Network – Running on more generic commodity OT hardware.
What problem are we trying to solve here exactly? PLCs have worked just fine without virtualization for the past 50 years… why would we want to mess with a good thing here?
Based on the huge benefits we have seen in the virtualization of the Personal Computer, one could assume the OT industry would enjoy similar benefits if we were to fully virtualize the PLC.
#1 Cost –
You are no longer locked into a single vendor for all your hardware – We break the vendor lock-in – Companies are no longer at the mercy of the PLC / DCS vendor for hardware.
Disrupt the electrical distributor model – by todays standards is actually pretty hard to buy a PLC – first of all the price is prohibitive – then you must also purchase through authorized re-sellers who have the special training to support the install of the hardware.
#2 Flexibility
Ability to move between product vendors seamlessly. For example a manufacturer might run rockwell for one product run and after they turn around the plant they might move to a Schnider Electric based system for the next product run.
Decoupling the physical I/O and computing capabilities allow for more compute power and scalable I/O.
#3 Support
Roll-back functionality.Virtualized testing environment. – Testing new Firmware – Easily create a simulated version of the running plant as the underlying software on the PLCs (Rslogix or UnityPro) will not even know that it is in a simulation.
#4 Performance:
DCS - centralized performance using modern processors. SCADA - edge compute power.As we have seen in the IT world it easier to scale CPU and Memory resources
lifecycle operations or change management protection
#3 Ease of Support:For instance: by creating a VM snapshot before applying a security patch, changes can be rolled back in case of failure; VMs can be cloned for sandboxed testing, prior to deployment into production; also, VM instances can be live migrated, allowing for reduced downtime every time a physical device needs to be stopped.
Snapshots and roll-back functionality.
Create a virtual twin of your running process and test process updates against live data to see how changes could impact the system.
When we announced this presentation I had a few people reach out in excitement about the possibility of PLC virtualization.
50 Years ago, way back in 1968, the foundational requirements of a PLC were laid out by the General Motors Standard Machine Controller RFP.
They were looking for a:
A solid-state system that was flexible like a computer but priced competitively with a like kind relay logic system.
Easily maintained and programmed in line with the already accepted relay ladder logic way of doing things.
It had to work in an industrial environment with all its dirt, moisture, electromagnetism and vibration.
It had to be modular in form to allow for easy exchange of components and expandability.
The requirements for a PLC have not changed that much over the past 50 years.
PLCs have become more scalable and user friendly to work on, the form factor has not evolved much.
PLCs typically run on an RTOS like: VxWorks, QNX,, Symbian OS, LynxOS, eCos, RTLinux
Unlike what happened in the IT domain, the use of virtualization technologies in OT has been a slow to take root.
It is becoming more and more common to find virtualization in SCADA and DCS greenfield environments:
Level 5 – 2 : Vendor support and extensive use of virtualization technologies.
Level 1 : Controllers are rarely virtualized in a production environment.
Using the Purdue model as a generic way of discussing ICS environments, we see virtualization in Level 5-2
In Level 1 - It is less common but not unheard of to virtualize controllers. Softlogix 5000, Steeplechase are PC based PLC solutions. Some fringe DCS systems leverage virtualized controllers to scale beyond the current limitations of their DCS controllers.
DCS systems yes - No SKU though
Edge case deployment
Level 0 – We are unlikely to virtualize a physical process unless you believe reality is a holographic projection in which case we could consider all things to be a virtual machine of sorts… that’s a discussion we all have after a few drinks later.
As plants expand and the demand for more data points increases, controller CPUs are pushed to their limits.
I have encountered sites that pushed the limits of what standard DCS controllers can provide. I have seen and heard about multiple DCS vendors do this, although it is NOT a product with SKU# you can buy off the shelf per se.
As plants expand and the demand for more data points increases, controller CPUs are pushed to their limits.
I have encountered sites that pushed the limits of what standard DCS controllers can provide. I have seen and heard about multiple DCS vendors do this, although it is NOT a product with SKU# you can buy off the shelf per se.
PLC SimulatorsPLC Simulation environments like STUDIO 5000 EMULATE and the UNITY PLC Simulator.
Most DCS vendors also offer a virtualized DCS controller
ResearchVulnerability research QEMU for Vxworks emulationYou can download a VMImage of VXWorks which runs most of the premium PLCs today.
DCS Controllers
Testing and development environments
PLC / Controller CPU virtualization is only 1/3 of the puzzle.
A PLC / Controller is not just 1 piece of hardware.
Its really 3.
-CPU
-Backplane(s)
-Cards
Going back to our definition of virtualization:
“software that looks and behave like specific hardware”
COMPLETE PLC virtualization would all you to run a Schneider Electric Unity XL programming environment and then migrate to a Rockwell Automation Studio 5000 environment without changing any hardware assets in the field.
To truly virtualize a PLC
The next challenge with PLC virtualization is the other 2/3s the Racks and the Cards
Deterministic nature of PLCs VS the indeterministic nature of virtualization.
Different sectors of course have different requirements – Water versus oil and gas versus electric grid.
Due to the deterministic nature of industrial control systems, this is an unacceptable tradeoff.
[1] L. Kean, “Microcontroller to Intel architecture conversion: PLC using Intel atom processor,” Intel Corp., Santa Clara, CA, USA, White Paper, 2010.
[2] S. Balacco and C. Lanfear, “The embedded software strategic market intelligence program 2002/2003 vol. I: Embedded systems market statistics,” Venture Develop. Corp., Mill Valley, CA, USA, Tech. Rep., 2003.
[3] ] C. E. Pereira and P. Neumann, Industrial Communication Protocols, S. Y. Nof, Ed. Heidelberg, Germany: Springer-Verlag, 2009.
For extreme cases, such as motion control applications, PLCs have to provide very low operation latencies, from 1ms to 250 µs (Class 3 RT Systems)
For example, and estimate interrupt and context switch latency requirements of 280 and 800 µs for electrical and process control industrial applications, respectively
for components on interconnected bus
A microsecond is exactly 1 x 10-6 seconds. 1 µs = 0.000,001 s. One millionth of a second.
To virtualize PLCs successfully in a production environment, you would want to create a Real-time Hypervisor and disable optimizations such as Hyperthreading that could impact the Latency.
Furthermore you would need to disable System Management Interrupts (SMI) that would otherwise suspend all normal program execution to switch to a special system management mode.
Tiago Cruz, Paulo Simões, and Edmundo Monteiro were able to achieve very low latency ~8 Microseconds using Commercial Off the Shelf Intel processors and multiple real-time VMs.
[1] Tiago Cruz, Paulo Simões, and Edmundo Monteiro “Virtualizing Programmable Logic Controllers: Toward a Convergent Approach” - IEEE EMBEDDED SYSTEMS LETTERS, VOL. 8, NO. 4, DECEMBER 2016
How would we go about truly virtualizing the PLC / DCS Controller? What would that look like?
One day you are running a Schneider electric system with Unity XL system – you go through a turnaround a move to Rockwell Studio 5000 without replacing any hardware. Completely decoupling of the hardware from the under lying software.
It is less common but not unheard of to virtualize controllers. Softlogix 5000, Steeplechase are PC based PLC solutions. Some fringe DCS systems leverage virtualized controllers to scale beyond the current limitations of their DCS controllers.
Fiberoptic Backplane - switched deterministic and/or real-time Ethernet fabric system
Centralized Virtual Controllers
Virtualized IO Cards – ARM based standalone endpoints
Commodity power supplies
Support for redundant power supplies
Support for redundant IO Cards / Controllers
Fiberoptic Backplane - switched deterministic and/or real-time Ethernet fabric system
DCS Centralized Virtual Controllers – Rackmount Enterprise Server
SCADA Controllers – Off the shelf industrial PCs
Virtualized IO Cards – Low-cost ARM based endpoints
Commodity power supplies
Support for redundant power supplies
Support for redundant IO Cards / Controllers
VMWare, Vbox, Qemu or any other Opensource or Commercial Off the Shelf (COTS) product is not going to cut it. A specialized Realtime Hypervisor is required.
Realtime Hypervisor: optimized for lowest possible latency.
System for automating deployment, scaling, and management of virtual PLCs and IO Modules.
Transparent redundancy and scalability
Transparent to the control system / engineering environment
The Commodity PLC
The virtualization and commoditization of the PLC would represent a significant shift in the business model of industrial automation.
We have seen this occur in other industries such as with the Personal Computer.
HP almost exited the PC market in 2012 but has since focused on providing premium products to the market and continues to be a market leader in the PC space and healthy growth.
IBM sold off their PC hardware business to focus on Server hardware and enterprise software.
Competitive Displacement
Virtualization could allow vendors to competitively bid on accounts that have traditionally been dedicated to a single vendor.
Focus on the Software
Less empathies on the hardware allows more resources to be put behind improving the software products and new licensable software solutions.
Market Share
The ability to leverage premium software and support structures provided by automation vendors with low-cost hardware can protect market share from low-cost automation hardware and open up new verticals and markets to sell into.
HE who controls the VM layer has influence on the market much like Vmware does today.