wp_smartplantssmartgrids_v3

1
IBM Software
Whitepaper Executive Summary
Smart Plants, Smart Grids.
January 2015
Contents
1 Summary
3 Requirements-Driven
Engineering
3 Box: A Biological Model for
Complex System Controls?
5 Systems Engineering
6 Standards and Non-
Functional Requirements
7 Box: Why Systems
and Software Fail
10 Professional Software
Engineer
11 Conclusion
12 References
Power and the Industrial
Internet of Things (IIoT)
Engineering complex systems to build
Smart Plants in a Smart Grid
Summary
The Internet of Things (IoT) is coming to the energy and manufac-
turing industries. The world’s energy infrastructure, in particular, is
undergoing a user-centered, software-driven, digital transformation—
a change from the material and mechanical innovations that have
sparked changes in other industries and other times.1
These changes,
moreover, are coming to an American infrastructure that earned
a barely passing grade of D+ from the American Society of Civil
Engineers—“an aging electrical grid and pipeline distribution systems,
some of which originated in the 1880s.”2
The Internet of Things (a.k.a., Cyber-Physical Systems, Industrie 4.0,
The Fourth Industrial Revolution, the Industrial Internet, Big Analog
Data Solutions, Smarter Planet, Intelligent Systems) has captured
the technological world’s imagination. Connecting computers, smart
phones, sensors, appliances, machinery, vehicles, utilities, and a host
of other elements into a reliable, efficient whole could be the biggest
engineering challenge we’ve ever faced.
Though the IoT is in its infancy, by the end of 2013 it already included
about 20 billion connected devices (out of about 187 billion connectable
devices). That number will grow to 30 billion by 2020.3
In 2014, most of those connected things belong to the Consumer IoT
(with smart phones topping the list). But another, less visible Industrial
Internet of Things (IIoT), with its heavy duty infrastructure (e.g.,
power, transportation) and applications (e.g., industrial equipment,
smart plants, smart automobiles, advanced medical devices), will pose
a bigger challenge and offer greater rewards.

2
IBM Software
January 2015
Industrial IoT systems will be bigger, heavier, costlier, far
more complex, operate at far higher energy levels with far
higher reliability than the products of the Consumer Internet
of Things. If a buggy smart-phone operating system or app
is rushed through beta and onto the market, its failure might
cause widespread nuisance and some real harm, but the actual
potential for damage is minuscule compared to failure in an
aircraft, a power plant, or an offshore oil rig.
Power and complexity are what really set the Industrial IoT
apart. High-power industrial systems have always put a
premium on safety, reliability, and stability. And power grid
engineers certainly have experience working with highly
complex systems. The complexity—the extent and intercon-
nectedness on many levels—of these new systems-of-systems,
though, will make them qualitatively different from what
most engineers have worked with in the past.
The elements of the system will change constantly—some-
times second-by-second. Industrial systems, moreover, can
have lifecycles measured in decades, far longer than the
months or years of consumer products. The design can never
be locked down in the familiar sense, and the cumulative
effect of changing composition and long product life can be
extremely daunting. The Industrial Internet of Things will
have to accommodate elements that are unknown—perhaps
even un-thought-of—when the system is first built. It will
include billions of components that are themselves complex,
components manufactured by different (and often competing)
organizations. They may come out of completely different
industries, with different standards, different design assump-
tions, and different business goals.
The joint effort of the automotive and electric power
industries to forge the link between plug-in electric vehicles
(EVs) and the Smart Grid offers a fascinating early example
of the coming convergence of industries in the Internet of
Things. It illustrates the prime factors future systems will
require: “cross-border” collaboration, technical integration,
and harmonization of business goals and methods.4
What
happens when hundreds of millions of electric cars plug in
each evening? Will utilities be able to use the batteries of
connected cars to store energy for peak use, to buffer their
generation demand? Will parked electric vehicles, indeed,
be able to supply their own power to the grid on sunny days?
What happens when the manufacturer of your software-
heavy EV decides to send out a firmware update? Will they
be able to keep track of all of the options and after-market
modifications—serial number by serial number—to push
down the right update? Most important, how well will con-
nected electric vehicles meet safety, reliability, and security
standards? Will outsiders be able to hack into it—or even
hack through it to tamper with the power grid? What happens
when you go out in the morning to start your drive to work?
Will your car start? Will it make it all the way to work? Will
the brakes fail or the motor race because of a software error?
(For insight onto how the two industries are cooperating,
see Thinking Across the Plug: The Coming Integration
of Electric Vehicles and Smart Grids, an IEEE Tech Insider
webinar sponsored by IBM®
Rational®
.)
Complexity
Even simple feedback control systems can produce emergent
behavior, those seemingly coordinated, large-scale patterns
that develop spontaneously, without any central direction:
the apparent structure of rippling desert sand, schools of fish,
and flocks of birds, for example. Designers of complex systems
of systems will routinely encounter emergent behavior, patterns
of variation in performance that, in many cases, can never
be entirely designed out of the system.
Yet even though such systems cannot be tightly controlled,
they can be highly reliable, robust, and safe. They are chaotic
in the mathematical sense. This does not mean disorder.
It is behavior so complex that the smallest variation in initial
conditions produces significantly different outputs. The
moment-by-moment state of a properly performing chaotic
system may be almost impossible to predict—though it will
range seemingly at random within a predictable operating
envelope. The beating of a healthy human heart, for example,
appears to be chaotic, while the heart of a patient with con-
gestive heart failure ticks as regular as a metronome.5
Thus,
as systems-of-systems grow more complex—with more layers
of communications, more checks and balances—they may
well come to resemble biological organisms more than a
steam engines. (See Box, “A Biological Model for Complex
System Controls?”)

3
IBM Software
January 2015
Yet we will expect these systems to be stable, and to share
common objectives. Ultimately, these devices will couple
real-time analyses with machine-to-machine, machine-to-
infrastructure, and user-to-machine communication, so
that they can adapt continually to changing circumstances.
(Consider: a “mere” 20 billion devices yield 400 quintillion—
4x1020
—potential communicating pairs.)
Developing a workable Industrial Internet of Things will
occupy us through at least 2020. It will require mastering
new design methodologies tailored to systems-of-systems.
And it will demand broad, flexible standards on an unprece-
dented scale.
Heavily regulated, high-energy, safety-critical operations—
nuclear power, aerospace, defense—have already realized
significant benefits from the systems engineering. They treat
a power plant or an aircraft as
an integrated collection of multiple sub-systems
made up of mechanical, electrical, chemical, soft-
ware, and human elements. Each of the sub-systems
meets a set of well-defined requirements that not
only define system performance, but also capture
what is necessary to meet regulatory mandates,
safety standards and governmental statutes.6
Soon, this will apply to any application involving high
energies or high hazards (such as power and energy, critical
biomedical devices, hazardous chemicals, large actuators
in public places).
Requirements-Driven Engineering
Systems engineering depends, first and foremost, on complete
and completely understood requirements7
:
Well defined requirements include not only functions and
subsystem dependencies, but all the factors that affect the
project. Requirements-gathering starts, as in any project, with
familiar engineering factors. In complex, intelligent systems
it is especially important to know how information and the
physical inputs and outputs pass from one subsystem to another
—up, down, and sideways.
A Biological Model for Complex
System Controls?
Figure 1: The MAPK/ERK (mitogen-activated protein kinases/extra-
cellular signal-regulated kinases) signaling pathway. It serves as an
example of the hundreds of control loops in a mammalian cell, each
of which can control biochemical processes or trigger switches in the
nucleus to turn on, turn off, or modify the 20,000-odd genes in the
body. (Creative Commons)
The human genome contains about 6 billion nucleic acids
—3 billion on each of the two complementary strands
of DNA. Just 1.2% of these are genes, templates for the
protein building blocks of the body. The lion’s share of
the remaining 98.8% was long considered “junk DNA,”
with no biological function.
In 2012, however, The Encode Project Consortium made
its first report on a comprehensive analysis of the entire
human genome. They found that at least 80% of the
genome seems to be chemically active. These regions bind
molecules that turn genes on and off, or produce short
RNAs that may influence protein production outside
the nucleus, or change the shape of the chromosome
by interacting with the DNA’s molecular bubble wrap.
The discovery surprised biologists around the world.31

4
IBM Software
January 2015
The analysis should go on to include relevant standards,
financing, and social influences (such as the organization’s
assumptions, culture, and processes). Each of these represents
a system that requires engineering.
Complex requirements make it essential to maintain a trace-
ability matrix, a linked list that allows project participants
to quickly and accurately log changes and determine how
a modification of one system will affect the performance of
all the other components in the network. This is particularly
important when the organization plans to strategically re-use
components. These elements will often be modified to accom-
modate new technology or special requirements. It becomes
vital to track changes in hardware, components, requirements,
and dependencies, to know which code version goes with what
hardware version, and how changes affect the inputs required
from upstream modules and outputs passed on to downstream
modules. Modularity can greatly accelerate product/application
development, but only if you can understand how any change
affects the rest of the system.
It’s easy to see that developing a complex system generates
a tremendous volume of work products.a
The best way to cope
with this flood is to adopt tools that automate the process.
Logging requirements and changes manually is error prone
in even modest-size systems. In systems with many thousands
of requirements (which some projects have), keeping up with
development with an Excel spreadsheet is impossible.
Experience shows that successful systems-engineering
programs invest heavily in defining the mission and gathering
requirements at the front end. Poor programs, on the other
hand, skimp on defining the mission, and have to compensate
by investing much more heavily in verification and validation
and system integration down the line.8
In other words, it looks as though 80% of the genome
is logic, and 1% is I/O. The genes, their associated
DNA binding areas for gene promoters and inhibitors,
segments that code for short RNA strands that seem to
modify protein production outside the nucleus, are just
part of the cell’s complex control system. Above them
in the system-of-systems hierarchy are the signaling
pathways inside the cell, which turn the genes on and off
(the illustration shows one pathway just to illustrate the
potential complexity). And the systems that send signals
between cells to trigger the signaling pathways. And
the body-wide controls, like the nervous and endocrine
systems, that can trigger lower-level signals throughout
the body.
Research indicates that these control systems are non-
linear and complex—chaotic in the mathematical sense.
To cite just one example: A healthy heart beats irregu-
larly, what one researcher called “robust beat-to-beat
heart rate variability.” The time between successive beats
changes unpredictably within a narrow range. Some
researchers contend that there is too much noise in the
measurement to confirm conclusively that the beat is
chaotic32
, others see strong empirical5
and theoretical33
evidence that it is. What is clear is that diseased or
stressed hearts—in patients suffering from congestive
heart failure, or under general anesthesia34
, or badly
injured—beat like clockwork. It’s tempting to interpret
this as a failsafe mode for a system in which some compo-
nents are not operating within their proper design limits,
sending out-of-range inputs to other subsystems and
disabling most of the feedback control loops.
Biology and engineering seem to be growing together
at their leading edges, with biology adopting ideas from
systems theory, and systems theory embracing models
from biology. The results are research efforts like the
Institute for Systems Biology in Seattle, and Harvard’s
Wyss Institute for Biologically Inspired Engineering in
Boston. As systems-of-systems get more complex, their
circuitry and software diagrams may come to resemble
cell-signaling pathways more than the linear flowcharts
of years gone by.
a
These can include safety standards, development plans, verification plans,
quality assurance plans, plans for certification, configuration management
plans, project schedules, system requirements, subsystem requirements,
electronic requirements, mechanical requirements, software requirements,
human/organizational requirements, safety analyses, hazard analyses,
environmental impact analyses, a system architectural design, system
interface specifications, electronic designs, mechanical designs, software
designs, software source code, test cases (both manual and automated),
test case results, formal mathematical analyses, results of various reviews,
quality assurance records, configuration management records, quality
assurance audits, and change requests.

5
IBM Software
January 2015
There are additional benefits: A properly governed system-
of-systems generates documentation useful in outside review
by regulators or customers—in, for example, power, defense,
aerospace, medical devices, or pharmaceuticals. And systems
engineering has been shown to reduce budget and schedule
overruns significantly.6 9 10
Requirements V Diagram
Figure 2: Requirements Driven Engineering (RDE) and process
improvement6
Systems Engineering
Successfully engineering Smart Plants in the Smart Grid
demands continuous engineering and innovation, the
same approach required in any complex system-of-systems.
It also demands technical competence in many engineering
disciplines.
According to the National Society of Professional Engineers,
Systems engineering (SE) is a hybrid engineering
and business discipline. Systems engineering seeks to
make the best use of personnel, material, equipment,
and energy; and focused on the characterization of
system properties, such as safety analysis, require-
ments, data integration, process transformation,
organizational dynamics, and architectural design
which can include technical topics such as software,
mechanical, nuclear, structural, chemical, petroleum,
and civil engineering. Systems engineering differs
from downstream engineering disciplines in that the
outcomes for the latter are implementations, while
the outcomes for the former are specifications.11
In addition to solid requirements-gathering, the essential
elements of good systems practice are:7
Engineering collaboration. IIoT products and projects will cross
boundaries that separate discipline from discipline, depart-
ment from department, company from company, industry
from industry, and nation from nation. The companies
building the Industrial Internet of Things will have to follow
suit, collaborating and demolishing organizational silos.
This applies to the whole organization, not just engineers.
The collaboration must also draw talent from the business
side, operations, maintenance, and even the receptionist who
greets everyone every day. Systems engineering happens in
three dimensions: physical (engineering’s traditional domain),
mathematical (the theoretical limits of what information and
algorithms can accomplish), and social (the realm of office
politics, personal communication, and company culture).12
Continuous validation and verification. These are the “VV”
of the now-ubiquitous V-diagrams. Validation confirms that
the designers have selected the right tasks to do and the
right requirements to fulfill. Verification confirms that they
have done the right task correctly. The two go hand in
hand. Too often, they are treated as independent activities:
As a result, the right task may sometimes be implemented
correctly. But uncoordinated, unsynchronized VV also
means that the organization can waste resources by perform-
ing the wrong task correctly, the right task incorrectly, and
even the wrong task incorrectly. Just as the component devices
and networks of the IIoT will communicate and react to one
another constantly, validation and verification will be continu-
ous and interactive. In both cases, constant feedback helps
the system be self-correcting in near-real time.
Strategic re-use (with the emphasis on “strategic”). All engineers
and programmers obviously try to re-use components—
hardware parts and subsystems, code libraries, and so on.
But few are strategic about it—planning, constructing, and
documenting the component for repeated use over the long-

6
IBM Software
January 2015
term. It can take time—up to an order of magnitude more,
in some cases—but the potential gains down the road in
lower costs, shorter time-to-market, and improved safety
and reliability far outweigh the initial investment.
The traditional approach may have been linear and sequential:
gather a list of all the requirements, write all the code, do all
the validation and verification, make all the changes, and then
ship the product. The contemporary systems engineering
process is an iterative, hierarchical, continuous, top-down
decomposition of system requirements.13
Systems Engineering V diagram
Figure 3: Forsberg14
describes the V-model, which links systems
engineering to the project cycle. Design explorations and analyses begin
system development, which ends with a completely integrated and
qualified finished product. On the left side of the V are decomposition
and definition activities; the specification is completed at the bottom
of the V; and the rising right arm outlines quantitative verification steps
to assure that requirements are met.
Automated tools help engineers get a handle on all this
slippery, daunting complexity. IBM®
Rational®
Software was
developed to help gather and organize requirements, trace
changes, and plot out modification paths for multiple projects.
To map subsystem interactions and dependencies, designers
can turn to tools like SysML (Systems Modeling Language,
http://www.omgsysml.org) from the Object Marketing Group
(www.omg.org, a non-profit standards consortium). SysML
is a visual modeling language that helps engineers with
specifying, analyzing, designing, and verifying com-
plex systems which may include hardware, software,
information, personnel, procedures, and facilities.
SysML provides visual semantic representations for
modeling system requirements, behavior, structure,
and parametrics, which is used to integrate with
other engineering analysis models.15
SE teams along
with system designers are responsible for verifying
that the developed systems meet all requirements
defined in the system specification documents.1
Standards and Non-Functional
Requirements
Requirements, of course, include not just the functional
requirements of your design customers. They also include
the standards and fundamental non-functional requirements
—what some in the field call the –ITY words: safety, security,
stability, reliability, maintainability, availability.
Re-Defining Failure, Safety, and Reliability
Standards for the IIoT
It’s hard to see how an Internet of Things can evolve without
an ambitious and widespread standards effort to bridge the
gaps between industries.16
IEEE is actively engaged in making standards for the IIoT.
Already published are standards for security in green commu-
nity networks, the WirelessMAN (Metropolitan Area Net-
work) air interface standard for machine-to-machine commu-
nication, and a protocol for utility industry metering.17
IEEE standards working group P2413 is constructing
a “Standard for an Architectural Framework for the Internet
of Things (IoT).”18
Many organizations, too many to list all of them, are working
on aspects of the IIoT standards challenge. These include
the Industrial Internet Consortium founded in 2014 by ATT,
Cisco, GE, Intel, and IBM®
and now grown to more than 100
member companies, and the Open Interconnect Consortium
(more than 40 members, including Cisco, Intel, Mediatek,
Samsung, ADT, Atmel, Dell, Eyeball Networks, and Hewlett
Packard).

7
IBM Software
January 2015
Other organizations have formed around single protocols.
These include the Thread Group (founded by ARM, Big As
Fans, Freescale, Google’s Nest, Samsung, Silicon Labs, and
Yale Security), and MQTT (the stripped-down MQ Telemetry
Transport protocol designed for reliability in low-bandwidth
or unreliable networks; it is an outgrowth of the SCADA
protocol and MQIsdp). The list goes on.
As they develop methods for ensuring safety, security, and
privacy in the design of electric vehicles and charging stations,
systems engineers might refer to a raft of standards beyond
the obvious standard for functional safety in road vehicles
(ISO 26262): for “functional safety of electrical/ electronic/
programmable electronic safety-related systems” (IEC 61508)
[19]; for programs in airborne systems (DO-178C) for cyber
security in nuclear plants (NEI 08/09), and perhaps even for
medical-device software (IEC 62304). [4]
Safety and Reliability
Like the other aspects of systems engineering, ensuring safety,
stability, and reliability in a system of systems goes beyond
code. It requires transformations of behavior and culture.
The more elaborate a system is, the more ways it can fail.
There are limits to the effectiveness of classical, bottom-up
methods, relying on fault trees, built-in redundancy, and
“Swiss cheese” defense. Yes, the engineer has to build all the
defenses possible into the system. But past a certain point,
redundancy simply guards over and over against the same failure
pathway, adding almost no additional assurance at great
expense.b
At some point, the systems engineer assumes that
a failure—of an algorithm, a circuit, a program, or a person—
is inevitable. The challenge is to figure out how to react then.
The answer is to analyze failure from the top down as well
as from the bottom up (the fault tree approach). In the top-
down approach, the designer focuses on outcomes rather than
causes. What happens if the GPS guided vehicle can’t find a
signal, or control software interprets completely normal noise
as a signal, or water enters the car deck of a ferry? At that point,
it doesn’t matter how it happened. What matters is what you
do now. Even at the highest level of integration, designers
b That point of diminishing returns is actually fairly well defined: two-fold
redundancy can provide significant improvement in reliability; three-fold
redundancy is generally not worth the additional overheads and expense.
Why Systems and Software Fail
Why do ambitious systems and software projects fail so
often, wasting billions of dollars every year? In both cas-
es, the causes are overwhelmingly faults in organizational
culture, processes, or management, and not technical
missteps.
Warning Signs of Physical System Catastrophe
The Center for Chemical Process Safety compiled a
list of 161 symptoms that a system may be in danger of
imminent catastrophic collapse. Though their catalog is,
of course, tailored to the chemical process industries, the
principles can apply to any complex, high-hazard system
of systems. The CCPS list breaks the warning signs into
nine general categories, listed here, along with some
examples and a rationale for some Rational®
solutions.23
Leadership and Culture
Indicators of faulty leadership and culture include: An
organization that tacitly or explicitly accepts operating
outside the envelope of safety. A place where everyone is
too busy to do things right. Employees who don’t know
about or don’t care about standards. The medicine for a
weak culture is a stronger one, inspired by Continuous
Engineering and Lean principles and supported by tools
that can make them work.
Training and Competency
Employees who are not trained or encouraged to recog-
nize building catastrophes. An “organization” in which
relatively minor system disruptions provoke disorganized
responses. These are some of the hallmarks of poor
training. In systems of systems, programs like the Profes-
sional Software Engineer licensure and IBM® Academic
Initiative in software engineering can help keep engineers
fully aware of the implications of their decisions. These
will help ensure that necessary operational and training
information reaches the workforce.

8
IBM Software
January 2015
need to consider possible modes of failure. As Nancy G.
Leveson notes in Engineering a Safer World: Systems Thinking
Applied to Safety, “All inputs should be checked for out-of-range
or unexpected values and a response designed into the control
algorithm. A surprising number of losses still occur due to
software not being programmed to handle unexpected inputs.”20
Post accident root-cause analyses often point to operator
error: a human mistake is almost always involved somewhere
in the chain of events. When one delves into the big engineer-
ing disasters—Chernobyl, Fukushima, Deepwater Horizon,
and Bhopal—one finds that failures of social engineering.
Usually, behind the human mistake, one finds a series organi-
zational and management actions that make that mistake
more and more likely. Cutbacks in staff mean workers who
are rushed and tired. Cutbacks in training leave workers
unprepared to handle emergencies. Deferred maintenance
means that alarms and sensors don’t work reliably. Poorly
placed indicators and controls mean seconds lost in an emer-
gency. Ignoring employees’ warnings about safety destroys
morale and fosters a culture of negligence. Operator error
may be the last straw, but it can only break a system that is
already bending under the burden of invisible organizational
mistakes.20 21 22 23
Cautionary Tales
Southwest U.S. Blackout, 2011:
A Misplaced Check Mark
On September 8, 2011, a phase imbalance took a bank of
capacitors at Arizona Public Service’s North Gila substation
offline. An experienced technician was dispatched to isolate
them from the rest of the system. He reviewed the 16-step
procedure, making a note on his clipboard as each was com-
pleted. Because he was simultaneously trying to direct members
of a work crew, the technician entered the completion time
for Step 6 on the line for Step 8. Because he had skipped two
steps, when he resumed the procedure at Step 9, the switch he
opened was still carrying current at 500 kilovolts. The switch
started to arc. As the FERC report on the incident notes, the
technician “stayed under the arcing 500 kV line, determined
to crank open the switch far enough to break the arc, thereby
preventing additional damage to the equipment.”
Process Safety Information
The indications of a flawed safety culture include system
documentation that does not reflect current changes
and status, safety data that is not readily available, and
the absence of any process for managing system alarms.
Automated tools, such as process plug-ins, patterns and
models for IBM® Rational®, can help organizations keep
process safety and other documentation up-to-date and
readily available.
Procedures
CCPS lists 13 signs that key procedures are not doing
their jobs, including poor documentation and lax adher-
ence that causes too many automatic trips and shutdowns.
(Here again, process plug-ins, patterns and models for
IBM® Rational® can help engineers manage procedures
and responses along with requirements and changes.)
Asset Integrity
Red flags that the physical assets are not being monitored
or maintained properly include deferred inspections and
maintenance, lack of a formal maintenance program, and
bypassing alarms and safety systems. Here, the integra-
tion of life-cycle and asset management tools can help
ease the symptoms on an Open Services for Lifecycle
Collaboration, OSLC, platform.
Analyzing Risk and Managing Change
The 21 signs of faulty risk- and change-management
systems include stand-by systems that are not working,
risk-assessments intended merely to confirm decisions
already made, instruments that are bypassed without a
real change-management process. Reporting and require-
ments gathering tools naturally support risk-assessment
and change-management. Decisions are easier, too, with
SysML tools that link mathematical engineering analyses
to structural and behavioral design models (tools such
as the Parametric Constraints Evaluator). Moreover,
organizations that embrace Continuous Engineering
principles (collaboration, continuous validation and ver-
ification, and strategic-reuse) are institutionally inclined
to find, acknowledge, and fix problems.

9
IBM Software
January 2015
As the switch opened, however, the arcs to both phases of
the disconnect switch lengthened…until they made contact.
This tripped a phase-to-phase fault. The phase angle difference
between the two power segments quickly increased to the
point that the quick reconnection was impossible. Eleven
minutes later, the surge of power had triggered a cascade of
automatic diversions and shutdowns that tripped the separation
switch for the San Onofre Nuclear Generating Station. This
isolated the entire region surrounding San Diego, CA, Yuma,
AZ, and Tijuana, Baja California, Mexico. This “power island”
collapsed almost immediately, leaving some 7 million people
completely without electricity and mostly without cell phone
service for six hours (and some as long as 12 hours).24 25
As one veteran systems engineer observed, “You can’t have
a system set up so that one guy can make a mistake to take
out seven million people’s power. There’s something wrong
with the system if that’s a possibility.”
Northeast Blackout, August 2003:
Secondary Software Errors
Between 3 p.m. and 4 p.m. on August 14, 2003, three 345-
kilovolt transmission lines in Ohio failed when overgrown
trees touched the wires. This triggered a cascade of failures
in an already stressed system, shutting down power to
55 million people in the Northeastern U.S. and Canada.
Though inadequate maintenance of the transmission right-
of-way caused the immediate failure, a series of software
failures earlier in the day had prevented the grid’s operators
from anticipating, recognizing, and correcting the other
problems that pushed the system past its breaking point.
The Midcontinent Independent System Operator (MISO,
Carmel, IN) control center’s listing of operating assets was
out of date: A 230 kV line was out of service, and the local
operator’s update of its status was not passed on to MISO’s
state estimator, software that monitors grid traffic and alerts
operators to impending problems. Unable to reconcile the
data it was receiving with its internal model. The system
started to generate solutions well outside acceptable limits.
MISO took the state estimator offline at 12:15 for trouble-
shooting. The problem was fixed by 1:00 p.m., but the techni-
cian did not reset the state estimator to poll the system and
report every five minutes. The state estimator stayed offline
until after 4 p.m., leaving MISO partially blind.
Audits
The 10 signs that the organization is not making
sufficient effort to identify and remedy its shortcomings
include: problems that come up again and again in audit
after audit; management that doesn’t review audits;
regulatory citations and fines; failure to let all involved
employees share audit results. Tools for managing require-
ments and changes aid greatly in reporting problems
and implementing changes, and the ability to incorporate
modeling allows engineers to analyze tradeoffs and
choose the most cost-effective fixes.
Learning from Experience
Down-bound organizations fail to learn from experience,
making the same mistakes over and over, and downplay-
ing the importance of each of their increasingly frequent
stumbles. Again, a Continuous Engineering culture based
on reusable assets such as object oriented requirements,
design specifics, and architectural work products will pro-
vide visibility and transparency opportunities, so they are
seen as occasions to understand and improve the system,
not as an embarrassments to be swept under the rug.
Physical Warning Signs
Finally, companies racing towards catastrophe simply
refuse to read the writing on the wall, ignoring the phys-
ical warning signs of coming failures: complaints from
workers or the surrounding community, equipment that
is visibly damaged, accumulations of dirt, missing safety
equipment, open flames, open electrical panels, and
loose fasteners on equipment. Yet again, a remedy is a
Continuous Engineering, coupled with regular inspec-
tions and a Lean Six Sigma mind-set intent on eliminating
waste and barriers to productivity.

10
IBM Software
January 2015
About two hours later, the First Energy’s SCADA (supervisory
control and data acquisition) alarm program, part of its key
Energy Management System (EMS), also failed, unnoticed
by anyone in the control room or IT. And about 20 minutes
after that, the server hosting the FE’s EMS also failed, perhaps
because the SCADA alarm had locked up. The system failed
over to a backup server, but it, too, failed after the stalled
SCADA alarm program was transferred to it. The backup
EMS crashed just before 4 p.m.—leaving the grid operators
partially deaf and blind to the condition of their system
as one heavy transmission line after another dropped off
the system.26
Mars Polar Lander, 1999: Blame the System
In December 1999, NASA’s Mars Polar Lander vanished
during decent to the Martian surface. Later reconstructions
indicated that descent braking engines had stopped firing
prematurely. The probe had been designed so that load
sensors in its legs would detect the impact of landing, and
the Lander’s flight control system would then shut down the
descent braking engines. As probe’s fall through the atmo-
sphere, however, apparently produced noise—a signal that
designers called “normal and expected.” The control system
interpreted noise as the signal that the probe had landed,
and shut down the engines in mid-air—an example of how
a system can malfunction even though all of its individual
components are working properly.20
Ariane 501, 1996: The Importance of Tracking Changes
On 4 June 1996, the new European Space Agency rocket
Ariane 5 tore itself apart and exploded 40 seconds after lift-
off, only 3700 meters above its launch site in French Guiana.
According to the Board of Inquiry convened to analyze the
failure, the problem lay in the software of Ariane 5’s inertial
reference system (IRS), a subsystem recycled from the earlier
Ariane 4. The Ariane 4 IRS expected signed 16-bit input;
the Ariane 5 was sending 64-bit floating point data. The
mismatch sent the processor into an error trap, the IRS failed,
and the vehicle lost stability.27
Earmarks of a Software Project
in Trouble
A 2005 IEEE Spectrum article, “Why Software Fails,”
listed the 12 most common causes for project collapse.
They are35
The project-killers include:
• Unrealistic or unarticulated project goals
• Inaccurate estimates of needed resources
• Badly defined system requirements
• Poor reporting of the project’s status
• Unmanaged risks
• Poor communication among customers,
developers, and users
• Use of immature technology
• Inability to handle the project’s complexity
• Sloppy development practices
• Poor project management
• Stakeholder politics
• Commercial pressures

11
IBM Software
January 2015
Security
Right now, security is the most prominent of the non-function-
al requirements that every design must address. The emphasis
changes, however. Today it’s security. Tomorrow it may
be reliability or maintainability. Later it may be availability
or safety. The important task is to balance all of the factors
to stand up over time.
In the power and energy sector particularly, legacy systems
often make up the bulk of the infrastructure. The sensors
and actuators were added later to permit some remote monitor-
ing and control. In many cases, security was an afterthought.
The situation should improve dramatically as systems meth-
odology and newer standards and protocols let engineers
design in security, identity verification, privacy protection,
etc. from the project’s inception.
There is no doubt, though, that Western energy infrastruc-
ture is under near-constant cyber assault. A 2013 survey
commissioned by two U.S. Congressmen included reports
from 69 American electric power utilities (36 independently
owned utilities, 25 municipal or regional suppliers, and 8
federal). They found that about 20% of the respondents
reported “‘daily,’ ‘constant,’ or ‘frequent’ attempted cyber-
attacks ranging from phishing to malware infection to
unfriendly probes. One utility reported that it was the target
of approximately 10,000 attempted cyber-attacks each month.”28
The Ponemon Institute, with sponsorship from Unisys,
conducted a larger security survey of 599 infrastructure
companies world-wide (electric, gas, and water utilities; oil
and gas producers and retailers; alternative energy producers;
chemical and industrial manufacturers). Two thirds of them
reported suffering “at least one security compromise that
led to the loss of confidential information or disruption”
during the preceding 12 months.29
Professional Software Engineer
As safety-critical systems get more complex, manufactur-
ers and customers alike need some way of ensuring that the
software is developed to a high standard, by developers who
understand not only the code, but also the physics or chem-
istry of the system they’re working with and the necessity
of working with organizational social structures to develop
requirements.
Software doesn’t fail of itself. It fails when it’s supplied with
input that’s inappropriate in the context of the physical system.
Though failure analyses may identify software as the root
cause, one too often finds that the program, in fact, did
exactly what it was supposed to do. The real cause was that the
requirements, the full definition of the problem to be solved,
were incomplete. The software engineer did not, in fact, write
the code wrong. He or she was never asked to write the right
code. Developing the real requirements and understanding
the full context in which the programs will operate is the true
foundation of software safety.
In April 2013, NCEES (National Council of Examiners
for Engineering and Surveying) began administering its
first exams for Professional Engineer licensure in Software
Engineering. The 80 questions in the 8-hour test put an
emphasis on requirements gathering, safety and security, and
design methodology. To earn a license, the applicant must
also demonstrate an understanding of the physical systems
with which the code interacts. As a prerequisite to the PE/SE
license, NCEES administers Fundamentals of Engineering
examinations in seven disciplines. The exams include topics
like engineering math, probability, circuit analysis, linear
systems, signal processing, electronics, chemistry, instru-
mentation, ethics, health and safety, engineering economics,
statics, dynamics, strength of materials, materials science,
fluid mechanics, and electricity and magnetism, and the
transfer of heat, mass, and energy.30
Topics on the NCEES Software Engineering Exam
Knowledge Area % of Exam
Requirements 17.5%
Design 13.75%
Construction 11.25%
Testing 12.5%
Maintenance 7.5%
Configuration Management 7.5%
Engineering Processes 7.5%
Quality Assurance 7.5%
Safety, Security, and Privacy 15%

12
IBM Software
January 2015
Conclusion
Whether or not it is consciously designed to the standards of
systems engineering, the coming Industrial Internet of Things
will be a system of systems. Adopting a unified engineering
approach that explicitly recognizes and tracks the interactions
of physical, organizational, and business factors will be the
key to developing these interlocking systems And it offers
the best hope for long-term safe and reliable operation of
constellations of billions of interacting elements that cannot
be fully defined at the design stage…if ever. This white paper
is intended to offer a glimpse of that future, and to help the
people who will build it get a sound start on the task.
By Douglas McCormick
About the Author
Douglas McCormick, principal of Runestone Associates, is a science
and technology writer and editor, an IEEE Spectrum webinar host,
and its test and measurement blogger.
This paper is based on interviews with IBM’s Ben Amaba and
Gavin Arthurs, and on papers written by Dr. Amaba, Paul
Fechtelkotter (also of IBM) and others.
References
1 T. Ahram, W. Karwowski, B. Amaba and P. Fechtelkotter, “A
System-of-Systems Engineering Approach for Power Energy
Management,” in Proceedings of the 2013 Industrial and Systems
Engineering Research Conference, 2013.
2 American Society of Civil Engineers, “2013 Report Card for Amer-
ica’s Infrastructure,” American Society of Civil Engineers, 2014.
[Online]. Available: http://www.infrastructurereportcard.org/.
3 EMC Digital Universe with Research Analysis by IDC, “The
Digital Univers of Opportunities: Rich Data and the Increasing Value
of the Internet of Things,” EMC, April 2014. [Online]. Available:
http://www.emc.com/leadership/digital-universe/2014iview/inter-
net-of-things.htm.
4 IBM®
, “Electric vehicles and the challenges of convergence,” IBM®
Corporation, Software Group, October 2014. [Online]. Avail-
able: http://www-01.ibm.com/common/ssi/cgi-bin/ssialias?sub-
type=WHinfotype=SAappname=SWGE_RA_VF_USENhtml-
fid=RAW14375USENattachment=RAW14375USEN.PDF#loaded.
5 A. L. Goldberger, “Giles F. Filley Lecture. Complex systems,” vol. 3,
no. 6, 2006.
6 P. Fechtelkotter, L. Bishop, D. Davis and B. Amaba, “Systems-of-Sys-
tems Engineering for Safety-Critical Projects,” in Mary Kay
O’Connor Process Safety Center International Symposium: Beyond
Regulatory Compliance, Making Safety Second Nature (October 28-
30, 2014), College Station, TX, 2014.
7 B. Amaba, “Industrial and Business Systems for Smart Cities,” in
EMASC ‘14, Nov 7-14, 2014, Orlando, FL, USA, 2014.
8 P. Fechtelkotter, G. Bleakley, B. P. Douglass and B. Amaba, “Mod-
el-Driven Development for Safety-Critical Projects in Intelligent
Energy,” in SPE Middle East Intelligent Energy Conference and
Exhibition, Dubai, UAE, October 28-30, 2013, 2013.
9 International Council on Systems Engineering, INCOSE Systems
Engineering Handbook, v2a, International Council on Systems Engi-
neering, 2004.
10 E. C. Honour, “Understanding the Value of Systems Engineering,”
2010. [Online]. Available: http://www.incose.org/secoe/0103/Val-
ueSE-INCOSE04.pdf.
11 Licensure and Qualifications for Practice Committee of the Naitonal
Society of Professional Engineers, Body of Knowledge, Licensure
and Qualifications for Practice, Alexandria, VA: National Society of
Professional Engineers, 2013.
12 T. Ahram, W. Karwowski and B. Amaba, “Collaborative systems
engineering and social-networking approach to design and modeling
of smarter products,” vol. 30, no. 1, 2011.
13 D. K. Hitchins, Systems Engineering: A 21st Century Systems Meth-
odology, Chichester, UK: John Wiley Sons, 2007.
14 K. Forsberg, H. Mooz and H. Cotterman, Visualizing Project Man-
agement: A Model for Business and Technical Success, New York,
NY: John Wiley Sons, 2000.

13
IBM Software
January 2015
15 S. Friedenthal, A. Moore and R. Steiner, A Practical Guide to SysML:
The Systems Modeling Language, Morgan Kaufmann: Elsevier Sci-
ence, 2008.
16 M. Anderson, “Is There Any Way to Avoid Standards Wars in the
Emerging Internet of Things?,” IEEE, 4 August 2014. [Online].
Available: http://spectrum.ieee.org/tech-talk/consumer-electronics/
standards/is-there-any-way-to-avoid-standards-wars-in-the-emerg-
ing-internet-of-things.
17 M. Rozenfeld, “Setting the Stage for the Internet of Things: Cov-
ering a spectrum of applications,” IEEE, 14 March 2014. [Online].
Available: http://theinstitute.ieee.org/benefits/standards/setting-the-
stage-for-the-internet-of-things.
18 IEEE P2413 Working Group, “Standard for an Architectural Frame-
work for the Internet of Things (IoT),” IEEE, [Online]. Available:
http://grouper.ieee.org/groups/2413/.
19 B. Amaba, “Ben Amaba Houston 08052012 v1.pdf,” IBM®
Software
Group, 2012. [Online]. Available: https://www-950.ibm.com/events/
wwe/grp/grp004.nsf/vLookupPDFs/Ben%20Amaba%20-%20
Houston%2008052012%20v1/$file/Ben%20Amaba%20-%20Hous-
ton%2008052012%20v1.pdf.
20 N. G. Leveson, Engineering a Safer World: Systems Thinking Ap-
plied to Safety, Cambridge, MA: MIT Press, 2012, p. 608pp.
21 J. Mahaffey, Atomic Accidents: A History of Nuclear Meltdowns and
Disasters: From the Ozark Mountains to Fukushima, Pegasus, 2014,
p. 460 pp.
22 C. Perrow, Normal Accidents: Living with High-Risk Technologies,
Basic Books, 1984, p. 386 pp.
23 Center for Chemical Process Safety (CCPS), Recognizing Cata-
strophic Incident Warning Signs in the Process Industries, Wiley,
2011, p. 264 pp.
24 Federal Energy Regulatory Commission and the North American
Electric Reliability Corporation, “Arizona-Southern California Out-
ages on September 8, 2011,,” Federal Energy Regulatory Commis-
sion and the North American Electric Reliability Corporation (April
2012). Arizona-Southern California Outages on September 8, 2011,
2012.
25 California Assembly Utilities and Commerce Committee and Joint
Legislative Committee on Emergency Management, “Briefing Paper
on September 8, 2011 Southwest Power Outage,” California Legisla-
ture.
26 U.S.-Canada Power System Outage Task Force, “Final Report on
the August 14, 2003 Blackout in the United States and Canada,”
U.S. Department of Energy, April 2004. [Online]. Available: http://
energy.gov/sites/prod/files/oeprod/DocumentsandMedia/BlackoutFi-
nal-Web.pdf.
27 Ariane 501 Inquiry Board, J.L. Lions, chair, “Ariane 501 - Presen-
tation of Inquiry Board report / Press Releases / For Media / ESA,”
European Space Agency, 23 July 1996. [Online]. Available: http://
esamultimedia.esa.int/docs/esa-x-1819eng.pdf.
28 Staffs of Congressmen Edward J. Markey (D-MA) and Henry A.
Waxman (D-CA), “Electric Grid Vulnerability: Industry Responses
Reveal Security Gaps,” U.S. House of Representatives, 2013.
29 Ponemon Institute LLC, “Critical Infrastructure: Security Prepared-
ness and Maturity,” Unisys Corporation, 2014.
30 NCEES, “NCEES Principles and Practice of Engineering Examina-
tion: Software Engineering Exam Specifications,” National Council
of Examiners for Engineering and Surveying, April 2013. [Online].
Available: http://cdn1.ncees.co/wp-content/uploads/2012/11/SWE-
Apr-2013.pdf.
31 The ENCODE Project Consortium, “An integrated encyclopedia of
DNA elements in the human genome,” vol. 489, pp. 57-74, 6 Septem-
ber 2012.
32 C. O. Tan, “Heart rate variability: are there complex patters?,” Fron-
tiersIn.org, 4 July 2013. [Online]. Available: http://journal.frontiersin.
org/Journal/10.3389/fphys.2013.00165/full.
33 C. C. Peng, A. C. Yang and A. L. Goldberger, “Statistical physics
approach to categorize biologic signals: From heart rate dynamics to
DNA sequences,” vol. 17, 2007.
34 G. Matchett and P. Wood, “General anesthesia suppresses normal
heart rate variability in humans,” vol. 24, no. 2, 2014.
35 R. N. Charette, “Why Software Fails,” IEEE, 2 September 2005.
[Online]. Available: http://spectrum.ieee.org/computing/software/
why-software-fails.

14
IBM Software
January 2015
RAL14120-USEN-00
© Copyright IBM Corporation 2015
IBM Corporation Software Group
Route 100
Somers, NY 10589
Produced in the United States of America
January 2015
IBM, the IBM logo, ibm.com, and Rational are trademarks of International
Business Machines Corp., registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other
companies. A current list of IBM trademarks is available on the web at
“Copyright and trademark information” at ibm.com/legal/copytrade.shtml
This document is current as of the initial date of publication and may
be changed by IBM at any time. Not all offerings are available in every
country in which IBM operates. It is the user’s responsibility to evaluate
and verify the operation of any other products or programs with IBM
products and programs.”
THE INFORMATION IN THIS DOCUMENT IS PROVIDED
“AS IS” WITHOUT ANY WARRANTY, EXPRESS OR IMPLIED,
INCLUDING WITHOUT ANY WARRANTIES OF MERCHANT-
ABILITY, FITNESS FOR A PARTICULAR PURPOSE AND ANY
WARRANTY OR CONDITION OF NON-INFRINGEMENT.
IBM products are warranted according to the terms and conditions
of the agreements under which they are provided.
The client is responsible for ensuring compliance with laws and regulations
applicable to it. IBM does not provide legal advice or represent or warrant
that its services or products will ensure that the client is in compliance
with any law or regulation.
Statements regarding IBM’s future direction and intent are subject to change
or withdrawal without notice, and represent goals and objectives only.

wp_smartplantssmartgrids_v3

Recommended

Recommended

More Related Content

What's hot

What's hot (17)

Viewers also liked

Viewers also liked (20)

Similar to wp_smartplantssmartgrids_v3

Similar to wp_smartplantssmartgrids_v3 (20)

wp_smartplantssmartgrids_v3