Engineering Large Scale Cyber-Physical Systems

Engineering Large-Scale
Cyber-Physical Systems (CPS)
Bob Marcus
Co-Chair NIST Big Data PWG
robert.marcus@et-strategies.com
Friday, April 29, 16

Key Points on Engineering Large Scale CPS - Initial Thoughts
• Many large scale CPS application (e.g. Smart City) will require the integration of
heterogenous systems into a system of systems
• These systems can be on the same horizontal level (e.g. networked embedded
systems) and on different vertical levels (e.g. IoT to Cloud)
• Standards will be needed for network, data, and command communication across
horizontal and vertical levels See the slide set on “Standards for CPS” at
http://www.slideshare.net/bobmarcus/standards-and-open-source-for-big-data-cloud-and-iot
• Enhanced design tools (e.g. modeling and simulation) will be needed to evaluate
system of system architecture and components
• Testbeds will be needed that can integrate real, virtual, and simulated systems at
multiple levels. See the slide set on “Research and Testbeds for CPS” at
http://www.slideshare.net/bobmarcus/research-and-testbeds-in-cyberphysical-systems
• Monitoring and management across multiple levels will be a major challenge

Outline of Presentation
• IoT Engineering for CPS
• Design Patterns for IoT from Michael Koster
• Systems Engineering for CPS
• CPS Projects from the European Union
• IoT European Research Cluster (IERC) Slides
• NIST CPS Framework
• NIST-NICT JointVision for Cyber-Physical Cloud Computing (CPCC)
• Large Scale CPS System of Systems
• Complex Systems
• Modeling and Simulation for Engineering CPS
• References

IoT Engineering for CPS

From http://www.slideshare.net/IntelSoftware/overview-of-the-intel-internet-of-things-developer-kit
Intel Development Kit for the Internet of Things

Intel Development Kit for the Internet of Things continued

Intel XDK IoT Device Daemon Overview

From www.anypresence.com/blog/2015/07/15/iot-microservices-mbaas-drive-next-wave-digital-transformation/
Microservices Gateway from AnyPresence

From http://intellyx.com/2015/06/19/challenges-of-microservices-architectures/
Microservices from Intellyx
Components of a Microservice Microservices in a Container environment
Contained Based Autoscaling
of Microservices

From http://blogs.gartner.com/gary-olliffe/2015/01/30/microservices-guts-on-the-outside/
Inner and Outer Microservice Architecture from Gartner

Design Patterns for IoT
from Michael Koster

From http://iot-datamodels.blogspot.com/2014/05/design-patterns-for-internet-of-things.html
Basic Design Patterns for the Internet of Things from Michael Koster
There is no one “reference” architecture for an Internet of
Things. Design Patterns are a way to construct architecture
solutions for specific use cases and use case classes.
The work to find system architecture solutions to Internet of Things problems
has led to an obvious conclusion that there is no single architecture appropriate
for most IoT use cases.The full spectrum of IoT presents a broad range of
diverse use cases and resource constraints, and thus motivates a range of
architecture solutions. Still, we would like to ground the discussion in a reference
set of technical concepts, to help promote a unified understanding and break
down the silos of thought around IoT architecture.We would also like to find
opportunities for standardization and commonality.
Different architecture solutions are appropriate for different use case classes, and
architecture is expected to be reusable within a particular class of use
cases.Therefore it makes more sense to talk about IoT architecture as a set of
Design Patterns, working together to achieve an end-to-end solution for some
problem.

Pattern: Software Networked to Things from Michael Koster
Software connected to thing via a network: the fundamental proposition of the IoT is
that connected software is of higher value than embedded software. Connected
software is less resource constrained and can integrate more diverse data sources.

Pattern: SoftwareVirtualization for IoT from Michael Koster
Virtualization: Software connecting to an abstract representation of the
thing. Virtualization makes it easier to create reusable software and devices.

Pattern: Middleware Abstraction for IoT from Michael Koster
Virtualization through middleware: allows many (web) applications to interact with
things. Middleware can cache the state of the thing and minimize network trafﬁc and
power drain on constrained devices, and can also serve as a persistent end point for
things that aren’t reachable over the network due to power cycling, ﬁrewalls, etc.

Pattern: Device to Device Connectivity from Michael Koster
• Devices talk to other devices peer-to-peer: local network connectivity enables
proximal ad-hoc networking, service federation and chaining, media stream continuity.
• Personal tracking device uses smartphone as gateway: common pattern for bluetooth
and WiFi connectivity.
• Smart home local application controller and gateway: application gateway pattern.

Pattern: Monitoring A Large Area of Things from Michael Koster
Monitor a large number of devices over a large area: collect, ﬁlter, analyze pattern used
in Smart City, surveillance systems, large scale resource monitoring.

Pattern: Human in the Control Loop from Michael Koster
People interacting with autonomic feedback loops: general purpose use case involving
people and automation in deﬁned roles.Autonomic control relieves people of the job
of monitoring, but lets them be involved in high level control and exception handling.

Design Patterns for IoT Security from Michael Koster
• Access control using data models: semantic hyperlinks control access to resources
based on the embedded metadata
• Social to physical graph relationship: well deﬁned concepts of ownership and
access delegation between people, entities, and things
• PGP and asymmetric public-key cryptography on devices: ways of creating SSL
sessions and signing data between devices and applications
• DTLS over UDP: security for resource constrained devices
• End-to-end encryption: transmitting and storing encrypted data independent of
channel encryption
• Device Management: using device identity, registration, and secure key exchange

Systems Engineering for CPS

Cyber-Physical Systems Structure and Behavior from U of Maryland
From http://www.iaria.org/conferences2014/ﬁlesICONS14/ICONS-Austin-Keynote2014-02-07.pdf

Systems Engineering Definition
From https://en.wikipedia.org/wiki/Systems_engineering
Systems engineering is an interdisciplinary field of engineering that focuses on how to
design and manage complex engineering systems over their life cycles. Issues such as
requirements engineering, reliability, logistics, coordination of different teams, testing and
evaluation, maintainability and many other disciplines necessary for successful system
development, design, implementation, and ultimate decommission become more difficult
when dealing with large or complex projects.
Systems Engineering Process with SIMILAR Method
From http://aaq.auburn.edu/node/125

SIMILAR Systems Engineering Method
From http://aaq.auburn.edu/node/125
• State the Problem: This should be in the form of a statement or a model. It is not used to describe how to fix the
problem; it simply describes what the problem is. It should include preferences and requirements stemming from
customer needs.
• Investigate Alternatives: Different options for a solution should be considered.They should be evaluated
on several criteria such as performance, cost, schedule, and risk.This reduces the risk of the project and helps clarify the
problem statement.
• Model the System: Models will be constructed for the different alternatives, then the model for the best alternative
will be further developed to help manage the system. Models will show the product and the process which produces
the product.They can be in the form of physical analogs, analytic equations, simulations, or diagrams.
• Integrate: Integrating the different facets of the systems means establishing logical flow and communication between
the subsystems. Ideally, a subsystem would finish its work on a product before sending it to another subsystem.A set up
which would require the least amount of information sharing between subsystems would be optimal.
• Launch the System: Launching the system is running through it to produce actual output.This requires more
detailed design as parts and new knowledge are integrated into the system. The systems engineers' products are: a
mission statement, a requirements document including verification (is the system built correctly?) and validation (is the
correct system being built?), a description of functions and objects, figures of merit, a test plan, a drawing of system
boundaries, an interface control document, a listing of deliverables, models, a sensitivity analysis, a tradeoff study, a risk
analysis, a life cycle analysis and a description of the physical architecture.
• Assess Performance: Performance is assessed through various metrics such as customer satisfaction, technical
performance measures, and figures of merit.To improve performance, you must be abke to control it.To control it, you
must be able to measure it. Resources should be budgeted and allocated to each subsystem, with a reserve amount
given to the project manager.
• Re-evaluate: Observe outputs from the system, and modify accordingly.This is a continual process that is constantly
repeated, as seen below.This is necessary for control and improvement.

SysML Definition
From https://en.wikipedia.org/wiki/Systems_engineering
The Systems Modeling Language (SysML) is a general-purpose modeling language for
systems engineering applications. It supports the specification, analysis, design,
verification and validation of a broad range of systems and systems-of-systems
From http://www.arscontrol.org/automationmenu/100-umlsysml

System Engineering Domain Elements using SysML
From http://www.jhuapl.edu/techdigest/TD/td3201/32_01-Topper.pdf

From www.engr.ncsu.edu/mechatronics/what-mech.php
From Mechatronics to Cyber-Physical Systems
In the last two decades, an intense shift from advanced mechatronics systems to cyber-
physical systems is taking place.The former systems, which integrate mechanical,
electronics, computing, control and situated reasoning components, are typically
implemented as closed, predeﬁned, controlled, and deterministic systems.The latter
systems are characterized by open system boundaries, large functional and structural
complexities, self-learning and -reasoning capabilities, partial autonomy, context-driven
adaptability, and decentralized decision making
Mechatronics is a multidisciplinary ﬁeld of engineering that includes a combination
of systems engineering, mechanical engineering, electrical engineering,
telecommunications engineering, control engineering and computer engineering.
From https://en.wikipedia.org/wiki/Mechatronics
From https://www.researchgate.net/publication/233687942_BEYOND_ADVANCED_MECHATRONICS_NEW_DESIGN_CHALLENGES_OF_SOCIAL-CYBER-PHYSICAL_SYSTEMS

System Engineering Process Design, Implementation, andVeriﬁcation
From https://www.quora.com/Can-Software-Architects-learn-anything-from-Systems-Engineering
based on https://www.fhwa.dot.gov/cadiv/segb/views/document/sections/section3/3_4_2.cfm

Cyber-Physical Systems Concept Map
Clickable diagram nodes at http://cyberphysicalsystems.org/

Engineering Resilient Cyber-Physical Systems from NEU
From http://www.coe.neu.edu/faculty-research/initiatives/ecsps

Dynamic Integrated CPS Platforms
From http://cps-vo.org/ﬁle/20908/download/60342

Nine Major Design Challenges for Cyber-Physical Systems
From https://www.researchgate.net/publication/233687942_BEYOND_ADVANCED_MECHATRONICS_NEW_DESIGN_CHALLENGES_OF_SOCIAL-CYBER-PHYSICAL_SYSTEMS
(i) handling aggregative complexity
(ii) static and dynamic compositional synergy
(iii) dynamic and evolutionary operation in time
(iv) multi-abstraction based modeling
(v) system integrity veriﬁcation and behavior validation
(vi) dynamic scalability towards meta-systems
(vii) transformation of big data
(viii) testable surrogate prototyping
(ix) robust social compliance

Innovative Features of CPS Systems from NIST
From https://s3.amazonaws.com/nist-sgcps/cpspwg/pwgglobal/CPS_PWG_Draft_Framework_for_Cyber-Physical_Systems_Release_0_8_September_2015.pdf

R&D Needs for High Conﬁdence CPS
From http://www.seas.upenn.edu/~lee/10cis541/lecs/lec-CPS-1x2.pdf

Six Propositions on the Design of Cyber-Physical Systems
From http://tinyurl.com/jyybtqe
Proposition 1: In general, we are still a considerable way away from having a transdisciplinary theoretical
framework for true CPSs and SCPSs, or even from elucidating the major principles by which they should
operate. (SCPS = Social-Cyber-Physical System)
Proposition 2: Several deﬁnitions of CPSs have been published and many systems have been realized, but
design, implementation and utilization of these systems are still perplexing, not to mention their possible
impacts on the society and the future implications.
Proposition 3:There is no consolidated design methodology known that could provide answer to the
discussed design challenges, systematize the consideration and management of the effects of long-term
learning and self-adaptation of CPSs, and to explain the principles of designing for semi-autonomous or
fully-autonomous operation.
Proposition 4:There is a huge knowledge gap concerning the design and engineering principles and
technologies of realizing high-end, non-linear CPSs (or sub-systems and components) that are
compositional, scalable, interoperable, and evolvable.
Proposition 5: New abstraction methodologies, as well as pre-implementation modeling, demonstration,
prototyping, and empirical testing methodologies are needed in particular for the investigation of
contextualized interactions with the human/social environment.
Proposition 6: Next generation CPSs are envisioned to be a horizontally and vertically heterogeneous
system of systems, having some level of reproductive intelligence. In order to advance the state-of-the-art,
both transdisciplinary insights and multidisciplinary operative knowledge synthesis are needed.

Principles for IoT Clouds
From http://www.infosys.tuwien.ac.at/research/viecom/papers/Truong2015Principles.pdf
Principle 1: Enable virtualization and composition of IoT components as units
Principle 2: Enable emulated/simulated IoT parts working with production cloud
services
Principle 3: Enable dynamic provisioning of IoT and cloud service units through
uniform marketplaces and repositories for multiple stakeholders
Principle 4: Provide multilevel software stack deployment and conﬁguration
Principle 5: Provide software-deﬁned elasticity and primitive governance
functions for all IoT and cloud service units
Principle 6: Provide monitoring and analysis for an end-to-end view on elasticity
and dependability properties
Principle 7: Coordinate elasticity to enable a coherent elastic execution
throughout the whole IoT cloud system

CPS Projects from the European Union

CPS Roadmaps from EU Projects
From http://road2cps.eu/events/wp-content/uploads/2015/09/01_Road2CPS_Intro_Methodology1.pdf

Road2CPS Objectives

Road2CPS Priority Themes

ElectronicComponentsandSystemsforEuropeanLeadership(ECSEL)
From http://www.ecsel-ju.eu/web/index.php

European Union CPS-Related Projects
From http://ec.europa.eu/newsroom/dae/document.cfm?doc_id=428

ICT1 - Smart Cyber-Physical Systems

ICT4 - Customized and Low Power Computing

ICT30 - IoT and Platforms for Connected Smart Objects

FoF1- Process Optimization of Manufacturing Assets

IoT European Research Cluster (IERC)

Implication of IoT for the Future Internet from EU IERC
From www.internet-of-things-research.eu/pdf/Building_the_Hyperconnected_Society_IERC_2015_Cluster_eBook_978-87-93237-98-8_P_Web.pdf

Billions of Connected Devices from EU IERC

Management of IoT for Robustness and Reliability from EU IERC

Intelligent Reasoning over IoT Data from EU IERC

IoT Timelines for Future Technology Developments from EU IERC

IoT Timeline from EU IERC continued

IoT Research Needs from EU IERC

IoT Research Needs from EU IERC continued

IoT Research Needs from EU IERC continued
•

NIST CPS Framework

NIST CPS Framework
From http://www.cpspwg.org/

CPS Domains, Facets, and Aspects from NIST
From http://tinyurl.com/oyo7h9n

Assurance Facet of CPS from NIST

Realization Facet of CPS from NIST

Recommendations for Time in CPS Design from NIST

NIST-NICT JointVision for
Cyber-Physical Cloud Computing (CPCC)

Cyber-Physical Systems Requirements from NIST and NICT
From http://www2.nict.go.jp/univ-com/isp/doc/NIST.IR.7951.pdf
● Conﬁgurable/Agile
● Real Time Operation
● Big Data Support
● Reliable
● Decoupled
● Core Components
● Standardized Interfaces
● Software Deﬁned Networks
● Secure & Private
● Social Interaction
● Synchronized Clocks

Cyber-Physical Systems Components from NIST and NICT

Cyber-Physical Systems Components continued

Issues in Managing and Creating Cyber-Physical Systems
• Virtualized Sensor/Actuators
• Interconnectivity
• Data Integration (Semantics and Formats)
• Resource Provisioning
• Security & Privacy
• Performance
• Reliability
• Measurements
• Standards for CPCC

Virtualized Sensor Actuators

Interconnectivity

Data Integration (Semantics and Formats)

Resource Provisioning

Security & Privacy

Performance

Reliability

Measurements

Cyber-Physical Cloud Systems Speciﬁcation Standards Needed

Large-Scale CPS
Systems of Systems (SoS)

System of Systems Deﬁnition
From http://www.slideshare.net/RealTimeInnovations/system-architecture-for-c4i-coalition-operations

System of Systems Deﬁnition
From https://en.wikipedia.org/wiki/System_of_systems
System of systems is a collection of task-oriented or dedicated systems that pool their
resources and capabilities together to create a new, more complex system which offers
more functionality and performance than simply the sum of the constituent systems
A Global Earth Observation example from 2016.sosengineering.org

System of Systems Engineering Definition
From https://en.wikipedia.org/wiki/System_of_systems_engineering
System-of-Systems Engineering and Systems Engineering are related but different fields
of study.Whereas systems engineering addresses the development and operations of
monolithic products, SoSE addresses the development and operations of evolving
programs. In other words, traditional systems engineering seeks to optimize an
individual system (i.e., the product), while SoSE seeks to optimize network of various
interacting legacy and new systems brought together to satisfy multiple objectives of the
program. SoSE should enable the decision-makers to understand the implications of
various choices on technical performance, costs, extensibility and flexibility over time;
thus, effective SoSE methodology should prepare decision-makers to design informed
architectural solutions for System-of-Systems problems.
Many cyber-physical systems are actually systems of systems, compositions of diverse
subsystems, typically developed by diverse teams, often from different organizations.
Modularity is the problem of designing subsystems (modules) with well-defined interfaces
that can be used in a variety of contexts. Composability is the ability to combine modules.
A related concern is compositionality, which refers to the ability to understand a composite
system by interstanding its components and how they are combined.
Modularity and Composability from CPS Concept Map at http://cyberphysicalsystems.org/

Cyber-Physical System of Systems (CPSoS)
From www.cpsos.eu/wp-content/uploads/2014/12/CPSoS-Initial-Research-and-Innovation-Priorities-Document-Nov.-2014.pdf

Hybrid and Heterogeneous Models
From Cyber-Physical Systems Concept Map at http://cyberphysicalsystems.org/
Cyber-physical systems are intrinsically heterogeneous.There are two distinct approaches
to modeling heterogeneous systems: (1) a grand unified theory (GUT) and (2) an abstract
semantics.The former is about developing a modeling language and conceptual framework
into which heterogeneous modeling languages and frameworks can be translated.The latter
is about developing interfaces between heterogeneous modeling languages that are
sufficient for interoperation, but not so rich that the interface language itself becomes a
modeling language.A GUT has the advantage of enabling model exchange between tools,
but the disadvantage that the semantic richness that is required to be able to encompass all
interesting heterogeneous modeling languages makes analysis of models difficult.An
abstract semantics has the advantage of enabling composition of domain-specific modeling
languages that are themselves sufficiently constrained that analysis is still possible, but the
disadvantage that engineers must learn a multiciplicity of modeling languages and must
understand how they interact within an abstract semantics.

From RFIDs to CPS System of Systems from IERC
From http://www.internet-of-things-research.eu/pdf/Building_the_Hyperconnected_Society_IERC_2015_Cluster_eBook_978-87-93237-98-8_P_Web.pdf

CPS Conceptual Domain Model from NIST

CPSView of System of Systems from NIST

CPS and Smart Grid as System of Systems from NIST
From www.nist.gov/smartgrid/upload/SGAC-Meeting-Presentations.pdf

Smart City as System of Systems from IBM
From http://www.urenio.org/2011/06/29/ibm-redbooks-smarter-cities-series/

Wave Model of System of Systems Development
From http://tinyurl.com/jlvsrng

Characteristics of System of Systems
From http://www.cpsos.eu/wp-content/uploads/2015/07/CPSoS-Scope-paper-vOct-26-2014.pdf
• Operational independence of the components of the overall system
• Managerial independence of the components of the overall system
• Geographical distribution
• Emerging behavior
• Evolutionary development process
Example: Global Earth Observation SoS from NOAA
http://www.esrl.noaa.gov/research/themes/observing/

Characteristics of Cyber-Physical System of Systems
From http://www.seas.upenn.edu/~lee/10cis541/lecs/lec-CPS-1x2.pdf

Characteristics of Cyber-Physical System of Systems
• Large, often spatially distributed physical systems with complex dynamics
• Distributed control, supervision and management
• Partial autonomy of the subsystems
• Dynamic reconﬁguration of the overall system on different time-scales
• Possibility of emerging behaviors
• Continuous evolution of the overall system during its operation.
Elements of Large-Scale CPS SoS (Bob Marcus)
• Heterogeneous sensors and actuators
• Human interfaces for data input, analysis, control, and collaboration
• Local networks and Internet
• Distributed data exchanges and messaging
• Multi-level analytics (streaming, historical)
• Cloud support for data integration, storage, and processing

16 Characteristics of Cyber-Physical System of Systems
1. CPSs are designed and implemented in order to support human activities and well- being by
decentralized cooperative problem solving, in harmony with the techno- econo-social environment,
2. CPSs consist of a digital cyber-part and an analog physical-part, which are supposed to work
together towards the highest possible level of functional and structural synergy,
3. CPSs are functionally decentralized and geographically distributed open systems with blurred overall
system boundaries,
4. CPSs are capable not only to dynamically reconfigure their internal structure and reorganize their
functionality/behavior, but also to change their boundaries,
5. CPSs are constructed of very heterogeneous sets of active components, which can enter and leave the
collective at any time, and may encounter other systems with similar or conflicting objectives,
6. CPSs, as well as their components, may work in extreme temporal ranges (from instantaneous to
quasi-infinite, and beyond), and manifest on various spatial scales (from intercontinental to nano-
scales),
7. Components are typically hybrid structures, encapsulating various compositions of hardware (e.g.
transformer and actuator) entities and embedded cyber (e.g. software and knowledge) entities,
8. Components may have predefined, emergent or ad-hoc functional connections, or all, with other
interoperable components at multiple levels,

16Characteristics of Cyber-Physical System of Systems continued
9. Components may operate according to different problem solving strategies (plans) towards achieving
the overall objective of the system,
10. Components are knowledge-intensive and able to handle built-in formal knowledge, knowledge
obtained by sensors, and knowledge generated by reasoning and learning mechanisms,
11. Components are able to make situated decisions and strive for automated problem solving by
gathering descriptive information and applying context-dependent causal and procedural reasoning,
12 Components are able to memorize and learn from history and situations in an unsupervised manner
and to specialize themselves based on smart software agents and emergent intelligence,
13 Components are able to adapt to unpredictable system states or emergent environmental
circumstances, as well as to execute non-planned functional interactions and to act proactively,
14 Overall decision-making is distributed over a large number of components (agents), and is based on
the reﬂexive interactions among the components and multi-criteria analysis (optimization),
15 Contrary to their distributed and decentralized nature, CPSs need to operate and communicate in
real-time and in a synchronized manner,
16 System resources are managed different sophisticated strategies and maintain security, integrity and
reliability of the components and the CPSs as a whole.

System of Systems Requirements Management
From http://tinyurl.com/hod9jt6

CPS System of Systems Security Characteristics
From http://www.slideshare.net/pfroberts/cyber-physical-systems-boston-2015-1

CP SoS Technologies, Methodologies and Areas for Future Research
• Requirements engineering and model-based systems engineering and validation and
verification over the system’s full life-cycle
• Modeling and large-scale simulation of heterogeneous systems of systems
• Partially autonomous decision making and system-wide control and coordination
• Collaborative decision making by computer systems and humans
• Fault detection, resilience, reconfiguration, and integration of new component
• Large-scale online data analysis and feature extraction (“artificial cognition”) for the
analysis and the dynamic management of systems of system
• Trust in large distributed systems
• Stability, structure formation and emergent behaviour in cyber-physical systems of systems

Managing Scale and Heterogeneity in CPS SoS
From http://cps-vo.org/ﬁle/20908/download/60342f

CPSOS Project Roadmap
From www.cpsos.eu/wp-content/uploads/2015/12/Roadmap-ICT-Info-Day-Brussels-Dec.-1-2015.pdf

CPS System of Systems from IMC-AESOP Project
From http://imc-aesop.org/html/ProjDes.html

Example: Vertical CPS SoS from Bob Marcus
Embedded
Systems
Networked
Embedded Systems
Internet of Things
Cyber-Physical Systems
Cyber-Physical-Social Systems
+ networking
+ multiple networks
+ analytics
+ collaboration
Devices
+ microprocessors

Example: Horizontal CPS SoS from Bob Marcus
Embedded
Systems
Locally Networked
Embedded Systems
Devices
Embedded
Systems
Locally Networked
Embedded Systems
Devices
Embedded
Systems
Locally Networked
Embedded Systems
Devices

Example: Horizontal andVertical CPS SoS from Bob Marcus
Embedded
Systems
Locally Networked
Embedded Systems
Devices
Embedded
Systems
Locally Networked
Embedded Systems
Devices
Embedded
Systems
Locally Networked
Embedded Systems
Devices
Embedded
Systems
LocallyNetworked
Embedded Systems
Devices
IP
IP
IP
IP
IP
IP IP
Internet of Things
Coordination
Internet of Things
Coordination
Cloud-based Analytics, Management, and Control for Cyber-Physical System of Systems

A Key Problem for Engineering Large Scale CPS from Bob Marcus
A key problem for the future of Large Scale CPS System of Systems will be the
robust interfacing of heterogeneous systems. Some initial success should be
achievable horizontally and vertically across systems in the same domain using
emerging standardizations and tools. However applications such as Smart Cities
will require the interfacing of systems from many different domains with diverse
application models, data processing and control structures.

Example: Horizontal andVertical CP SoS Smart City Architecture
From http://www.slideshare.net/SchneiderElectric/smart-city-anditregislargilier

Complex Systems

A complex system is any system featuring a large number of interacting
components (agents, processes, etc.) whose aggregate activity is nonlinear (not
derivable from the summations of the activity of individual components) and
typically exhibits hierarchical self-organization under selective pressures.
Complex Systems Deﬁnition
From http://www.informatics.indiana.edu/rocha/publications/complex/csm.html
From doursat.free.fr/pubs.html

Scalability and Complexity Management
Cyber-physical systems are inherently heterogeneous, since at a minimum they combine
physical dynamics with computational processes. But they are often heterogeneous even
within the physical and cyber domains.The physical domain may be multi-physics, combining
for example mechanical motion control, chemical processes, biological processes, and
human operators.The cyber domain may combine networking technologies, programming
languages, software component models, and concurrency mechanisms. Software in CPS
applications can grow to very large systems.The challenge, therefore, is to provide design
methodologies and tools that support those methodologies, that scale to large designs,
facilitate analysis, and promote understanding of complex systems.The problems include:
• Systems engineering;
• Software engineering processes;
• Software engineering technologies (refactoring tools, program analysis, etc.)
• Design tools;
• Co-simulation technologies;
• Model exchange.

Tradition System Engineering vs. Complex Systems Engineering
From https://www.mitre.org/sites/default/ﬁles/pdf/norman_engineering.pdf

Some Possible Complex Systems Characteristics from Wikipedia
From https://en.wikipedia.org/wiki/Complex_system
• Feedback loops
• Some degree of spontaneous order
• Robustness of the order
• Emergent organization
• Numerosity [large scale]
• Hierarchical organization

Complex Systems Properties from Bob Marcus
From http://www.slideshare.net/bobmarcus/2010-complex-systems-engineering-for-the-global-information-grid

Complex Systems Characteristics
-
From http://www.acq.osd.mil/se/webinars/2010-03-09-SoSECIE-MS-Emergent-Behaviors-Zentner-etal-brief.pdf

Complex Systems Interaction Patterns from Bob Marcus

Complex Systems Questions from 1995 from Bob Marcus

Definition of Complex Adaptive Systems from MIT
From http://web.mit.edu/esd.83/www/notebook/Complex%20Adaptive%20Systems.pdf
Complex Adaptive Systems include many natural systems (e.g., brains, immune systems,
ecologies, societies) and increasingly, many artificial systems (parallel and distributed
computing systems, artificial intelligence systems, artificial neural networks, evolutionary
programs) that are characterized by apparently complex behaviors that emerge as a result of
often nonlinear spatio-temporal interactions among a large number of component systems at
different levels of organization.

Complex Systems Organization Map from Wikipedia
From https://en.wikipedia.org/wiki/File:Complex_systems_organizational_map.jpg

Complex Adaptive Behavior
From http://basreus.nl/tag/complex-adaptive-systems/

Modeling and Simulation (M&S)
for Engineering CPS

Speciﬁcation, Modeling, and Analysis for CPS
Cyber-physical systems are intrinsically concurrent.At a minimum, the cyber and the
physical subsystems coexist in time, but even within these subsystems, concurrent
processes are common. Models of concurrency in the physical world (coexisting physical
dynamics in a time continuum) are very different from models of concurrency in software
(arbitrary interleaving of sequences of atomic actions), and very different from models of
concurrency in networks (asynchronous, partially-ordered discrete actions or clock-driven
time slots). Reconciling these divergent models of concurrency, and ensuring
interoperability and communication between components that have divergent models of
concurrency, is a central problem in CPS.

Adaptive and Predictive Control for CPS
CPS systems are typically closed-loop systems, where sensors make measurements
of physical processes, the measurements are processed in the cyber subsystems,
which then drive actuators that affect the physical processes.The control strategies
implemented in the cyber subsystems need to be adaptive (responding to changing
conditions) and predictive (anticipating changes in the physical processes).
Model Reference Adaptive Control (MRAC) from Drexel U.
From http://www.pages.drexel.edu/~kws23/tutorials/MRAC/MRAC.html

Model-Based Systems Engineering
From http://www.lboro.ac.uk/research/avrrc/research/currentprojects/bigdata/platform-independent-model-driven-architectures-for-future-vehicle-systems.html

Aspects of Modeling CPS
From https://modelica.org/events/modelica2011/Proceedings/pages/papers/20_1_ID_121_a_fv.pdf
• The scale and complexity of CPS require compositional methods for integrated design
and modeling
• Most CPS applications are built up from controlled sub-systems, systems-of-systems,
with local interaction between physical systems and controllers affecting global
performance.
• Distributed sensing, actuation, and control needs to be modeled and simulated.
• The interfaces between the cyber and physical needs to be identiﬁed and properties of the
interfaces should be easy to specify.
• New hybrid models may be needed to completely model all aspects of CPS

Open META CyPhy Toolchain from DARPA
From http://www.slideshare.net/joe_porter/veriﬁcation-cps

Model-Based Design for CPS fromVanderbilt U
From http://www.nist.gov/el/msid/upload/1Neema_Model-Based-Tools-for-Design-releaseversion.pdf

Model Based Systems Engineering (MBSE)

Mixing Formal and Semi-Formal Approaches in MBSE

Layered Diagram for Standardized Systems and Service Development
From http://www.et-strategies.com/great-global-grid/SE-SOA-MS.pdf

MultiScale Modeling Deﬁnition
From https://en.wikipedia.org/wiki/Multiscale_modeling
In engineering, mathematics, physics, meteorology and computer science, multiscale
modeling or multiscale mathematics is the ﬁeld of solving problems which have
important features at multiple scales of time and/or space
Physical Scales for CPSfrom http://tinyurl.com/jyybtqe

Multiscale Coupling Library and Environment
From http://www.qoscosgrid.org/trac/muscle

Features of a CPS Design and M&S Framework
From http://www.cs.virginia.edu/sigbed/archives/2008-01/JKim.pdf
• Heterogeneous (in terms of various types of sensors/actuators) applications support: CPS usually consists of non-
homogeneous applications.Thus, it should be able to simulate heterogeneous application logics simultaneously.
• Various physical modeling environments: the physical modeling environment should support mathematical
expressions and incorporate domain specific physical modeling descriptions (e.g. floor plan of buildings) by
extracting relevant information from them.
• Scalability support: support for the development and simulation ranging from small scale (tens) to large scale
(thousands of) sensors and actuators.
• Mobility support: support for modeling systems using relevant properties (e.g., communication, signal strength).
• Integration of existing simulation tools: easy-to-use support to link to existing simulation tools is required.
• Integration of proprietary solutions and open standard support: proprietary solutions and open standards including
protocols, infrastructures and existing software should be able to be easily incorporated into a generic framework.
• Software reuse: a generic framework should support software reuse either by exploiting code generation
techniques (which can also use proprietary infrastructure), linking libraries or using configurable components.
• Usability: Graphical representation of modeling and simulation environment can enable easy development of new
applications. Domain-specific3D modeling environments can also be supported depending upon requirements.

Modeling and Simulation Enhanced SoS Engineering by Bob Marcus
From http://www.et-strategies.com/great-global-grid/emerging-trends.pdf

Simulation for CPS
Simulation is the process of validating a design by imitating its behavior for a given set of
inputs. Particular CPS challenges include
• Heterogeneous simulation: co-simulation of diverse physical and cyber subsystems;
• Multiresolution simulation: cosimulating subsystems expressed at different levels of
abstraction or with different time scales and precisions;
• Models of time: distributed cyber-physical systems cannot precisely share a single
measurement of time, and discrepancies in their measurements can lead to unexpected
artifacts, so simulators need to accurately models these discrepancies.
• Hardware in the loop simulation:This is where a subsystem simulation interacts in real
time with hardware realizing either a physical or a cyber susbsystem.

Need for IoT Simulations
From http://futureinstruments.ornl.gov/pdfs/T1_N%201420%20IOT_FIIW15_050415.pdf

Unique Requirements of IoT Simulations

SimpleIoTSimulator Supported Protocols
From http://www.smplsft.com/SimpleIoTSimulator.html
• CoAP is an IETF proposed standard for retrieving and managing information for sensors and devices in a
constrained environment.The simulator can "learn" from existing CoAP sensors/devices to duplicate customer
environments, or use the learnt data as template to create thousands of sensors and gateways.
• MQTT is a publish/subscribe based protocol. Both MQTT ver 3.1 and ver 3.1.1 clients are supported and simulated
sensors can be setup to periodically publish messages to a specified broker.A built-in learner utility is also included
that subscribes to a broker and learns messages for subseqent replay.
• MQTT-SN is a variation of MQTT for Sensor Networks that has a more compact packet encoding. Like MQTT,
simualted sensors can be setup to periodically publish MQTT-SN client messages to a specified broker and a built-in
learner utility is included to learn messages for subsequent replay.
• MQTT-Broker receives MQTT subscribe requests from applications within the cloud/platform and sends publish
messages to them.The simulator supports this functionality to simulate thousands of gateways.
• HTTP/s client sends periodic XML/REST requests to cloud/platform servers.The simulator includes learner
applications to learn http requests and periodically send them to specified servers for simulating thousands of
gateways.
• HTTP/s server responds to incoming HTTP requests with responses.The simulator can be setup using learnt
data for simulating gateways that support this functionality.

Simulation Testbed Model from ORNL

Federated CPS Simulations from NIST
From http://www.nist.gov/smartgrid/upload/SGAC-Meeting-Presentations.pdf

M&S for Complex Networked Systems (Research Areas from DOE)
From http://science.energy.gov/~/media/ascr/pdf/program-documents/docs/Complex_networked_systems_program_ﬁnal.pdf

FIESTA IoT Testbed from Europe
From www.com4innov.com/var/input/FileManager/PAGES_WEB/FIESTA/FIESTA_Fact_Sheet 2015.pdf?PHPSESSID=0onnd2irqan1o82b9pc4sg9bj6

FIESTA IoT Testbed from Europe
From www.com4innov.com/var/input/FileManager/PAGES_WEB/FIESTA/FIESTA_Fact_Sheet 2015.pdf?PHPSESSID=0onnd2irqan1o82b9pc4sg9bj6
Federation Model for IoT Testing
Experiment as a Service Scenario
IoT Functional Blocks Architecture

References
Inventory of all Bob Marcus CPS Slides on Slideshare
http://www.slideshare.net/bobmarcus/inventory-of-my-cps-slide-sets

Reference Links (Systems Engineering for CPS)
Institute for Software Integrated Systems (ISIS) Lab Vanderbilt CPS Research
http://www.isis.vanderbilt.edu/research/SST
Workshop on Cyber-Physical System Engineering
https://es-static.fbk.eu/events/eitcpse2013/program.php
Cyber-Physical Systems Design Challenges
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.156.9348&rep=rep1&type=pdf
Cyber-Physical Society
http://www.computer.org/csdl/proceedings/wetice/2015/7692/00/7692a114.pdf
Engineering Cyber-Physical Systems Book
http://link.springer.com/chapter/10.1007%2F978-3-642-34404-6_1#page-1
Cyber Physical Systems: The Next Computing Revolution
http://www.seas.upenn.edu/~lee/10cis541/lecs/lec-CPS-1x2.pdf
Cyber-Physical Systems Design Challenges
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.156.9348&rep=rep1&type=pd
Mechatronics for Cyber-Physical Systems
http://ade.sagepub.com/content/6/591629.full
Top 49 tools for the Internet of Things
https://blog.proﬁtbricks.com/top-49-tools-internet-of-things/
IoT Open Platforms
http://open-platforms.eu/libraries/
Patterns for building IoT systems from Iron.Io
http://resources.idgenterprise.com/original/AST-0162936_Ironio-IoT-Whitepaper-Jun2015_4.pdf
AT&T Collaboration with Cisco, Microsoft, and Intel
http://about.att.com/story/att_grows_iot_ecosystem.html

Reference Links (CPS System of Systems and Consortia)
Cyber-Physical Systems Engineering with Examples of Systems of Systems
http://www.utexas.edu/research/cem/smartgrid_images/UTSA%20Forum/Jamshidi%20-%20UT_Austin_Lec.pdf
AMADEOS (Architecture for Multi-Critically Agile Dependable Evolutionary Open System of Systems)
http://amadeos-project.eu/project/
System of Systems Project from Europe
https://ec.europa.eu/digital-agenda/en/system-systems
Cyber-Physical Virtual Organization
http://cps-vo.org/
Road2CPS Initiative in Europe
http://www.road2cps.eu/#
CPS Engineering Labs in Europe
https://ec.europa.eu/digital-agenda/en/news/cyber-physical-systems-engineering-labs-ﬁrst-open-call-innovation-projects
http://www.cpse-labs.eu/
Smart Anything Everywhere Initiative in Euope
https://smartanythingeverywhere.eu/
EuroCPS Projects
https://www.eurocps.org/
European Research Cluster on IoT (IERC)
http://www.internet-of-things-research.eu/
Downloadable Book on Building the Hyper-Connected Society from IERC
http://www.internet-of-things-research.eu/pdf/Building_the_Hyperconnected_Society_IERC_2015_Cluster_eBook_978-87-93237-98-8_P_Web.pdf

Reference Links (CPS as Complex Systems)
Building Self-Adaptive Cyber-Physical Systems using Unreliable Components
http://www.ee.washington.edu/research/nsl/aar-cps/WeisongShi-20081015085116.pdf
Complex Adaptive System of Systems Engineering
www.sandia.gov/casosengineering/
Complex Adaptive Systems Modeling
http://www.casmodeling.com/
Complex Systems Engineering for the Global Information Grid
http://www.slideshare.net/bobmarcus/2010-complex-systems-engineering-for-the-global-information-grid
Progress on Complex Cyber-Physical Systems Engineering
http://www.slideshare.net/ghackenberg/research-group-seminar-progress-on-complex-cyberphysical-systems-engineering
Multiple Drones Collision Avoidance
haoyi.io/projects/short-term-conf-reso.pdf

Reference Links (Modeling and Simulation for CPS)
Modelica for CPS
https://modelica.org/events/modelica2011/Proceedings/pages/papers/20_1_ID_121_a_fv.pdf
Model Based Systems Engineering for System of Systems using Agent-Based Modeling
http://tinyurl.com/jlvsrng
DPWSim: A Simulation Toolkit for IoT Applications
http://servicearchitecture.wp.tem-tsp.eu/ﬁles/2014/10/DPWSim-A-Simulation-Toolkit-for-IoT-Applications-using-
A Framework for Multiscale-Multiscience Modeling and Simulation from University of Geneva
http://www.lorentzcenter.nl/lc/web/2013/569/presentations/03.%20Chopard.pdf
Using Complex Event Processing for the Modeling and Simulation of Cyber-Physical Systems
https://www.researchgate.net/publication/264815514_Using_complex_event_processing_for_modelling_and_simulation_of_cyber-physical_systems
Model Based Design of Cyber-Physical Systems using MATLAB and Simulink
www.mathworks.com/discovery/cyber-physical-systems.html?requestedDomain=www.mathworks.com
Methodology for Design, Integration, Modeling, and Simulation of Cyber-Physical Systems SIG
https://www.designsociety.org/ds_group/30/methodology_for_design_integration_modeling_and_simulation_of_cyber_physical_systems_design_cps
Modeling and Simulation of Complex Systems
http://eco83.econ.unito.it/dottorato/michele_sonnessa/sonnessa_thesis.pdf
CoSMo Complex Systems Modeling
http://www.thecosmocompany.com/technology
CoSMoS Process for Complex Systems Modeling and Simulation from University of York
http://offog.org/publications/york2010-cosmos.pdf
Federated Interoperable Semantic IoT/cloud Testbeds and Applications (FIESTA) in Europe
http://www.com4innov.com/var/input/FileManager/PAGES_WEB/FIESTA/FIESTA_Fact_Sheet%202015.pdf?PHPSESSID=0onnd2irqan1o82b9pc4sg9bj6

Engineering Large Scale Cyber-Physical Systems

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (13)

Similar to Engineering Large Scale Cyber-Physical Systems

Similar to Engineering Large Scale Cyber-Physical Systems (20)

Recently uploaded

Recently uploaded (20)

Engineering Large Scale Cyber-Physical Systems