This document provides a good management-lever introduction to the Data-Distribution Service (DDS) technology and capabilities. It was prepared by the OMG at the request of the US Navy in order to educate on the data-centric software architectural principles of DDS and how they can help meet its agility and cost-control requirements.
Design and implementation of a personal super Computerijcsit
Resources of personal devices, whether mobile or stationary, can be productively leveraged to service their
users. By doing so, personal users will be able to ubiquitously run relatively complex computational jobs,
which cannot be accommodated in their individual personal devices or while they are on the move. To this
end, the paper proposes a Personal Super Computer (PSC) that superimpose grid functionality over
networked personal devices. In this paper, architectural designs of (PSC) were developed and evaluated
thoroughly through a strictly controlled empirical evaluation framework. The results showed that this
system has successfully maintained high speedup over regular personal computers under different running
conditions.
A survey of peer-to-peer content distribution technologiessharefish
This document provides a survey of peer-to-peer content distribution technologies. It begins with defining key concepts of peer-to-peer computing and classifying peer-to-peer systems. The focus is on content distribution systems, which allow personal computers to function as a distributed storage medium for digital content. The document proposes a framework for analyzing nonfunctional characteristics and architectural designs of current peer-to-peer content distribution systems.
INTRUSION DETECTION AND MARKING TRANSACTIONS IN A CLOUD OF DATABASES ENVIRONMENTijccsa
The cloud computing is a paradigm for large scale distributed computing that includes several existing
technologies. A database management is a collection of programs that enables you to store, modify and
extract information from a database. Now, the database has moved to cloud computing, but it introduces at
the same time a set of threats that target a cloud of database system. The unification of transaction based
application in these environments present also a set of vulnerabilities and threats that target a cloud of
database environment. In this context, we propose an intrusion detection and marking transactions for a
cloud of database environment.
This document proposes an integrated cloud-based framework for collecting and processing sensory data from mobile phones to support diverse people-centric applications. The framework includes modules for user adaptation, storage, application interfaces, and mobile cloud engines. A prototype is implemented to demonstrate how the framework can reduce mobile device energy consumption while meeting application requirements such as for emergency response systems.
This document discusses the need for adaptive and dynamic software development that can adjust to changing runtime environments and fault conditions. It argues that traditional static approaches to fault tolerance, like using fixed levels of redundancy, are inadequate as the threat environment may vary. The document then introduces an adaptive data integrity tool that allows the level of redundancy to change dynamically based on faults detected at runtime. This provides an example of the new approach called for, termed "New Software Development," that is more adaptive, maintainable and reconfigurable like New Product Development concepts.
Quality of Service in Publish/Subscribe MiddlewareAngelo Corsaro
During the last decade the publish/subscribe communication paradigm gained a central role in the design and development of a large class of applications ranging from stock exchange systems to news tickers, from air traffic control to defense systems. This success is mainly due to the capacity of publish/subscribe to completely decouple communication participants, thus allowing the development of applications that are more tolerant to communications asynchrony. This chapter introduces the publish/subscribe communication paradigm, stressing those charac- teristics that have a stronger impact on the quality of service provided to partic- ipants. The chapter also introduce the reader to two widely recognized industrial standards for publish/subscribe systems: the Java Message Service (JMS) and the Data Distribution Service (DDS).
The document proposes a Cloud Information Accountability (CIA) framework to provide accountability for data sharing in the cloud. The framework uses a decentralized, object-centered approach where data owners can enclose data and policies within programmable JAR files. Any access to the data will trigger automated logging stored locally within the JARs. The framework provides efficient, scalable and granular accountability while meeting the dynamic needs of the cloud. Experiments demonstrate the framework's performance.
IRJET- A Survey on File Storage and Retrieval using Blockchain TechnologyIRJET Journal
This document discusses using blockchain technology for secure file storage and retrieval. It first describes existing technologies like distributed file systems, InterPlanetary File System (IPFS), storing file hashes on blockchain, Filecoin, and Storj. It then proposes a system using Ethereum, Swarm, and Whisper that encrypts files before storing encrypted blocks on Swarm and recording hashes on blockchain. File access permissions are shared via Whisper messages. This decentralized system improves security, accessibility, and avoids data redundancy compared to traditional methods.
Design and implementation of a personal super Computerijcsit
Resources of personal devices, whether mobile or stationary, can be productively leveraged to service their
users. By doing so, personal users will be able to ubiquitously run relatively complex computational jobs,
which cannot be accommodated in their individual personal devices or while they are on the move. To this
end, the paper proposes a Personal Super Computer (PSC) that superimpose grid functionality over
networked personal devices. In this paper, architectural designs of (PSC) were developed and evaluated
thoroughly through a strictly controlled empirical evaluation framework. The results showed that this
system has successfully maintained high speedup over regular personal computers under different running
conditions.
A survey of peer-to-peer content distribution technologiessharefish
This document provides a survey of peer-to-peer content distribution technologies. It begins with defining key concepts of peer-to-peer computing and classifying peer-to-peer systems. The focus is on content distribution systems, which allow personal computers to function as a distributed storage medium for digital content. The document proposes a framework for analyzing nonfunctional characteristics and architectural designs of current peer-to-peer content distribution systems.
INTRUSION DETECTION AND MARKING TRANSACTIONS IN A CLOUD OF DATABASES ENVIRONMENTijccsa
The cloud computing is a paradigm for large scale distributed computing that includes several existing
technologies. A database management is a collection of programs that enables you to store, modify and
extract information from a database. Now, the database has moved to cloud computing, but it introduces at
the same time a set of threats that target a cloud of database system. The unification of transaction based
application in these environments present also a set of vulnerabilities and threats that target a cloud of
database environment. In this context, we propose an intrusion detection and marking transactions for a
cloud of database environment.
This document proposes an integrated cloud-based framework for collecting and processing sensory data from mobile phones to support diverse people-centric applications. The framework includes modules for user adaptation, storage, application interfaces, and mobile cloud engines. A prototype is implemented to demonstrate how the framework can reduce mobile device energy consumption while meeting application requirements such as for emergency response systems.
This document discusses the need for adaptive and dynamic software development that can adjust to changing runtime environments and fault conditions. It argues that traditional static approaches to fault tolerance, like using fixed levels of redundancy, are inadequate as the threat environment may vary. The document then introduces an adaptive data integrity tool that allows the level of redundancy to change dynamically based on faults detected at runtime. This provides an example of the new approach called for, termed "New Software Development," that is more adaptive, maintainable and reconfigurable like New Product Development concepts.
Quality of Service in Publish/Subscribe MiddlewareAngelo Corsaro
During the last decade the publish/subscribe communication paradigm gained a central role in the design and development of a large class of applications ranging from stock exchange systems to news tickers, from air traffic control to defense systems. This success is mainly due to the capacity of publish/subscribe to completely decouple communication participants, thus allowing the development of applications that are more tolerant to communications asynchrony. This chapter introduces the publish/subscribe communication paradigm, stressing those charac- teristics that have a stronger impact on the quality of service provided to partic- ipants. The chapter also introduce the reader to two widely recognized industrial standards for publish/subscribe systems: the Java Message Service (JMS) and the Data Distribution Service (DDS).
The document proposes a Cloud Information Accountability (CIA) framework to provide accountability for data sharing in the cloud. The framework uses a decentralized, object-centered approach where data owners can enclose data and policies within programmable JAR files. Any access to the data will trigger automated logging stored locally within the JARs. The framework provides efficient, scalable and granular accountability while meeting the dynamic needs of the cloud. Experiments demonstrate the framework's performance.
IRJET- A Survey on File Storage and Retrieval using Blockchain TechnologyIRJET Journal
This document discusses using blockchain technology for secure file storage and retrieval. It first describes existing technologies like distributed file systems, InterPlanetary File System (IPFS), storing file hashes on blockchain, Filecoin, and Storj. It then proposes a system using Ethereum, Swarm, and Whisper that encrypts files before storing encrypted blocks on Swarm and recording hashes on blockchain. File access permissions are shared via Whisper messages. This decentralized system improves security, accessibility, and avoids data redundancy compared to traditional methods.
The advent of Big Data has seen the emergence of new processing and storage challenges. These challenges are often solved by distributed processing. Distributed systems are inherently dynamic and unstable, so it is realistic to expect that some resources will fail during use. Load balancing and task scheduling is an important step in determining the performance of parallel applications. Hence the need to design load balancing algorithms adapted to grid computing. In this paper, we propose a dynamic and hierarchical load balancing strategy at two levels: Intrascheduler load balancing, in order to avoid the use of the large-scale communication network, and interscheduler load balancing, for a load regulation of our whole system. The strategy allows improving the average response time of CLOAK-Reduce application tasks with minimal communication. We first focus on the three performance indicators, namely response time, process latency and running time of MapReduce tasks.
Survey on Division and Replication of Data in Cloud for Optimal Performance a...IJSRD
Outsourcing information to an outsider authoritative control, as is done in distributed computing, offers ascend to security concerns. The information trade off may happen because of assaults by different clients and hubs inside of the cloud. Hence, high efforts to establish safety are required to secure information inside of the cloud. On the other hand, the utilized security procedure should likewise consider the advancement of the information recovery time. In this paper, we propose Division and Replication of Data in the Cloud for Optimal Performance and Security (DROPS) that all in all methodologies the security and execution issues. In the DROPS procedure, we partition a record into sections, and reproduce the divided information over the cloud hubs. Each of the hubs stores just a itary part of a specific information record that guarantees that even in the event of a fruitful assault, no important data is uncovered to the assailant. Additionally, the hubs putting away the sections are isolated with certain separation by method for diagram T-shading to restrict an assailant of speculating the areas of the sections. Moreover, the DROPS procedure does not depend on the customary cryptographic procedures for the information security; in this way alleviating the arrangement of computationally costly approaches. We demonstrate that the likelihood to find and bargain the greater part of the hubs putting away the sections of a solitary record is to a great degree low. We likewise analyze the execution of the DROPS system with ten different plans. The more elevated amount of security with slight execution overhead was watched.
How to Develop True Distributed Simulations? HLA & DDS InteroperabilityJose Carlos Diaz
This document discusses two publish/subscribe communication standards for distributed simulations: HLA and DDS. It introduces NCWare, a middleware developed by Nextel Engineering that unifies HLA and DDS. NCWare allows applications to interoperate between HLA and DDS by mapping data between their models and providing a single API. It also improves performance over HLA by using DDS as the underlying transport protocol while maintaining HLA semantics and services. NCWare provides a simple interface and real-time QoS capabilities to enable true distributed simulations using both HLA and DDS standards.
Addressing the Challenges of Tactical Information Management in Net-Centric S...Angelo Corsaro
This paper provides an overview of the advantages provided by the OMG Data Distribution Service for Real-Time Systemts (DDS) for addressing the challenges associated with Tactical Information distribution.
CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...ijcsit
In supporting its large scale, multidisciplinary scientific research efforts across all the university campuses and by the research personnel spread over literally every corner of the state, the state of Nevada needs to build and leverage its own Cyber infrastructure. Following the well-established as-a-service model, this state-wide Cyber infrastructure that consists of data acquisition, data storage, advanced instruments, visualization, computing and information processing systems, and people, all seamlessly linked together through a high-speed network, is designed and operated to deliver the benefits of Cyber infrastructure-as-aService (CaaS).There are three major service groups in this CaaS, namely (i) supporting infrastructural
services that comprise sensors, computing/storage/networking hardware, operating system, management tools, virtualization and message passing interface (MPI); (ii) data transmission and storage services that provide connectivity to various big data sources, as well as cached and stored datasets in a distributed
storage backend; and (iii) processing and visualization services that provide user access to rich processing and visualization tools and packages essential to various scientific research workflows. Built on commodity hardware and open source software packages, the Southern Nevada Research Cloud(SNRC)and a data repository in a separate location constitute a low cost solution to deliver all these services around CaaS. The service-oriented architecture and implementation of the SNRC are geared to encapsulate as much detail of big data processing and cloud computing as possible away from end users; rather scientists only need to learn and access an interactive web-based interface to conduct their collaborative, multidisciplinary, dataintensive research. The capability and easy-to-use features of the SNRC are demonstrated through a use case that attempts to derive a solar radiation model from a large data set by regression analysis.
The huge volume of text documents available on the internet has made it difficult to find valuable
information for specific users. In fact, the need for efficient applications to extract interested knowledge
from textual documents is vitally important. This paper addresses the problem of responding to user
queries by fetching the most relevant documents from a clustered set of documents. For this purpose, a
cluster-based information retrieval framework was proposed in this paper, in order to design and develop
a system for analysing and extracting useful patterns from text documents. In this approach, a pre-
processing step is first performed to find frequent and high-utility patterns in the data set. Then a Vector
Space Model (VSM) is performed to represent the dataset. The system was implemented through two main
phases. In phase 1, the clustering analysis process is designed and implemented to group documents into
several clusters, while in phase 2, an information retrieval process was implemented to rank clusters
according to the user queries in order to retrieve the relevant documents from specific clusters deemed
relevant to the query. Then the results are evaluated according to evaluation criteria. Recall and Precision
(P@5, P@10) of the retrieved results. P@5 was 0.660 and P@10 was 0.655.
Paper MIE2016 from Proceedings pags 122-126vilaltajo
This document proposes using a Document Archiving and Communication System (DACS) to provide secure storage and access to clinical documents in a standardized way. A DACS would function similarly to a PACS for medical images. It describes the key components of a DACS including a web portal, server, and databases to store documents formatted according to HL7 CDA standards. Privacy rules and security protocols like SAML and XACML are used to control access to documents based on user roles. Clinical documents can be queried through the DACS using a finder application to retrieve time-series data, summaries, and facilitate complex searches across repositories.
Implementing data-driven decision support system based on independent educati...IJECEIAES
Decision makers in the educational field always seek new technologies and tools, which provide solid, fast answers that can support decision-making process. They need a platform that utilize the students’ academic data and turn them into knowledge to make the right strategic decisions. In this paper, a roadmap for implementing a data driven decision support system (DSS) is presented based on an educational data mart. The independent data mart is implemented on the students’ degrees in 8 subjects in a private school (AlIskandaria Primary School in Basrah province, Iraq). The DSS implementation roadmap is started from pre-processing paper-based data source and ended with providing three categories of online analytical processing (OLAP) queries (multidimensional OLAP, desktop OLAP and web OLAP). Key performance indicator (KPI) is implemented as an essential part of educational DSS to measure school performance. The static evaluation method shows that the proposed DSS follows the privacy, security and performance aspects with no errors after inspecting the DSS knowledge base. The evaluation shows that the data driven DSS based on independent data mart with KPI, OLAP is one of the best platforms to support short-tolong term academic decisions.
Multi-Agent systems (Autonomous agents or agents) and knowledge discovery (or data mining) are two active
areas in information technology. A profound insight of bringing these two communities together has unveiled a tremendous
potential for new opportunities and wider applications through the synergy of agents and data mining. Multi-agent systems
(MAS) often deal with complex applications that require distributed problem solving. In many applications the individual and
collective behavior of the agents depends on the observed data from distributed data sources. Data mining technology has
emerged, for identifying patterns and trends from large quantities of data. The increasing demand to scale up to massive data sets
inherently distributed over a network with limited band width and computational resources available motivated the development of
distributed data mining (DDM).Distributed data mining is originated from the need of mining over decentralized data
sources. DDM is expected to perform partial analysis of data at individual sites and then to send the outcome as partial result
to other sites where it sometimes required to be aggregated to the global result
The document proposes a Cloud Information Accountability (CIA) framework to provide distributed accountability for data sharing in the cloud. The CIA framework uses Java ARchive (JAR) files to automatically log any access to user data in the cloud. When user data is sent to cloud service providers, it is enclosed in JAR files along with access policies and a logging mechanism. Any access to the data will then trigger authenticated logging local to the JAR files. This provides a decentralized yet enforceable way to track how user data is used throughout the dynamic cloud environment. The authors implement and test the CIA framework on a cloud testbed and find that it efficiently and effectively provides accountability for user data in distributed cloud systems.
This document discusses distributed databases. It begins by introducing distributed database systems and their structure. Key points include that the database is split across multiple computers that communicate over a network. It then discusses the tradeoffs of distributing a database, such as increased availability but also higher complexity. The document outlines two approaches to distributing data - replication, where copies of data are stored at different sites, and fragmentation, where relations are split into pieces stored at different sites. It provides examples to illustrate these concepts.
This document summarizes a research paper that proposes a scheme for ensuring security and reliability of data stored in the cloud. The scheme utilizes erasure coding to redundantly store encrypted data fragments across multiple cloud servers. It generates homomorphic tokens that allow auditing of the data storage and identification of any misbehaving servers. The scheme supports secure dynamic operations like modification, deletion and append of cloud data files. Analysis shows the scheme is efficient and resilient against various security threats like server compromises or failures. It ensures storage correctness and fast localization of data errors in the cloud.
The AIRCC's International Journal of Computer Science and Information Technology (IJCSIT) is devoted to fields of Computer Science and Information Systems. The IJCSIT is a open access peer-reviewed scientific journal published in electronic form as well as print form. The mission of this journal is to publish original contributions in its field in order to propagate knowledge amongst its readers and to be a reference publication.
Blockchain enabled task and time sheet management for accounting services pro...Conference Papers
This document describes a blockchain-enabled timesheet management system for accounting firms. It aims to improve on traditional centralized timesheet databases which are vulnerable to tampering. The proposed system uses blockchain to immutably store task and timesheet data, including check-in/out times. This ensures accuracy and avoids issues like overclaiming hours. The document outlines the system architecture, which features a frontend app and blockchain backend on Hyperledger Fabric. Timesheet records are added to the blockchain using smart contracts. Preliminary results found blockchain improved aspects like organizational management, cost savings, transparency and data security compared to traditional methods.
This document discusses using Hidden Markov Model (HMM) forward chaining techniques for prefetching in distributed file systems (DFS) for cloud computing. It begins by introducing DFS for cloud storage and issues like load balancing. It then discusses using HMM to analyze client I/O and predict future requests to prefetch relevant data. The HMM forward algorithm would be used to prefetch data from storage servers to clients proactively. This could improve performance by reducing client wait times for requested data in DFS for cloud applications.
iaetsd Controlling data deuplication in cloud storageIaetsd Iaetsd
This document discusses controlling data deduplication in cloud storage. It proposes an architecture that provides duplicate check procedures with minimal overhead compared to normal cloud storage operations. The key aspects of the proposed system are:
1) It uses convergent encryption to encrypt data for privacy while still allowing for deduplication of duplicate files.
2) It introduces a private cloud that manages user privileges and generates tokens for authorized duplicate checking in a hybrid cloud architecture.
3) It evaluates the overhead of the proposed authorized duplicate checking scheme and finds it incurs negligible overhead compared to normal cloud storage operations.
IRJET- A Novel Framework for Three Level Isolation in Cloud System based ...IRJET Journal
This document proposes a novel three-level isolation framework for cloud storage based on fog computing. The framework aims to address privacy and security issues in cloud storage by distributing user data across three layers - cloud servers, fog servers, and local machines. It uses a hash-Solomon encoding algorithm to split user data into multiple shares and store each share in a different layer. This provides three-way redundancy to protect against data loss and enhances security by isolating data across multiple environments. Theoretical analysis and experimental evaluation demonstrate the feasibility and security improvements of the proposed three-level isolation framework compared to existing cloud storage schemes.
1) The document proposes a privacy-preserving public auditing scheme for cloud storage. It allows a third party auditor (TPA) to efficiently check the integrity of outsourced data in the cloud without learning anything about the data contents.
2) The scheme utilizes homomorphic linear authenticators to generate proofs of data storage correctness, enabling TPA to perform audits without accessing the entire data files. It also supports batch auditing so that TPA can simultaneously audit multiple users' delegated tasks.
3) The scheme aims to achieve public auditability, storage correctness, privacy-preserving audits, and lightweight computation overhead during the auditing process. It analyzes the security and performance of the proposed privacy
Privacy preserving public auditing for secure cloud storageMuthu Sybian
1) The document proposes a privacy-preserving public auditing scheme for cloud storage. It allows a third party auditor (TPA) to efficiently check the integrity of outsourced data in the cloud without learning anything about the data contents.
2) The scheme utilizes homomorphic linear authenticators to generate proofs of data storage correctness, enabling TPA to audit multiple tasks simultaneously in a batch manner with minimal overhead.
3) Extensive analysis shows the scheme is provably secure and efficient, addressing key issues of public auditability, data privacy, and lightweight verification.
Interoperability demonstration between 5 different products that implement the OMG DDS Interoperability Wire Protocol (DDS-RTPS).
The demonstration took place at the March 2011 OMG technical meeting in Washington DC.
The following companies demonstrated interoperability between their products: RTI (Connext DDS). TwinOaks Computing (CoreDX), PrismTech (OpenSpliceDDS), Gallium Visual Systems/Kongsberg (Compass DDS), IBM.
High-level introduction to the OMG Data Distribution Service (DDS) standard and how it provides values beyond what is possible with traditional messaging middleware such as JMS or AMQP.
Interoperability demonstration between 7 different products that implement the OMG DDS Interoperability Wire Protocol (DDS-RTPS).
The demonstration took place at the June 2013 OMG technical meeting in Berlin Germany.
The following companies demonstrated interoperability between their products: RTI (Connext DDS). TwinOaks Computing (CoreDX), PrismTech (OpenSpliceDDS), OCI (OpenDDS), ETRI (ETRI DDS), NADS, and RemedyRT.
The advent of Big Data has seen the emergence of new processing and storage challenges. These challenges are often solved by distributed processing. Distributed systems are inherently dynamic and unstable, so it is realistic to expect that some resources will fail during use. Load balancing and task scheduling is an important step in determining the performance of parallel applications. Hence the need to design load balancing algorithms adapted to grid computing. In this paper, we propose a dynamic and hierarchical load balancing strategy at two levels: Intrascheduler load balancing, in order to avoid the use of the large-scale communication network, and interscheduler load balancing, for a load regulation of our whole system. The strategy allows improving the average response time of CLOAK-Reduce application tasks with minimal communication. We first focus on the three performance indicators, namely response time, process latency and running time of MapReduce tasks.
Survey on Division and Replication of Data in Cloud for Optimal Performance a...IJSRD
Outsourcing information to an outsider authoritative control, as is done in distributed computing, offers ascend to security concerns. The information trade off may happen because of assaults by different clients and hubs inside of the cloud. Hence, high efforts to establish safety are required to secure information inside of the cloud. On the other hand, the utilized security procedure should likewise consider the advancement of the information recovery time. In this paper, we propose Division and Replication of Data in the Cloud for Optimal Performance and Security (DROPS) that all in all methodologies the security and execution issues. In the DROPS procedure, we partition a record into sections, and reproduce the divided information over the cloud hubs. Each of the hubs stores just a itary part of a specific information record that guarantees that even in the event of a fruitful assault, no important data is uncovered to the assailant. Additionally, the hubs putting away the sections are isolated with certain separation by method for diagram T-shading to restrict an assailant of speculating the areas of the sections. Moreover, the DROPS procedure does not depend on the customary cryptographic procedures for the information security; in this way alleviating the arrangement of computationally costly approaches. We demonstrate that the likelihood to find and bargain the greater part of the hubs putting away the sections of a solitary record is to a great degree low. We likewise analyze the execution of the DROPS system with ten different plans. The more elevated amount of security with slight execution overhead was watched.
How to Develop True Distributed Simulations? HLA & DDS InteroperabilityJose Carlos Diaz
This document discusses two publish/subscribe communication standards for distributed simulations: HLA and DDS. It introduces NCWare, a middleware developed by Nextel Engineering that unifies HLA and DDS. NCWare allows applications to interoperate between HLA and DDS by mapping data between their models and providing a single API. It also improves performance over HLA by using DDS as the underlying transport protocol while maintaining HLA semantics and services. NCWare provides a simple interface and real-time QoS capabilities to enable true distributed simulations using both HLA and DDS standards.
Addressing the Challenges of Tactical Information Management in Net-Centric S...Angelo Corsaro
This paper provides an overview of the advantages provided by the OMG Data Distribution Service for Real-Time Systemts (DDS) for addressing the challenges associated with Tactical Information distribution.
CYBER INFRASTRUCTURE AS A SERVICE TO EMPOWER MULTIDISCIPLINARY, DATA-DRIVEN S...ijcsit
In supporting its large scale, multidisciplinary scientific research efforts across all the university campuses and by the research personnel spread over literally every corner of the state, the state of Nevada needs to build and leverage its own Cyber infrastructure. Following the well-established as-a-service model, this state-wide Cyber infrastructure that consists of data acquisition, data storage, advanced instruments, visualization, computing and information processing systems, and people, all seamlessly linked together through a high-speed network, is designed and operated to deliver the benefits of Cyber infrastructure-as-aService (CaaS).There are three major service groups in this CaaS, namely (i) supporting infrastructural
services that comprise sensors, computing/storage/networking hardware, operating system, management tools, virtualization and message passing interface (MPI); (ii) data transmission and storage services that provide connectivity to various big data sources, as well as cached and stored datasets in a distributed
storage backend; and (iii) processing and visualization services that provide user access to rich processing and visualization tools and packages essential to various scientific research workflows. Built on commodity hardware and open source software packages, the Southern Nevada Research Cloud(SNRC)and a data repository in a separate location constitute a low cost solution to deliver all these services around CaaS. The service-oriented architecture and implementation of the SNRC are geared to encapsulate as much detail of big data processing and cloud computing as possible away from end users; rather scientists only need to learn and access an interactive web-based interface to conduct their collaborative, multidisciplinary, dataintensive research. The capability and easy-to-use features of the SNRC are demonstrated through a use case that attempts to derive a solar radiation model from a large data set by regression analysis.
The huge volume of text documents available on the internet has made it difficult to find valuable
information for specific users. In fact, the need for efficient applications to extract interested knowledge
from textual documents is vitally important. This paper addresses the problem of responding to user
queries by fetching the most relevant documents from a clustered set of documents. For this purpose, a
cluster-based information retrieval framework was proposed in this paper, in order to design and develop
a system for analysing and extracting useful patterns from text documents. In this approach, a pre-
processing step is first performed to find frequent and high-utility patterns in the data set. Then a Vector
Space Model (VSM) is performed to represent the dataset. The system was implemented through two main
phases. In phase 1, the clustering analysis process is designed and implemented to group documents into
several clusters, while in phase 2, an information retrieval process was implemented to rank clusters
according to the user queries in order to retrieve the relevant documents from specific clusters deemed
relevant to the query. Then the results are evaluated according to evaluation criteria. Recall and Precision
(P@5, P@10) of the retrieved results. P@5 was 0.660 and P@10 was 0.655.
Paper MIE2016 from Proceedings pags 122-126vilaltajo
This document proposes using a Document Archiving and Communication System (DACS) to provide secure storage and access to clinical documents in a standardized way. A DACS would function similarly to a PACS for medical images. It describes the key components of a DACS including a web portal, server, and databases to store documents formatted according to HL7 CDA standards. Privacy rules and security protocols like SAML and XACML are used to control access to documents based on user roles. Clinical documents can be queried through the DACS using a finder application to retrieve time-series data, summaries, and facilitate complex searches across repositories.
Implementing data-driven decision support system based on independent educati...IJECEIAES
Decision makers in the educational field always seek new technologies and tools, which provide solid, fast answers that can support decision-making process. They need a platform that utilize the students’ academic data and turn them into knowledge to make the right strategic decisions. In this paper, a roadmap for implementing a data driven decision support system (DSS) is presented based on an educational data mart. The independent data mart is implemented on the students’ degrees in 8 subjects in a private school (AlIskandaria Primary School in Basrah province, Iraq). The DSS implementation roadmap is started from pre-processing paper-based data source and ended with providing three categories of online analytical processing (OLAP) queries (multidimensional OLAP, desktop OLAP and web OLAP). Key performance indicator (KPI) is implemented as an essential part of educational DSS to measure school performance. The static evaluation method shows that the proposed DSS follows the privacy, security and performance aspects with no errors after inspecting the DSS knowledge base. The evaluation shows that the data driven DSS based on independent data mart with KPI, OLAP is one of the best platforms to support short-tolong term academic decisions.
Multi-Agent systems (Autonomous agents or agents) and knowledge discovery (or data mining) are two active
areas in information technology. A profound insight of bringing these two communities together has unveiled a tremendous
potential for new opportunities and wider applications through the synergy of agents and data mining. Multi-agent systems
(MAS) often deal with complex applications that require distributed problem solving. In many applications the individual and
collective behavior of the agents depends on the observed data from distributed data sources. Data mining technology has
emerged, for identifying patterns and trends from large quantities of data. The increasing demand to scale up to massive data sets
inherently distributed over a network with limited band width and computational resources available motivated the development of
distributed data mining (DDM).Distributed data mining is originated from the need of mining over decentralized data
sources. DDM is expected to perform partial analysis of data at individual sites and then to send the outcome as partial result
to other sites where it sometimes required to be aggregated to the global result
The document proposes a Cloud Information Accountability (CIA) framework to provide distributed accountability for data sharing in the cloud. The CIA framework uses Java ARchive (JAR) files to automatically log any access to user data in the cloud. When user data is sent to cloud service providers, it is enclosed in JAR files along with access policies and a logging mechanism. Any access to the data will then trigger authenticated logging local to the JAR files. This provides a decentralized yet enforceable way to track how user data is used throughout the dynamic cloud environment. The authors implement and test the CIA framework on a cloud testbed and find that it efficiently and effectively provides accountability for user data in distributed cloud systems.
This document discusses distributed databases. It begins by introducing distributed database systems and their structure. Key points include that the database is split across multiple computers that communicate over a network. It then discusses the tradeoffs of distributing a database, such as increased availability but also higher complexity. The document outlines two approaches to distributing data - replication, where copies of data are stored at different sites, and fragmentation, where relations are split into pieces stored at different sites. It provides examples to illustrate these concepts.
This document summarizes a research paper that proposes a scheme for ensuring security and reliability of data stored in the cloud. The scheme utilizes erasure coding to redundantly store encrypted data fragments across multiple cloud servers. It generates homomorphic tokens that allow auditing of the data storage and identification of any misbehaving servers. The scheme supports secure dynamic operations like modification, deletion and append of cloud data files. Analysis shows the scheme is efficient and resilient against various security threats like server compromises or failures. It ensures storage correctness and fast localization of data errors in the cloud.
The AIRCC's International Journal of Computer Science and Information Technology (IJCSIT) is devoted to fields of Computer Science and Information Systems. The IJCSIT is a open access peer-reviewed scientific journal published in electronic form as well as print form. The mission of this journal is to publish original contributions in its field in order to propagate knowledge amongst its readers and to be a reference publication.
Blockchain enabled task and time sheet management for accounting services pro...Conference Papers
This document describes a blockchain-enabled timesheet management system for accounting firms. It aims to improve on traditional centralized timesheet databases which are vulnerable to tampering. The proposed system uses blockchain to immutably store task and timesheet data, including check-in/out times. This ensures accuracy and avoids issues like overclaiming hours. The document outlines the system architecture, which features a frontend app and blockchain backend on Hyperledger Fabric. Timesheet records are added to the blockchain using smart contracts. Preliminary results found blockchain improved aspects like organizational management, cost savings, transparency and data security compared to traditional methods.
This document discusses using Hidden Markov Model (HMM) forward chaining techniques for prefetching in distributed file systems (DFS) for cloud computing. It begins by introducing DFS for cloud storage and issues like load balancing. It then discusses using HMM to analyze client I/O and predict future requests to prefetch relevant data. The HMM forward algorithm would be used to prefetch data from storage servers to clients proactively. This could improve performance by reducing client wait times for requested data in DFS for cloud applications.
iaetsd Controlling data deuplication in cloud storageIaetsd Iaetsd
This document discusses controlling data deduplication in cloud storage. It proposes an architecture that provides duplicate check procedures with minimal overhead compared to normal cloud storage operations. The key aspects of the proposed system are:
1) It uses convergent encryption to encrypt data for privacy while still allowing for deduplication of duplicate files.
2) It introduces a private cloud that manages user privileges and generates tokens for authorized duplicate checking in a hybrid cloud architecture.
3) It evaluates the overhead of the proposed authorized duplicate checking scheme and finds it incurs negligible overhead compared to normal cloud storage operations.
IRJET- A Novel Framework for Three Level Isolation in Cloud System based ...IRJET Journal
This document proposes a novel three-level isolation framework for cloud storage based on fog computing. The framework aims to address privacy and security issues in cloud storage by distributing user data across three layers - cloud servers, fog servers, and local machines. It uses a hash-Solomon encoding algorithm to split user data into multiple shares and store each share in a different layer. This provides three-way redundancy to protect against data loss and enhances security by isolating data across multiple environments. Theoretical analysis and experimental evaluation demonstrate the feasibility and security improvements of the proposed three-level isolation framework compared to existing cloud storage schemes.
1) The document proposes a privacy-preserving public auditing scheme for cloud storage. It allows a third party auditor (TPA) to efficiently check the integrity of outsourced data in the cloud without learning anything about the data contents.
2) The scheme utilizes homomorphic linear authenticators to generate proofs of data storage correctness, enabling TPA to perform audits without accessing the entire data files. It also supports batch auditing so that TPA can simultaneously audit multiple users' delegated tasks.
3) The scheme aims to achieve public auditability, storage correctness, privacy-preserving audits, and lightweight computation overhead during the auditing process. It analyzes the security and performance of the proposed privacy
Privacy preserving public auditing for secure cloud storageMuthu Sybian
1) The document proposes a privacy-preserving public auditing scheme for cloud storage. It allows a third party auditor (TPA) to efficiently check the integrity of outsourced data in the cloud without learning anything about the data contents.
2) The scheme utilizes homomorphic linear authenticators to generate proofs of data storage correctness, enabling TPA to audit multiple tasks simultaneously in a batch manner with minimal overhead.
3) Extensive analysis shows the scheme is provably secure and efficient, addressing key issues of public auditability, data privacy, and lightweight verification.
Interoperability demonstration between 5 different products that implement the OMG DDS Interoperability Wire Protocol (DDS-RTPS).
The demonstration took place at the March 2011 OMG technical meeting in Washington DC.
The following companies demonstrated interoperability between their products: RTI (Connext DDS). TwinOaks Computing (CoreDX), PrismTech (OpenSpliceDDS), Gallium Visual Systems/Kongsberg (Compass DDS), IBM.
High-level introduction to the OMG Data Distribution Service (DDS) standard and how it provides values beyond what is possible with traditional messaging middleware such as JMS or AMQP.
Interoperability demonstration between 7 different products that implement the OMG DDS Interoperability Wire Protocol (DDS-RTPS).
The demonstration took place at the June 2013 OMG technical meeting in Berlin Germany.
The following companies demonstrated interoperability between their products: RTI (Connext DDS). TwinOaks Computing (CoreDX), PrismTech (OpenSpliceDDS), OCI (OpenDDS), ETRI (ETRI DDS), NADS, and RemedyRT.
Presentation of the DDS Interoperability demo performed in Washington DC between RTI, TwinOaks and PrismTech.
This demonstration shows the use of the DDS-RTPS interoperability protocol in 9 different scenarios.
This document discusses how data-centric systems are increasingly needed to manage growing data volumes and velocities. It presents the Data Distribution Service (DDS) as a solution, which defines a common data model and quality of service contracts. DDS provides a standardized publish-subscribe architecture around the shared data model for improved integration and interoperability compared to traditional message-based approaches. It has been adopted for several major systems due to reducing programming effort, costs, and risks.
Interoperability demonstration between 6 different products that implement the OMG DDS Interoperability Wire Protocol (DDS-RTPS).
The demonstration took place at the March 2012 OMG technical meeting in Washington DC.
The following companies demonstrated interoperability between their products: RTI (Connext DDS). TwinOaks Computing (CoreDX), PrismTech (OpenSpliceDDS), OCI (OpenDDS), ETRI (ETRI DDS), IBM.
Use of the Data-Distribution Service (DDS) --a publish-subscribe middleware standard from OMG -- as a communication infrastructure for Event Processing Engines.
This whitepaper focuses on “real-world” systems, that is, systems that interact with the external physical world and must live within the constraints imposed by real-world physics. Good examples include air-traffic control systems, real-time stock trading, command and control (C2) systems, unmanned vehicles, robotic and vetronics, and Supervisory Control and Data Acquisition (SCADA) systems.
More and more these “real-world” systems are integrated using a Data-Centric Publish- Subscribe approach, specifically the programming model defined by the Object Management Group (OMG) Data Distribution Service (DDS) specification.
This whitepaper describes the basic characteristics of real-world systems programming, reasons why DDS is the best standard middleware technology to use to integrate these systems, and a set of “best practices” guidelines that should be applied when using DDS to implement these systems.
Reactive Stream Processing for Data-centric Publish/SubscribeSumant Tambe
The document discusses the Industrial Internet of Things (IIoT) and key challenges in developing a dataflow programming model and middleware for IIoT systems. It notes that IIoT systems involve large-scale distributed data publishing and processing streams in a parallel manner. Existing pub-sub middleware like DDS can handle data distribution but lack support for composable local data processing. The document proposes combining DDS with reactive programming using Rx.NET to provide a unified dataflow model for both local processing and distribution.
A CLOUD BASED ARCHITECTURE FOR WORKING ON BIG DATA WITH WORKFLOW MANAGEMENTIJwest
In real environment there is a collection of many noisy and vague data, called Big Data. On the other hand,
to work on the data middleware have been developed and is now very widely used. The challenge of
working on Big Data is its processing and management. Here, integrated management system is required
to provide a solution for integrating data from multiple sensors and maximize the target success. This is in
situation that the system has constant time constrains for processing, and real-time decision-making
processes. A reliable data fusion model must meet this requirement and steadily let the user monitor data
stream. With widespread using of workflow interfaces, this requirement can be addressed. But, the work
with Big Data is also challenging. We provide a multi-agent cloud-based architecture for a higher vision to
solve this problem. This architecture provides the ability to Big Data Fusion using a workflow management
interface. The proposed system is capable of self-repair in the presence of risks and its risk is low.
Intrusion Detection and Marking Transactions in a Cloud of Databases Environm...neirew J
The cloud computing is a paradigm for large scale distributed computing that includes several existing
technologies. A database management is a collection of programs that enables you to store, modify and
extract information from a database. Now, the database has moved to cloud computing, but it introduces at
the same time a set of threats that target a cloud of database system. The unification of transaction based
application in these environments present also a set of vulnerabilities and threats that target a cloud of
database environment. In this context, we propose an intrusion detection and marking transactions for a
cloud of database environment.
The document discusses several security issues related to the collection and storage of metadata. It notes that while most metadata is random and anonymous, it can also contain personally identifiable information. This makes metadata a desirable target for cybercriminals. Additionally, dishonest employees may leak valuable marketing data or embarrassing personal information. Poorly trained employees are also more risk of falling for social engineering attacks. Finally, outside attacks could be used to disrupt services or steal stored information. The document argues that while anonymous metadata poses few risks, the collection of large amounts of data presents serious security challenges.
Implementation of Agent Based Dynamic Distributed ServiceCSCJournals
This document proposes a design for agent migration between distributed systems using ACL (Agent Communication Language) messages. It involves serializing an agent's code and state into an ACL message that is sent from one system to another. The receiving system deserializes the agent to restore its execution. The design includes defining an ontology for migration messages, a migration protocol specifying the message flow, and components for handling class loading, agent migration, and conversation protocols. The performance of this distributed agent migration approach is evaluated by applying it to a distributed prime number calculation application.
This document summarizes the software integration strategy used for an Alaska oil regulatory system. The strategy integrated disparate legacy database applications through a service-oriented architecture with autonomous components. A key part was establishing a "system of record" to store standardized data in one location with services providing CRUD functionality. An example application built was a Non Tank Vessel Contingency Plan system that integrated affiliation, document, and vessel data from independent services and objects. The integration improved data access, quality and decision support capabilities for the oil industry regulations.
The document outlines 10 guiding principles for designing computer-based decision support systems to assist human decision makers with complex problems. The principles emphasize partnership between humans and computers, with the system taking a distributed and open architecture. The system should provide tools rather than solutions, utilize high-level representations, embed knowledge, and decentralize decision making. A key principle is identifying conflicts rather than automatically resolving them. The goal is to design highly interactive systems that integrate planning and execution functions.
The document discusses security issues in distributed database systems. It begins by defining distributed databases and their architecture. It then discusses three main security aspects: access control, authentication, and encryption. The document also discusses distributed database system design considerations like concurrency control and data fragmentation. Emerging security tools for distributed databases mentioned include data warehousing, data mining, collaborative computing, distributed object systems, and web applications. Maintaining security when building and querying data warehouses from multiple sources is highlighted as a key challenge.
A Reconfigurable Component-Based Problem Solving EnvironmentSheila Sinclair
This technical report describes a reconfigurable component-based problem solving environment called DISCWorld. The key features discussed are:
1) DISCWorld uses a data flow model represented as directed acyclic graphs (DAGs) of operators to integrate distributed computing components across networks.
2) It supports both long running simulations and parameter search applications by allowing complex processing requests to be composed graphically or through scripting and executed on heterogeneous platforms.
3) Operators can be simple "pure Java" implementations or wrappers to fast platform-specific implementations, and some operators may represent sub-graphs that can be reconfigured to run across multiple servers for faster execution.
Ontology Tutorial: Semantic Technology for Intelligence, Defense and SecurityBarry Smith
Dr. Barry Smith is the director of the National Center for Ontological Research. He discussed how semantic technology can help solve the problem of data silos by enabling data from different sources to be integrated and analyzed together. Ontologies, or controlled vocabularies, can be used to semantically enhance data by tagging it in an interoperable way. This allows the data to be retrieved, understood, and used by others even if they were not involved in creating the data. The semantic enhancement approach aims to break down silos incrementally by coordinating the creation of ontologies and linking datasets through shared terms.
Ambiences on the-fly usage of available resources through personal devicesijasuc
In smart spaces such as smart homes, computation is
embedded everywhere: in toys, appliances, or the
home’s infrastructure. Most of these devices provid
e a pool of available resources which the user can
take
advantage, interacting and creating a friendly envi
ronment. The inherent composability of these system
s
and other unique characteristics such as low-cost e
nergy, simplicity in module programming, and even
their small size, make them a suitable candidate fo
r dynamic and adaptive ambient systems. This resear
ch
work focuses on what is defined as an “ambience”, a
space with a user-defined set of computational
devices. A smart-home is modeled as a collection of
ambiences, where every ambience is capable of
providing a pool of available resources to the user
. In turn, the user is supposed to carry one or sev
eral
personal devices able to interact with the ambience
s, taking advantage of his inherent mobility. In th
is way,
the whole system can benefit from resources discove
red in the spatial proximity. A software architectu
re is
designed, which is based on the implementation of l
ow-cost algorithms able to detect and update the sy
stem
when changes in an ambience occur. Ambience middlew
are implementation works in a wide range of
architectures and OSs, while showing a negligible o
verhead in the time to perform the basic output
operations.
The document summarizes the study of distributed systems technologies conducted as part of an external practice. It begins with an introduction to distributed systems, defining them and discussing properties like heterogeneity, openness, security, and transparency. Next, it examines communication protocols used in distributed systems and various architectures. For both protocols and architectures, it analyzes the state of the art and provides a summary table comparing the key aspects. The document aims to provide relevant background information on technologies related to improving energy efficiency in smart grids.
This document proposes a technical approach to building decentralized computer systems. It suggests distributing functions as independent "servers" that abstract data through high-level operations and communicate with "requestors" via standard message protocols. This requestor-server model has advantages over traditional distributed databases by allowing for modularity and performance improvements. It argues this approach is better suited for decentralized organizations by giving autonomy over function and data to individual parts of the system, while still allowing integration through common messaging standards.
This document provides an introduction to Eclipse Zenoh, an open source project that unifies data in motion, data at rest, and computations in a distributed system. Zenoh elegantly blends traditional publish-subscribe with geo-distributed storage, queries, and computations. The presentation will demonstrate Zenoh's advantages for enabling typical edge computing scenarios and simplifying large-scale distributed applications through real-world use cases. It will also provide an overview of Zenoh's architecture, performance, and APIs.
Data Structures in the Multicore Age : NotesSubhajit Sahu
The document discusses the challenges of designing concurrent data structures for multicore processors. It begins by explaining Amdahl's Law, which states that the speedup gained from parallelization is limited by the sequential fraction of a program. For mainstream applications, the sequential fraction often involves coordinating concurrent access to shared data structures.
It then presents an example of designing a concurrent stack. It starts with a simple lock-based stack protected by a single lock. While this guarantees linearizability, it suffers from poor scalability due to the centralized locking bottleneck. It also relies on strong scheduling assumptions. The document indicates that future concurrent data structures will need to be more distributed and relaxed in their consistency requirements to achieve scalability on multicore
Data and Computation Interoperability in Internet ServicesSergey Boldyrev
This document discusses the need for a framework to enable interoperability between heterogeneous cloud infrastructures and systems. It proposes representing data and computation semantically so they can be transmitted and executed across different environments. It also emphasizes the importance of analyzing system behavior and performance to achieve accountability and manage privacy, security, and latency requirements in distributed cloud systems.
Managing, searching, and accessing iot devicesIJCNCJournal
In this paper a new method is proposed for management of REST-based services acting as proxies for Internet-of-Things devices. The method is based on a novel way of monitoring REST resources by hierarchical set of directories, with the possibility of smart searching for “the best” device according to atthe- place devices’ availability and functionality, overall context (including geo-location), and personal preferences. The system is resistant to changes of network addresses of the devices and their services, as well as core system points such as directories. Thus, we successfully deal with the problem of
(dis)connectivity and mobility of network nodes, and the problem of a “newcomer” device trying to connect
to the network at an incidental place/time.
Main novelty of the approach is a summary of three basic achievements. Firstly, the system introduces
unifying tools for efficient monitoring. On one hand, we may control an availability and load (statistics) of
devices/services. On the other hand, we are able to search for “the best” device/service with different criteria, also formulated ad-hoc and personalized. Secondly, the system is resistant to sudden changes of network topology and connections (basically IP addressing), and frequent disconnections of any system element, including core nodes such as central directories. As a result, we may have a common view to the whole system at any time/place and with respect to its current state, even if the elements of the system are distributed across a wider area. Thirdly, any element of the system, from simple devices to global directories, is able to self-adjust to evolving parameters of the environment (including other devices as a part of this environment). In particular, it is possible for a mobile “newcomer” device to interact with the system at any place and time without a need for prior installation, re-programming, determination of
actual parameters, etc. The presented approach is a coherent all-in-one solution to basic problems related
with efficient usage of IoT devices and services, well suited to the hardware- and software-restricted world
of Internet of Things and Services. Fully implemented, the system is now being applied for an “intelligent”
home and workplace with user-centric e-comfort management.
A Survey of File Replication Techniques In Grid SystemsEditor IJCATR
Grid is a type of parallel and distributed systems that is designed to provide reliable access to data
and computational resources in wide area networks. These resources are distributed in different geographical
locations. Efficient data sharing in global networks is complicated by erratic node failure, unreliable network
connectivity and limited bandwidth. Replication is a technique used in grid systems to improve the
applications’ response time and to reduce the bandwidth consumption. In this paper, we present a survey on
basic and new replication techniques that have been proposed by other researchers. After that, we have a full
comparative study on these replication strategies.
This document provides a survey of file replication techniques used in grid systems. It begins with an introduction to grid systems and discusses their use of replication to improve response times and reduce bandwidth consumption. It then categorizes replication techniques as static or dynamic and describes challenges of replication including maintaining consistency and overhead. The document surveys various replication strategies for different grid topologies like peer-to-peer, tree and hybrid. It evaluates strategies based on factors like access latency, bandwidth consumption and fault tolerance. Specific replication techniques are discussed for peer-to-peer architectures aimed at availability, placement strategies and balancing workloads.
Similar to Management High-level overview of the OMG Data Distribution Service (DDS) (20)
DDS Security Version 1.2 was adopted in 2024. This revision strengthens support for long runnings systems adding new cryptographic algorithms, certificate revocation, and hardness against DoS attacks.
From its first use case that enabled distributed communications for US Navy ships to the autonomous systems of today, the DDS family of standards has enabled new generations of applications to run reliably, rapidly and securely, regardless of distance or scale.
To commemorate the 20th year milestone, the DDS Foundation is creating presentations that highlight the 14 specifications in the DDS standard, along with selected real-world use cases.
This presentation introduces some of the original use-cases and experiments, along with a brief history of the Standards.
A recorded video of the presentation is available at this URL
https://www.brighttalk.com/webcast/12231/602966
Introduction to DDS: Context, Information Model, Security, and Applications.Gerardo Pardo-Castellote
Introduction to the Data-Distribution Service (DDS): Context and Applications.
This 50 minute presentation summarizes the main features of DDS including the information model, the type system, and security as well as how typical applications use DDS.
It was presented at the Canadian Government Information Day in Ottawa on September 2018.
There is also a video of this presentation at https://www.youtube.com/watch?v=6iICap5G7rw.
This Object Management Group (OMG) RFP solicits submissions identifying and defining mechanisms to achieve integration between DDS infrastructures and TSN networks. The goal is to provide all artifacts needed to support the design, deployment and execution of DDS systems over TSN networks.
The DDS-TSN integration specification sought shall realize the following functionality:
● Define mechanisms that provide the information required for TSN-enabled networks to calculate any network schedules needed to deploy a DDS system.
OMG RFP
● Identify those parts of the set of the IEEE TSN standards that are relevant for a DDS-TSN integration and indicate how the DDS aspects are mapped onto, or related to, the associated TSN aspects. Examples include TSN- standardized information models for calculating system-wide schedules and configuring network equipment.
● Identify and specify necessary extensions to the [DDSI-RTPS] and [DDS- SECURITY] specifications, if any, to allow DDS infrastructures to use TSN- enabled networks as their transport while maintaining interoperability between different DDS implementations.
● Identify and specify necessary extensions to the DDS and DDS- XML specification, if any, to allow declaration of TSN-specific properties or quality of service attributes.
A NEW ARCHITECTURE PROPOSAL TO INTEGRATE OPC UA, DDS & TSN.
Suppliers and end users need a complete solution to address the complexity of future industrial automation systems. These systems require:
• Interoperability to allow devices and independent software applications from multiple suppliers to work together seamlessly
• Extensibility to incorporate future large or intelligent systems
• Performance and flexibility to handle challenging deployments and use cases
• Robustness to guarantee continuity of operation despite partial failures
• Integrity and fine-grained security to protect against cyber attacks
• Widespread support for an industry standard
This document proposes a new technical architecture to build this future. The design combines the best of the OPC Unified Architecture (OPC UA), Data Distribution Service (DDS), and Time-Sensitive Networking (TSN) standards. It will connect the factory floor to the enterprise, sensors to cloud, and real-time devices to work cells. This proposal aims to define and standardize the architecture to unify the industry.
The document provides an overview of the DDS-XRCE specification, which defines an Agent-Client communication model to enable the use of the DDS data distribution service (DDS) in extremely resource-constrained networks. It describes the motivation for DDS-XRCE and its key aspects, including the message structure, interaction model, supported deployment scenarios, and how it provides security through the use of a client key.
The document describes a demonstration of interoperability between 5 vendor DDS security implementations using a shapes demo application. The demo consists of 6 scenarios that illustrate different aspects of DDS security configuration and functionality, including controlling access to the domain, enabling open access to selected topics, comparing data integrity vs encryption, protecting metadata, securing discovery, and fine-grained access control at the topic level. Each scenario varies the security governance and permission files to achieve the desired access control configuration.
Applying MBSE to the Industrial IoT: Using SysML with Connext DDS and SimulinkGerardo Pardo-Castellote
This document summarizes a presentation about applying model-based systems engineering (MBSE) to industrial internet of things (IIoT) systems using the SysML modeling language, Connext DDS middleware, and Simulink. It discusses how SysML can be used to design interfaces, applications, and quality of service policies for DDS-connected systems. The presentation also provides examples of integrating MagicDraw, Simulink, and Connext DDS to enable translating SysML models into implementations and deployments of distributed IIoT applications and components.
One of the most important challenges that system designers and system integrators face when deploying complex Industrial Internet of Things (IoT) systems is the integration of different connectivity solutions and standards. At RTI, we are constantly working to accelerate the Industrial IoT revolution. Over the past few years, we have developed standard connectivity gateways to ensure that DDS systems can easily integrate with other core connectivity frameworks.
This year, we developed a standard OPC UA/DDS Gateway, a bridge between two of the most well-known Industrial IoT connectivity frameworks. We are excited to announce that the gateway was just adopted by the Object Management Group (OMG).
In this webinar, we will dive deeper into the importance of choosing a baseline core connectivity standard for the Industrial IoT and how to ensure all system components are fully integrated. Attendees will also learn:
How the OPC UA/DDS Gateway specification was developed and how it works
How to leverage the Gateway to enable DDS and OPC UA applications to interoperate transparently
About the first standard connectivity gateway released with RTI Web Integration Service in Connext DDS 5.3
Gateways are a critical component of system interoperability and we will keep working to help companies accelerate Industrial IoT adoption.
This document defines an OPC UA/DDS gateway specification. It specifies how to bridge the OPC UA and DDS protocols by defining mappings between their data models, type systems and core services. This includes mapping OPC UA data types, services and subscriptions to DDS data types and topics as well as mapping DDS data types and the global data space to OPC UA address space objects. Configuration formats are also defined to allow configuration of OPC UA to DDS and DDS to OPC UA bridges.
This is the DDS-XRCE 1.0 Beta specification adopted by the OMG March 2018.
The purpose of DDS-XRCE is to enable resource-constrained devices to participate in DDS communication, while at the same time allowing those devices to be disconnected for long periods of time but still be discoverable by other DDS applications.
DDS-XRCE defines a wire protocol, the DDS-XRCE protocol, to be used between an XRCE Client and XRCE Agent. The XRCE Agent is a DDS Participant in the DDS Global Data Space. The DDS-XRCE protocol allows the client to use the XRCE Agent as a proxy in order to produce and consume data in the DDS Global Data Space.
Demonstrates interoperability of 5 independent products that implement the Data-Distribution Service (DDS) Security Standard
(https://www.omg.org/spec/DDS-SECURITY/).
Tests the following implementations: RTI Connext DDS, Twin Oaks Computing CoreDX DDS, Kongsberg InterComm DDS, ADLink Vortex DDS Cafe, and Object Computing Inc OpenDDS.
Demonstrates interoperability of 3 independent products that implement the Data-Distribution Service (DDS) Security Standard
(https://www.omg.org/spec/DDS-SECURITY/).
Tests the following implementations: RTI Connext DDS, Twin Oaks Computing CoreDX DDS, and Kongsberg InterComm DDS.
This specification provides the following additional facilities to DDS [DDS] implementations and users:
* Type System. The specification defines a model of the data types that can be used for DDS Topics. The type system is formally defined using UML. The Type System is de- fined in section 7.2 and its subsections. The structural model of this system is defined in the Type System Model in section 7.2.2. The framework under which types can be modi- fied over time is summarized in section 7.2.3, “Type Extensibility and Mutability.” The concrete rules under which the concepts from 7.2.2 and 7.2.3 come together to define compatibility in the face of such modifications are defined in section 7.2.4, “Type Com- patibility.”
* Type Representations. The specification defines the ways in which types described by the Type System may be externalized such that they can be stored in a file or communi- cated over a network. The specification adds additional Type Representations beyond the
DDS-XTypes version 1.2 1
one (IDL [IDL41]) already implied by the DDS specification. Several Type Representa- tions are specified in the subsections of section 7.3. These include IDL (7.3.1), XML (7.3.2), XML Schema (XSD) (7.3.3), and TypeObject (7.3.4).
* Data Representation. The specification defines multiple ways in which objects of the types defined by the Type System may be externalized such that they can be stored in a file or communicated over a network. (This is also commonly referred as “data serializa- tion” or “data marshaling.”) The specification extends and generalizes the mechanisms already defined by the DDS Interoperability specification [RTPS]. The specification in- cludes Data Representations that support data type evolution, that is, allow a data type to change in certain well-defined ways without breaking communication. Two Data Repre- sentations are specified in the subsections of section 7.4. These are Extended CDR (7.4.1, 7.4.2, and 7.4.3) and XML (7.4.4).
* Language Binding. The specification defines multiple ways in which applications can access the state of objects defined by the Type System. The submission extends and gen- eralizes the mechanism currently implied by the DDS specification (“Plain Language Binding”) and adds a Dynamic Language Binding that allows application to access data without compile-time knowledge of its type. The specification also defines an API to de- fine and manipulate data types programmatically. Two Language Bindings are specified in the subsections of section 7.5. These are the Plain Language Binding and the Dynamic Language Binding.
The document describes version 1.1 of the DDS Security specification which defines a security model and plugin architecture to provide information assurance capabilities to DDS implementations, including defining builtin plugins for authentication, access control, encryption, and logging; it also lists normative references and provides an overview of the specification's scope and conformance points.
This document describes the Interface Definition Language (IDL) version 4.2 specification published by the Object Management Group (OMG). It defines the syntax and semantics of IDL, which is used to define interfaces, data types, exceptions, modules and other elements used in CORBA, CCM, and other OMG specifications. The document includes sections on lexical conventions, grammar, scoping rules, standardized annotations, and CORBA/CCM profiles supported by IDL. It is intended to provide a standard way to define interfaces that are independent of specific programming languages.
This the the formal version 1.0 of the DDS Security specification released September 2016. OMG document number formal/2016-08-01.
DDS-Security defines the Security Model and Service Plugin Interface (SPI) architecture for compliant DDS implementations.
The DDS Security Model is enforced by the invocation of these SPIs by the DDS implementation. This specification also defines a set of builtin implementations of these SPIs.
* The specified builtin SPI implementations enable out-of-the box security and interoperability between compliant DDS applications.
* The use of SPIs allows DDS users to customize the behavior and technologies that the DDS implementations use for Information Assurance, specifically customization of Authentication, Access Control, Encryption, Message Authentication, Digital Signing, Logging and Data Tagging.
This specification is a response to the OMG RFP "eXtremely Resource Constrained Environments DDS (DDS- XRCE)"
It defines a DDS-XRCE Service based on a client-server protocol between a resource constrained, low-powered device (client) and an Agent (the server) that enables the device to communicate with a DDS network and publish and subscribe to topics in a DDS domain. The specifications purpose and scope is to ensure that applications based on different vendor’ implementations of the DDS-XRCE Service are compatible and interoperable.
This is the Joint submission by RTI, TwinOaks, and eProsima. Updated September 2017, OMG document number mars/2017-09-18.
Management High-level overview of the OMG Data Distribution Service (DDS)
1. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
Data Distribution Service
(DDS) Brief
Standards-Based Data-Centric Messaging
from the Object Management Group (OMG)
1 Executive Summary
U.S. Navy SPAWAR-ATL Engineering Competency requested the Object Management Group
(OMG) to deliver a paper describing the technical capabilities of its Data Distribution Service
(DDS). This paper complements an earlier one prepared in collaboration with several DDS
vendors for the Office of the Secretary of Defense (OSD), which describes DDS adoption in
military and commercial applications. That paper, “The Data Distribution Service: Reducing
Cost through Agile Integration,” is hosted online by the UAS Control Segment (UCS) program1.
Navy decision makers are being asked to respond more quickly on the basis of increasing
volumes of information. This information is sourced from multiple systems of systems executing
on heterogeneous platforms and networks. To face this challenge, the Navy needs to increase its
leverage from proven technology and increase the integration between existing systems. Navy
leadership has embraced these requirements with mandates for Open Architecture integration
based on open standards and off-the-shelf products. These principles help the Navy to align its
technology roadmap with broader industry directions and to empower competitive markets that
reduce vendor lock-in and drive down costs.
The OMG has long been a favored venue for the collaboration of Navy interests with industry
thought leaders around the promulgation of relevant standards. Navy Surface Warfare Center
(NSWC), Navy Undersea Warfare Center (NUWC), Boeing, Lockheed Martin, General
Dynamics, Northrop Grumman, and other U.S. and allied organizations are all active
participants. DDS technology in particular has been rapidly and widely adopted by these
organizations. This adoption has been driven by the ease and flexibility with which it can be used
to develop, maintain, and integrate complex systems while maintaining strong performance and
governance. DDS is supported by a large vendor community and has been called out in U.S.
DoD guidance from Navy Open Architecture, DISA, NESI, UCS, and other U.S. and allied
organizations. This guidance has been born out in hundreds of defense and civilian programs,
and DDS implementations exist at Technology Readiness Level (TRL) 9.
This paper describes the software architectural principles that can help the Navy to meet its
agility and cost-control requirements. It further describes how DDS technology in particular
supports this architecture—not just hypothetically but in real-world systems—in unique and
powerful ways.
1
See http://www.ucsarchitecture.org/downloads/DDS%20Exec%20Brief%20v20l-public.pdf.
—1 of 20—
2. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
Table of Contents
1
Executive Summary.................................................................................................................1
2
Step 1: System Architecture ...................................................................................................2
2.1
Benefits.................................................................................................................................................... 4
2.2
Challenges
Facing
Traditional
Implementations ...................................................................... 5
2.3
An
Improved
Approach
to
Managing
Data-Centricity.............................................................. 7
3
Step 2: Supporting the Architecture ......................................................................................8
3.1
Data-Centric
Messaging ..................................................................................................................... 8
3.2
DDS............................................................................................................................................................ 9
4
Step 3: Instantiating the Architecture .................................................................................11
4.1
Topology ...............................................................................................................................................12
4.2
Disadvantaged
Networks ................................................................................................................13
4.3
Scalability .............................................................................................................................................14
4.4
Security..................................................................................................................................................15
5
Conclusion ..............................................................................................................................17
6
Appendix: Technology Comparison ....................................................................................17
6.1
Specification
Comparison................................................................................................................18
6.2
Vendor
Comparison ..........................................................................................................................19
2 Step 1: System Architecture
Industry has grappled for over a decade
with the problem of deploying and
maintaining groups of applications that
on the one hand need to integrate with
one another, but at the same time need
to remain decoupled, so that they can
join and leave the network
dynamically, and so that they can
evolve according to their independent
life cycles.
The architecture they have followed is
data-centric. Data-centric architecture
is often instantiated in so-called “n-
layer” or “n-tier” enterprise systems.
Stateful data is maintained by
infrastructure, and applications remain
Figure 1—Schematic of a data-centric, n-layer architecture
—2 of 20—
3. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
stateless. Applications do not communicate directly with one another; instead their interactions
are mediated by the data and expressed in terms of changes to that data.
This architecture is described as “data-centric” because it organizes applications and their
interactions in terms of stateful data rather than in terms of operations to be performed. It
conforms to the following principles:
1. The structure, changes, and motion of stateful data must be well defined and
discoverable, both for and by humans as well as automation. What do we mean by
“state”? State consists of the information about the application, the system, and the
external world that an application needs in order to interpret events correctly. For
example, suppose there is an announcement, “the score is four to three”. What game is
being played? Who are the players? Which one of them has four points and which three?
The answers to these questions comprise the state that is necessary to understand the
message.
2. State must be managed by the infrastructure, and applications must be stateless.
(This is also a recognized SOA pattern called “State Repository”2.)
3. State must be accessed and manipulated by a set of uniform operations. What do we
mean by “operations”? Operations express attempts to change the state. In a data-centric
architecture, the operations are uniform3. These operations are often referred to by the
acronym CRUD, for Create, Read, Update, and Delete, because most supporting
technologies define parallels for these concepts4.
Multiple technologies directly support this architecture, including relational databases, RESTful
web services5, and OMG Data Distribution Service.
Consider a hypothetical distributed game of chess.
• A non-data-centric implementation might assume that all parties understand the initial
layout of the game. Then players would send their moves to one another—“pawn 4 to c3”
for example. Such an implementation further assumes that each recipient has out-of-band
access to its own copy of the current state of the board so that it can change that state
accordingly and that each player receives every message so that different copies don’t get
out of synch with one another.
• A data-centric implementation would present a common picture of the board to
authorized parties and allow them to query and modify it—to not only say that pawn 4
should move to c3, but also to ask what is at c3 beforehand. This state is maintained by
the infrastructure; applications do not need to maintain their own copies. And
applications act of this state independently of which other applications may or may not be
2
See http://soapatterns.org/state_repository.php for an introduction to this pattern.
3
Computer science uses the term “polymorphism” to describe a situation in which a common interface may be used
to access different kinds of resources. Polymorphism helps software fit together like puzzle pieces: a component that
understands a particular interface can communicate with any other component that understands the same interface.
Data-centric architecture takes polymorphism to its logical conclusion: all state shares a single common set of
operations.
4
In SQL, the uniform operations are INSERT, SELECT, UPDATE, and DELETE. In HTTP, they are POST, GET,
PUT, and DELETE. In DDS, they are write, read, dispose, and unregister.
5
For a brief introduction to the Representation State Transfer (REST) pattern, see
http://en.wikipedia.org/wiki/Representational_State_Transfer.
—3 of 20—
4. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
observing. (Note that a distributed infrastructure may communicate within itself using
messages, but applications are written to a higher level of abstraction.)
The following sections describe the benefits of a data-centric approach, the challenges faced in
traditional implementations of the approach, and how data-centric messaging technologies like
DDS overcome those challenges.
2.1 Benefits
The benefits of data-centricity derive from the loose coupling between applications. They do not
communicate directly; instead, one modifies a given data object, and another observes the
change.
2.1.1 Reliability and Resiliency
Applications have the ability to obtain from the infrastructure the current state of the world in
which they’re interested. Therefore, if an application fails and has to be restarted, it can recover
very quickly. In contrast, if the application is not stateless, a restart is expensive and risky.
Message senders must store all messages that they sent during the failure and replay them upon
reconnection, because if the recovering application misses even a single message, its state will be
out of synch, and it will act on incorrect information. If message rates are high relative to the
recovery time, storing these messages will become infeasible.
For example, consider an intermittent network link, such as a satellite or line-of-sight radio.
Applications separated by such a link and architected in a data-centric way will be able to resume
communication by sending only the differences between the relevant pre-disconnection state and
the current post-reconnection state. This data volume is bounded in size and often much less than
the sum of all messages that might have been exchanged in the mean time.
2.1.2 Integration Complexity
It is best, when integrating multiple
90
elements (applications or entire
subsystems), to avoid mediating 80
Lingua
every element to every other. Such a 70
Franca
design requires (n * (n – 1)) 60
Pattern
integrations per n elements—the 50
complexity, effort, and cost of the 40
integration increase with the square Point-‐
30
of the number of elements. Instead, to-‐
employ the well-known Lingua 20
Point
Franca architectural pattern: design a 10
Integ'ns
normalized model, and integrate each 0
element with that model. Each 1
2
3
4
5
6
7
8
9
element need only be integrated once, Figure 2—Relative complexity of point-to-point integration vs.
and the complexity of the integration applying the Lingua Franca Pattern, in the worst case
therefore increases linearly with the
number of elements.
—4 of 20—
5. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
This pattern applies to state as well as to behavior. A set of point-to-point integrations in which
neither state nor operations is normalized will consequently scale even worse than n2. This
additional complexity can be reclaimed by normalizing the programming interface and the
message set with an ESB; complexity growth returns to n2. A data-centric architecture takes the
next step: operations are uniform, and state is normalized. The growth in the complexity is
therefore linear.
2.1.3 System Evolution Cost
Because applications have no awareness of one another, they can be added to or removed from
the system with much lower impact. Changes are limited in scope—replacing one component
with another does not require that all other components be updated as well.
Consider again the chess example above. What if I want to add a new application to the game—
perhaps to provide move recommendations, a turn timer, a GUI display, or other capability? Any
of these can be built based on the state of the board that I already have, and no other application
that uses that state needs to know or care that it’s being used for one more thing. A stateless ESB
cannot provide the same benefit, because it provides applications with no ability to query the
current state of the board—it deals only with stateless messages, not with stateful data.
2.1.4 Acquisition Flexibility
A standards-based data-centric system provides interoperability not only at the level of messages
on the network but also at the level of an operational picture. This higher level of interoperability
decreases lock-in not only to middleware vendors but to integrators as well, because the
integration is fundamentally governed. The information about which information is to be
exchanged under which conditions is captured in explicit configuration, not buried in application
code, and is accessible to any authorized vendor using industry-standard tools.
2.2 Challenges Facing Traditional Implementations
Before the advent of data-centric messaging, data-centric designs were primarily based on
proprietary and/or web-based protocols connecting “client” applications to relational databases.
Such implementations remain valid for many systems, but they also face significant challenges—
challenges that tempt some applications to abandon the architecture. This section describes some
of those challenges and the unfortunate result.
2.2.1 Challenges: Scalability, Reliability, Latency, and Management
Challenge #1: Vulnerabilities of shared infrastructure. Shared infrastructure, such as
databases and servers, can become a performance bottleneck and single point of failure.
Challenge #2: Synchronization of federated state. All applications may not have access to a
single common state repository. In such cases, it’s necessary to maintain copies of the relevant
state in multiple locations and to synchronize them independently of the applications. This is a
difficult task that not all teams are equipped to tackle.
Challenge #3: Data access latency. Messaging between the application that wants a piece of
state and the repository that has it can be slow. Response times may be acceptable in cases where
nothing changes faster than a human can process it—a person with a web browser is a good
—5 of 20—
6. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
example. But for machine-to-machine interactions, or machine-to-human interactions, this
latency proves much too high.
2.2.2 Typical Result: Brittle, Unmanageable Systems
Unfortunately, too often the challenges above lead architects to abandon data-centricity for ad
hoc approaches.
Figure 3—Tangled communication resulting from the application of messaging technology without a
governing architecture
• Rather than allowing their actions to be mediated by the data, applications send messages
directly to one another. They may use abstractions like messages queues, but these
patterns remain fundamentally point-to-point.
• Rather than relying on the state repository to manage their data, every application
maintains its own state.
In effect, system-level state management is neglected. Does our experience lead us to believe
that when we don’t design something, it will nevertheless work well? The result instead is
systems that are brittle and difficult to manage.
Why brittle?
• Applications are coupled to one another, so they can’t come and go, or evolve over
time, independently.
Implications: Decreased operational flexibility and increased costs for maintenance and
integration.
• State is coupled to individual applications, not maintained independently, so new
applications can’t reuse the state that’s already there, and existing applications can’t
recover their state if they restart or relocate on the network.
Implications: Decreased reliability and resiliency and increased cost to develop and
integrate incremental functionality.
—6 of 20—
7. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
Why unmanageable?
• Data structure is ad hoc, so it’s impossible to detect whether a piece of information is
malformed until a particular application tries to access it. By then, it’s too late to avoid
and hard to debug.
Implication: Error detection will occur later in the process, when it’s more expensive to
fix and has a bigger impact on schedules.
• Data movement around the network is ad hoc, so as each application maintains its own
view of state, these views can get out of synch. Applications act on incorrect or obsolete
information and can’t respond in a timely way to new information.
Implication: Decreased reliability.
• Data change is ad hoc, so making sure that the right parties see the right changes in an
acceptable amount of time is the responsibility of every application—or else everything
has to be sent to a single central party on the network, and that one has to know
everything and never fail.
Implications: Increased upfront cost due to duplicated application-development effort,
increased maintenance costs due to inter-application coupling, and decreased reliability if
a single point of failure is introduced.
2.3 An Improved Approach to Managing Data-Centricity
The challenges described above can be solved in a more scalable way while retaining the data-
centric architecture. This preferred approach relies on data-centric messaging, which is described
in section 3.1 below.
Figure 4—Data-centric messaging improves scalability of data-centric architecture
Challenge #1: Vulnerabilities of shared infrastructure. Federate state management to where
it’s needed. Each portion of the network has independent access to exactly the state it needs at
that moment and no more. This is the logical conclusion of server federation: the more broadly
you federate, the smaller the burden is on any one party; each can remain lightweight. And there
are no longer any single points that can take out a whole network.
—7 of 20—
8. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
Challenge #2: Synchronization of federated state. This burden should not be on the
applications; it should be on a best-in-class infrastructure. Point-to-point messaging between
applications is replaced by data-centric publish-subscribe messaging within that infrastructure.
Instead of full consistency, it seeks eventual consistency. That guarantee is easier to maintain
under challenging network conditions, and the implementation can be orders of magnitude faster.
Challenge #3: Data access latency. Because we’re employing a solution based on distributed
state with eventual consistency, we can treat large-scale, long-term persistence as a separate
concern from application access. We can eliminate most databases and instead place lightweight
in-memory caches very close to each application—on the same node, or even within the same
process—to maximally reduce this latency. Meanwhile, we can place high-availability
persistence stores on appropriately provisioned nodes elsewhere on the network.
3 Step 2: Supporting the Architecture
To support this architecture, we need technology that can govern data contents, as a database
can, as well as governing communication flows within complex networks, as messaging buses
can. And it must go beyond conventional message buses—it must tie messages back to the
underlying data objects and formally describe how those objects will be synchronized across the
network as they change.
3.1 Data-Centric Messaging
Data-centric messaging is the application of messaging to the distribution, access, and
manipulation of stateful data. Data-centric messaging supports data-centricity for data in motion,
just as a relational database does for data at rest. The vendor community has been supporting
such technology, called data-centric messaging, for over ten years. This section describes the
technology generally.
As described in section 2.3 above, an architecture
that employs data-centric messaging offers
significant benefits over one instantiated based on
solely on the basis of databases or solely on the
basis of non-data-centric messaging.
• Reduced integration and maintenance costs,
as with any data-centric technology.
• Improved performance. Where a web
service connected to a database might
deliver a dozen data updates per second, an Figuredata. database stores data. A data bus
moves
5—A
efficient data-centric messaging bus can
deliver tens or even hundreds of thousands.
• Improved scalability. Where centralized infrastructure might support a few connected
applications, a decentralized data bus can support hundreds or thousands—on less
hardware.
• Improved reliability and resiliency. Single points of failure have been eliminated.
• Improved manageability. The infrastructure enforces explicit contracts on the structure of
data, how it moves across the network, and how it changes over time. And if and when
—8 of 20—
9. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
these contracts are violated, it prevents incorrect communication, takes mitigating action,
and notifies the relevant applications.
Between 2001 and 2002, several of these vendors came together at the OMG to begin work on
standardizing data-centric messaging technology. The result, the Data Distribution Service
(DDS), is the subject of the next section.
3.2 DDS
The Data Distribution Service is the standard for data-centric messaging. Adopted by the OMG
in 20036, DDS now comprises a comprehensive suite of layered specifications. In particular, it is
the only widely deployed standards-based middleware technology to define code portability as
well as on-the-network interoperability.
• DDS itself, which defines the behavior of the bus itself as well as programming interfaces
in several languages. Thirty-six companies voted to adopt the original specification,
including Ericsson, Fujitsu, IBM, Lockheed Martin, MITRE, Mercury Computer
Systems, Nokia, Objective Interface Systems, Oracle, PrismTech, RTI, Rockwell Collins,
and THALES. The Navy Surface Warfare Center (NSWC) played a significant
supporting role. Today, approximately ten vendors support the specification.
• A network protocol, called Real-Time Publish-Subscribe (DDS-RTPS), which provides
interoperability among DDS implementations. This specification became available
through the OMG in 2008 at version 2.0. (It was based on an earlier specification, RTPS
1.0, which was standardized through the IEC in 2004.) The current version, 2.1, became
available in January 2009. Most vendors now support this protocol, and interoperability
has been publicly demonstrated at a number of OMG-hosted events.
• Integration with UML models to bridge the gap from design to implementation. This
UML profile was adopted in 2008.
• Enhancements to the type system to address system evolution and more flexible data
views. This specification was adopted in 2010 and is currently in the process of
finalization and implementation with the
involvement of multiple vendors. Founded in 1989, OMG is now the
• Improvements to the C++ and Java largest and longest-standing not-for-
programming interfaces to enhance profit, open-membership consortium
portability, performance, and ease of use. developing and maintains computer
These specifications were adopted in 2010 industry specifications with more than
and are currently in the process of 470 member companies. It is
finalization and implementation with the continuously evolving to remain current
involvement of multiple vendors. while retaining a position of thought
• etc. leadership.
DDS continues to define one of the most active All OMG specifications are freely
communities within the OMG. In addition to available to the public from
ongoing direct collaboration among member www.omg.org.
organizations, the OMG hosts quarterly in-person
6
OMG issued an RFP for the definition of a data-centric publish-subscribe messaging bus in late 2001. Initial
proposals were received from several vendors in 2002. The first version of the specification was preliminarily
adopted in 2003 and finalized in 2004. The current version, 1.2, became available in January 2007.
—9 of 20—
10. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
technical meetings. OMG also hosts an annual workshop on time-critical systems that in recent
years has become increasingly focused on DDS technology. And the ecosystem continues to
grow, with new vendors joining the community and specifications for security enhancements and
web connectivity in progress.
3.2.1 Adoption
DDS has been adopted and/or mandated by many military and civilian organizations.
DDS plays a major role in naval combat systems in the U.S. and worldwide. It has been designed
into the Aegis, SSDS, and DDG 1000 programs and is deployed by allied navies, including those
of Germany, Japan, the Netherland, and over a dozen more. DDS has been adopted by the
following organizations:
• U.S. Navy—Open Architecture, FORCEnet
• Defense Information Systems Agency
(DISA)—Mandated standard within the DoD
Information Technology Standards and Profile
Registry (DISR)
• U.S. Air Force, Navy—Net-centric Enterprise
Solutions for Interoperability (NESI)
• U.S. Army, OSD—UAS Control Segment
(UCS)
• U.S. intelligence community
• UK Ministry of Defence—Generic Vehicle
Architecture, an interoperable open architecture
for unmanned vehicles
DDS is also used commercially in a number of industries, including communication,
transportation, financial services, SCADA, industrial automation, agriculture, power generation,
air traffic control, mobile asset tracking, and medicine. A number of universities worldwide are
using DDS in active research projects, including MIT, Carnegie Mellon University, and
Vanderbilt University in the U.S. and ETH Zurich, Technical University of Munich, and Korea
Advanced Institute of Science and Technology internationally.
The following sections describe some of the capabilities of DDS. These are not capabilities
specific to a particular vendor; they are specified within the DDS standard.
—10 of 20—
11. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
3.2.2 Managed Data Distribution
Non-data-centric messaging technologies just provide Standards-Based Governance
ways to send messages from A to B. Architects must
Data structure
develop their own idioms on top.
Data value history
DDS is different. Like HTTP, DDS uses the technique of
messaging to support system architecture. Because the Rate of data value change
data model is clear and explicit rather than implicit in
Data ordering and coherency
static code, DDS can define, propagate, and govern data
flows more comprehensively and more efficiently. DDS Lifespan/TTL
provides: Network partitions
• Formal data design and integration to avoid lock-
Resource utilization
in to vendors and integrators
• Strong enforcement of data structure and quality Priority
of service to make propagation more efficient and Reliability
catch errors sooner
• Comprehensive status monitoring to detect and Durability, persistence, and high
mitigate potential problems at run time availability
• Flexible security with a comprehensive road map Fault tolerance and fail-over
3.2.3 Flexible Deployment Filters based on contents and time
DDS is the only widely deployed messaging technology Publication/subscription matching
to scale from embedded systems to data centers to global
networks. Connection health
• DDS implementations support both peer-to-peer and brokered message-flow topologies,
which can be combined as needed for local, wide-area, and disadvantaged network
environments. See section 4.1, “Topology”, below for more information.
• DDS is interoperable across multiple programming languages, real-time and non-real-
time constraints, and enterprise and embedded platforms.
DDS is compatible with enterprise environments. In addition to support for higher-level
languages like Java, it has an API that is similar to other messaging technologies7. Vendors also
provide a variety of connectors to other standards-based messaging and storage technologies,
including JMS, databases, and web services.
4 Step 3: Instantiating the Architecture
This section applies the architectural principles described above, and the technologies that
support them, to describe the construction of flexible, performant, and affordable systems. It
focuses on three areas: topology, scalability, and security.
7
OMG is the standards body responsible for both the DDS and CORBA specifications. However, these two
technologies work differently and do not depend on one another.
—11 of 20—
12. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
4.1 Topology
Peer-to-peer communication is a fundamental building block of any network communication.
Other topologies—such as brokered
communication—are constructed from it. For
example, a network of data producer communicating
with a data consumer by way of a broker consists of
three peers, one in between the other two. It so
happens that the middle peer is typically provided by Figure 6—Peer-to-peer communication is a
the messaging vendor and provides services to the fundamental building block
other two peers.
4.1.1 Composing Brokered Networks
DDS specifications are defined peer-to-peer8 in order to provide implementers with maximum
flexibility. Most vendors take advantage of this and support peer-to-peer communication as an
option within their products. However, other DDS vendors support only brokered configurations,
while some support peer-to-peer communication but also ship message brokers, so that users can
compose their systems however is most appropriate.
Figure 7—A brokered network is composed of multiple peer-to-peer connections. However, whether that is
reflected in a given messaging implementation varies.
Most vendors of traditional non-data-centric messaging technologies support only brokered
configurations.
4.1.2 Composing Local and Global Networks
Different networks have different characteristics and requirements.
• Local networks support deterministic, low-latency communication. They can often take
advantage of efficient IP multicast. Applications running here may also be more trusted.
• Wide-area networks have higher latencies and may or may not support multicast. They
may route different transport protocols (e.g. TCP but not UDP). Applications connected
across such networks may be less trusted than those running on a LAN.
• Disadvantaged wireless networks have significantly different reliability and performance
characteristics. Applications running here may be the least trusted, given that wireless
connections may be easier to intercept than wired connections.
• Any one of these physical networks, or a logical “subnet” within it, may represent an
independent security domain or subsystem.
8
This situation is not unlike that of other messaging technologies. For example, the non-data-centric Advanced
Message Queuing Protocol (AMQP) is also specified peer-to-peer.
—12 of 20—
13. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
Connecting these heterogeneous environments requires mediation—a broker to filter and
transform the data, cleanse it to meet IA requirements, and ultimately forward it to the other side.
However, brokers may not be needed within the LAN, because it is a more controlled
environment. These opportunities and constraints lead us to design networks such as is depicted
in the following figure.
Figure 8—A composite network taking advantage of peer-to-peer communication on the LAN and brokered
communication across the WAN
Such networks can take advantage of peer-to-peer performance and resilience on the LAN while
mediating data and enforcing security policies at subsystem or network boundaries with data
routers. Persistence and other services can be deployed and relocated as appropriate.
4.2 Disadvantaged Networks
It is never acceptable for applications to act upon obsolete information. When networks are
disconnected, intermittent, or limited in their bandwidth (DIL), this challenge is even more
significant. Messaging technologies have the opportunity to either mitigate or exacerbate it. The
following are a few of the factors to consider:
• Data compactness: The more limited the network’s bandwidth, the more important it is
that the messaging layer does not bloat its payloads with an inefficient data
representation. Larger payloads also take longer to send, increasing the chance that a
network drop will hit in the middle, preventing successful transmission. System designers
sometimes rely on XML to provide data transparency; unfortunately, XML can be bulky.
DDS does support XML payloads but also provides similar benefits using a very compact
binary data representation.
• Protocol responsiveness: The protocol must recover from losses and disconnections
quickly, and while the network is connected, it must use it efficiently. TCP—and
protocols layered on top of it—suffers in this area. While it can provide excellent
performance when connectivity is good, it can be slow to respond to changing network
conditions. And its head-of-line blocking behavior and global timeouts can cause
multiple message streams to halt delivery for an extended period if any one of them
suffers a transitory loss of synchronization.
—13 of 20—
14. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
The DDS-RTPS protocol can be layered on top of TCP. However, most typically it runs
on top of UDP, where it provides reliability and independent quality-of-service control on
a per-stream basis, avoiding cross-stream interference and extended blocking.
• Bounded resource use: A typical durable messaging technology operating over an
intermittent link must store every message that was sent while the link was down and
replay those messages when the link is restored. If the link goes down for an extended
period, the resources needed to store these messages can grow in an unbounded fashion.
And upon reconnection, replaying those messages will take an increasing amount of time.
At some point, the relationship between the data rate, the available bandwidth, and the
likelihood of network disconnection will reach a tipping point, at which it will be
impossible to replay the messages cached from the previous disconnection before the
next disconnection occurs. At this point, the network, while connected, will be
continually full, but receiving applications will be permanently behind.
A data-centric message design eliminates dependencies between messages, allowing
durable implementations to cache safely only a bounded number of messages rather than
all that were ever sent, reducing both local resource use as well as network bandwidth
requirements. DDS can express such a design directly using standard QoS policies, and
the DDS-RTPS/UDP protocol stack can support these policies all the way down to the
network level.
• Graceful degradation: In some cases, if it’s not possible to deliver every message, it’s
best to deliver none of them. In other cases, graceful degradation is more desirable:
deliver as much data as possible within the reliability and timeliness requirements of the
applications involved, but allow other messages to be dropped in the interest of allowing
those applications to continue processing the most up-to-date information. Paradoxically,
a middleware that expects to be able to deliver everything over a network that can’t fulfill
that expectation will often end up delivering very little—and at great expense, as it
continually floods the network with negative acknowledgements and resends of messages
that were previously dropped by the network.
A data-centric message design enables graceful degradation by eliminating dependencies
among messages in the same stream. DDS provides standard QoS policies and a flexible
protocol that allow this design to be realized in a portable and interoperable way. These
policies allow administrators to specify the strict reliability guarantees some message
streams require. But they also allow more relaxed contracts when and where appropriate,
including dropping unacknowledged messages that have been superseded by a certain
number of subsequent messages, down-sampling rapid-fire streams, and so on.
4.3 Scalability
DDS networks such as that shown in Figure 8 above enable scalability in each portion of the
overall system.
Peers are lightweight. Each application participating in the local network needs to keep only an
in-memory cache of the recent updates to the data it is publishing or has subscribed to. It does
not need a traditional database or persistent storage. Furthermore, a single UDP socket can
communicate with an arbitrary number of remote peers, so IP resources are kept to a minimum.
—14 of 20—
15. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
These efficiencies allow DDS implementations to run in embedded systems in addition to
enterprise-class workstations and servers.
Peer-to-peer networks are reliable and efficient. Peer-to-peer communication avoids artificial
bottlenecks and single points of failure. The DDS on-the-network format in particular is designed
to be highly compact, and multicast communication is supported (though not required).
Thousands of applications can communicate in a single network, exchanging hundreds of
thousands of data updates per second per producer-consumer pair, or many millions in aggregate.
(These same properties make DDS well suited for disadvantaged, limited-bandwidth, and/or
intermittent links.)
Wide-area networks require flexibility. Unlike TCP-based protocols, DDS-RTPS offers per-
channel quality-of-service control and avoids head-of-line blocking. These characteristics
improve performance and make the infrastructure’s performance more predictable, even over
challenging links. When connecting local and wide-area nodes, a broker can forward data and
shape traffic appropriately for each side.
4.4 Security
Secure messaging requires a comprehensive approach.
• Implementations must run on secure platforms to prevent errant code from exploiting the
network.
• Applications must communicate over secure transports to prevent unauthorized parties
from snooping their data.
• Data must remain confidential even when passing through brokers, persistence services,
and other infrastructure components.
• Data must be properly attributed such that recipients can understand from whence it
came.
• Middleware must enforce system-level access-control policies to limit the production and
consumption of data to authorized parties.
OMG has published a complete security roadmap for DDS. This document describes where the
specification is currently and where it is going.
4.4.1 DDS Security—In Production
Standard interception APIs for policy enforcement. The DDS API notifies applications when
remote peers attempt to communicate with them—to join the same network, publish data to the
application, or subscribe to data from the application. These notifications carry with them
metadata about the remote application, publication, or subscription—including, if desired,
security credentials. Applications then have the opportunity to reject communication with
unauthorized peers.
—15 of 20—
16. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
Figure 9—DDS access control
Vendor-supplied secure transport. OMG is currently working with vendors on the
specification of an interoperable secure transport for DDS (see below). In the mean time, secure
transport support based on the IETF-standard TLS9 protocol (over TCP, or DTLS over UDP) is
available from the vendor community.
Deep packet inspection. In DDS, data types are discoverable and data formats are standardized,
allowing data updates to be introspected at run time by the infrastructure. This can be done
without or without the use of XML—there is no need to give up the compactness or performance
of binary data.
Secure operating-system support. DDS implementations run on secure enterprise platforms
such as SE Linux as well as secure partitioned operating systems such as VxWorks MILS.
4.4.2 DDS Security—In Progress
OMG is currently working with vendors on a comprehensive security specification that will take
DDS to the forefront of middleware and messaging technologies. This specification will address
the following scope:
• Interoperable secure transport, such as TLS, for security in transit
• Data-level tagging, signing, and encrypting for non-repudiation and confidentiality, even
as data updates traverse brokers or are persisted to disk
• Authentication and authorization at the domain (network) and topic level to enforce
system access-control policies
• A richer set of pluggable service-provider interfaces to allow users to integrate security-
enforcement mechanisms across multiple platforms and technologies in a system—
without locking themselves into a vendor-proprietary stack
OMG issued the RFP for this specification late last year and is currently processing initial
proposals. An adopted specification is expected late this year or early next year.
9
Transport-Layer Security (TLS) is the current name of the specification previously known as Secure Socket Layer
(SSL).
—16 of 20—
17. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
Figure 10—Overview of the in-progress DDS security specification (source: OMG DDS Security RFP)
5 Conclusion
Interoperability based on open standards fosters the growth of a competitive marketplace to
empower innovation while driving down costs. A monolithic “common” infrastructure by itself
can achieve neither of those ends. The customer community, the vendor community, and
independent standards bodies like OMG must work together.
This is what OMG, the Navy, its integrators, and its supporting vendors have done around DDS
technology. DDS applies long-proven architectural principles in new ways to enable rapid
development and insertion of new capability, lower-risk system integration and evolution, and
more reliable operations. At the same time, multiple vendors actively compete for Navy
business, lowering acquisition and life-cycle costs.
6 Appendix: Technology Comparison
The following tables compare specifications and vendor implementations of several
technologies.
—17 of 20—
18. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
6.1 Specification Comparison
DDS WS-N AMQP
Governing Body OMG W3C AMQP Working
Independent standards body; Independent Group
standards body; Industry consortium;
Open membership under published
rules Open membership Ad hoc membership
under published
rules
Participation 12+ (DDS), 12+ (C4I, A&D specs Unknown 12+;
atop DDS);
Weekly conference
Quarterly in-person calls (minutes available)
meetings (minutes available to
OMG members)
Vendors About 6+ About 3 About 4
Primary Defense (prod’n) Defense (prod’n) Finance (prod’n)
Adoption Communication (prod’n) Defense (dev’t)
SCADA/Industrial (prod’n) Transportation
(unknown)
Transportation (prod’n)
Finance (prod’n)
Integration Data-Centric Message- Message-Centric
Architecture Centric
Portable API DDS 1.2 (Java, C++, C); WSDL-based None;
JMS 1.1 (Java; vendor-specific) JMS 1.1 (Java; vendor-
specific)
Data/Message Formal Formal Informal or formal
Definition W3C XSD or OMG IDL W3C XSD AMQP-specific language
—18 of 20—
19. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
DDS WS-N AMQP
Interoperable Real-Time Publish- SOAP 1.1, 1.2 AMQP 1-0r0
Protocol Subscribe 2.1 Transport: Pre-v1 release candidate;
Transport: UDP ucast & mcast; HTTP/TCP
Transport: TCP
Transport: TCP (in progress +
vendor-specific)
Throughput 10Ks–100Ks msgs/s10 10s msgs/s 100s–1Ks msgs/s11
(one-to-one)
Security AuthN/AuthZ interception Secure Secure transport;
pts; transport; AuthN/AuthZ
Improved AuthN/AuthZ (in Data signing, (vendor-specific)
progress); encryption
Secure transport (in progress +
vendor-specific);
Data tagging, signing,
encryption (in progress)
6.2 Vendor Comparison
IBM PrismTech Red Hat NCES
(R3) (OpenSplice) RTI (MRG-M) (JUM)
API JMS 1.1 DDS 1.2 (Java, DDS 1.2 (Java, Vendor- WSDL-
(Java) C, C++, .NET) C, C++, .NET, specific (Java, based
Ada); Python, C++,
.NET);
JMS 1.1
(Java) JMS 1.1
(Java)
10
Source: http://www.rti.com/products/dds/benchmarks-cpp-linux.html#MSGRATE. This data is presented for one-
to-one connections.
11
Source: “Reference Architecture Red Hat Enterprise MRG Messaging” whitepaper linked from
http://www.redhat.com/mrg/messaging/. This data is presented in aggregate across 60 applications. The test
methodology describes how to derive one-to-one measurements from it.
—19 of 20—
20. Copyright 2011, Object Management Group (OMG). All Rights Reserved.
IBM PrismTech Red Hat NCES
(R3) (OpenSplice) RTI (MRG-M) (JUM)
Protocol Real-Time Real-Time Real-Time AMQP 0-10 WS-N
Publish- Publish- Publish- Pre-v1
Subscribe Subscribe 2.1; Subscribe
2.1 RT- 2.1
Networking
(vendor-specific)
Enterprise Yes Yes Yes Yes No
Support
Option
License Comm’l Open Source; Comm’l Comm’l Comm’l
Comm’l Free for eval, Based on open-
IR&D; source Apache
(free for eval) qpid
Source avail for
purchase
Topology Brokered Peer-to-peer; Peer-to-peer; Brokered Brokered
Brokered Brokered
Redundancy Unknown Hot producer Hot Clustered Unknown
fail-over producer brokers
Per DDS fail-over Vendor-specific
Ownership spec Per DDS
Ownership spec
Persistence Broker Per-node Broker; Broker Broker
Location daemon Standalone
service
Data Caching None Yes Yes None None
In-memory; In-memory;
Persistent Persistent
—20 of 20—