SOFIA - Smart Objects For Intelligent Applications. INDRA/ESI
SOFIA project: Smart Objects For Intelligent Applications Jesús Fernández Gómez-Pimpollo Raúl Otaolea Indra ESI Tecnalia Summary — The SOFIA project (http://www.sofiaproject.eu/), a three-year ARTEMIS project started in January 2009 andinvolving partners from four different EU countries, is built around the concept of smart environment, intended as an ecosystem ofinteracting objects – e.g. sensors, devices, appliances and embedded systems in general – able to self-organize, offer federated servicesand process or provide complex data. The mission of the SOFIA is to create a semantic interoperability platform and a selected set of vertical applications to form smartenvironments based on embedded systems. The key factors in these smart environments will be an open and distributed informationstorage and common search procedures for all the embedded systems, implemented with different specific technologies. In this vision,local mash-up applications will be built on open data and using a range of devices. I. INTRODUCTION SOFIA envisions that the ongoing digital convergence will fundamentally change the way embedded systems are used toconstruct not only physical devices and but also services. First, consumer goods are more and more composed of multipleembedded systems. In this respect, car industry is a leader but the phenomenon is rapidly gaining ground also in mobile,consumer electronics, home automation and other industries. Second, the internet networking has revolutionised the way digitalservices are provided to the consumers. For example, an embedded GPS device can well calculate its location by synchronisingwith the satellites but the calculation of initial co-ordinates will be much more efficient if the embedded device provides initialinformation to a dedicated server that possesses more information and computing power. SOFIA believes that the digitalconvergence has still a huge unleashed potential for consumers, industry and research but it requires a common understandingof the information exchanged between embedded systems through heterogeneous systems and communication technologies. Central to SOFIA project is the notion of smart environment. A smart environment is an ecosystem of interacting objects -e.g. sensors, devices, appliances and embedded systems in general - that have the capability to self-organize, to provide servicesand manipulate/publish complex data. The mission of SOFIA project is to create a semantic interoperability platform and selected set of vertical applications toform an embedded system based smart environment. The key factor in these smart environments will be common, openinformation storage and search extent for all embedded systems, regardless of their specific implementation technology. In thisvision, simple, local mash-up applications will be built on open data and devices. The motivations behind this approach are twofold: • connecting real physical world with information world enriches the user experience; • an open interoperability platform will foster innovation and will guarantee the future evolution of embedded systembased smart environment, both from a scientific/technological point of view and in terms of business. With these concepts, SOFIA proposes an Internet-like revolution in physical space. Main goal of the project is to make "embedded information" in the physical world available for smart services - connectingphysical world with information world. Common targets are to enable and maintains cross-industry interoperability, to fosterinnovation while maintaining value of existing legacy and to create new user interaction and interface concepts to enable usersto benefit from smart environments. In particular, a major target is multi-vendor interoperability platform as platform for newservices. The project addresses three application areas or “verticals” which represent different kind of spaces, in terms of scale andpotential applications and services. Spanning from very local personal spaces, e.g. the vehicle, to smart indoor spaces andfurther to smart city the project captures the specific aspects of smart spaces and combines the requirements for commonsolutions. The key outcomes of the project relate to user interaction paradigms for interacting in smart environments, the commoninterpretability solution between many heterogeneous devices and embedded systems, and on the application developmentschemes that can mobilise new developers for smart environments. The principal SOFIA concepts and uses cases included within this contribution are: SOFIA ADK, Smart Spaces in theSOFIA concept and the unified monitoring and surveillance systems.
II. SOFIA ADK The ADK is a set of tools that provides several functionalities to the whole development life cycle. In the Ontology DrivenDevelopment (ODD)  based on Ontology Driven Software Engineering approach there are three different phases: design,implementation, testing and deployment. Figure 1. SOFIA development life cycle.A. Design The Wizard plug-in centralizes the programmer decisions about the Knowledge Processor (KP). The wizard guides thedeveloper to choose in between ontology concepts to use and their profile (producers and/or consumers), platform andprogramming language, communication protocol and project details like name, package names, etc. A new SOFIA project is created in the last wizard´s step where all necessary middleware (all layers in the proper language,an editor for that language) is added and finally all the connectors selected; Smart Space Access Protocol (SSAP) orcommunication protocol knowledge is not needed to start coding the logic of a KP. This plug-in selects a proper editor for theprogramming language chosen by the KP developer, and assign it to the SOFIA project generated. Connectors are alreadycoded for a variety of languages (partially programmed for Java) and will be stored in a centralized server where ADK connectsto and downloads locally. Openness and scalability are major factors, therefore connectors for different communicationprotocols and different languages are feasible. The Smart Application Wizard includes the OWL2Classes facility, whichtranslates several ontologies into programming languages classes in order to be used with the SOFIA middleware. When thisclass is used, all its interactions are sent to the SIB; therefore programmers do not need to have expertise knowledge aboutontologies.B. Implementation A KP is divided in four layers: 1. Logic. A developer is only responsible for programming this layer and represents thefunctionalities a final user or a company want to create. 2. Semantic model. This layer is automatically generated by theOWL2Classes tool of the ADK. It is an API with the classes in a specific programming language representing ontologiesselected in the wizard tool. Thus, they are the concepts to exchange with the smart space. 3. SSAP. This layer gathers all APIsdedicated to deal with the SIB. The upper layer knows how to invoke the appropriate methods of this API. . Connection. Thedifferent connectors are plugged in this layer to connect with the SIB. It is also responsible for Semantic Information Broker(SIB) discovery.
KP KPI Figure 2. Knowledge Processor Structure Hence, the complexity related with ontologies and communication protocols are totally encapsulated in a very easy APIformed by plain classes (model layer). Developers only have to instantiate them and use their getters/setters in order to interactwith the smart space.C. Testing and Deployment The Semantic Information Broker (SIB) is the responsible for the interoperability between heterogeneous smart applicationsand devices. Its core is differentiated in three parts: 1.Sessions. Refers to the KP connected to the SIB at a given moment. 2.Subscriptions. Subscribed nodes are notified when certain conditions in the semantic model are produced. 3. Semantic Modelprovides the interoperability level needed to connect different devices, platforms, sensors or data using the SOFIA Smart Spacecontext. It also enables the knowledge processing, sharing and reuse between applications. The Semantic Model is based onJena (Semantic Web Framework), which provides an API to allow the storage of ontologies, classes, individuals, properties andall data related to ontologies. It is assumed that languages can be used for the interaction with the model are OWL (or RDF-XML) and N3. Decisions are based on the definition of the SSAP Message Protocol defined in Description of XML encodingof SSAP document (internal Wiki). This model maintains a full class hierarchy of already connected nodes and the individuals(instances or triples) that has joined the SIB. All KPs must send in their first ontology model connection to the SIB, so that, allindividuals defined in that KP can be understood by the SIB and other KPs in the Smart Space. When a KP is removed from theSmart Space all the information about this KP is also removed from the SIB Semantic Model, including all the insertedindividuals. The SIB OSGi Bundle exposes a set of services using the public interfaces ISIB and IViewer; the first shows all themethods to interact with the SIB using SSAP Message type communication, Gateways and KP must communicate with the SIBthrough ISIB. The second exposes methods to observe the content of the SIB and to register the changes that are producedinside the SIB. There are two managers; Subscription Manager is responsible for maintaining the information about the subscribed KPsand Session Manager manages the KPs connected to the SIB at a given moment. Only connected KP-nodes are allowed toperform the SSAP operations in the SIB. A KP-node connects to the SIB using the JOIN-REQUEST message and disconnectsto the SIB using the LEAVE-REQUEST message or when some communication error arises for that connection. BothManagers include queries to that are subscribed the KPs, the already sent data to the subscribed KPs, the check-process whensomething is changed in the semantic model and the indication throwing process. The SIB into a Sofia smart space needs to interoperate with the KPs of several smart applications that are running into thesmart space, which enables the KP to change information into the SIB and the SIB communicates those change to several KPssubscribed to them. KP can be executed in any physical device with communication capabilities to interoperate with a SIB ofthe smart space. The architecture could become a problem while considering having devices with different communicationprotocols; the solution to this issue is the use of Gateways. The OSGI platform is considered as a module container to achieve a modular and extendible architecture, where either theSIB or the different Gateways are OSGI bundles and changes within the SIB bundle will not affect to any Gateway bundle. Thisarchitecture will be enhanced with new Gateways supporting new communication protocols and increasing the amount ofdevices where smart applications can be deployed. Every Gateway will manage the opened sessions between the KPs and theSIB, for this fragment bundle will be extended by every new Gateway, freeing new Gateways developers of any responsibilitytowards this task. Moreover, two specific Gateway OSGI Bundles are been looked at the moment to support TCP/IP andBluetooth connections. The platform is also open to develop new Gateways supporting more connection protocols. The SIB server and viewer are both Eclipse plug-ins. The server represents any server visually and its two mainfunctionalities are start and stop the SIB. The viewer visualizes the content of the embedded SIB and has three tabs: 1. Sessionsrepresents a KP or KPs connected. 2. Subscriptions visualize KP subscriptions (query and results). 3. Semantic model checks ifinsertions and removals are correct between the connected KPs and the SIB, which is a key tool to test KP.
D. Future and conclusions The main goal of the ADK is to help developers programming high quality smart applications very fast. Even though theactual progress of the project meets this objective, we want to push this concept to the limit and even allow final users to buildtheir own applications. To reach this ambitious goal, we are currently defining a smart application model (as a part of the ODD)to design easy applications visually linking the different parts semantically . This approach will gain high dynamicity andcomposability inside a smart space, because the dependencies are not established statically but semantically so that they can besolved at runtime based on semantics. A version with this feature will be ready by the second version of the ADK. II. SMART SPACES IN THE SOFIA CONCEPT In SOFIA, a smart space is an information search extent where semantic information is presented in the form of an ontologygraph or, equivalently, of triples (subject-predicate object). SIBs are the information stores of the smart space. They provide aservice interface for agents (KPs). KPs can access the information within the smart space by connecting to the SIB andinvoking the operations offered by the SSAP interface. Through SSAP operations, KPs can perform session management (join, leave) and triple governance transactions (insert,remove, update, query, subscribe, unsubscribe) in order to insert new information into the smart space, remove obsolete orotherwise undesirable information from the smart space, retrieve specific data on-demand and be dynamically notified of thechanges in the information content of the smart space by a subscription/notification mechanism. A KP, that has discovered a smart space it wants to interact with, must first join the smart space by sending it its credentials.If the access control policies are met, the KP is allowed to establish the connection. Service discovery is accomplished usingthe mechanisms available at the service level (e.g. NoTA ). To enable their reactive behavior the KPs make subscriptions inthe smart space. The subscriptions are persistent queries that notify the subscribing KP every time that the query results change.This allows the KP to react appropriately and timely to changes in the smart space, while avoiding unnecessary communication. Depending on its behaviour, a KP can be a producer KP (i.e. a KP able to generate new information and insert them into theSIB), a consumer KP (i.e. a KP able to retrieve and react on specific information from the SIB through queries orsubscriptions) or both at the same time. The complete SIB interface is a union of the consumer and producer interfaces. III. THE UNIFIED MONITORING AND VIDEO SURVEILLANCE SYSTEM This section reports on the results achieved in the application areas related to Smart City spaces, focusing in particular on ourUnified Monitoring and Video Surveillance (UMVS) system. Among other possible scenarios, the subway station has beenselected, as it is a public and densely populated area that needs to be continuously and effectively monitored throughheterogeneous and distributed sensors and video cameras and a dynamic environment with several categories of users(passengers, operators, security staff) interested in different events to be delivered at the right time. Figure 3. Use Case diagram
Some solutions have been proposed in the past for event detection based on Wireless Sensor Networks (WSNs)  and videosurveillance . The SOFIA IOP enables the combined processing of both WSN and video surveillance data resulting in moreefficient event detection. This concept is the starting point for our UMVS architecture. The UMVS architecture follows a multi-layer approach and is based on a common, distributed UMVS middleware able togenerate heterogeneous low-layer data, called raw events that describe various aspects of the surrounding environment. Theyare combined into a set of complex higher-layer information, called composite events, and made available to the UMVSapplication layer through the SOFIA IOP. The UMVS middleware includes two types of modules: the producers of the raw events and the data correlators (UMVSEvent Managers) in charge of processing the low level data in order to produce more detailed events. Currently the raw eventsoriginate from producer KPs such as Closed-Circuit TV (CCTV) servers handling video cameras distributed in a large area, orWSNs characterized by different kinds of sensors (e.g. temperature, smoke, presence) , but the UMVS middleware can beeasily extended in order to support further data sources, including third-party services. The information flow of our UMVS system is described through a structured ontology. The main advantage of this approachis that although a uniform mechanism is used throughout, individual subsystems can be treated separately. For example, theKPs inside the CCTV server and the WSN managers need to adhere to only a fraction of the ontology and need not beconcerned about the rest. This is helping the implementation on resource-constrained devices, and at the same time is beneficialfor the extensibility of the system. The data exchange among all the UMVS modules, as well as the consistency and thesynchronization of the information stored in the SIBs, relies on the SOFIA IOP mechanisms. At the top of the UMVS architecture, the application layer includes interactive modules that inform the users of events oralerts and allow them to trigger further actions. All the UMVS applications interact with the SOFIA IOP in order to receive andexploit the data provided by the UMVS middleware, including both raw and composite events. On the other hand, some trusted applications are also able to insert into the SIBs some specific events (i.e. alerts), so that theycan be propagated towards their own peers, or commands (i.e. configuration parameters or pan/tilt/zoom (PTZ) commands) totrigger some specific actions in a given device. The UMVS applications provide services for various user groups, characterizedby specific roles and enabled to perform different activities, depending on their user profile and the associated access policies. The UMVS application layer comprises a set of applications deployed on a variety of mobile and stationary devices. Centralto the UMVS is a monitoring station application that allows “first-level operators” to visualize all the events and alertsexchanged through the UMVS middleware, generate new alerts, and send configuration commands to the WSN managers andthe CCTV server. The monitoring station also receives video streaming directly from the CCTV server, bypassing the IOP forperformance reasons. “Second-level operators” use a mobile or wearable device to receive a subset of the events/alerts generated by the system andto generate a subset of alerts addressed to other operators. Mobile devices with sufficient resources can interact directly withthe CCTV server to receive video streams and control remote PTZ cameras. A third type of UMVS applications deals withusers that need to be informed of exceptional events such as an evacuation order. In this class, there are modules that changethe public displays accordingly, but also commercial signage displays are integrated into the emergency signaling. They switchto a pre-programmed alternative content that leads passengers more prominently to the emergency exits than it is possible withtraditional signs. Finally an UMVS application informs passengers with suitable personal mobile devices. IV. ACKNOWLEDGMENTS SOFIA project is partially funded by the European Commission, within the framework of the ARTEMIS JU SP3 SOFIAproject (http://www.sofiaproject.eu/) and the Spanish Ministry of Industry, Tourism and Trade. The authors would like to thankall project partners who are contributing to the definition and implementation of SOFIA targets. V. REFERENCES W3C, Ontology Driven Architectures and Potential Uses of the Semantic Web in Systems and Software Eng., 2006,online:http://www.w3.org/2001/sw/BestPractices/SE/ODA/. K. M. de Oliveira, K. Villela, A. R. Rocha, and G. H. Travassos, “Use of ontologies in software development environments,” in Calero,C., Ruiz, F., Piattini, M. (eds) Ontologies for Software Engineering and Software, 2006. M. Vanden Bossche, P. Ross, I. MacLarty, B. Van Nuffelen, and N. Pelov, “Ontology driven software engineering for real lifeapplications,” in Proc. 3rd Intl. Workshop on Semantic Web Enabled Software Engineering, 2007. NoTA, http://www.notaworld.org Toninelli, A., Bellavista, P., Pantsar-Syv¨aniemi, S., Ovaska E.: Supporting Context Awareness in Smart Environments: a ScalableApproach to Information Interoperability. In International Workshop on Middleware for Pervasive Mobile and Embedded Computing(M-MPAC’09), within the ACM/IFIP/USENIX 10th International Middleware Conference (Middleware’09) Chatzigiannakis I., Koninis C., Mylonas G., Colesanti U., Vitaletti A. (2009). A Peer-to-Peer Framework for Globally-Available SensorNetworks and its Application in Building Management. In: 2nd InternationalWorkshop on Sensor Network Engineering (IWSNE’09) SanMiguel, J.C., Martinez, J.M., Garcia A.: ontology for event detection and its application in surveillance video, Sixth IEEEInternational Conference on Advanced Video and Signal Based Surveillance, Genova 2009.