SOFIA - Experiences in Implementing a Cross-domain Use Case by Combining Semantic and Service Level Platforms - NOKIA/CRF/VTT

Uploaded on


More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Experiences in Implementing a Cross-domain Use Case by Combining Semantic and Service Level PlatformsVesa Luukkala, Dirk-Jan Binnema, Mihaly Börzsei Alessio Corongiu Nokia Research Center Centro Ricerche Fiat Finland Orbassano, Italy {vesa.luukkala,dirk- jan.binnema,mihaly.borzsei} Pasi Hyttinen Valtion Teknillinen Tutkimuskeskus Kuopio, Finland pasi.hyttinen@vtt.fiAbstract— We describe experiences of implementing a mash- mechanisms which allow handling and representing semi-up of two independent use cases, one operating in the structured information. At the lowest level, the Resourceembedded domain and the other in the internet messaging Description Framework [4] (RDF) is used to present thedomain. We combine these service implementations on anexisting Service Oriented Architecture (SOA) system with information as a set of triples. The structures which are builtagents executing on a novel smart space architecture. Based on on top of RDF can be specified by knowledge representationthe lessons learned from this work, we sketch an approach for languages such as RDF-Schema [5] (RDFS) or a webimplementing similar use cases in a scalable and future proof ontology language [6] (OWL). These languages are used tomanner by defining simple ontological concepts and associated define ontologies which describe the shared vocabulary forbehavior. modeling a particular domain. Exhaustively defining and Keywords: semantic interoperability, semantic web, smart agreeing ontologies for all or most domains and participantsspaces, RDF, ontologies, embedded systems is a similar effort to standardization, but the Semantic Web approach gives the possibility of leaving information only I. INTRODUCTION partially defined. Furthermore, the information described by semantic web technologies is to a degree self-describing,A smart space can be defined as an ecosystem of interacting allowing participating entities to interpret the information atcomputational objects. The physical manifestation of a runtime. To minimize the standardization problem, wesmart space can consist of sensors, devices, and appliances expect to canonically define the smallest building blockswhich communicate with each other by means provided by (RDF, RDFS, OWL) as well as selected higher levelthe smart space. To guarantee interoperability between the concepts as core ontologies. Use of ontologies and evenparticipants, a common ground needs to be defined and this core ontologies is not enforced; the participating nodesis typically done by means of standardization. However, the define their own, local interpretation of the used ontologies.promise of smart space environments is that all kinds of Eventually this may lead to emergence of de-facto standardinformation can be shared and accessed via the smart space ontologies.and thus the number of participating devices, vendors, The Artemis SOFIA project [7] aims to provide a solutionproduct domains and produced information is large. The to the interoperability problem by creating an opengamut runs from sensors measuring a physical property like innovation platform which is a framework of middleware,temperature, to mobile devices providing a list of favorite methods and tools to create ecosystems of interactingrestaurants and their addresses. The downside of this variety objects. The presented work is part of the SOFIA projectis that creating a common standard is hard, as is changing an effort. We have used the Smart-M3 [8] middlewareexisting standard [1]. The existing SOA-based [2] produced by SOFIA to implement use cases in the SOFIAapproaches offer tools for defining service interfaces, but “Personal Spaces” work package. “Personal Spaces” aimsthey also suffer from the same problem of standardization. to create a smart space around a moving person in order toThis slows down innovation which would otherwise benefit enable this person to access and organize relevant servicesfrom the available multitude of information. Semantic Web and data around him. Here we use the term “service level”[3] approaches offer one solution for the above problem by
  • 2. to mean implementations based on SOA [2] principles (e.g. cases and we have selected two from these to beWeb-services, UPnP, NoTA), which typically consist of an implemented and then combined, even though they competeinterface description and possibly a way of advertising the for same at a registry. By “semantic level” we mean the The driving force in our particular environment is thatinformation sharing occurring in a blackboard-like Smart- existing and future devices and embedded systemsM3 system by means of the above Semantic Web participating in the personal space have very different lifetechnologies. cycles. A car model has a life cycle of eight to twelve years, We have selected two use cases: “Music follows user” whereas a consumer electronic device has a life cycle ofand “Read aloud message”, which we are using as trialing only a few years. Smart space ties these differentplatforms for SOFIA results as well as parts of the “Personal interoperating systems and devices closer together. Smartspaces” offerings for the user. Our specific target is to space interoperation has a net effect that products nowcombine or mash-up the two use cases and be able to repeat having short life cycles can be longer – new features appearthe same process for future use cases. A more generic goal from smart space. Also product having now long life cyclesis to be able to execute the same use case in many can be shorter via smart space application. We expect thatenvironments using the best possible resources. For example similar situations arise elsewhere as well. The combinationin our case we want to reify the abstract notion of “music of on-board facilities and nomadic devices as well aslistening” to a semantic level service, which may be services here presented will be offered in the near futureinstantiated to seamlessly use various available resources. will narrow this gap, allowing for a fast market growth for We present a naive solution and, based on our these applications. The possibility of bidirectionalexperiences from that work, we propose to introduce and communication among the vehicle and the personal devicesdefine ontologically a semantic level entity corresponding to enables the use of vehicle system and nomadic devicethe semantic service and the resources it uses. Any service applications and controls in coordination for example,implementations or agents which contribute to this semantic sharing displays and controls in both systems. Furthermoreservice must adhere to a small set of commands given by the it enables the access to vehicle data via mobile device in asemantic level. The managing of these semantic service way fully transparent to the user. The service andinstances must be implemented separately. application will not be coupled strictly to one device but This approach resembles service composition or rather to the environment.orchestration [9] and it can also be viewed as a simple and The mash-up of the two selected use cases aims to showstatic implementation layer for a multi-stakeholder how a user can enjoy music from different devices whiledistributed system [10]. moving around between different locations and given the Section 2 describes the environment we are operating in, opportunity, use resources from the personal environment tosection 3 describes the architecture and the implementation, enhance the experience of playing music. User can startSection 4 describes our findings, and solutions to solve the listening to music on the personal mobile device (MD),observed problems and section 5 offers conclusions and our picking a song. User can control the playing from thefuture plans. mobile device user interface and the music plays from its loudspeakers. When entering a car with steering wheel II. BACKGROUND controls and a rear seat entertainment system (RSE), the For our “personal space” we expect to have the following user can switch the rendering between the mobile deviceresources available: a Mobile Device (MD) with multiple and the Rear Seat Entertainment system at will, with aconnectivity options, an embedded car dashboard unit with dedicated user interface element. The user can also controlBluetooth connectivity and a Rear Seat Entertainment the rendering both from the steering wheel as well from theSystem (RSE) with WLAN connectivity. These devices mobile device user interface.implement services on an embedded SOA system [11] and Note that the mobile device and the on-board telematicuse a common logical blackboard for sharing of semantic unit share the control of the playback by means of thelevel information (the Smart-M3). Smart-M3 is a platform Steering Wheel Buttons, while audio handover happensinspired by the tuple space paradigm [12]. It provides a between speakers of the mobile and the rear seatsingle logical information store by which multiple entertainment system.independent nodes interact with each other. Store and the The user is also subscribed to a particular twitter feed andassociated nodes can be thought to implement a “smart when a new message is received during music playback, thespace” [13]. We expect that later on new services and music is paused, the message is read out aloud and then theinformation will be available for the “personal space”, for music rendering resumes. This is useful when the messagesexample via the other SOFIA work packages. In order to describe for instance traffic conditions. While the use casesshow the value for the user we have defined several use are as such independent from each other, they both require
  • 3. exclusive use of audio resources. We expect similar kind of smart applications to program using ontology conceptsof conflicts over same resources to arise more generally in relevant to the application area. This Ontology API mayother use cases. either be provided by a separate ontology library generated Also, the components we are using as building blocks of at design-time, or by the Sofia Application Development Kitthe use cases are a suitably diverse set of service (ADK).implementations, applications using the services and The service architecture is based on the NoTA servicesemantic level agents. These give a good overview of based concept. In NoTA-based systems, the servicetypical challenges in integrating semantic-level entities to implementations are called Service Nodes (SNs) and usersservice-level and other functionality. of those services are called Application Nodes (ANs). A At the time of this implementation, no core ontologies NoTA service implementation is typically accompanied by awere available, so we used our own simple domain service description file, which enumerates the serviceontologies and tools as a starting point. messages that are used to communicate with that service. The service description file is the canonical definition of III. LOGICAL ARCHITECTURE AND IMPLEMENTATION the service interface, but not the service behavior. These files can also be used to generate stub code to access the A. Architecture service. To enable run-time discovery, NoTA services can The functional architecture of the use cases is illustrated either register themselves with a Resource Manager (RM)in Figure 1. The two main elements at the semantic level are or, alternatively, count on pre-existing knowledge regardingthe Semantic Information Broker (SIB), an entity the service ID (SID). The resource manager can be thenperforming triple governance in possible co-operation with queried by clients for service descriptions and connected toother SIBs for one Smart Space and the Knowledge the service based on the results.Processor (KP), an entity contributing to insert/remove)and/or consuming (query/subscribe) content according to B. Service Level Implementationsontology relevant to its defined functionality. Figure 2 depicts the service level architecture implemented for the “Music Follows User” use case. The remote music rendering use case is implemented directly over the NoTA service-level and requires the NoTA infrastructure to operate. The audio rendering service implementations rely on the OpenMAX-IL-IL standardOpenMAX-IL. The Steering Wheel service implementation is basically a keypad service implementation. Figure 2 - The "Music follows user" service architecture 1) The player The mediaplayer acts as the control point for the use case “Music follows user”. The user can start playing an audio file and control the playing with the usual player controls on Figure 1 - Service level and semantic level architecture the mobile device. The player acts as a client to existing The KPs can connect to the SIB with any suitable services (keypad, OpenMAX-ILRemote Audio Renderer,transport using the Smart Space Access Protocol (SSAP). SIB as described below). The player itself is not a service inThe components handling different transports may either be the SOA sense as it does not offer a service interface. It stillincluded in the SIB or KPs at compile-time or be dynamic, implements the functionality for music rendering and is inplugin-like components. Both SIB and KPs may have charge of the interoperability between the Remote Audiomultiple transports. Renderer OpenMAX-ILand Keypad services. Note that we The architecture also contains the notion of an Ontology could (re-)build services using the same services as theAPI, which is an ontology-specific API allowing developers
  • 4. player uses, but this would mean unnecessary duplication of C. Semantic level implementationwork and in case of handover, fairly involved engineering After this discussion of the service level implementations,work. The semantic part of the mediaplayer is that it we continue with the semantic level – that is, the knowledgeconnects to the SIB service and then places a subscription processors (KPs). We have three KPs: the “tweet-on the arrival of a message, which then causes pausing and notification”, “media-control” and “speech synthesizer”.resuming. Firstly, the “tweet-notification” refers to the detection of 2) Steering Wheel new Twitter messages (“tweets”). It is backed by a specially The Steering Wheel service implementation makes the crafted version of the Mauku Twitter-client, and thebutton presses on the wheel available for remote clients. The “MessagingClient” NoTA service, which along withsteering wheel buttons are mapped to the usual player “speech synthesizer” allows for retrieving the arrivedcontrols. The transmitted key symbols are compatible with “tweets” and performing text-to-speech of their contents.the X Window system keycodes. Whenever “tweet-notification” detects that a new tweet 3) ILProxy has been added to the Twitter feed, it then creates an OpenMAX™ is a royalty-free, cross-platform API that instance of a Message (as defined in domain specificprovides comprehensive streaming media codec and messaging ontology) to the Smart Space. The message classapplication portability. The OpenMAX IL (Integration definition is as follows:Layer) API defines a standardized media component wp1:msg a owl:class;interface [14]. ILProxy is a drop-in replacement for a local wp1:senderId xsd:string;OpenMAX-IL library, which makes a remote OpenMAX-IL wp1:msgId xsd_string; wp1:arrivalTime xsd:string;library implementation available locally. In this case, the wp1:content xsd:string.“service description” is the C header file for the library. The “tweet-notification” fills in all the properties, except the Upon request, the client streams this music to the remote content playback device. In the demonstration setup, we use Secondly, the “speech synthesizer” has subscribed itself toan NXP media rendering device for that, but the same role instances of type msg and for new messages it first obtainscould be played by any computer with the necessary the message contents corresponding to the msgId from thesoftware. The client creates a distributed rendering pipeline NoTA messaging service. Then it assumes that the playingstarting at the mobile device and ending in the remote is stopped and initiates the reading of the message content.playback service. The client feeds the music files into this After the message has been read, it adds thepipeline, and it allows for switching between local and msg:hasBeenRead property with value “yes” to theremote playback. particular message instance. This is used to keep track of 4) NoTA Messaging service messages which have been already processed. The messaging service makes the content of received Thirdly, the “media-control” has subscribed itself toTwitter messages on a certain Twitter feed available.However, the full service counts on the semantic level being instances of type msg and will be notified of appearance ofavailable. In this implementation, the notification of the message triggering pausing of playback and anotherreceived message along with an ID is published via the SIB subscription described soon below. Messages that have theservice and the content is obtainable only through the NoTA msg:hasBeenRead with value “yes” property areservice level interface. The mapping of the ID to the ignored. Here we have in effect defined an extension (themessage content is internal data of this service. hasBeenRead property) to the messaging domain 5) SIB Service ontology, which is shared by the three entities. Any other The SIB service implementation is part of the M3 entity which only knows about the message definition canbaseline for SOFIA IOP. The semantic activities of this ignore the extension and operate without knowledge of that.demo are all handled through the SIB by means of nodes Also, using this representation we maintain part of the statereading and writing information to SIB service. There is a of the whole use case (the messages which have beensingle SIB implementation which exposes both a NoTA processed) on the semantic side.service interface as well as an XML based interface over IV. ANOTHER APPROACHTCP/IP. The SIB registers itself to Resource Manager likeany other service. For interoperability between entities, we need to share Any semantic level node needs to connect to this service well-defined information and agree on operationalimplementation to be able to share semantic content. From semantics according to which the information is operated onthe SSAP protocol point of view the messages are lists of [15]. The implementation described in previous section isRDF-triples. straightforward from the ontological perspective, but it has some flaws: 1. the relation between new instance of a Message
  • 5. appearing and the player pausing, is not obvious Activity. We have two devices, a mobile device and a 2. if we want to add another type of use case which dashboard device, both of which have the same capabilities needs to pause the player, then we need another for audio output and keyboard input. Even though they are ontology that must be understood by the player or contained in devices, in this example they have been even worse, we must abuse the Message mechanism. published to the SIB as individual capabilities. An This is detrimental for scalability. unspecified actor has inserted an Activity, which uses In short, the operational semantics have been some of the capabilities published on the SIB and it haspurposefully left informally defined. To solve the problems published that a more general way requires more overhead: a descriptionof the behavior in addition to a description of theinformation representation. A general solution is sketched in[10], which suggests that each participating agent publishesa description of its behavior. Other participants can interpretthis information and then deduce based on that how to hookinto the existing system. Alternatively, we could take theother extreme, which is the standardization approach bydefining operational semantics for all information. Thisapproach is not realistic due to the open ended nature of the Figure 3: Example of an Activityinformation and the amount of work required. We propose an in-between solution which is more The activity instances are managed by arbiters, which arelimited, yet simpler mechanism. We define an ontology, dedicated to a single instance of an activity. An arbiter is awhich includes an Activity class that indicates the program which monitors instances of activities. The mainresources (which may include service implementations) purpose of an arbiter is to schedule different activities bywhich are to be used by some functionality: setting the wp1:active property to True or False. Note that wp1:Activity a rdfs:Class; even though an arbiter most likely resides in the device wp1:requires rdfs:Class; which originates the Activity, it is a logical entity and wp1:uses dcs:Capability; wp1:active xsd:string; thus can reside in any node. Arbiters are able to detect on wp1:conflicts owl:Thing; their own whether different activities are conflicting by wp1:importance xsd:string; using the same resources or same types of resources. In this wp1:additional owl:Thing. case the arbiters can try to resolve the conflict by comparing The activity may be active or not active, it may have a preference information which has been related to thefree form importance tag added to it and it may also link to activity instance by the additional property. At simplestadditional information. In addition to the indication of usedresource instances, it is also possible to require a class of conflict can be resolved by comparing the importanceresources. In our use cases both the “media-control” and property along with a preference list of values. On the other“speech-synthesizer” KPs create activity instances which hand if an activity instance has not defined resources to berequire a resource of type AudioOutput and also point to used the arbiter may assign existing free resources for it.known resources. This mechanism relies on the resource For the example in Figure 3, suppose that another actordescriptions being available and described on the semantic would insert new Activity, which would use at least oneside. Any entity which publishes an Activity instance is of the same capabilities as the existing Activity. It wouldexpected to commit to some functionality and rules: then be up to the arbiter to detect that a conflict exists and 1. it observes changes to the Activity instance, secondly to resolve it based on the available information on especially the active property the SIB. For decisions which cannot be resolved by just 2. it honors the changes to the active property, when looking at the Activity instances there may be a more active has value “no”, it pauses its use of general arbiter, which is able to schedule the conflicting resources so that they can be used by other entities, entities. when active has value “yes”, it can use the resources As mentioned before this relies on the fact that the 3. it does not change the Activity instance, expect for resources which can be used are also maintained on the semantic side faithfully, so that their existence implies the uses properties readiness to participate or be directed by semantic level 4. it is able to access the indicated resource instances activities. This requires integration with the service level by information in the its descriptor resource manager. Like for the activities the resources may Here we are separating and making explicit the have their own arbiters which independently decide whethercoordination of the activity from the domain specific they comply with requests from the activities or not. Weinformation. Figure 3 shows and example of use of an
  • 6. expect that resource specific arbiters with an independent V. CONCLUSION AND FUTURE WORKarbiter for multiple activities can make essentially local Based on our experiences on implementing a mash-up ofdecisions without ending up in a heavy multiparty conflict two use cases, we have designed an ontology-basedresolution process. mechanism for abstracting away access to shared resources. The activity and the behavior experienced by the user We expect to re-implement existing use cases using thisshould reflect the desires and wishes of the user. In a mechanism to gain scalability and future-proofness of thesesimplified setting these can be expressed as an ordered list use cases. We have trialed the mechanism on semantic-levelof preferences. Preference can be defined as a set of simulator. The activity abstraction itself is fairly simple, butactivities that does not produce resource conflicts. If the it can be parameterized by producing different kinds ofactivities do not use the same resources they can be easily schedulers, which can take into account the preferencesput into the same preference set since there is never conflict. associated with activities and resources. We expect that thisIf activities use the same resource then we need to notify the mechanism will be included in SOFIA core ontologies.nature of the usage and define rules for using it. Exclusiveusage of the resource can be handled by using user defined ACKNOWLEDGMENTpriorities. Potentially conflicting activities in a preference This work was supported by SOFIA (Smart Objects Forare ordered. Activities may use several resources that Intelligent Applications) project and funded by Tekes (thepotentially are always conflicting. In this case the user Finnish Funding Agency for Technology and Innovation)accepts only one activity. and the European Commission. From the user point of view the preferred smart space REFERENCESbehavior and therefore activity is also context and situation [1] J. Farrell and G. Saloner, "Standardization, Compatibility, anddependent. User will have different audio preferences for Innovation", The RAND Journal of Economics, Vol. 16, No. 1day and night situations, for example. Full context aware (Spring, 1985), pp. 70-83and situation conscious behavior is the ultimate target but [2] T. Erl, Service-Oriented Architecture, Prentice-Hall, Englewoodvery difficult to achieve. Therefore we simplify again and Cliffs, 2005define context as all information in the smart space and [3] Tim Berners-Lee, James Hendler, and Ora Lassila. The semantic web. Scientific American, 2001.reduce the situation to set of current activities. Smart space [4] Resource description framework. is series of transitions where predefined user Referenced March 4th 2010preference is changed to next preference. This is controlled [5] RDF vocabulary description language, preference scheduler that using the resource availability schemainformation switches between preferences. [6] Web ontology language. Referenced March 4th 2010 We note that the simplest approach for combining the [7] SOFIA project. Referenced March 4thservice and semantic level is to have the service 2010implementations publishing information on the SIB where [8] OpenM3 demonstration release. information can then be freely operated on in any m3/files/OpenM3.tar.gz/download. Referenced March 4th 2010.manner by other actors in the system. This write-only [9] D. Chakraborty and A. Joshi, "Dynamic Service Composition: State-approach enables easy mash-ups as any entity operating on of-the-Art and Research Directions,"University Of Maryland, Baltimore, December 19 2001.the information will not affect the producers at all. In this [10] R. J. Hall, "Open Modeling in Multi-stakeholder Distributedway the interoperability can be achieved without definition Systems" presented at Proc. First Workshop on the State of the Art inof operational semantics. However, it also sidesteps the Automated Software Engineering, 2003.problem we discussed in this paper, and we think that the [11] Suoranta, R.: Modular service-oriented platform architecture - a key enabler to soc design quality. In: 7th International Symposium ondynamic environment where state needs to be updated on Quality of Electronic Design (ISQED 2006) (March 27-29, 2006),the SIB requires a mechanism which is similar to ours. San Jose, CA, USA, pp. 11–13. IEEE Computer Society, Los We also hope that the relative simplicity of the Activity Alamitos (2006)mechanism would make it easy to modify existing service- [12] Gelernter, David. "Generative communication in Linda". ACM Transactions on Programming Languages and Systems, volume 7,level entities to participate. Also the limited amount of number 1, January 1985requirements may be easier to fulfill, even when the [13] Ian Oliver and Jukka Honkola. Personal Semantic Web Through Abehavior of the service level is not fully understood. This is Space Based Computing Environment. In Second IEEE Interntionala situation which typically arises when using an existing Conference on Semantic Computing, August 4, 2008, Santa Clara, CA, USA. IEEE, 2008piece of software as a component of a smart space. [14] OpenMAX APIs. Referenced Finally, as the activity is an abstract entity, it can exist on March 4th 2010its own, independently of the underlying resources. This [15] L. Brownsword, et al, "Current Perspectives onallows the resources to be reassigned based on need. Interoperability",Software Engineering Institute, Carnegie Mellon University, Pittsburgh, PA, 2004.