0.4pt0pt
NRC-TR-2009-M3-1
M3 Information SmartSpaces
Technology Overview
Editors: Ian Oliver
Nokia Research Center Helsink...
2
NRC-TR-2009-M3-1 Copyright c 2009 Nokia
Contents
1 Introduction 7
2 Information Spaces As A Basis for Personalising The Semantic Web 9
2.1 Introduction . . . . . ...
4
4.2 The M3 Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
4.3 The Case Study . . . . . . . . ....
5
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77
8.2 Background . . . . . . . . ....
6
NRC-TR-2009-M3-1 Copyright c 2009 Nokia
Chapter 1
Introduction
This book contains a cross-section of work made during the M3-Sedvice SmartSpaces
project. Each ind...
8
NRC-TR-2009-M3-1 Copyright c 2009 Nokia
Chapter 2
Information Spaces As A Basis for
Personalising The Semantic Web
Ian Oliver
The future of the Semantic Web lies ...
10
We elaborate on the ideas of ubiquitous information, the role of reasoning and knowledge,
the location of the informati...
11
Personalisation is achieved through explicitly demarking a space in which information is
stored and agents have access....
12
These individual corpii of information may be overlapping in terms of their content. Alice can
interact simultaneously ...
13
Information Spaces As A Basis for Personalising The Semantic Web The second form of
interaction is a variation of this:...
14
Graph provenance and the related semantics problem are the major problems in any
Semantic Web/interoperability system a...
15
Finally the problem of uncertain, incomplete, inconsistent and undefined information re-
quires a much more formal appro...
16
interfaces through the space - this has been successfully used in coordinating media streaming
and storage devices via ...
Chapter 3
Context Gathering in Meetings:
Business Processes Meet Agents and
The Semantic Web
Ian Oliver, Seppo T¨orma, Esk...
18
3.1 Introduction
In practice, the complex cognitive tasks faced in organizations – including coordination, problem
solv...
19
figure 3.1.
Space "A" Space "B"
Individual Brokers +
Information Stores
connections to spaces
Agents +
Control Flow
Mech...
20
visually (or by some other means, eg: audio) so that their combined results may be perceived
as a single whole.
3.3 Sim...
21
Figure 3.2: Meeting Ontology Class Hierarchy
Figure 3.3: Devices and Presentation Ontology Class Hierarchy
location: Me...
22
ontologies can be reasonably simple and does not require the power of OWL5
reasoner.
Furthermore, most current business...
23
Figure 3.4: RDF graph of a meeting
bility (from simple atomic actions to implementing complete functionality) which can...
24
any information. For example, one could link some IRC conversation with an agenda
point and use that information later ...
25
Γ, x type TKK :: Meeting
Γ, x type Nokia :: Meeting
type inference2
(3.2)
A more sophisticated variation on this is to ...
26
The crucial aspect in our approach is the flexible and easy interleaving of meeting
conversation with visual presentatio...
27
page on a window in each laptop. Using PresentationControllerAgent, Alice moves forward
in the presentation. The AVDevi...
28
tions into the space. Eve has also set up her web browser agent to link these presentations
directly into the current m...
29
way to time stamp events. This can easily be added to the agents.
The software for MeetingMintesPresentationAgent and t...
30
NRC-TR-2009-M3-1 Copyright c 2009 Nokia
Chapter 4
Cross-Domain Interoperability: A Case
Study
Jukka Honkola, Hannu Laine, Ronald Brown, Ian Oliver
We describe a c...
32
Technologies such as the Semantic Web [10] provide enablers for solutions to these prob-
lems. Within the Semantic Web ...
33
SIB
SIB
SIB
Agent
Agent
Agent
Agent
M3 Space
Figure 4.1: A diagram of the general system architecture: Agents, M3 space...
34
the agents see best. Strict ontology enforcement would make this kind of activity extremely
difficult as all new ways of ...
35
internet tablets. The M3 space infrastructure runs on a Linux laptop.
4.3.1 Application Development Vision
The case stu...
36
m3:phonelinestatus
OperatorAccount
m3:user
m3:playerm3:mobileaccount
m3:username
m3:totaldistance
m3:totaltime
m3:stopt...
37
4.3.3 Agents
Next we describe the operation of agents making up the case study. It should be noted that
the agents are ...
38
Telephony status observer
The telephony voice line status is part of a OperatorAccount class. Operator accounts are bou...
39
BEAT” according to a threshold configuration on the number of lives. When the game ends,
audio rendering is stopped.
The...
40
this case the mood renderer represents such resource (audio player), and implicitly it is usually
the game that uses it...
Chapter 5
Agent/Space-Based Computing and
Radio Frequency Memory Tag
Interaction
Joni Jantunen, Sergey Boldyrev, Ian Olive...
42
Mobile devices, such as mobile telephones, have access to ubiquitous connectivity solutions, eg:
WLAN, GSM, UMTS networ...
43
To address this issue we have implemented a system based around larger, more user or
personal focussed structured which...
44
trends especially on the field of non-volatile memory technologies according to which the power
consumption, physical si...
45
may be necessary to have sufficient processing capabilities in the tag itself. If fully active tags are
excluded from the...
46
data/information within the RF memory tags, it also becomes possible to revoke the system
once it would be demanded by ...
47
The first two categories are the ‘classic’ ways of passively interacting with a tag and constitute
simple information re...
48
5.3.2 Agent Interactions
The “simple” read case is useful for applications where the tags provide relatively static loc...
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
M3 Information SmartSpaces Technology Overview
Upcoming SlideShare
Loading in …5
×

M3 Information SmartSpaces Technology Overview

1,228 views
1,158 views

Published on

This book contains a cross-section of work made during the M3-Sedvice SmartSpaces project. Each individual paper stands as an independent text which when taken together describe the ideas, concepts and historical aspects - the subtle change and clarification
of ideas - of the work as a whole.

The papers chosen present various aspects ranging from the positional works through to examples, solutions and the academic and scentific basis underpinning our notion of SmartSpaces.

The following persons have been directly involved either as collaborators or project team members in this work. The project members are: Ian Oliver, Sergey Boldyrev, Sergey Balandin, Samuli Silanto, Vesa Luukkala, Olli Tyrkko, Jurgen Ziegler, Jukka Honkola, Antti Lappetelainen, Pekka Luoma, Hannu Laine, Ronald Brown. Nokia persons who have contributed: Ora Lassila, Deepali Krushraj, Timo Erikson, Risto Suoranta, Petri Liuha, Jarmo Arponen, Joni Jantunen. Externally to Nokia: Seppo Torma, Esko Nuutila, Ilkka Niemela, Heikki Saikkonen from The University of Helsinki, Daniel Jackson and Felix Change from MIT, Boston, Alessanra Tonninieli from University of Bologna.

This work has been partially supported by TEKES through the ICT SHOK DIEM (www.diem.fi) project and the EU Artemis Sofia project. and the FP6 Minami project.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,228
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

M3 Information SmartSpaces Technology Overview

  1. 1. 0.4pt0pt NRC-TR-2009-M3-1 M3 Information SmartSpaces Technology Overview Editors: Ian Oliver Nokia Research Center Helsinki, Finland http://research.nokia.com June, 2009entries Abstract: Index Terms: end-user programming Smart Spaces, M3, Sedvice Semantic Web, RDF, OWL Service Composition, Service Interoperability Triple-Space and Cloud-Computing, Information and Data Clouds Agents, Distributed and Mobile Computing NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  2. 2. 2 NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  3. 3. Contents 1 Introduction 7 2 Information Spaces As A Basis for Personalising The Semantic Web 9 2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 2.2 Personalisation and Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . 10 2.3 Interaction and Sharing with Spaces . . . . . . . . . . . . . . . . . . . . . 11 2.4 Information and Semantic Issues . . . . . . . . . . . . . . . . . . . . . . . 13 2.5 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Context Gathering in Meetings: Business Processes Meet Agents and The Semantic Web 17 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2 Space-Based Computing Framework . . . . . . . . . . . . . . . . . . . . . 18 3.3 Simple Contextual Model of a Meeting . . . . . . . . . . . . . . . . . . . 20 3.4 Content-Driven Meetings Scenario . . . . . . . . . . . . . . . . . . . . . . 25 3.5 Discussion, Conclusions and Future Work . . . . . . . . . . . . . . . . . . 29 4 Cross-Domain Interoperability: A Case Study 31 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3
  4. 4. 4 4.2 The M3 Concept . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 4.3 The Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 4.4 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 5 Agent/Space-Based Computing and Radio Frequency Memory Tag In- teraction 41 5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 5.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 5.3 Interaction with Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 5.4 Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 5.5 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . 53 6 A Mechanism for Managing and Distributing Information and Queries in a Smart Space Environment 55 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 6.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 6.3 Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6.4 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 6.5 Discussion and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 63 7 Network and Content Aware Information Management 65 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 7.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 7.3 Description . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 7.4 Discussion and Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . 75 8 Operations on Spaces of Information 77 NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  5. 5. 5 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 8.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 8.3 Basic Operations on Spaces . . . . . . . . . . . . . . . . . . . . . . . . . 79 8.4 Histories and Extended Queries . . . . . . . . . . . . . . . . . . . . . . . 82 8.5 Example Usages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 8.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 9 Design and Validation of a Distributed Computation Environment for Mobile Devices 91 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 9.2 Smart Spaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 9.3 Simulation and Validation of the Domain Model . . . . . . . . . . . . . . 94 9.4 Demarcation Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 9.5 Integration of Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 9.6 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 9.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . . . . 103 Bibliography 105 NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  6. 6. 6 NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  7. 7. Chapter 1 Introduction This book contains a cross-section of work made during the M3-Sedvice SmartSpaces project. Each individual paper stands as an independent text which when taken together describe the ideas, concepts and historical aspects - the subtle change and clarification of ideas - of the work as a whole. The papers chosen present various aspects ranging from the positional works through to examples, solutions and the academic and scentific basis underpinning our notion of SmartSpaces. The following persons have been directly involved either as collaborators or project team members in this work. The project members are: Ian Oliver, Sergey Boldyrev, Sergey Balandin, Samuli Silanto, Vesa Luukkala, Olli Tyrkko, Jurgen Ziegler, Jukka Honkola, Antti Lappetelainen, Pekka Luoma, Hannu Laine, Ronald Brown. Nokia per- sons who have contributed: Ora Lassila, Deepali Krushraj, Timo Erikson, Risto Suoranta, Petri Liuha, Jarmo Arponen, Joni Jantunen. Externally to Nokia: Seppo Torma, Esko Nuutila, Ilkka Niemela, Heikki Saikkonen from The University of Helsinki, Daniel Jackson and Felix Change from MIT, Boston, Alessanra Tonninieli from University of Bologna. This work has been partially supported by TEKES through the ICT SHOK DIEM (www.diem.fi) project and the EU Artemis Sofia project. and the FP6 Minami project. 7
  8. 8. 8 NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  9. 9. Chapter 2 Information Spaces As A Basis for Personalising The Semantic Web Ian Oliver The future of the Semantic Web lies not in the ubiquity, addressability and global sharing of information but rather in localised, information spaces and their interactions. These information spaces will be made at a much more personal level and not necessarily adhere to globally agreed semantics and structures but rely more upon ad hoc and evolving semantic structures. Originally published: Interational Conference on Enterprise Infomration Systems (ICEIS’09) 2.1 Introduction In this paper we position our vision of the continuation of the development or evolution of the Semantic Web [10]. This is best visualised as the Giant Global Graph concept popularised by Tim Berners-Lee1. Most information however is not ubiquitous but personalised, hidden, private and inter- preted locally - this information tends to be the personal, highly dynamic information that one stores about oneself: contact lists, friends, media files, ‘my’ current context, ‘my’ family, ‘my’ home etc and the interweaving and linking between these entities through ad hoc personal structures. 1 http : //en.wikipedia.org/wiki/Giant Global Graph 9
  10. 10. 10 We elaborate on the ideas of ubiquitous information, the role of reasoning and knowledge, the location of the information with relation to its ubiquity through the concept of projections from the Giant Global Graph called spaces. We then describe an implementation of an envi- ronment supporting these ideas in a mobile and personal context as well as many of the issues that this directly brings up with regards to what are semantics and how information is going to be dealt with in this context. In the following sections we outline our position and areas of research relating to notions of personalisation, Semantic Web, information and its meaing and semantics as well as our implementation. 2.2 Personalisation and Spaces The Semantic Web is succeeding in relatively small-scale, specific situations which are restricted to a given domain. If we expand the notion of a domain in a more orthogonal sense to encompass personal level then this suggests that we have a notion of a ‘Personal Semantic Web’ in which one can organise their own information according to these principles. The advantages of a Semantic Web based approach is that certain structures, schemata and semantics can be fixed enabling some - and this is an important point, we should not (and can not?) try to attempt everything - meaningful communication, reasoning and interoperability to take place. Mobile devices with various methods of connectivity which now constitute for many as being the primary gateway to the internet and also being a major storage point for much personal information [48, 61]. This is in addition to the normal range of personal computers and furthermore sensor devices plus ‘internet’ based providers. Combining these devices together and lately the applications and the information stored by those applications is a major challenge of interoperability [100, 104]. This is achieved through numerous, individual and personal spaces in which persons, groups of persons etc can place, share, interact and manipulate webs of information [57, 55] with their own locally agreed semantics without necessarily conforming to an unobtainable, global whole. These spaces are projections of the ‘Giant Global Graph’ in which one can apply semantics and reasoning at a local level. A detailed survey of such space-based systems is given in [78]. This approach we feel addresses at least two of the counter-arguments against the Semantic Web vision: feasibility and privacy by directly addressing notions of locality or ubiquity and ownership. Feasibility because we are changing the problem to address much smaller-scale structures through setting clear boundaries in terms of computation. In order to apply reasoning and other manipulations (such as sharing) of that information we are required to construct processes which have access to that information - typically these are known as agents in the ‘traditional’ sense of the word although we tend towards the classification given in [40]. Agents are either personal in that they perform tasks either directly decided upon by the user or autonomously for or on behalf of the user, monitor particular situations or reason/data-mine the existing information. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  11. 11. 11 Personalisation is achieved through explicitly demarking a space in which information is stored and agents have access. Within each space information is organised according to the owner (or owners) of that space. For an agent to obtain entry to that space then it is made on the terms of that space. Similarly for two spaces to interact directly similar contracts must also hold. Interactions with spaces is described in section 2.3. The kinds of information that are stored in a personal space vary but initially contacts lists, media files (links to media), personal information management data (calendars etc), email and other personal communication etc. This easily expands to information feeds such as those provided through RSS and even WWW interfaces, family or community information, social networking and so on. These kinds of information can be then further augumented by tagging, internal links and more sophisticated equivalence relationships such as might be seen between social networking contacts, contacts lists, calendars etc. In addition more static and thus more externalisable information can be stored or referenced in the same manner - such information might be census records, telephone directories or even cultural information [47]. Of course there are issues regarding the interpretation of information and how the meaning or semantics is preserved across spaces and agents; this also includes deciding whether two independent structures actually represent the same piece of information and can be merged or coalesced. Furthermore issues regarding trust and security need to be addressed - we do not specifically discuss this problem in this paper. 2.3 Interaction and Sharing with Spaces Consider the following scenarios: in figure 2.1 Alice interacts with her personal space - through ‘agents’2 running on a multitude of devices. This space contains a corpus of information A which through local reasoning and deductive closure algorithms - a feature of our spaces - provides her with the corpus R(A). Information is represented using Semantic Web standards, ie: RDF, RDFS, OWL, FOAF etc, rule sets in RuleML etc. Alice A R(A) Alice’s "agents" executing on interacting with uses ...and others... Figure 2.1: Alice’s Agents, Devices, Spaces and Information In this case the boundary of R(A) is the limit of Alice’s personal space. If Alice has two spaces or corpii of information she might bind these together to produce a much larger space. 2 Agent is rather a loaded word, but alternatives aren’t numerous: executives, nodes, UIs, programs, etc NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  12. 12. 12 These individual corpii of information may be overlapping in terms of their content. Alice can interact simultaneously with many discrete spaces. These situations are visualised in figure 2.2; for simplicity we only show the spaces and closures. Here the total information available for a given space is the union of the deductive closure over all the individual corpii. A B C Alice D R(C) R(D) R(A u B) Figure 2.2: Alice and Alice’s Spaces Alice can decide to break and reconfigure her current spaces into many smaller spaces. This may be made in any manner including removing all information and creating multiple individual smaller or even empty spaces to making a complete copies of the current space. We now introduce Bob, who has a space of his own that is constructed from a single corpus of information. At this point we can say nothing about the relationship between the contents of the corpii A and B with C; they may potentially all contain the same information. Interaction between Alice and Bob can be made in three different forms. The first is simple in that Bob only needs to give Alice’s ‘agents’ access to his space as visualised in figure 2.3. Bob of course has his own ideas about privacy and grants Alice access to only a portion of his space. Alice has direct access to a subset of Bob’s space - if she has write access then potentially this could have an effect on the space as a whole. Privacy is asymmetric - it is on the sharer’s terms only thus precluding the need for a globally agreed privacy mechanism. If Alice just so happens to be able to satisfy the criteria for accessing Bob’s space then Alice is granted access at whatever level Bob’s privacy mechanisms allow. Alice’s own privacy mechanisms do not affect Bob’s mechanisms and vice versa. Alice A B Bob C Figure 2.3: Alice Interaction with Bob’s Space NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  13. 13. 13 Information Spaces As A Basis for Personalising The Semantic Web The second form of interaction is a variation of this: Bob to partitions off a given subspace to which Alice has access, then any changes can be kept local and the merge back controlled by Bob explicitly. In both of these cases access policy including trust mechanisms are local to Bob - there is no need for Alice to know about what mechanisms are in place. The fist two forms we expect to be the most common methods of interaction, the third offers a set of different possibilities based around the merging of the spaces. This is more complicated as Alice and Bob must both agree to the merge (fig. 2.4) both in terms of personally agreeing through their trust mechanisms (might just be personal trust) but also through the shared semantics of the information. Alice A B C Bob Figure 2.4: Alice’s and Bob’s (Merged) Space This then constitutes how spaces are related but has not yet addressed certain specific ideas about information and the semantics of that information and these are discussed in section 2.4. 2.4 Information and Semantic Issues We can classify the issues with this approach as: • Non-monotonicity of Deductive Closure and Rules • Graph Provenance • Semantics of the Information • Uncertainty, Incompleteness, Inconsistency and Undefinedness of Information Given a single space s, the information contained in that space is i(s), the rules ρ(s) and the deductive closure calculated as R(i(s), ρ(s)). A merge of two spaces s1 and s2 results in a single space sm where sm is calculated as R(i(s1) ∪ i(s2), δ(ρ(s1), ρ(s2)). The function δ determines the set of rules to apply and is constructed from a mechanism which prioritises the rules somehow - the exact mechanism would of course vary but we envisage would be similar to those found in certain kinds of non-monotonic calculi, eg. [76]. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  14. 14. 14 Graph provenance and the related semantics problem are the major problems in any Semantic Web/interoperability system as there, beyond pre-agreed addressing and strict, stan- dardised ontologies, no definitive method for relating structures representing the same informa- tion together exists. While information about typing etc is carried within the space, deeper semantics is not. Identification of larger meaningful structures such as RDF molecules [26] provide at this time the strongest basis for provenance analysis and addressing deeper semantically meaningful struc- tures. In the most part we must rely upon local convention between any two merging or interacting spaces with the hope that this leads to a more global convention [3, 97]. The meaning of the information to Alice might be very different to the meaning ascribed by Bob (see also provenance above). The meaning or semantics of the information can only be made by the reader [17]; the writer of the information only gives hints through typing and tagging and other relationships to what the intended meaning might be. This hints to the question how do we guarantee that two agents understand the informa- tion in the same way, our response to this is that we don’t care; at least to the point where there is no internal mechanism for this [56]. As it is, the meaning of some ontological structure evolves over time and there appears to be no reasonable mechanism for communicating deeper level semantic structures. For true, reliable communication how many semantic levels are re- quired is unknown. However, since that if access to a space is granted or a merge made then this is implicit agreement between the parties and particularly the readers over the intended semantics of the information. In [35] is provided a detailed discussion of the style of intentional semantics [56] we propose here that it is not justing typing but the whole construct of properties of an object - and then the scope over which we define an object - that must be taken into consideration when deciding how an agent interprets a given structure. This also applies when dealing with the aforementioned graph provenance issue. To complicate matters further, the notion of semantics embodied within these kinds of information structures is little more than meta-data whereas one really needs to describe a further relationship into the ‘real-world’ [98]. Solutions based upon lexical and semantic anal- ysis through resources such as Wordnet[29] appear to be the most promising here with regard to issues surrounding semantics similarity [50, 94]. However, despite this we can (an no cur- rent computer system) never be sure that any two agents actually act upon their respective interpretations information in the same way. Currently we are seeing two mechanisms for semantic agreement: the first is through standardisation of ontologies (W3C) and the second is through folksonomic evolution of initially personal and informal structures into ad hoc ontologies which become more concretised as social agreements form. Even in the strict, formal ontology development scenario evolution of the ontology takes place as usage changes and develops [95], however this is much slower than the folksonomic cases. A fairly common mechanism for agreeing on semantics is the upper ontology approach, but again this suffers from the problem of many upper ontologies (for example: [46])- an interesting discussion of this is made in [106] NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  15. 15. 15 Finally the problem of uncertain, incomplete, inconsistent and undefined information re- quires a much more formal approach within the various agent and reasoning structures [64]. Within the Semantic Web we must further explore notions of undefinedness, modality, proba- bility and non-monotonicity. This is left for future work though mechanisms are already present within our implementation for attaching and modality properties. 2.5 Implementation Supporting these ideas we already have the computing and networking infrastructure. Our par- ticular solution builds a space-based computing framework [83, 86] based upon the Piglet/Wilbur [63] RDF++ engine [62]. The notion of space being constructed out of a number of individual, linked (totally routable) brokers. Interaction with the space is via a agents which may reside on any suitable device with suitable connectivity and computing capabilities; similarly the brokers. Figure 2.5 shows one possible configuration. Space "A" Space "B" Individual Brokers + Information Stores connections to spaces Agents + Control Flow Mechanisms explicitPotential Figure 2.5: An Example Implementation Configuration Agents may connect to one or more spaces at a time and to which spaces may vary over the lifetime of an agent. Mobility in this sense is provided in a ‘pi-calculus’ manner [73] in that links may move rather than the physical running process. Agents can save state and become ‘mobile’ when another agent restores that state. We have not addressed code mobility in the current implementation and this remains low priority at this moment. Agents themselves are anonymous and independent of each other - there is no explicit control flow other than that provided through preconditions to agent actions. A coordination model based around expressing coordination structures as first-order entities is being investi- gated, however we are more focussed on collecting and reasoning over context. Control flow can be made outside of the space through agents explicitly sharing details of their external NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  16. 16. 16 interfaces through the space - this has been successfully used in coordinating media streaming and storage devices via UPnP3 and NoTA4 for example. The brokers each contain a corpus of information and when linked together to form a space distribute the information in an asymmetric manner - some information is not replicated because of computational resources, connectivity, storage reasons and even legal issues such as copyright. This distribution layer is also responsible for the efficient distributed computation of queries and marshalling the resultant responses such that calculating the deductive closure can be made. In this manner we can distribute complex computation away from the original agent over the whole space. This distribution can be used in a number of ways and one of the most interesting for us is giving the user temporary access to a larger body of information which can be used to selectively enhance their current corpus of information, for example, a body of linking information based around the owl:sameAs construct [91]. Other examples include personal access to more dynamic corpii such as news, weather or similar services. We make no attempt to enforce consistency of the information rather letting the writer of the information have freedom to express what they want and leave the interpretation to the reader. The semantics of the information is merely intensional in that the writer provides ‘clues’ through typing, tagging and other means. Repair of information according to schemata or other criterion can be enforced within the space and for some kinds of information would even be desirable. If consistency of particular structures is required then this can be achieved through agent implementation and specific belief revision models. 2.6 Conclusion In summary we believe that the Semantic Web will move from being a global information corpus to multiple, individual, linked and personal corpii. Semantics, reasoning and processing about the information will be localised and personalised within these corpii. As corpii are shared, linked, merged and split, certain schemata will coalesce and evolute into fixed or standard structures in an evolutionary form. There are still questions regarding what precisely is semantics and how this is preserved across information structures but techniques do exist for reasoning about this and the related graph/information provenance problems and these are currently being implemented and trialled within the framework we have described. 3 Universal Plug and Play http://upnp.org 4 Network on Terminal Architecture http://www.notaworld.org/ NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  17. 17. Chapter 3 Context Gathering in Meetings: Business Processes Meet Agents and The Semantic Web Ian Oliver, Seppo T¨orma, Esko Nuutila1 Semantic Web, Space Based Computing and agent technologies provide for a more sophisticated and personalised context gathering and information sharing experience. Environments which benefit from these properties are often found in direct social situations such as meetings. In the business world such meetings are often artificially structured according to a process which does not necessarily fit the physical (and virtual) environment in which a meeting takes place. By providing simple ontologies to structure information, agents and space-based information shar- ing environment with reasoning capabilities we can provide for a much richer social interaction environment. Originally published: The 4th International Workshop on Technologies for Context-Aware Business Process Manage- ment (TCoB 2009) 1 Seppo T¨orma, Esko Nuutila - Helsinki University of Technology, Finland 17
  18. 18. 18 3.1 Introduction In practice, the complex cognitive tasks faced in organizations – including coordination, problem solving, design, planning, ideation, and decision making – need to be tackled in meetings where different perspectives and disciplines can be combined and shared understanding can be built [22, 72]. In these kind of content-centered meetings, the formal structure - or predefined process - is much less important than the wealth of information contained in a meeting. The content and development of the information has a significant impact on the course of the meeting, while the official agenda often provides only superficial names to the phases of the conversation. A typical meeting both uses and produces information. The participants need to share existing information with each other, often in an ad hoc manner and interact with the meeting in varying ways as situations arise. This includes varying amounts and configurations of typical computing and display equipment such as mobile phones, displays, personal computers etc. Increasingly the ‘conversation’ of the meeting is also supplemented with slide presenta- tions, documents, diagrams, and other visual material as well as references to material not explicitly included in the meeting. During the meeting new information is usually produced, even if it not always explicitly written down. It should be noted that a large part of the in- formation is related to the context of a meeting itself, its participants, as well as the materials presented. Much of this information could nowadays be automatically gathered for later use. There is a lot of diversity in the actual arrangements of meetings. At one extreme are well- organized meetings kept in special meeting rooms. At the other extreme are ad hoc gatherings to tackle urgent problem that can take place in office rooms or even cafeterias. Regardless of the type, it is nowadays typical to have lots of available displays. There can be shared displays such as video projectors or large screens as well as personal displays such as laptop computers or PDAs. In this paper we will focus specifically on the problem of sharing (media-based) informa- tion in a easy and flexible manner during a meeting. We are especially interested in increasing the interaction among participants by making the granularity of presentations smaller and by allowing simultaneous presentation of information. We will present a case study of this func- tionality. Our solution builds on the Semantic Web and context gathering and reasoning techniques [10, 55, 7] that enable rich information to be stored, linked and reasoned about from a variety of sources. In this paper we show how the interaction between a few simple ontologies and users of those ontologies frees a meeting scenario from the confines of a strict process enabling ad hoc linking and reasoning about the information being gathered in that context. 3.2 Space-Based Computing Framework The Sedvice architecture [83, 86] supports distributed agent based interactions over a declarative space-based infrastructure [78]. A diagrammatic representation of the architecture is shown in NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  19. 19. 19 figure 3.1. Space "A" Space "B" Individual Brokers + Information Stores connections to spaces Agents + Control Flow Mechanisms explicitPotential Figure 3.1: An Example Implementation Configuration Spaces are virtual entities which contain the information; these are represented by a distributed, totally routable set of information brokers (known as SIBs). A Space at its sim- plest contains: listeners for one or more transport/connectivity protocols (eg: HTTP, TCP/IP, UMTS, SOAP etc), the logic for processing messages, policy/security and handling subscrip- tions, a deductive closure mechanism and an information store (an RDF store in our case). Computation is placed primarily in agents [40] which interact with the space (physically via individual SIBs). The responsibilities of agents range from user-interaction to reasoning and performing tasks such as truth maintenance, belief revision, information consistency man- agement etc. Agents are typically anonymous and autonomous from each other. Control flow is either provided through first-class mechanisms inside the spaces themselves or via agents communicating externally through specific control-based environments such as the Network on Terminal Architecture2 or Universal Plug and Play3. Agents are necessarily anonymous and independent of each other unless the agent specifically shares information about itself through the space. Information is stored as a graph using the Resource Description Framework (RDF) [62]. The semantics of the information is implied by the writer by way of what typing information is provided, the amount of information and what ontologies are used. The reader of any piece of information interprets the information according to its own criteria which is influenced by the typing, tagging and ontologies. Information is semi-structured deliberately allowing inconsis- tency and ad hoc expansion of the information schemata. What the user perceives as being an application [93] is constructed from a number of independently operating agents which — if they have user-interfaces — may be grouped together 2 www.notaworld.org 3 upnp.org NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  20. 20. 20 visually (or by some other means, eg: audio) so that their combined results may be perceived as a single whole. 3.3 Simple Contextual Model of a Meeting We present a simple contextual model of a meeting, which considers two major aspects [89]. The first aspect captures the notion of meeting as an assembly of group of people — the participants — gathering together at a specific time and place to handle a specific topic using a set of documents. The second aspect captures the dynamic notion of a meeting as set of acts that are done in a meeting using equipment. This model can easily be extended to capture other aspects. For an implementation of this contextual model we create a number of individual agents that interact with small subsets of the ontologies defined below. Within a meeting there exist an agenda, notes on the meeting, documents or presentation to be made in that meeting and the participants of that meeting etc. Outside of this we have notions of physical devices that might exist in the location of a meeting (though we don’t enforce physical presence). We also show how these aspects, that of the meeting and its presentations and that of physical devices can be combined. 3.3.1 Ontologies The basic information defining a meeting itself can be fairly simple: a set of participants, documents and an agenda. Within a meeting however the actual information structure can be extremely complex involving individual comments, ad hoc changes to the agenda, participants etc and then further links not only within the meeting information but outside of that to provide additional context and content to the meeting. For example, the participants of a meeting in many tools are just names; using the Semantic Web we can extend this out to business contacts (eg: LinkedIn, Facebook etc4) and the possibilities that arise from including that information [104]. Our basic design consists of two separate ontologies; we do not espouse the ‘one true ontology’ view and it is highly likely that differing formats for various aspects of the whole system will be used and thus need to interoperate [44]. In Figures 3.2 and 3.3 we present the classes that are used in these ontologies. We differentiate between the ontologies when necessary in this text by prefixing with shortened namespace declarations: Nokia and TKK respectively. We also define a set of properties between these classes, for example in the Nokia meeting ontology we have the following properties 4 http://www.linkedin.com/ http://www.facebook.com NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  21. 21. 21 Figure 3.2: Meeting Ontology Class Hierarchy Figure 3.3: Devices and Presentation Ontology Class Hierarchy location: Meeting → Location functional property meetingHost: Meeting → Participant subrole of participants, cardinality 1 participants: Meeting → Participant cardinality: 0..∞ topic: Meeting → Topic cardinality: 1..∞ belongsTo: Presentation → Meeting functional property agenda: Meeting → AgendaItem cardinality: 1..∞ responsibleOf: AgendaItem → Participant functional property presentations: Meeting → Presentation cardinality: 0..∞ participatesIn: Controller → Meeting functional property controllers: Meeting → Controller cardinality: 0..∞ shows: AVDeviceController → Presentation cardinality: 0..∞ controls: PresentationController → Presentation cardinality: 0..∞ document: Presentation → Document functional property Data properties include the start and end times of the meetings, agenda item de- scriptions and the name of the participants: these are all of type ‘date/time’ and ‘string’ accordingly. Similar properties for the presentation and device ontology can be see in the example RDF graph in figure 3.4. The expressiveness required of the description logic for these ontologies is relatively simple, in the meeting ontology case it is ALCHQ(D) and in the device/presentation case it is ALCF(D). This means that any reasoning that is requires to be made over these NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  22. 22. 22 ontologies can be reasonably simple and does not require the power of OWL5 reasoner. Furthermore, most current business information models have been specified using entity- relationship models which are easily mapped to simple description logics of much less complexity than OWL [8]. Note that the construction of the ontologies are very much a compromise between the parties creating those ontologies. There are properties and even classes which do not necessarily accurately reflect ‘real-life’. Ontologies will change over time and the aforementioned semi-structured/ontology agnostic nature of RDF assists in representing this. Some classes represent rather technical matters. Class AVDevice represents a thing that is able to present audiovisual content, e.g., a video projector, a display of a laptop, a loudspeaker, a window on a computer desktop, etc. Class AVDeviceController rep- resents a software component that can render output to an AVDevice. Class SKIM is an example subclass of AVDeviceController. It represents a software component that uses the SKIM viewer [1] to show a PDF document on a display. Class Controller is just an abstraction of the controller components. The class MultiPagePresentation is an important subclass of Presentation and it represents a presentation consisting of several pages, e.g., a presentation of a PDF document. In Figure 3.4 we present an RDF graph of an instance of these ontologies. The un- derlined x’s are URI’s that represent individual objects: participants, presentations, con- trollers, etc. For each URI we present its type, i.e., the class the instance of which the URI is. In addition we present several relations between objects, e.g., the relation shows be- tween and the instance x7 of class SKIM and instance x5 of class MultiPagePresentation, which tells that a presentation is shown by a software component on a specific audiovisual device. The concepts of a ‘meeting’ and a ‘document’ occurs in both ontologies which suggests some kind of semantic similarity. As we shall see we can take advantage of these kinds of semantic similarity to enable interoperability between the various ontologies [16, 69]. The information models are linked together by the classes Meeting and Document. We can see the separation of the ontologies: the left-hand side presents the devices- and-presentations aspect and the right-hand side the participants-topic-location-and-time aspect of a meeting. 3.3.2 Agents and Interaction To effect a meeting we require actions over the defined ontologies. These actions are implemented in our system through simple agents [40] with varying degrees of responsi- 5 http://www.w3.org/TR/owl-features/ NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  23. 23. 23 Figure 3.4: RDF graph of a meeting bility (from simple atomic actions to implementing complete functionality) which can be distributed across many devices. Consider the case where a meeting is being set up, this involves the creation of an instance of class Meeting and to ensure consistency, one participant which initially must be the meeting host, the topic and at least one agenda item. We defined the data properties and other object properties to be optional, though one should also specify the location, start and end times etc. Whether these properties should be optional (and others not) is a matter for the ontology designer and what particular aspects they are trying to capture. To implement this, we need a software agent, say MeetingBuilderAgent, that is able to create the meeting in the Sedvice RDF store, add agenda items and participants of the meeting to the RDF store etc. This software agent communicates with Sedvice and has a user interface, e.g., a set of web pages, some widget-style application etc. As the information store contains only the current context there is no necessity for presentations, participants etc to be bound to the meeting when it is created. In fact as we are only manipulating context we have a divorce between the temporal aspect of when entities have been created and linked to each other. This allows us to be free from any given specific structure and process associated with the setting up of a meeting. As a meeting progresses the interactions between the agents with the body of shared information or the context of the meeting more accurately reflects how this particular interaction between users progresses. Information ranging from simple RDF triples to larger bodies of information can be linked to the entities defined in our ontologies as necessary. Additionally as we are ‘free’ from specific, rigid structures we can in effect link NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  24. 24. 24 any information. For example, one could link some IRC conversation with an agenda point and use that information later when reasoning and processing about the overall context of a meeting such as effected by automatic meeting minutes generation and/or summarisation. The agents controlling the presentation of the material work thus: we need at least two software agents, say PresentationControllerAgent and SKIMAgent. The former is used by the participants of the meeting for controlling presentations: starting presentations, moving forward and backward in the documents that are presented, etc. It communicates with the Sedvice RDF store adding there RDF triples representing the presentations, their corresponding documents, the current pages in a presentation etc. It also has a user interface, e.g. a set of web pages to be used by a meeting participant for controlling the presentation. Several PresentationControllerAgent instances may exist simultaneously, if several participants control presentations at the same time. The SKIMAgent component listens to modifications in the RDF store. Whenever a new presentation is to be shown by the SKIM agent, it awakes the local SKIM application and tells it to show the corresponding document6 . Similarly, if a change of the current page appears in the RDF store, the SKIM agent tells the local SKIM application to change the page. If several displays are available, we should have several SKIM agents running. 3.3.3 Reasoning and Information Integration Given a corpii of information such as that in figure 3.4 we can construct agents that can analyse and reason about this information. In particular we require agents that can: 1. relate two elements together 2. relate two (or more) information structures together 3. search for and generate additional information based on current context The first case is relatively trivial and can be realised using agents that look for in- stances of certain types and link these together. The simplest version of this is an agent that simply notices an instance of one type and asserts that it also takes symmetrically a second type: Γ, x type Nokia :: Meeting Γ, x type TKK :: Meeting type inference1 (3.1) 6 We chose SKIM as the PDF viewer, since it can easily be commanded outside using AppleScript NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  25. 25. 25 Γ, x type TKK :: Meeting Γ, x type Nokia :: Meeting type inference2 (3.2) A more sophisticated variation on this is to utilise the owl:sameAs predicate [91] but this relies upon reasoning over a larger set of information instead of indiscriminate linking of instances (nodes in the RDF graph) together. For example, one could imple- ment an agent with the following rule for some properties p1 and p2: Γ, x type Nokia :: Meeting ∧ y type TKK :: Meeting ∧ x.p1 = y.p2 Γ, x owl : sameAs y (3.3) The above are of course monotonic rules though in our experience rule sets within agents and typical information corpii tend to be non-monotonic in nature. This then leave various problems regarding rule conflict and aspects of negation and missing information - open world vs closed world assumptions. Investigation is progressing in these ares but at this time is not mature enough to meaningfully discuss in this context. This then leads to analysing larger information structures and deciding, potentially at run-time what constitutes the required structure: [26] defines the notion of RDF- molecule which denotes a larger structure based upon the necessary set of properties aggregated together to make a more meaningful element or object [98]. How larger structures are mapped together or made equivalent beyond the above syntactic methods is open for development [97]. Techniques based upon semantic simi- larity in the linguistic sense probably provide the most reasonable method for performing this matching [94, 27]. In simple examples such as the case study in this paper the three explicitly stated rules above implemented over a number of agents suffices to link the information struc- tured by the given ontologies together. In many cases simple rule set suffice although as we have noted there is the need for non-monotonic reasoning in some cases. 3.4 Content-Driven Meetings Scenario In this section we show how meetings work using our framework. In content-centered, in- formal meetings the activity is not driven by an agenda or predefined process; instead, the activity emerges in a dynamic and opportunistic manner as reaction to the development of the information content. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  26. 26. 26 The crucial aspect in our approach is the flexible and easy interleaving of meeting conversation with visual presentations. The conversation itself is flexible and can lead into vivid and dynamic interaction between participants. However, the difficulties to make visual presentations often lead to static arrangements, over-prepared presentation materials, and as a result, in non-interactive presentation events. Such meetings give little room for successful building of shared knowledge. Below we go through meeting scenarios to show how flexible presentation interoper- ability tools have a potential to provide more freedom to ordinary meetings. It should be stressed that the presentation framework is based on open and free participation: each participants must decide whether to follow the presentation by subscribing to it. In ad- dition, these scenarios should not be confused with “smart meeting room” scenarios; we are focusing on ubiquitous, easy, and flexible meetings. Scenario (Setting up and Working in the Meeting) Three researchers, Alice, Bob and Eve, have to prepare a conference paper. Alice has written a draft for sections 1 and 2, Bob and Eve drafts for sections 3 and 4. They should meet to discuss the paper. Using MeetingBuilderAgent, Alice creates in a ‘smart space’ (or just space) a meet- ing with agenda items Go through Alice’s draft, Go through Eve’s and Bob’s draft, and Producing the final version. Later, Alice creates a presentation for document s12.pdf and binds it to the meeting in the space. Here one might see a structure similar to the right-hand-side of the diagram in figure 3.4. Alice, Bob and Eve agree on a meeting at university cafeteria. Like Alice before, Bob creates a presentation for document s34.pdf. This can be made in the same space or in a different space with the two spaces being merged or one projected into the other - for simplicity we assume the same space. Alice, Bob and Eve meet at university cafeteria. They have their laptops with them. To show his presentation, Alice launches a PresentationControllerAgent, which creates a the meeting and presentation to be controller (but of different types to the earlier meeting and presentation) in the space and starts to control the presentation. One of the reasoner agents recognises this and infers that these can be unified in the semantic sense - for example using the rules 3.1,3.2 and 3.3 described earlier and possibly enhanced with the more sophisticated semantic pattern matching also described. Now the various agents that work across the different ontologies do not see any difference in the separate information structures. To see the presentation at their laptops, Alice, Eve and Bob launch AVDeviceCont- rollerAgents. Using PresentationControllerAgent, Alice starts the presentation on the first page. The AVDeviceControllerAgents detect the change of page and show the corresponding NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  27. 27. 27 page on a window in each laptop. Using PresentationControllerAgent, Alice moves forward in the presentation. The AVDeviceControllerAgents detect the page changes and show the corresponding pages. At some point, Eve notes that there is an inconsistency between two document drafts. He launches another PresentationControllerAgent, makes it join the meeting and control the presentation of Bob’s and Eve’s draft. The participants set their AVDevice- ControllerAgents to show this presentation too. Using his PresentationControllerAgent Eve moves to page three of Bob’s and Eve’s draft. AVDeviceControllerAgents show the corresponding page on the laptops. Alice adds into the space an annotation about the inconsistency between the drafts. At this point, the participants decide that it is better to simultaneously go through the both presentations to check the inconsistencies. Alice and Eve control the presen- tations, and the participants add annotations about detected inconsistencies in the pre- sentations. After working like this for a while, the participants decide that they should write new versions of their drafts. They decide to meet again later to finalize the paper using the new versions of the drafts. Note how the situation developed. In the original agenda, the idea was to go through the drafts one at a time and then decide how to produce the final version. However, when an inconsistency between the drafts was detected, the meeting took a different path. In traditional, rigid meeting situations, this may be considered as a fault in the process. However, when we have a content-driven approach, this is very natural and causes no problems. The required software agents can be rather simple. MeetingBuilderAgent and Pre- sentationControllerAgent can be implemented as web pages that access the RDF store. AVDeviceControllerAgent can be implemented as a couple of scripts that monitor changes in the space and command a PDF viewer. The SKIM controller mentioned above was implemented this way. Scenario Continued (Note taking) Eve makes copious notes during the meet- ing using an unrelated agent. As Eve makes these notes they are linked both to the meeting context and to the current presentation. Furthermore as the note contain valu- able information regarding what is currently being discussed Eve starts a further search agent that can analyse the text and make searching for other interesting presentation via some suitable search technology (Citeseer, Google etc). Eve notices that some of these suggested presentations are highly relevant and selects them from her web-browser which then writes the details of the selected presenta- NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  28. 28. 28 tions into the space. Eve has also set up her web browser agent to link these presentations directly into the current meeting so that they can become both part of the current meet- ing context and because of the reasoning agents described earlier (cf: eqns.3.1,3.2,3.3) are also available for show during the meeting. This kind of interoperability and specifically the ability to link in information into the meeting in a very ad hoc manner allows not only for the overall meeting context to be expanded but also that items such as searches for material made ‘outside’ the meeting scope not to be lost. There are interesting privacy issues here such as what happens if much of the above is fully automatic and one of the meeting participants is not fully concentrating on the meeting and starts searching for ‘other’ material - an all too common occurrence in many meetings. However we espouse the use of much more direct interaction with the user thus empowering the user with more control over how his or her information is being used and the context gathered. Scenario continued (Minutes) Later at a meeting room in TKK, Bob and Eve have another meeting to edit their draft. Eve creates a new meeting and adds the presentations of the previous meeting to this meeting. He also adds a new presentation for the new draft that they are going to edit. In addition to their laptops, they have a computer that is connected to a video projector. To see what happened in the last meeting, Bob creates another presentation “last meeting minutes”. This is a special kind of presentation that can be controlled by MeetingMintesPresentationAgent, a software agent that is capable of going through the events of the previous meeting. Such an agent might even be working continuously such that the minutes are being continuously generated. For showing the minutes, Bob also launches an AVDeviceControllerAgent that shows a time-line of the last meeting on the video projector. Using these agents, Bob and Eve go through the annotations of in the previous meeting. For each annotation, the meeting minutes agents show the pages of the presentations that were shown in the previous meeting when the annotation was made. Here we see how the context of the previous meeting is seamlessly merged to the context of the new meeting. There was no need to separately write down the minutes of the meeting; they were created during the meeting. The information that the agents record in the RDF store is a much richer and more accurate presentation of what really happened. What is needed in addition to the classes that we described in Section 3 is a NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  29. 29. 29 way to time stamp events. This can easily be added to the agents. The software for MeetingMintesPresentationAgent and the corresponding AVDevice- ControllerAgent can be simple. The former can be implemented as web pages that access the space to detect the course of action in the previous meeting and provides a way to go through this information. In addition, the agent acts as a controller for the presentations that were shown in the previous meeting to show how the presentations proceeded in the previous meeting. 3.5 Discussion, Conclusions and Future Work We have described the ontologies, the typical reasoning over ontologies and the actions might be performed by participants of a meeting. Our approach does not explicitly imply a strict process nor ordering of actions but rather enables the actions (implemented as agents) to be made under the control of the users at the appropriate times according to the context of the whole meeting. The following points are specifically raised and are supported by the distributed, information sharing nature of our system: • Individual actions implemented as agents • User use the useful set of agents at any given point in time • The context of the meeting is independent from the actual meeting itself • The physical and temporal location of the meeting is independent from the content and thus context of the meeting This benefits the user in more realistic and complex situations where people make annotations, make multiple presentations, link to external content, are physically mobile (temporally and physically) and so on. One of the particular features that supports a much freer environment for the users is that there is no flow of control explicitly enforced between agents - the environment is declarative in nature. This of course has certain problems such as agents getting access to and potentially modifying data before a particular information structure is ready. These are however handled by suitable guards in the agent design and even by external signalling mechanisms as described earlier. Issues relating to security, privacy and trust are not addressed by our framework at this time. However the nature of a space is such that the architecture support policy based agent joining to a space: that is for an agent to have access to a space then it must supply the correct set of credentials before being admitted. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  30. 30. 30 NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  31. 31. Chapter 4 Cross-Domain Interoperability: A Case Study Jukka Honkola, Hannu Laine, Ronald Brown, Ian Oliver We describe a case study of the behaviour of four agents using a space based communication architecture. We demonstrate that interoperability may be achieved by the agents merely de- scribing information about themselves using an agreed upon common ontology. The case study consists of an exercise logger, a game, a mood rendered attached to audio player and phone sta- tus observing agents. The desired scenario emerges from the independent actions of the agents Originally published: 2nd Russian Conference on Smart Spaces, ruSMART 2009 4.1 Introduction Interoperability between devices and the software and applications that run on those devices is probably the major goal for many developers of such systems. This can be achieved in a number of ways: particularly through open, agreed standards or often through monopoly. Whichever particular ‘interoperability’ route is taken there will always be a plurality of standards relating to how such systems communicate [67][9]. For example, the Universal Plug and Play interoperability standard is plagued with man- ufacturer and device specific variations which complicate and nullify to some extent the long and complex standardization process employed to avoid this. 31
  32. 32. 32 Technologies such as the Semantic Web [10] provide enablers for solutions to these prob- lems. Within the Semantic Web exist information representation formats such as the Resource Description Framework (RDF) [92] and the web ontology language OWL [88] which themselves build upon enablers such as XML. Using these technologies we can address and provide solu- tions1 to the interoperability problem. At one level there are existing solutions such as Web Services (SOAP, WSDL, UDDI etc) and various stacks of middleware for processing representation formats and preservation and reasoning about the semantics of messages. At the other, there are solutions based around more declarative mechanisms such as TripCom [99] and the space-based solutions, for example Java Spaces [33] for the transport and processing of messages and information to assist in interoperability. We do not believe that interoperability will be achieved through standardization com- mittees, nor through standard, globally accepted semantics and ontologies but rather by the unification of semantics in a localized manner, as described in previous work [84]. In this paper we describe an approach to solving the interoperability problem through a combination of context gathering, reasoning and agents within a space-based infrastructure taking advantage of technologies such as RDF. We demonstrate this through the ad hoc inte- gration or mash-up of a number of distinct applications to demonstrate the principles of this approach. 4.2 The M3 Concept The M3 system consists of a space based communication mechanism for independent agents. The agents communicate implicitly by inserting information to the space and querying the information in the space. The space is represented by one or more semantic information brokers (SIBs), which store the information as an RDF graph. The agents can access the space by connecting to any of the SIBs making up the space by whatever connectivity mechanims the SIBs offer. Usually, the connection will be over some network, and the agents will be running on various devices. The information in the space is the union of the information contained in the participating SIBs. Thus, the agent sees the same information content regardless of the SIB to which it is connected. The high-level system architecture is shown in Figure 4.1. The agents may use five different operations to access the information stored in the space: Insert : Insert information in the space Remove : Remove information from the space Update : Atomically update the information, i.e. a combination of insert and re- move executed atomically Query : Query for information in the space Subscribe : Set up a persistent query in the space; changes to the query results are reported to the subscriber 1 plural, never singular! NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  33. 33. 33 SIB SIB SIB Agent Agent Agent Agent M3 Space Figure 4.1: A diagram of the general system architecture: Agents, M3 spaces and SIBs. In addition to the four access operations there are Join and Leave operations. An agent must have joined the space in order to access the information in the space. The join and leave operations can thus be used to provide access control and encrypted sessions, though the exact mechanisms for these are still undefined. In its basic form the M3 space does not restrict the structure or semantics of the infor- mation in any way. Thus, we do not enforce nor guarantee adherence to any specific ontologies, neither do we provide any complex reasoning2. Furthermore, information consistency is not guaranteed. The agents accessing the space are free to interpret the information in whatever way they want. We are planning to provide, though, a mechanism to attach agents directly to the SIBs. These agents have a more powerful interface to access the information and can be e.g. guaranteed exclusive access to the information for series of operations. Such agents may perform more complex reasoning, for example ontology repair or translation between different ontologies. However, they may not join any other spaces but are fixed to a single SIB and thus a single space. The M3 spaces are of local and dynamic nature, in contrast to semantic web which embodies Tim Berners-Lee’s idea of semantic web [10] as a “giant global graph”. The locality and dynamicity—we envision that the spaces will store very dynamic context information, for example—poses different challenges than the internet-wide semantic web. For example, in order to provide a true interoperability for local ubiquitous agents, the space (i.e. SIBs) will have to provide a multitude of connectivity options in addition to http: plain TCP/IP, NoTA [79], Bluetooth,. . . Furthermore, the space should be fairly responsive. While we do not aim for real-time or near real-time system, even half minute long response times for operations are unacceptable. The responsiveness is one of the factors behind the fundamental decision to not enforce any specific ontologies and allowing the agents to interpret the information freely, as it lessens the computational burden of the infrastructure. Another, and more important reason is that we explicitly want to allow mashing up information from different domains in whatever way 2 The current implementation of the concept understands the owl:sameAs concept NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  34. 34. 34 the agents see best. Strict ontology enforcement would make this kind of activity extremely difficult as all new ways of mashing up the information would require approval from some ontology governance committee. However, as mentioned above, we still plan to provide means for ontology enforcement for cases where the space provider explicitly wishes to restrict the ways the information is used as there are bound to be also situations where this is the best approach. The information content in a M3 space may be distributed over several SIBs. The dis- tribution mechanism assumes that the set of SIBs forming a M3 space are totally routable but not necessarily totally connected. The information content that the agents see is the same regardless the SIB where they are connected. 4.2.1 Applications in M3 Spaces The notion of application in M3 space is differs radically from the traditional notion of a monolithic application. Rather, as a long term vision, we see the applications as possible scenarios which are enabled by certain sets of agents. Thus, we do not see an email application running in M3 space, but we could have a collection of agents present which allow for sending, receiving, composing and reading email. For this kind of scenario based notion of application, we also would like to know whether the available agents can succesfully execute the scenario. The envisioned model of using this system is that the user has a set of agents which are capable of executing certain scenarios. If a user needs to perform a new scenario that the current set of agents are not capable of executing, she could go and find a suitable agent from some directory by describing the desired scenario and the agents she already has. Thus, we need some formal or semi-formal way of describing agent behavior both with respect to the M3 space and to the environment. While there exists research addressing behavior in multi-agent systems, for example by Herlea, Jonker, Treur and Wijngaards [42], this kind of ad-hoc assembly of agents in order to execute a certain scenario seems to be quite unaddressed in current research. However, slightly similar problems have been addressed in e.g. web service orchestration research [31], but these still seem to concentrate on design-time analysis rather than run-time analysis. As for shorter term, our vision is that sets of existing applications would be enhanced by being able to interoperate and thus allow execution of (automatic) scenarios that would have been impossible or required extensive work to implement without the M3 approach. 4.3 The Case Study The case study consists of four agents, Exercise logger, SuperTux game, Phone line observer, and Mood renderer, running on several devices. The exercise logger and phone line observer run on Nokia S60 platform phones, and the SuperTux and mood renderer run on Nokia N800 NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  35. 35. 35 internet tablets. The M3 space infrastructure runs on a Linux laptop. 4.3.1 Application Development Vision The case study setup illustrates a “legacy enhancement” approach to application development in M3 spaces. Conceptually, we added M3 space capability to existing applications (Super- Tux, Mood renderer, SportsTracker, Telephony) even if the SportsTracker and Telephony were implemented as standalone simple applications for practical reasons. The starting point for the case study was a single person (as defined in Person ontol- ogy, section 4.3.2) who would be exercising and tracking the workouts with an exercise logger, playing computer games and listening to music. The goal was to demonstrate the benefits of interoperability between existing application, that is, to be able to execute a scenario where the SuperTux game would award extra lives for exercising, a mood renderer embedded in a media player would play suitable music depending on a game state, and the game and media player would react accordingly if the person receives a call. 4.3.2 Ontologies The different ontologies have been defined informally, and are not enforced in any way. The different components of the case study assume that they are used correctly. A pseudo-UML diagram of the complete ontology is shown in Figure 4.2, and an example instantiation of the ontology when a game is being played and the phoneline is idle is shown in Figure 4.3. As a general note, the needs of the scenario studied drove the modeling of the ontologies. Most of the choices of what to include and what to exclude were based on the need of the infor- mation in executing the chosen scenario. A broader study might need to revisit the ontologies and generalize end expand them. Person The person ontology describes the information about a person necessary for this case study. A person can have a name (m3:username) and an operator account (m3:mobileaccount). Workout The workout ontology describes a generic distance-oriented workout. Thus, we include infor- mation about the length of the workout, distance traveled, person who has done the workout etc. but no information related to, for example, weightlifting where we probably would like to know the number of repeats and weights instead of time used and distance. The choice of what to include in the ontology was influenced by the information available in the SportsTracker application. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  36. 36. 36 m3:phonelinestatus OperatorAccount m3:user m3:playerm3:mobileaccount m3:username m3:totaldistance m3:totaltime m3:stoptime m3:username m3:starttime m3:sessionstart m3:sessionend m3:lives Exercise Person SuperTux Figure 4.2: An informal description of used ontologies ?3 ?2 m3:supertux m3:user m3:player m3:livesm3:sessionstart "2008−10−30T11:30:02" m3:workout rdf:type m3:stoptime m3:totaldistance m3:totaltime "SportsTrackerUser" m3:username ?1 m3:mobileaccount rdf:type m3:phonelinestatus "3738786" ?4 "2008−10−29T08:05:12" rdf:type rdf:type m3:person "4" "idle" m3:operatoraccount "2053132" "2008−10−29T07:02:30" m3:starttime Figure 4.3: An example RDF graph during the execution of scenario Game The Supertux game ontology includes information about the gaming session start time, stop time, lives left and player. The session start is defined to be the time when the game executable was started, showing a welcome screen. The lives value is inserted when a player starts a game, new or saved, from the welcome screen. When the player quits the game completely, that is, exits the welcome screen, the session stop time is inserted. The ontology could include also information about current score, amount of coins, field being played etc. which the mood renderer could use. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  37. 37. 37 4.3.3 Agents Next we describe the operation of agents making up the case study. It should be noted that the agents are only communicating implicitly through the M3 space by reading the information inserted to it by other agents. Workout Monitor The workout monitor reads SportsTracker workout files and writes a summary of the workouts to M3 space. We currently read the information related to general description of the workout and insert that in the M3 space. Any path information is ignored as we have no use for it in the current scenarios. The workout information is related to the person doing the workouts. If a person with same name as the SportsTracker username is found, we attach the workout in question to that person, otherwise we create a new person. We use an SportsTracker internal workout identifier to identify workouts so that we do not insert duplicate workouts if the monitor is run again with same workouts as before. Supertux Supertux is a classic jump’n’run sidescroller game which has taken strong inspiration from the original SuperMario games for Nintendo. Supertux is open source and available under GPL. The ontologies related to Supertux game are the Person ontology, the Workout ontology and the Supertux ontology. Each Supertux game is assigned with player information. If the person class instance of the player can not be found from the M3 space the game creates one and inserts player information according to the Person ontology. The person class instance also binds the player to the workout information inserted to the M3 space. Prior starting the game supertux queries workout information from the M3 space. The first query searches for all instances of the Workout class. For each found workout class instance a new query is performed in order to find out the details of the workout. If there are long and recent enough workouts performed by the player the game awards two extra lives to a game level. The game creates also an instance of itself and inserts information such as reference to the person class of the player, session start and stop time to the M3 space. In addition, the game updates how many lives the player has left during playing a game level. Every time the player looses or gains one life in the game, the game removes old livesinformation and inserts the new information to the M3 space. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  38. 38. 38 Telephony status observer The telephony voice line status is part of a OperatorAccount class. Operator accounts are bound to a person i.e. it is assumed that each account belongs to a person3. The voice line status only has two states, namely active and idle. The phone line status monitor subscribes to voice line status changes provided by Symbian’s CTelephony API. When the user dials a phone number to establish voice connection or when there is incoming call to the phone, CTelephony API provides the current voice line status to the observer application. The observer application transforms the CTelephony states (e.g. EStatusRinging, EStatusDialling, EStatusIdle, . . . ) to corresponding values defined by the OperatorAccount ontology and updates the information to the M3 space. Mood renderer A mood renderer conceptually would have interfaces to sensory, i.e. mood rendering devices of a smart space invironment in order to cooridinate their controls according to a mood determined by its knowlege processing. The questions would be many about such knowlege for determing a mood and how to render it for a given environment. The mood renderer of the case study is a very simple start on the problem, which side-steps many issues for practical cases, while showing the essence of knowlege processing for the given concept. The mood renderer in this case study is based on music as the only rendering device, and thus, the simple assumption that music, more precisely some audio track, maps to a de- terminable mood, and still further, that the mapping stems solely from the state of a partical game being played and applies invariently with regard to time. This dodges questions about personal tastes and more than one person contributing to the smart space mood. In choosing a track to play, volume is not controlled, but play, pause and stop are used where appropriate. Mood rendering is based solely on two sources of information in the smart space environment. The first is an operating supertux game and the second is phone status. Note then, that this mood renderer is solely reactive to information about a dynamic smart space environment to which it belongs—for simplicity, it does not contribute information even though it clearly impacts it. Given the simplifications stated, and that the mood renderer understands the ontologies of the supertux game and telephony status observer. It follows their two independent contributions to the smart space environment and combines them to render a mood appropriate for the state of game play, while maintaining the courtesy of pausing an audio track, if being played while any phone status of the smart space is active. The mood renderer is implemented as two persistent queries for smart space information. The first is looking for an active game in order to render its mood, and the second is looking for any phone with an active status. When an active game is found, a third persistent query, particular to this game, looks for its level of m3:lives and renders a mood as CALM, for no lives before the real play starts, and then when m3:lives exist, either “UP-BEAT” or “DOWN- 3 or a group of persons—we do not prevent that NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  39. 39. 39 BEAT” according to a threshold configuration on the number of lives. When the game ends, audio rendering is stopped. The mood renderer’s behavior can be described as follows: • Start with renderer.setTrack (NONE), renderer.setMode (STOP), and gameSelected=FALSE. • Subscribe to the call-status attribute for all phone-status-observers. Monitor subscription results, for any attibute value of “active”, if currentMode is PLAY, then do renderer.setMode (PAUSE); otherwise, if currentMode is PAUSE, do renderer.setMode (PLAY) i.e. any playing pauses during any phone call, or resumes when no phone call if paused during a game. • Subscribe to a supertux game, which has a session-start and no session-end, or session- start after session-end. Monitor subscription results, and, whenever fulfilled, first ensure that rendering is stopped and end subscriptions for any earlier game. Then render mu- sic/mood mapping for the selected game using the next subscription. • Subscribe to session-end and lives attributes of the selected game. When fulfilled, set renderer.setMood (mood), for mood according to lives as follows: – no lives: CALM i.e. game started, but not yet played. – lives above threshold: UP-BEAT – below threshold: DOWN-BEAT When session-end of game is after session-start, stop rendering and end subscriptions about this game. 4.4 Conclusions The idea of agents interoperating by communicating implicitly through M3 spaces worked well for this case study. We were able to execute the scenario in mind and add the phone agent to it without modifying the existing ontology. However, this work was performed inside a single team in a single company. This kind of close cooperation between people implementing the agents obviously makes things easier. Most of the work required in integrating the different agents into a coherent scenario was related to the mood renderer observing the state of the SuperTux game. The ontology did not directly provide information about all the state changes in the game. For example, when starting to play a level in the game, the mood renderer deduce this by observing the appearance of the lives attribute in the M3 space. This is however something that will probably be a very common case as the ontologies can not and should not try to describe all possible information that some agent may need. It should suffice that the information is deducible from the explicitly stated information. Another aspect of local pervasive computing environments that was not really addressed in this case study was the handling of resources that can only be used by one agent at a time. In NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  40. 40. 40 this case the mood renderer represents such resource (audio player), and implicitly it is usually the game that uses it as the mood renderer deduces the mood based on the game state alone. When the phone rings, the mood renderer will stop the music and continue after the call is finished. However, in a more complex case there needs to be (preferably) clear policy for each resource on who can use it. We currently think that this can be handled at the ontology level, with suitable coordination structures and an agent acting as a gatekeeper. In general, this is however a subject for further study. The basic principles of agent-dependent interpretation of information and implicit com- munication proved useful when the phoneline status information was added to the case study. When we only describe the information and not the actions to be taken by the agents, the agent can select suitable course of action based on the detailed knowledge of the agent in question. In this particular case, the correct course of action for the SuperTux game is to pause when the phoneline is active but not automatically resume the game when it goes idle, as the player is not necessarly ready to continue immediately after ending the call. On the other hand, the mood renderer should continue playing the music immediately after the distraction (in this case call) has ended. If we expanded the case study with more persons with each having mobile phones, the difference with mood renderer and SuperTux regarding the behavior when a call arrives would be even more. Sensible behavior for mood renderer would be to stop the music whenever anyone in the same (acoustic) location receives a call, while the game should only react to calls to the player of the game. Of course, in some cases it might be desirable to have coordinated response from a group of agents to certain events. We believe that this is best handled outside the system, by e.g. requirements in the scenario descriptions or ontology documentation. Anyhow, such coordinated responses would not apply to all agents but rather to only certain groups of agents involved in executing a given scenario. An obvious next step related to the interoperability hypothesis is to have external research partners extend the demo with agents implemented outside Nokia. This would clarify the required level of ontology and agent behavior description as the communication would then approximate more the real world case where the agents would be implemented by multiple vendors. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  41. 41. Chapter 5 Agent/Space-Based Computing and Radio Frequency Memory Tag Interaction Joni Jantunen, Sergey Boldyrev, Ian Oliver, Jukka Honkola As devices become more ubiquitous and information more pervasive technologies such as Se- mantic Web and Space-Based computing come to the fore as a platform for user-interaction with their environment. Integrating technologies such as advanced RF memory Tag with mobile devices and combining this with computation platforms such as those supporting Semantic Web principles provides for the user innovative and novel applications. Originally published: The 3rd International Workshop on RFID Technology - Concepts, Applications, Challenges (IWRT 2009) 5.1 Introduction The integration of mobile devices, RF memory tags and semantic web style information sharing and management will open new opportunities for the ad hoc gathering, sharing of information and for more interactive applications built on top of these concepts. RF memory tag systems will have the ability to store large quantities of information coupled with high-bandwidth radio connections and even contain local processing capabilities. 41
  42. 42. 42 Mobile devices, such as mobile telephones, have access to ubiquitous connectivity solutions, eg: WLAN, GSM, UMTS networks, as well as sophisticated media and processing capabilities as well as many local sensors such as GPS etc. Semantic Web technologies provide canonical information representation formats, in par- ticular XML for the representation, RDF for the expression of semi-structured, linked and linkable information and languages such as OWL for the specification of those structures. This allows much easier integration and interoperability between applications, agents and devices through standardising many features and providing a basis for reasoning technologies leading to, at least, weak-AI capabilities. Combining these technologies together provides a platform for highly interactive, viral information sharing applications that can be applied in many situations: such a platform has been implemented as in [83, 86]. For example: mass memory tags are used as a distribution point of information in a certain locations such as office or shopping center information points, news stands, tourist attractions etc; even individual physical items can be tagged - an application particular relevant in the medial sector. Applications of a system would provide enahancements to existing service-oriented technologies. In this paper we describe the technologies and an implementation which leads to frame- work that supports such applications and interactions. Firstly we provide descriptions of the technologies, then an overview of the types of applications with examples, then a discussion on a specific interaction case which highlights particular interesting areas of the RFID-Device-Space interaction and finally a discussion of the future direction and issues of this technology. 5.2 Background We focus on the integration of three major technologies: the Semantic Web, RF memory tags and mobile devices. Within these categories we specialise on the notion of ‘Smart Spaces’, the distribution and integration of information and a particular mass-memory, high-bandwidth form of RF memory Tag. 5.2.1 Semantic Web and ‘Smart Spaces’ The Semantic Web [10] is a vision for the improvement of personal computing through the exposition and sharing of information based upon common representation formats, ontologies and semantics. The idea is that information would become globally ubiquitous and interoperable. However much information is not ubiquitous in the global sense; it is personalised, hidden, private and is interpreted locally - this information tends to be the personal, highly dynamic information that one stores about oneself: contact lists, friends, media files, ‘my’ current context, ‘my’ family, ‘my’ home etc and the interweaving and linking between these entities through ad hoc personal structures. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  43. 43. 43 To address this issue we have implemented a system based around larger, more user or personal focussed structured which we term ‘Smart Spaces’ [86]. Within a ‘Smart Space’ (or just space) a user can encapsulate all of their information and interact with that information according to their personal semantics and needs. Spaces have the ability to be distributed across the user’s devices (mobile phone, media center, personal computer etc) as well as more centralised providers. Synchronisation between these distributed, individual repositories of information is asymmetric to address device and network capabilities as well as the user’s needs in terms of security etc. Interaction with spaces is nominally by ‘agents’ which encapsulate fine grained function- ality which themselves may be distributed across any number of devices that have access to the user’s space. Spaces themselves can interact through merging and projection enabling larger spaces to be constructed either on permanent or temporary basis. Further interaction between users is enabled through one user granting access to their space (or spaces) to another user’s agents or even by directly sharing the contents of their spaces through the asymmetric distribution mechanisms. A space can transcend over many of the user’s devices leading to the distribution of information and queries upon that information. For any agent accessing this information the physical location of it and the information is irrelevant: an agent sees the ‘totality’ of all information in that space. This requires sophisticated distribution algorithms that actually preserve a degree of assymetry of information depending upon the stability, connectivity and other properties of a particular information repository. 5.2.2 From RFID to RF memory tags Radio frequency identification (RFID) technologies have been used for decades in extensive variety of applications [59]. In the simplest RFID applications the tags embedded into objects only provided 1-bit presence information when exposed to the electromagnetic field transmitted by a reader. Identification of tags with individual ID codes started from active tags and read only memories and it has become commodity also for passive tags. Recently, especially the mainstream development of passive RFID technologies has been steered by applications such as logistics [28], and mobile payment [90]. In these applications the tags may include also some amount of non-volatile memory. Thus, a bi-directional communication link between reader and tags is required to enable also writing of data to tags instead of read-only access. Recently, respective reader capabilities have been also integrated into mobile phones [96]. That provides a possibility to use RFID tags as part of Smart Space applications by using mobile phone readers as a gateway. One development branch of RFID technologies is focusing on applications where passive RF memory tags will contain increasingly large non-volatile memory capacity (in scale of mega- to gigabytes) [51]. Such tags would enable storage of digital content exactly to the point where it is used e.g. into objects and physical locations. This approach is totally opposite to the network databases traditionally used in RFID systems where tags may store only a link to the databases. The visions behind this research are based on the foreseen development NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  44. 44. 44 trends especially on the field of non-volatile memory technologies according to which the power consumption, physical size and price of (non-volatile) memories are continuously decreasing. TSuch his enables development of passive RF memory tags which can be accessed wirelessly, for example, with mobile phones and which can be used as ubiquitous storage capacity in Smart Space applications. In RFID systems the physical interaction is usually based on back-scattering of narrow- band RF signals [30]. The reader device provides a continuous wave signal which is used by the tag to extract supply voltage. The communication from the tag to reader is possible thanks to modulation of back- scattered signal from the tag, whereas the communication from reader to tag can be done for example with simple amplitude or phase modulation methods respecting the power extraction needs in the tag. Small memory capacity can be implemented with EEPROM and the whole functionality is usually controlled by a simple finite state-machine (FSM). However, when the storage capacity of tags increases significantly back-scattering method based on a single fre- quency band may be insufficient to achieve data-rates high enough. According to current regulations wide enough frequency bands are not available on low frequencies and the efficiency of wireless power transfer decreases on high RF frequencies. Thus, either data-rate or commu- nication range is somewhat limited. Therefore dual-band systems have been proposed for data centric Ambient Intelligence (AmI) applications, such as, the ones used in Smart Space context [54]. An exemplary (complete) block diagram of an RF memory tag is presented in Figure 5.1. In such a system one frequency band is dedicated for wireless power transfer and one wider band for high data-rate communication. Figure 5.1: Block Diagram of RF memory Tag Section 5.3 categorizes groups of interactions between RF memory tags and Smart Spaces. Depending on the category the required computational power of tag may range from mere memory access (a simple FSM is enough) to hosting of an agent or a space (requiring MCU or even CPU). In the two ‘classic’ ways it is sufficient that the tag is capable of transferring the data defined by the reader with the highest possible efficiency to achieve user-friendly interaction. The reader device is responsible of defining which data is read and/or written, and basically in the tag there is no need to process the content itself. Even the file-system can be maintained in the reader side if it is first copied from tag to reader in the beginning of interaction and then updated version is written to tag after the payload data in the tag has been modified. In more sophisticated cases where tags are capable of hosting an agent or running a space it NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  45. 45. 45 may be necessary to have sufficient processing capabilities in the tag itself. If fully active tags are excluded from the study, to achieve easy and low cost maintenance, the remaining alternatives are fully passive tags and semi-passive tags. In the passive solution the data is processed fast in the tag during the reader-tag interaction. In the semi-passive solution the data is processed over a long period of time (as a background process) by using a dedicated power source e.g. the power harvested from the environment and only the high data-rate communication requires wireless powering provided by the reader. In semi-passive case the tag may also function as a data-logger by slowly collecting a huge amount of samples from a low-power sensor over a long measurement period. However, these applications require extremely low power consumption what comes to the memory accesses, data processor and sensors but also required data-rate is extremely slow. Altogether, the ongoing development on many technological fields will in the future enable utilization of RF memory tag systems in Smart Spaces. Actually, Smart Space applications may become one of the main drivers for the technological development of aforementioned RF memory tag systems. 5.2.3 Distribution The scope of distribution and corresponding infrastructures in case of RF memory tag systems can be analysed in terms of two areas: • physical environment, e.g. associating of the RF memory tag with direct or indirect functionality provided by the physical environment, and • ubiquitous device architecture, e.g. impact to the internals of a particular mobile device Considering further, the following can be seen: • RF memory tag is dumb memory block and has just mechanisms for bidirectional wireless operations (read/write) • RF memory tag has enough computational power on board, but still external power is needed, or it can be provided on a volatile basis Therefore, the range of potential application can be variate from simple “just-in-place” and “just-in-time” file storages coming to distributed ad hoc storage systems [11] and going beyond asynchronous computation engines. Projecting such features to the Smart Spaces ar- chitecture RF memory tag system enables use cases when information related to the particular physical environment can be delivered in-place on-demand or even by estimation, in a very power and computation efficient manner. By saying “delivered” the whole process of extrac- tion, recovery, storing and computing should be understood. Since using RF memory tag system it is feasible to produce a distributed snapshot of Smart Space which is tied with the particular physical environment by storing the run-time environment and corresponding contexts with NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  46. 46. 46 data/information within the RF memory tags, it also becomes possible to revoke the system once it would be demanded by any user. In other words the distributed process of booting, sus- pending and resuming of the Smart Space now becomes more flexible, reliable and less physical context dependant. Smart Space application infrastructure benefits from RF memory tag system as well. Since RF memory tags infrastructure provides distributed grained approach for execution context and information handling, Smart Space application can be constructed by traversing the RF memory tags and retrieving demanded parts of execution context and corresponding information. In terms of ubiquitous device architecture, as it is presented by Figure 5.1, RF memory tag system can enable computationally constraint device with memory extension, distributed memory architecture, or, rich context data analysis and corresponding information generation, which then can be uploaded to RF memory tag system. The problem of memory management (both, execution and mass memory) in distributed environment, when substantial amount of the devices are using RF memory tag system tech- nology, converges to the efficient and sustained context execution which is essential part of whole computing environment, such as Smart Space. Especially because of high dynamics and constrains the amount of memory on board the mobile devices in Smart Space. The approach of efficient and sustained distributed memory utilization by means of RF memory tag illustrates the solution of such problem. Consider the computing environment where memory is integrated with RF memory tag based architecture, and several distributed memory modules are dispersed around the Smart Space(s). Therefore they are considered as one stackable memory unit by means of corresponding combination of memory blocks of the environment. Putting environment where memory is integrated with RF based memory tag architecture and several distributed memory modules are dispersed around the Smart Space(s). From the Smart Space perspectives and corresponding functionality is provided by distributed RF memort tag based infrastructure, at least following cases described in section 5.3 can be seen. 5.3 Interaction with Tags In this section we describe the various modes by which spaces and agents interact with the RF Memory tags. We define four categories: • Agents read from Tag • Agents write to Tag • Agents on Tag • Space on Tag NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  47. 47. 47 The first two categories are the ‘classic’ ways of passively interacting with a tag and constitute simple information retrieval and sharing. The third and fourth cases involve much more ‘active’ interaction based requiring specific features of the agent/space architecture. We explain these in the following sections with an addition discussion of the actual physical interaction between devices and the RF memory tags. 5.3.1 User Interaction As mentioned in the previous section, in the context of Smart Spaces many new requirements for the future enhanced RFID-like technologies are set. In Smart Space environment also the user interaction is in a critical role since the fast data connection is enabled only if user voluntarily accesses RF memory tags. Therefore the system must be fast, flexible and reliable in addition to many other requirements. The need of bi-directional high speed data transfer is obvious to achieve convenient user experience when the storage capacity of tags increases. Flexibility can be understood as a requirement for practical communication range. Typical physical interaction is shown in figure 5.2. Figure 5.2: Physical Interaction of Device and RF memory Tag From user’s perspective it is needed especially to allow intuitive and convenient control through user interface of a mobile reader. Figure 5.3 presents a situation where mobile phone is used to interact with a tag which is part of a Smart Space. As can be expected, the com- munication range must be long enough to avoid need for accurate pointing of reader to tag in varying usage conditions. The communication link should also allow movement of the reader while the user makes selections through the user interface. Reliability of the user interaction can be understood as the quality of data transfers i.e. low probability of errors on different levels but also as security and privacy aspects since obviously users cannot tolerate such infringements if the content stored to RF memory tags is private. NRC-TR-2009-M3-1 Copyright c 2009 Nokia
  48. 48. 48 5.3.2 Agent Interactions The “simple” read case is useful for applications where the tags provide relatively static local information, for example, maps, positioning information or even marketing information such as might be found in a shopping mall style scenario. One existing application is that found in museum environments [47, 95] and this is currently being investigated as an initial deployment of the technology described here. The “simple” write allows the user to write to the tag. This enables sharing of user’s information through copying information from the user’s space to the RF memory tag. This case would be almost invariably used in conjunction with the read case. Typical applications here might be viral sharing of media or other information, for example, highly localized blogging and messaging. These classic cases are shown in figure 5.3. A A RF Memory Tag Information flow −−−> A A RF Memory Tag <−−− Information flow Figure 5.3: Classic Interaction The third case shown in figure 5.4 requires the tag to have some computational capabilities. When the device with the reader is presented to the tag, it activates agents on the tag itself which then either request to join the user’s space or are invited to the user’s space for the duration of the time while the reader is in contact with the RF memory tag. The agents on the tag may provide additional reasoning or other data gathering/processing operations as well as supporting the classic read and write use cases described earlier. The agents can access a Smart Space with five basic operations: • Insert: Insert information into the Smart Space. • Remove: Remove information from the Smart Space. • Update: Update the information in the Smart Space. This is effectively an atomic remove and insert combination. • Query: Query for information in a Smart Space. NRC-TR-2009-M3-1 Copyright c 2009 Nokia

×