Data Modelling and Knowledge Engineering for the Internet of Things


Published on

Tutorial on Data Modelling and Knowledge Engineering for the Internet of Things, presented at EKAW 2012, Galway City, Ireland, October 8-12, 2012

Published in: Technology
1 Comment
  • The internet of Things (IoT) is one of the most important areas of a Future Internet with significant potential to positively impact the Norwegian economy and society.
    I found a website which is also providing devices which work on Internet of things technology to monitor industrial data and share it between different devices. For more information related to it view
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Connecting physical world to the internet
  • One gateway; multiple gateways (1:n); many nodes to many gateways or end users (n:n, bi-directional; not only passing info, but also processing)
  • Scalability and interoperability problems
  • Intelligence at the edge or hub; still no good answer; could be app or design dependent
  • Traditional networking: host to host Now: data-oriented communication; looking for data, not the host providing the data unless you want to manage the node
  • IoT data different from traditional content; transient, small. The more important thing is how to find the service that can provide the data; could be huge stream of data
  • - Powerful commercial-off the shelf (cots) sensors
  • Take about something on the web of data
  • Since then bring in the semantic web technologies
  • Images:,r:0,s:0
  • Images:
  • Four characteristics of perceptual inference
  • Images:
  • Images:
  • Data Modelling and Knowledge Engineering for the Internet of Things

    1. 1. Data Modelling and KnowledgeEngineering for the Internet of Things Wei Wang1, Cory Henson2, Payam Barnaghi1Centre for Communication Systems Research, University of Surrey Kno.e.sis Center, Wright State University Galway City, Ireland, October 8-12, 2012
    2. 2. Part 1: Introduction to Internet of “Things”Image source: CISCO 2
    3. 3. Internet of Things “sensors and actuators embedded in physical objects — from containers to pacemakers — are linked through both wired and wireless networks to the Internet.” “When objects in the IoT can sense the environment, interpret the data, and communicate with each other, they become tools for understanding complexity and for responding to events and irregularities swiftly” source:
    4. 4. “Thing” connected to the internetSource: CISCO 4
    5. 5. Future Internet - A new dimension 55
    6. 6. Internet of Things - definition  “A world where physical objects are seamlessly integrated into the information network, and where the physical objects can become active participants in business processes.”  “Services are available to interact with these “smart objects” over the Internet, query and change their state and any information associated with them, taking into account security and privacy issues. ’”.Source: Stephan Haller, Internet of Things: An integral Part of the Future Internet, SAP Research, 2009. 6
    7. 7. What “Things” can be connected?Home/daily-life devicesBusiness andPublic infrastructureHealth-care…
    8. 8. Sensor devices are becoming widely available- Programmable devices- Off-the-shelf gadgets/tools
    9. 9. Application domain
    10. 10. Why is IoT important?
    11. 11. Observation and measurement dataAdapted from: W3C Semantic Sensor Networks, SSN Ontology presentation,
    12. 12. Data is important and IoT will producelots of it! Sensors and devices provide data about the physical world objects. The observation and measurement data related to an “object” can be related to an event, situation in the physical world. The processing of turning this data into knowledge/ perception and using it for decision making, automated control, etc. is another important phase. Huge amount of data related to our physical world that need to be  Published  Stored (temporary or for longer term)  Discovered  Accessed  Proceeded  Utilised in different applications
    13. 13. Turning Data into Wisdom
    14. 14. The “Things” Embedded device + physical world objects  Sensor nodes (e.g. SunSPOT, TelOSB, WASPmote).  Mobile devices (e.g. mobile phones, tablets)  A set of these that provide information about (a feature of interest of) a physical world object (e.g. water level in a tank, temperature of a room).
    15. 15. Components related to “Things” Physical world objects  e.g. A room, a car, A person; Feature of Interest  e.g. Temperature of the room, Location of the car, heart-rate of the person; Sensors  e.g. Temperature sensor, GPS, pulse sensor Embedded device  e.g. WASPmote, SunSPOT, …
    16. 16. Sensors Active & Passive Sensors Energy Efficiency Processing capabilities Network communications  hardware platforms  software platforms
    17. 17. RFID Active Tags and Passive Tags Applications: supply chain, inventory tracking, tools collection, etc. Limitations:  Technology  Reading range  Physical limitations  Interference  Security and Privacy
    18. 18. Hardware components of sensornodes Controller Memory Communication device Sensors (or actuators) Power supply
    19. 19. Example: Radiation Sensor Board (Libelium) WaspmoteSource: Wireless Sensor Networks to Control Radiation Levels, David Gascón, Marcos Yarza, Libelium, April 2011.
    20. 20. Energy consumption of the nodes Batteries have small capacity and recharging could be complex (if not impossible) in some cases. The main consumers of the energy are: the controller, radio, to some extent memory and depending on the type, the sensor(s). A controller can go to:  “active”, “idle” and “sleep” A radio modem could turn transmitter, receiver, or both on or off, sensors and memory can be also turned on and off.
    21. 21. Beyond common sensors  Human as a sensor  e.g. tweeting real world data and/or events  Virtual sensors  e.g. Software agents generating dataAdapted from: The Web of Things, Marko Grobelnik, Carolina Fortuna, Jožef Stefan Institute.
    22. 22. Actuators [2] Stepper Motor [1][4] [3] Image credits: [1] [2] [3] [4]
    23. 23. Wireless Sensor Networks (WSN)Image source: Protocols and Architectures for Wireless Sensor Networks, Protocols and Architectures for Wireless Sensor NetworksHolger Karl, Andreas Willig, chapter 3, Wiley, 2005 .
    24. 24. Wireless Sensor Networks (WSN)-gateway connection
    25. 25. Distributed WSN
    26. 26. What are the main issues? Heterogeneity Interoperability Mobility Energy efficiency Scalability Security
    27. 27. What is important? Robustness Quality of Service Scalability Seamless integration Security, privacy, Trust
    28. 28. In-network processing  Mobile Ad-hoc Networks are supposed to deliver bits from one end to the other  WSNs, on the other end, are expected to provide information, not necessarily original bits  Gives addition options  E.g., manipulate or process the data in the network  Main example: aggregation  Applying aggregation functions to a obtain an average value of measurement data  Typical functions: minimum, maximum, average, sum, …  Not amenable functions: mediansource: Protocols and Architectures for Wireless Sensor Networks, Protocols and Architectures for Wireless Sensor NetworksHolger Karl, Andreas Willig, chapter 3, Wiley, 2005 .
    29. 29. In-network processing- exampleApplying Symbolic Aggregate Approximation (SAX)SAX Pattern (blue) with word length of 20 and a vocabulary of 10 symbolsover the original sensor time-series data (green)
    30. 30. Data-centric networking  In typical networks (including ad hoc networks), network transactions are addressed to the identities of specific nodes  A “node-centric” or “address-centric” networking paradigm  In a redundantly deployed sensor networks, specific source of an event, alarm, etc. might not be important  Redundancy: e.g., several nodes can observe the same area  Thus: focus networking transactions on the data directly instead of their senders and transmitters ! d a ta -c e ntric ne two rking  Principal design changesource: Protocols and Architectures for Wireless Sensor Networks, Protocols and Architectures for Wireless Sensor NetworksHolger Karl, Andreas Willig, chapter 3, Wiley, 2005 .
    31. 31. Implementation options for data-centric networking  Overlay networks & distributed hash tables (DHT)  Hash table: content-addressable memory  Retrieve data from an unknown source, like in peer-to-peer networking – with efficient implementation  Some disparities remain  Static key in DHT, dynamic changes in WSN  DHTs typically ignore issues like hop count or distance between nodes when performing a lookup operation  Publish/subscribe  Different interaction paradigm  Nodes can publish data, can subscribe to any particular kind of data  Once data of a certain type has been published, it is delivered to all subscribes  Subscription and publication are decoupled in time; subscriber and published are agnostic of each other (decoupled in identity);  There is concepts of Semantic Sensor Networks- to annotate sensor resources and observation and measurement data!Adapted from: Protocols and Architectures for Wireless Sensor Networks, Protocols and Architectures for Wireless Sensor NetworksHolger Karl, Andreas Willig, chapter 3, Wiley, 2005 .
    32. 32. IoT and Semantic technologies The sensors (and in general “Things”) are increasingly being integrated into the Internet/Web. This can be supported by embedded devices that directly support IP and web-based connection (e.g. 6LowPAN and CoAp) or devices that are connected via gateway components.  Broadening the IoT to the concept of “Web of Things” There are already Sensor Web Enablement (SWE) standards developed by the Open Geospatial Consortium that are widely being adopted in industry, government and academia. While such frameworks provide some interoperability, semantic technologies are increasingly seen as key enabler for integration of IoT data and broader Web information systems.
    33. 33. Semantics and IoT resources anddata Semantics are machine-interpretable metadata (for mark-up), logical inference mechanisms, query mechanism, linked data solutions For IoT this means:  ontologies for: resource (e.g. sensors), observation and measurement data (e.g. sensor readings), domain concepts (e.g. unit of measurement, location), services (e.g. IoT services) and other data sources (e.g. those available on linked open data) Semantic annotation should also supports data represented using existing forms Reasoning /processing to infer relationships and hierarchies between different resources, data Semantics (/ontologies) as meta-data (to describe the IoT resources/data) / knowledge bases (domain knowledge).
    34. 34. A Few Words onSemantic Web 34
    35. 35. Semantic Web SSW Introduction (a c c o rd ing to Fa rs id e ) Concrete Facts Concrete Facts Re ssoourc ee De ssccrip tio nn Fra m eewo rk Re urc De rip tio Fra m wo rk lives in General Knowledge General Knowledge We bb O nto lo ggyy La ng ua ggee We O nto lo La ng ua has pet Person Person Animal Animal has pet is a is a“Now! – That should clear up a few things around here!”
    36. 36. Semantic Web Stack
    37. 37. Linked Open Data
    38. 38. Linked Open Data ~ 50 Billion Statements ~ 50 Billion Statements
    39. 39. SW is moving from academiato industry
    40. 40. In the last few years, we haveseen many successes … Apple Siri Watson Knowledge Graph
    41. 41. Google Knowledge Graph
    42. 42. Sensors and the Web 42
    43. 43. Sensors are ubiquitous
    44. 44. Sensors are small and inexpensive
    45. 45. Digitization of the physical world
    46. 46. Leading to … Improved situational awareness Advanced cyber-physical systems / applications Enabling the Internet of Things
    47. 47. Enabling the Internet of Things Situational awareness enables:  Devices/things to function and adapt within their environment  Devices/things to work together
    48. 48. Sensor systems aretoo often s to ve p ip e d .Closed centralizedmanagement of sensingresourcesClosed inaccessibledata and sensors
    49. 49. We want to set this data freeWith freedom comesresponsibilityDiscovery, access, and searchIntegration and interpretationScalability
    50. 50. Drowning in DataA cross-country flight from New York to Los Angeles on aBoeing 737 plane generates a massive 240 terabytes ofdata - G ig a O m ni M d ia e
    51. 51. Drowning in Data In the next few years, sensor networks will produce 10-20 time the amount of data generated by social media. - GigaOmni Media
    52. 52. Drowning in Data
    53. 53. ChallengesTo fulfill this vision, there are difficult challenges to overcome suchas the discovery, access, search, integration, and interpretation ofsensors and sensor data at scaleDiscovery finding appropriate sensing resources and data sourcesAccess sensing resources and data are open and availableSearch querying for sensor dataIntegration dealing with heterogeneous sensors and sensor dataInterpretation translating sensor data to knowledge usable by peopleand applicationsScalability dealing with data overload and computational complexity of interpreting the data
    54. 54. Solution Semantic Sensor Web Internet Computing, July/Aug. 2008 Uses the Web as platform for managing sensor resources and data  Uses semantic technologies for representing data and knowledge, integration, and interpretation
    55. 55. SolutionDiscovery, access, and search  Using standard Web services  OGC Sensor Web Enablement
    56. 56. SolutionIntegration  Using shared domain models / data representation  OGC Sensor Web Enablement  W3C Semantic Sensor Networks
    57. 57. SolutionInterpretation  Abstraction – converting low-level data to high-level knowledge  Machine Perception – w/ prior knowledge and abductive reasoning  IntellegO – Ontology of Perception
    58. 58. SolutionScalability  Data overload – sensors produce too much data  Computational complexity of semantic interpretation  “Intelligence at the edge” – local and distributed integration and interpretation of sensor data
    59. 59. SSW Adoption and Applications
    60. 60. Part 2: Semantic Modelling for the Internet of “Things”Image source:; CISCO 60
    61. 61. Recall of the Internet of Things A primary goal of interconnecting devices and collecting/processing data from them is to create situation awareness and enable applications, machines, and human users to better understand their surrounding environments. The understanding of a situation, or context, potentially enables services and applications to make intelligent decisions and to respond to the dynamics of their environments.Barnaghi et al 2012, “Semantics for the Internet of Things: early progress and back to the future”
    62. 62. IoT challenges Numbers of devices and different users and interactions required.  Challenge: Scalability Heterogeneity of enabling devices and platforms  Challenge: Interoperability Low power sensors, wireless transceivers, communication, and networking for M2M  Challenge: Efficiency in communications Huge volumes of data emerging from the physical world, M2M and new communications  Challenge: Processing and mining the data, Providing secure access and preserving and controlling privacy. Timeliness of data  Challenge: Freshness of the data and supporting temporal requirements in accessing the data Ubiquity  Challenge: addressing mobility, ad-hoc access and service continuity Global access and discovery  Challenge: Naming, Resolution and discovery
    63. 63. IoT: one paradigm, many visionsDiagram adapted from L. Atzori et al., 2010, “the Internet of Things: a Survey”
    64. 64. Semantic oriented vision “The object unique addressing and the representation and storing of the exchanged information become the most challenging issues, bringing directly to a ‘‘Semantic oriented”, perspective of IoT”, [Atzori et al., 2010] Data collected by different sensors and devices is usually multi-modal (temperature, light, sound, video, etc.) and diverse in nature (quality of data can vary with different devices through time and it is mostly location and time dependent [Barnaghi et al, 2012] some of challenging issues: representation, storage, and search/discovery/query/addressing, and processing IoT resources and data.
    65. 65. What is expected? Unified access to data: unified descriptions Deriving additional knowledge (data mining) Reasoning support and association to other entities and resources Self-descriptive data an re-usable knowledge In general: Large-scale platforms to support discovery and access to the resources, to enable autonomous interactions with the resources, to provide self- descriptive data and association mechanisms to reason the emerging data and to integrate it into the existing applications and services.
    66. 66. Semantic technologies and IoT There are already Sensor Web Enablement (SWE) standards developed by the Open Geospatial Consortium that are widely adopted. While such frameworks provide certain levels of interoperability, semantic technologies are seen as key enabler for integration of IoT data and and existing business information systems. Semantic technologies provide potential support for:  Interoperability and machine automation  IoT resource and data annotation, logical inference, query and discovery, linked IoT data
    67. 67. Identify IoT domain concepts Users Physical entities Virtual entities Devices Resource Services …Diagram adapted from IoT-A project D2.1
    68. 68. IoT domain concepts - Entity P hysical entities (or entity of interests): objects in the physical world, features of interest that are of interests to users (human users or any digital artifacts).  Virtual entities: virtual representation of the physical entities.  Entities are the main focus of interactions between humans and/or software agents.  This interaction is made possible by a hardware component called Device.Definition adapted from De et al, 2012, “Service modeling for the Internet of Things”
    69. 69. IoT domain concepts –Device, Resource and Service A Device mediates the interactions between users and entities. The software component that provides information on the entity or enables controlling of the device, is called a R esource. AS ervice provides well-defined and standardised interfaces, offering all necessary functionalities for interacting with entities and related processes.Definition adapted from De et al, 2012, “Service modeling for the Internet of Things”
    70. 70. Other concepts need to considered Gateways Directories Platforms Systems Subsystems … Relationships among them A links to e x is ting kno wle d g e ba s e a nd nd linke d d a ta
    71. 71. Don’t forget the IoT data Sensors and devices provide observation and measurement data about the physical world objects which also need to be semantically described and can be related to an event, situation in the physical world. The processing of data into knowledge/ perception and using it for decision making, automated control, etc. Huge amount of data from our physical world that need to be  Annotated  Published  Stored (temporary or for longer term)  Discovered  Accessed  Proceeded  Utilised in different applications
    72. 72. Semantics for IoT resources and data Semantics are machine-interpretable metadata, logical inference mechanisms, query and search mechanism, linked data… For IoT this means:  ontologies for: resource (e.g. sensors), observation and measurement data (e.g. sensor readings), services (e.g. IoT services), domain concepts (e.g. unit of measurement, location) and other data sources (e.g. those available on linked open data) Semantic annotation should also supports data represented using existing forms Reasoning/processing to infer relationships between different resources and services, detecting patterns from IoT data
    73. 73. Characteristics of IoT resources Extraordinarily large number Limited computing capabilities Limited memory Resource constrained environments (e.g., battery life, signal coverage) Location is important Dynamism in the physical environments Unexpected disruption of services …
    74. 74. Characteristics of IoT data Stream data (depends on time) Transient nature Almost always related to a phenomenon or quality in our physical environments Large amount Quality in many situations cannot be assured (e.g., accuracy and precision) Abstraction levels (e.g., raw, inferred or derived) …
    75. 75. Utilise semantics Find all available resources (which can provide data) and data related to “Ro o m A (which is an ” object in the linked data)?  What is “Room A”? What is its location? returns “location” data  What type of data is available for “Room A” or that “location”? (s e ns o r c a te g o ry ty p e s ) Predefined Rules can be applied based on available data  (TempRoom_A > 80°C) AND (SmokeDetectedRoom_A position==TRUE)  FireEventRoom_A  Learning these rules needs data mining or pattern recognition techniques
    76. 76. Semantic modelling Lightweight: experiences show that a lightweight ontology model that well balances expressiveness and inference complexity is more likely to be widely adopted and reused; also large number of IoT resources and huge amount of data need efficient processing Compatibility: an ontology needs to be consistent with those well designed, existing ontologies to ensure compatibility wherever possible. Modularity: modular approach to facilitate ontology evolution, extension and integration with external ontologies.
    77. 77. Existing models for resources and data W3C Semantic Sensor Network Incubator Group’s S N ontology (mainly for sensors and S sensor networks, observation and measurement, and platforms and systems) Quantity Kinds and Units  Used together with the SSN ontology  based on QUDV model OMG SysML(TM)  Working group of the SysML 1.2 Revision Task Force (RTF) and W3C Semantic Sensor Network Incubator Group
    78. 78. Existing models for services OWL-S and WSMO are heavy weight models: practical use? Minimal service model  Deprecated  Procedure-Oriented Service Model (POSM) and Resource- Oriented Service Model (ROSM): two different models for different service technologies  Defines Operations and Messages  No profile, no grounding SAWSDL: mixture of XML, XML schema, RDF and OWL hRESTS and SA-REST: mixture of HTML and reference to a semantic model; sensor services are not anticipated to have HTML
    79. 79. W3C’S SSN ontologyDiagram adapted from SSN report
    80. 80. Some existing IoT models andontologies FP7 IoT-A project’s Entity-Resource-Service ontology  A set of ontologies for entities, resources, devices and services  Based on the SSN and OWL-S ontology FP7 IoT.est project’s service description framework  A modular approach for designing a description framework  A set of ontologies for IoT services, testing and QoS/QoI
    81. 81. IoT-A resource modelDiagram adapted from IoT-A project D2.1
    82. 82. IoT-A resource descriptionDiagram adapted from IoT-A project D2.1
    83. 83. IoT-A service modelDiagram adapted from IoT-A project D2.1
    84. 84. IoT-A service descriptionDiagram adapted from IoT-A project D2.1
    85. 85. Service modelling in IoT.estDiagrams adapted from Iot.est D3.1
    86. 86. IoT.est service profile highlight ServiceType class represents the service technologies: RESTful and SOAP/WSDL services. serviceQos and serviceQoI are defined as subproperty of serviceParameter; they link to concepts in the QoS/QoI ontology. serviceArea: the area where the service is provided; different from the sensor observation area Links to the IoT resources through “exposedB property y” Future extension:  serviceNetwork, servicePlatform and serviceDeployment  Service lifecycle, SLA…
    87. 87. Linked data principles  using URI’s as names for things: Everything is addressed using unique URI’s.  using HTTP URI’s to enable people to look up those names: All the URI’s are accessible via HTTP interfaces.  provide useful RDF information related to URI’s that are looked up by machine or people;  including RDF statements that link to other URI’s to enable discovery of other related concepts of the Web of Data: The URI’s are linked to other URI’s.
    88. 88. Linked data in IoT Using URI’s as names for things; - URI’s for naming M2M resources and data (and also streaming data); Using HTTP URI’s to enable people to look up those names; - Web-level access to low level sensor data and real world resource descriptions (gateway and middleware solutions); Providing useful RDF information related to URI’s that are looked up by machine or people; - publishing semantically enriched resource and data descriptions in the form of linked RDF data; Including RDF statements that link to other URI’s to enable discovery of other related things of the web of data; - linking and associating the real world data to the existing data on the Web;
    89. 89. Linked data layer for not only IoT…Images from Stefan Decker,; linked data diagram:
    90. 90. Creating and using linked sensor data
    91. 91. Sensor discovery using linked sensordata
    92. 92. Semantics in IoT - reality If we create an Ontology our data is interoperable  Reality: there are/could be a number of ontologies for a domain  Ontology mapping  Reference ontologies  Standardisation efforts Semantic data will make my data machine-understandable and my system will be intelligent.  Reality: it is still meta-data, machines don’t understand it but can interpret it. It still does need intelligent processing, reasoning mechanism to process and interpret the data. It’s a Hype! Ontologies and semantic data are too much overhead; we deal with tiny devices in IoT.  Reality: Ontologies are a way to share and agree on a common vocabulary and knowledge; at the same time there are machine-interpretable and represented in interoperable and re-usable forms;  You don’t necessarily need to add semantic metadata in the source- it could be added to the data at a later stage (e.g. in a gateway);
    93. 93. Part 3: Semantic Sensor Web and PerceptionImage source:; CISCO 93
    94. 94. Introducing the Sensor Web
    95. 95. What is the Sensor Web? Sensor Web is an additional layer connecting sensor networks to the World Wide Web. Enables an interoperable usage of sensor resources by enabling web based discovery, access, tasking, and alerting. Enables the advancement of cyber-physical applications through improved situation awareness.
    96. 96. Why is the Sensor Web important? In general  Enable tight coupling of the cyber and physical world In relation to IoT  Enable shared situation awareness (or context) between devices/things
    97. 97. Bridging the Cyber-Physical DividePsyleron’s Mind-Lamp (Princeton U),connections between the mind and thephysical world. MIT’s Fluid Interface Group: wearable device with a projector for deep interactions with the environment Neuro Skys mind-controlled headset to play a video game.
    98. 98. Bridging the Cyber-Physical Divide FitBit Community allows the automated collection and sharing of health-related data, goals, and achievementsFoursquare is an online application whichintegrates a persons physical location andsocial network. Community of enthusiasts that share experiences of self-tracking and measurement.
    99. 99. Bridging the Cyber-Physical Divide Tweeting Sensors sensors are becoming social
    100. 100. How do we design the Sensor Web? Integration through shared semantics  OGC Sensor Web Enablement  W3C SSN ontology and Semantic Annotation Interpretation through integration of heterogeneous data and reasoning with prior knowledge  Semantic Perception/Abstraction  Linked Open Data as prior knowledge Scale through distributed local interpretation  “intelligence at the edge”
    101. 101. OGC Sensor Web Enablement
    102. 102. Role of OGC SWE
    103. 103. Vision of Sensor Web Quickly discover sensors (secure or public) that can meet my needs – location, observables, quality, ability to task Obtain sensor information in a standard encoding that is understandable by me and my software Readily access sensor observations in a common manner, and in a form specific to my needs Task sensors, when possible, to meet my specific needs Subscribe to and receive alerts when a sensor measures a particular phenomenon
    104. 104. Principles of Sensor Web Sensors will be web accessible Sensors and sensor data will be discoverable Sensors will be self-describing to humans and software (using a standard encoding) Most sensor observations will be easily accessible in real time over the web
    105. 105. OGC SWE Services Sensor Observation Service (SOS)  access sensor information (SensorML) and sensor observations (O&M Sensor Planning Service (SPS)  task sensors or sensor systems Sensor Alert Service (SAS)  asynchronous notification of sensor events (tasks, observation of phenomena) Sensor Registries  discovery of sensors and sensor data
    106. 106. OGC SWE Services
    107. 107. OGC SWE Languages Sensor Model Language (SensorML)  Models and schema for describing sensor characteristics Observation & Measurement (O&M)  Models and schema for encoding sensor observations
    108. 108. OCG SWE Observation
    109. 109. Semantic Sensor WebOGC Sensor Web Enablement RDF OWL
    110. 110. Sensor Web + Semantic WebSemantic Web Sensor WebThe web of data where web content is processed The internet of things made up of Wireless Sensorby machines, with human actors at the end of the Networks, RFID, stream gauges, orbiting satellites,chain. weather stations, GPS, traffic sensors, ocean buoys, animal and fish tags, cameras, habitat monitors,The web as a huge, dynamic, evolving database recording data from the physical world.of facts, rather than pages, that can be interpretedand presented in many ways (mashups). Today there are 4 billion mobile sensing devices plus even more fixed sensors. The US NationalFundamental importance of ontologies to describe Research Council predicts that this may grow tothe fact that represents the data. RDF(S) trillions by 2020, and they are increasingly connectedemphasises labelled links as the source of meaning: by internet and Web protocols.essentially a graph model . A label (URI) uniquelyidentifies a concept. Record observations of a wide variety of modalities: but a big part is time-series‟ of numericOWL emphasises inference as the source of measurements.meaning: a label also refers to a package of logicalaxioms with a proof theory. The Open Geospatial Consortium has some web- service standards for shared data access (SensorUsually, the two notions of meaning fit. Web Enablement).Goal to combine information and services for Goal is to open up access to real-time and archivaltargeted purpose and new knowledge data, and to combine in applications.
    111. 111. So, what is a Semantic Sensor Web? Reduce the difficulty and open up sensor networks by:  Allowing high-level specification of the data collection process;  Across separately deployed sensor networks;  Across heterogeneous sensor types; and  Across heterogeneous sensor network platforms;  Using high-level descriptions of sensor network capability; and  Interfacing to data integration methods using similar query and capability descriptions. To create a Web of Real Time Meaning!
    112. 112. W3C SSN Incubator Group SSN-XG commenced: 1 March 2009 Chairs:  Amit Sheth, Kno.e.sis Center, Wright State University  Kerry Taylor, CSIRO  Amit Parashar  Holger Neuhaus  Laurent Lefort, CSIRO Participants: 39 people from 20 organizations, including:  Universities in: US, Germany, Finland, Spain, Britain, Ireland  Multinationals: Boeing, Ericsson  Small companies in semantics, communications, software  Research institutes: DERI (Ireland), Fraunhofer (Germany), ETRI (Korea), MBARI (US), SRI International (US), MITRE (US), US Defense, CTIC (Spain), CSIRO (Australia), CESI (China)
    113. 113. W3C SSN Incubator GroupTwo main objectives:The development of an ontology for describingsensing resources and data, andThe extension of the SWE languages to supportsemantic annotations.
    114. 114. Sensor Standards Landscape
    115. 115. SSN Ontology  OWL 2 DL ontology  Authored by the XG participants  Edited by Michael Compton  Driven by Use Cases  Terminology carefully tracked to sources through annotation properties  Metrics  Classes: 117  Properties: 148  DL Expressivity:SSN Ontology – SIQ(D)
    116. 116. SSN Use Cases
    117. 117. SSN Use Cases
    118. 118. SSN Ontology
    119. 119. Stimulus-Sensor-Observation The SSO Ontology Design Pattern is developed following the principle of minimal ontological commitments to make it reusable for a variety of application areas. Introduces a minimal set of classes and relations centered around the notions of stimuli, sensor, and observations. Defines stimuli as the (only) link to the physical environment. Empirical science observes these stimuli using sensors to infer information about environmental properties and construct features of interest.
    120. 120. SSN Ontology Modules
    121. 121. SSN Ontology Modules
    122. 122. SSN Sensor A sensor can do (implements) sensing: that is, a sensor is any entity that can follow a sensing method and thus observe some Property of a FeatureOfInterest. Sensors may be physical devices, computational methods, a laboratory setup with a person following a method, or any other thing that can follow a Sensing
    123. 123. SSN Measurement Capability Collects together measurement properties (accuracy, range, precision, etc) and the environmental conditions in which those properties hold, representing a specification of a sensors capability in those conditions.
    124. 124. SSN Observation An Observation is a Situation in which a Sensing method has been used to estimate or calculate a value of a Property. Links to Sensing and Sensor describe what made the Observation and how; links to Property and Feature detail what was sensed; the result is the output of a Sensor; other metadata gives the time(s) and the quality. Different from OGC’s O&M, in which an “observation” is an act or event, although it also provides the record of the event.
    125. 125. Alignment with DOLCE
    126. 126. What SSN does not model Sensor types and models Networks: communication, topology Representation of data and units of measurement Location, mobility or other dynamic behaviours Animate sensors Control and actuation ….
    127. 127. Semantic Annotation of SWERecommendedtechnique via Xlinkattributes requires nochange to SWExlink:href - link toontology individualxlink:role - link toontology classxlink:arcrole - link toontology object property
    128. 128. How do we design the Sensor Web? Integration through shared semantics  OGC Sensor Web Enablement  W3C SSN ontology and Semantic Annotation Interpretation through integration of heterogeneous data and reasoning with prior knowledge  Semantic Perception/Abstraction  Linked Open Data as prior knowledge Scale through distributed local interpretation  “intelligence at the edge”
    129. 129. AbstractionAbstraction provides the ability to interpret and synthesize informationin a way that affords effective understanding and communication ofideas, feelings, perceptions, etc. between machines and people.
    130. 130. Abstraction  People are excellent at abstraction; of sensing and interpreting stimuli to understand and interact with the world.  The process of interpreting stimuli is called perception; and studying this extraordinary human capability can lead to insights for developing effective machine perception.
    131. 131. Abstraction conceptualization of “real-world” observe perceive “real-world”
    132. 132. Semantic Perception/AbstractionFundamental QuestionsWhat is perception, and how canwe design machines to perceive?What can we learn from cognitivemodels of perception?Is the Semantic Web up to the taskof modeling perception?
    133. 133. What is Perception? Perception is the act of  Abstracting  Explaining  Discriminating  Choosing
    134. 134. What can we learn fromCognitive Models ofPerception? Ulric Neisser (1976) Ulric Neisser (1976) Richard Gregory (1997) Richard Gregory (1997)  A-priori background knowledge is a key enabler  Perception is a cyclical, active process
    135. 135. Is Semantic Web up to the taskof modeling perception? Representation Heterogeneous sensors, sensing, and observation records Background knowledge (observable properties, objects/events, etc.) Inference Explain observations (hypothesis building) Focus attention by seeking additional stimuli (that discriminate between explanations) Difficult Issues to Overcome Perception is an infe re nc e to the be s t e x p la na tio n Handle streaming data Real-time processing (or nearly)
    136. 136. Both people and machines are capable of observingqualities, such as redness. observes Observer Quality * Formally described in a sensor/ontology (SSN ontology)
    137. 137. The ability to perceive is afforded through the use ofbackground knowledge, relating observable qualities toentities in the world. Quality * Formally described in inheres in domain ontologies (and knowledge bases) Entity
    138. 138. With the help of sophisticated inference, both people andmachines are also capable of perceiving entities, such asapples. perceives Perceiver Entity  the ability to degrade gracefully with incomplete information  the ability to minimize explanations based on new information  the ability to reason over data on the Web  fast (tractable)
    139. 139. Perceptual Inference Abductive Logic (e.g., Deductive Logic (e.g., PCT) OWL) high complexity (relatively) low complexity minimize explanations tractabl e Web degrade gracefully reasoning Perceptual Inference (i.e., abstraction)
    140. 140. The ability to perceive e ffic ie ntly is afforded through thecyclical exchange of information between observers andperceivers. Observer sends sends Traditionally called the observation focus Perceptual Cycle (or Active Perception) Perceiver
    141. 141. Neisser’s Perceptual Cycle
    142. 142. Cognitive Theories of Perception 1970’s – Perception is an active, cyclical process of exploration and interpretation. - N s s ie r’s Pe rc e p tio n Cy c le e 1980’s – The perception cycle is driven by background knowledge in order to generate and test hypotheses. - Ric ha rd G re g o ry (o p tic a l illus io ns ) 1990’s – In order to effectively test hypotheses, some observations are more informative than others. - N rwic h’s Entro p y The o ry o f Pe rc e p tio n o
    143. 143. Key InsightsBackground knowledge plays a crucial role in perception; what weknow (or think we know/believe) influences our perception of theworld.Semantics will allow us to realize computational models ofperception based on background knowledge.Contemporary IssuesInternet/Web expands our background knowledge to a globalscope; thus our perception is global in scopeSocial networks influence our knowledge and beliefs, thusinfluencing our perception
    144. 144. Integrated together, we have an general model – capable ofabstraction – relating observers, perceivers, and backgroundknowledge. observes Observer Quality sends sends observation inheres in focus perceives Perceiver Entity
    145. 145.  Ontology of Perception – as an extension of SSN Provides abstraction of sensor data through perceptual inference of semantically annotated data
    146. 146. Prior Knowledge W3C SSN Ontology Bi-partite Graph Prior knowledge conformant to SSN ontology (left),structured as a bipartite graph (right)
    147. 147. Semantics of ExplanationEx p la na tio n is the act of accounting for sensory observations (i.e.,abstraction); often referred to as hypothesis building.Observed Property: A property that has been observed.   ObservedProperty ≡ ∃ssn:observedProperty—.{o1} ⊔ … ⊔ ∃ssn:observedProperty—.{on} Explanatory Feature: A feature that explains the set of observedproperties.  ExplanatoryFeature ≡ ∃ssn:isPropertyOf—.{p1} ⊓ … ⊓ ∃ssn:isPropertyOf—.{pn} 
    148. 148. Semantics of Explanation Example Assume the properties elevated blood pressure and palpitations have been observed, and encoded in RDF (conformant with SSN):   ssn:Observation(o1), ssn:observedProperty(o1, elevated blood pressure) ssn:Observation(o2), ssn:observedProperty(o2, palpitations)   Given these observations, the following ExplanatoryFeature class is constructed: ExplanatoryFeature ≡ ∃ssn:isPropertyOf—.{elevated blood pressure} ⊓ ∃ssn:isPropertyOf—.{palpitations} Given the KB, executing the query ExplanatoryFeature(?y) can infer the features, Hypertension and Hyperthyroidism, as explanations: ExplanatoryFeature(Hypertension) ExplanatoryFeature(Hyperthyroidism)
    149. 149. Semantics of DiscriminationDis c rim ina tio n is the act of deciding how to narrow down the multitude ofexplanatory features through further observation.Expected Property: A property is e x p e c te d with respect to (w.r.t.) a setof features if it is a property of every feature in the set. ExpectedProperty ≡ ∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ∃ssn:isPropertyOf.{fn}  NotApplicable Property: A property is no t-a p p lic a ble w.r.t. a set offeatures if it is not a property of any feature in the set. NotApplicableProperty ≡ ¬∃ssn:isPropertyOf.{f1} ⊓ … ⊓ ¬∃ssn:isPropertyOf.{fn}Discriminating Property: A property is d is c rim ina ting w.r.t. a set offeatures if it is neither expected nor not-applicable. DiscriminatingProperty ≡ ¬ExpectedProperty ⊓ ¬NotApplicableProperty
    150. 150. Semantics of Discrimination Example Given the explanatory features from the previous example, Hypertension and Hyperthyroidism, the following classes are constructed: ExpectedProperty ≡ ∃ssn:isPropertyOf.{Hypertension} ⊓ ∃ssn:isPropertyOf.{Hyperthyroidism}   NotApplicableProperty ≡ ¬∃ssn:isPropertyOf.{Hypertension} ⊓ ¬∃ssn:isPropertyOf.{Hyperthyroidism}   Given the KB, executing the query DiscriminatingProperty(?x) can infer the property clammy skin as discriminating:   DiscriminatingProperty(clammy skin)
    151. 151. How do we design the Sensor Web? Integration through shared semantics  OGC Sensor Web Enablement  W3C SSN ontology and Semantic Annotation Interpretation through integration of heterogeneous data and reasoning with prior knowledge  Semantic Perception/Abstraction  Linked Open Data as prior knowledge Scale through distributed local interpretation  “intelligence at the edge”
    152. 152. Efficient Algorithms for IntellegO Use of OWL-DL reasoner too resource-intensive for use in resource constrained devices (such as sensor nodes, mobile phones, IoT devices)  Runs out of resources for problem size (prior knowledge) > 20 concepts  Asymptotic complexity: O(n3) [Experimentally determined] To enable their use on resource-constrained devices, we now describe algorithms for efficient inference of explanation and discrimination. These algorithms use bit vector encodings and operations, leveraging a-priori knowledge of the environment.
    153. 153. Efficient Algorithms for IntellegO Semantic (RDF) Bit Vector Encoding Encoding Lower Lift First, developed lifting and lowering algorithms to translate between RDF and bit vector encodings of observations.
    154. 154. Efficient Algorithms for IntellegO Explanation Algorithm Utilize bit vector operators to efficiently compute explanation and discrimination Explanation: Use of the bit vector AND operation to discover and d is m is s those features that cannot explain the set ofDiscrimination Algorithm observed properties Discrimination: Use of the bit vector AND operation to discover and indirectly a s s e m ble those properties that discriminate between a set of explanatory features. The discriminating properties are those that are determined to be neither expected nor not-applicable
    155. 155. Efficient Algorithms for IntellegOEvaluation: The bit vector encodings and algorithms yield significant andnecessary computational enhancements – including asymptotic order ofmagnitude improvement, with running times reduced from minutes tomilliseconds, and problem size increased from 10’s to 1000’s.
    156. 156. Adoption of SSN
    157. 157. SSN Applications
    158. 158. Linked Sensor Data Linked Sensor Data (~2 Billion Statements)
    159. 159. Sensor Discovery Application Query w/ location name to find nearby sensors
    160. 160. SSN Applications Applications of SSN Weather Rescue Healthcare
    161. 161. SSN Application: Weather 50% savings in sensing resource requirements during the detection of a blizzard Order of magnitude resource savings between storing observations vs. relevant abstractions
    162. 162. SSN Application: Fire Detection Weather Application SECURE: Semantics-empowered Rescue Environment (detect different types of fires) DEMO:
    163. 163. SSN Application: Health Care MOBILEMD: Mobile app to help reduce re- admission of patients with Chronic Heart Failure
    164. 164. SSN Application: Health CarePassive Monitoring Phase Passive Monitoring Phase Observed Symptoms Possible Explanations • Abnormal heart rate • Panic Disorder • Clammy skin • Hypoglycemia • Hyperthyroidism • Heart Attack • Septic Shock Passive Sensors – heart rate, galvanic skin response
    165. 165. SSN Application: Health CareActive Monitoring PhaseActive Monitoring Phase Are you feeling lightheaded? Are you feeling lightheaded? yes yes Are you have trouble taking deep Are you have trouble taking deep Observed Symptoms Possible Explanations breaths? breaths? • Abnormal heart rate • Panic Disorder yes yes • Hypoglycemia • Clammy skin • Lightheaded • Hyperthyroidism Do you have low blood pressure? • Trouble breathing • Heart Attack Do you have low blood pressure? • Low blood pressure • Septic Shock yes yes Have you taken your Methimazole Have you taken your Methimazole medication? medication? no no Active Sensors – blood pressure, weight scale, pulse
    166. 166. Future work Creating ontologies and defining data models are not enough  tools to create and annotate data  Tools for publishing linked IoT data Designing lightweight versions for constrained environments  think of practical issues  make it as much as possible compatible and/or link it to the other existing ontologies Linking to domain knowledge and other resources  Location, unit of measurement, type, theme, …  Linked-data  URIs and naming
    167. 167. Some of the open issues Efficient real-time IoT resource/service query/discovery  Directory  Indexing Abstraction of IoT data  Pattern extraction  Perception creation IoT service composition and compensation  Integration with existing Web services  Service adaptation
    168. 168. Selected references Payam Barnaghi, Wei Wang, Cory Henson, Kerry Taylor, "Semantics for the Internet of Things: early progress and back to the future", (to appear) International Journal on Semantic Web and Information Systems (special issue on sensor networks, Internet of Things and smart devices), 2012. Atzori, L., Iera, A. & Morabito, G. , “The Internet of Things: A survey”, Computer Networks, Volume 54, Issue 15, 28 October 2010, 2787- 2805. Suparna De, Tarek Elsaleh, Payam Barnaghi , Stefan Meissner, "An Internet of Things Platform for Real-World and Digital Objects", Journal of Scalable Computing: Practice and Experience, vol 13, no.1, 2012. Suparna De, Payam Barnaghi, Martin Bauer, Stefan Meissner, "Service modelling for the Internet of Things", in Proceedings of the Conference on Computer Science and Information Systems (FedCSIS), pp.949-955, Sept. 2011. Cory Henson, Krishnaprasad Thirunarayan, and Amit Sheth, “An Efficient Bit Vector Approach to Semantics-based Machine Perception in Resource-Constrained Devices,‘” In: Proceedings of 11th International Semantic Web Conference (ISWC 2012), Boston, Massachusetts, USA, November 11-25, 2012. Cory Henson, Amit Sheth, and Krishnaprasad Thirunarayan, “Semantic Perception: Converting Sensory Observations to Abstractions”, IEEE Internet Computing, Special Issue on Context-Aware Computing, March/April 2012. Cory Henson, Krishnaprasad Thirunarayan, Amit Sheth, “An Ontological Approach to Focusing Attention and Enhancing Machine Perception on the Web.,” Applied Ontology, vol. 6(4), pp.345-376, 2011. Payam Barnaghi, Frieder Ganz, Cory Henson, Amit Sheth, “Computing Perception from Sensor Data”, In proceedings of the 2012 IEEE Sensors Conference, Taipei, Taiwan, October 28-31, 2012. Michael Compton et al, “The SSN Ontology of the W3C Semantic Sensor Network Incubator Group”, Journal of Web Semantics, 2012. Harshal Patni, Cory Henson, and Amit Sheth , “Linked Sensor Data”, in Proceedings of 2010 International Symposium on Collaborative Technologies and Systems (CTS 2010), Chicago, IL, May 17-21, 2010. Amit Sheth, Cory Henson, and Satya Sahoo , “Semantic Sensor Web IEEE Internet Computing”, vol. 12, no. 4, July/August 2008, pp. 78- 83. Wei Wang, Payam Barnaghi, Gilbert Cassar, Frieder Ganz, Pirabakaran Navaratnam, "Semantic Sensor Service Networks", (to appear) in Proceedings of the IEEE Sensors 2012 Conference, Taipei, Taiwan, October 2012. Wang W, De S, Toenjes R, Reetz E, Moessner K, "A Comprehensive Ontology for Knowledge Representation in the Internet of Things", International Workshop on Knowledge Acquisition and Management in the Internet of Things (KAMIoT 2012) in conjunction with IEE
    169. 169. Some useful links related to IoT Internet of Things, ITU  IoT Comic Book  Internet of Things Europe, Internet of Things Architecture (IOT-A)  W3C Semantic Sensor Networks  Kno.e.sis Semantic Sensor Web Group 