Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Tutorial on RDF Stream
Processing 2016
M.I. Ali, J-P Calbimonte, D. Dell'Aglio,
E. Della Valle, and A. Mauri
http://stream...
http://streamreasoning.org/events/rsp2016
Share, Remix, Reuse — Legally
 This work is licensed under the Creative Commons...
http://streamreasoning.org/events/rsp2016
RSP for developers
• RDF Streams in practice
• RSP Query Engines
• Developing wi...
http://streamreasoning.org/events/rsp2016
RDF Streams in Practice
4
http://streamreasoning.org/events/rsp2016
RSP: Keep the data moving
5
Process data in-stream
Not required to store
Active ...
http://streamreasoning.org/events/rsp2016 6
RDF Stream
…
Gi
Gi+1
Gi+2
…
Gi+n
…
unboundedsequence
Gi {(s1,p1,o1),
(s2,p2,o2...
http://streamreasoning.org/events/rsp2016 7
Linked Data on the Web
Web of Data
Linked Data
W3C Standards: RDF, SPARQL, etc.
http://streamreasoning.org/events/rsp2016 8
Linked Data principles for RDF streams?
e.g. publish sensor data as RDF/Linked...
http://streamreasoning.org/events/rsp2016 9
(Sensor) Data Streams on the Web
9
http://mesowest.utah.edu/
http://earthquake...
http://streamreasoning.org/events/rsp2016 10
RDF Streams before RDF Streams
http://richard.cyganiak.de/2007/10/lod/
2011
L...
http://streamreasoning.org/events/rsp2016 11
Sensor Data & Linked Data
11
Zip Files
Number of Triples
Example: Nevada data...
http://streamreasoning.org/events/rsp2016 12
Sensor Data & Linked Data
12
<http://knoesis.wright.edu/ssw/MeasureData_Preci...
http://streamreasoning.org/events/rsp2016 13
RDF Streams before RDF Streams
i.e. just use RDF
:observation1 rdf:type om-ow...
http://streamreasoning.org/events/rsp2016 14
Feed an RDF Stream to a RSP engine
Ad-hoc
Conversion to
RDF
Live Non-RDF Stre...
http://streamreasoning.org/events/rsp2016 15
Feed an RDF Stream to C-SPARQL
public class SensorsStreamer extends RdfStream...
http://streamreasoning.org/events/rsp2016 16
Actor Model
Actor
1
Actor
2
m No shared mutable state
Avoid blocking operator...
http://streamreasoning.org/events/rsp2016 17
RDF Stream
object DemoStreams {
...
def streamTriples={
Iterator.from(1) map{...
http://streamreasoning.org/events/rsp2016 18
RDF Stream
… other issues:
Graph implementation?
Timestamps: application vs s...
http://streamreasoning.org/events/rsp2016 19
Data stream characteristics
19
Data regularity
• Raw data typically collected...
http://streamreasoning.org/events/rsp2016 20
Feed an RDF Stream to a RSP engine
Conversion to
RDF
Live Non-RDF Streams
RDF...
http://streamreasoning.org/events/rsp2016 21
R2RML Mappings
21
:ObsValueMap
rr:subjectMap [
rr:template "http://opensense....
http://streamreasoning.org/events/rsp2016 22
An example: TripleWave
Running modes
Sources
http://streamreasoning.org/events/rsp2016 23
RDF Streams in W3C RSP
:g1 {:axel :isIn :RedRoom. :darko :isIn :RedRoom}
{:g1...
http://streamreasoning.org/events/rsp2016 24
RSP Engine Implementations
http://streamreasoning.org/events/rsp2016
Existing RSP systems (oversimplified!)
 C-SPARQL: RDF Store + Stream processor
...
http://streamreasoning.org/events/rsp2016
Existing RSP systems (oversimplified!)
 SPARQLstream: Ontology-based stream que...
http://streamreasoning.org/events/rsp2016
Classification of existing systems
Model
Continuous
execution
Union,Join,
Option...
http://streamreasoning.org/events/rsp2016 28
C-SPARQL
http://streamreasoning.org/events/rsp2016 29
A Reminder of SPARQL
http://streamreasoning.org/events/rsp2016 30
Where C-SPARQL Extends SPARQL
http://streamreasoning.org/events/rsp2016 31
C-SPARQL Language
Query and Stream Registration
http://streamreasoning.org/events/rsp2016 32
C-SPARQL Language
Query and Stream Registration
 All C-SPARQL queries over R...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
Query registration - Example
 Using the social stream fb, Who...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
Stream registration - Example
 Results of a C-SPARQL query ca...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
Stream Registration - Notes
 The output is constructed in the...
http://streamreasoning.org/events/rsp2016 36
C-SPARQL Language
FROM STREAM Clause
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
FROM STREAM Clause
 FROM STREAM clauses are similar to SPARQL...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
FROM STREAM Clause - windows
 physical: a given number of tri...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
FROM STREAM Clause - Example
 Using the social stream fb, how...
http://streamreasoning.org/events/rsp2016 41
C-SPARQL Language
C-SPARQL reports only snapshots
t
t+10
t+20
t+30
t+40
t+50
...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
Multiple FROM STREAM Clause - Example
 Using the social strea...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
Query Chaining
 A C-SPARQL query Q1 registered using the STRE...
http://streamreasoning.org/events/rsp2016 44
C-SPARQL Language
TimeStamp Function
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
TimeStamp Function – Syntax and Semantics
 The timestamp of a...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
TimeStamp Function - Example
 Who is “following” whom?
REGIST...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
Accessing background Information
 C-SPARQL allows for asking ...
http://streamreasoning.org/events/rsp2016
C-SPARQL Language
C-SPARQL queries and reasoning - example
 Memo
• posts is a s...
http://streamreasoning.org/events/rsp2016 49
Introduction
C-SPARQL Engine Architecture
 Simple, modular
architecture
 It...
http://streamreasoning.org/events/rsp2016
Introduction
C-SPARQL Engine Features at a glance 1/3
 In-memory RDF stream Pro...
http://streamreasoning.org/events/rsp2016
Introduction
C-SPARQL Engine Features at a glance 2/3
 Extensible Middleware
• ...
http://streamreasoning.org/events/rsp2016
Introduction
C-SPARQL Engine Features at a glance 3/3
 Known limitations
• larg...
http://streamreasoning.org/events/rsp2016
Introduction
C-SPARQL Engine as general RSP
 RSP-services proposes a unified in...
http://streamreasoning.org/events/rsp2016
Resources
 Read out more
• C-SPARQL semantics
– Davide Francesco Barbieri, Dani...
http://streamreasoning.org/events/rsp2016 55
SPARQL Stream &
Morph-streams
http://streamreasoning.org/events/rsp2016
Morph-streams: Overview
56
Query
rewriting
Query
Processing
Client
SPARQLStream
...
http://streamreasoning.org/events/rsp2016
SPARQLStream Language
57
FROM NAMED STREAM
ISTREAM
DSTREAM
RSTREAM
WINDOW
Underl...
http://streamreasoning.org/events/rsp2016
SPARQLStream Language
 NamedStream  ‘FROM’ [‘NAMED’] ‘STREAM’ StreamIRI ‘[’ Wi...
http://streamreasoning.org/events/rsp2016
SPARQLStream: examples
59
PREFIX sr4ld: <http://www.streamreasoning.org/ontologi...
http://streamreasoning.org/events/rsp2016
Underlying Query Processors
Esper
• CEP/DSMS
• EPL language
SNEE
• DSMS/Sensor N...
http://streamreasoning.org/events/rsp2016
Morph-streams: Overview
61
Query
rewriting
Query
Processing
Client
SPARQLStream
...
http://streamreasoning.org/events/rsp2016
3rd: Mapping the two models
62
Observation
Sensor
Person
Roomwhere
who
observes
...
http://streamreasoning.org/events/rsp2016
R2RML – There is a recommendation!
63
We can use the W3C
recommendation
http://streamreasoning.org/events/rsp2016
R2RML - Overview
64
http://streamreasoning.org/events/rsp2016
Encoding in R2RML
65
:triplesMap a rr:TriplesMap;
rr:logicalTable [ rr:tableName...
http://streamreasoning.org/events/rsp2016
Underlying Query Processors
66
SELECT ?proximity
FROM STREAM <http://streamreaso...
http://streamreasoning.org/events/rsp2016
Underlying query processors
Features Esper SNEE GSN Cosm/Xively
Projection ✔ ✔ ✔...
http://streamreasoning.org/events/rsp2016
Morph-streams: With reasoning!
68
Query
rewriting
Query
Processing
Client
SPARQL...
http://streamreasoning.org/events/rsp2016
Reasoning with Morph-streams
 Rewriting the SPARQLStream queries:
69
SELECT ?x
...
http://streamreasoning.org/events/rsp2016
Morph-streams: With reasoning!
70
Query
rewriting
Query
Processing
Client
SPARQL...
http://streamreasoning.org/events/rsp2016
Now some code
Morph-streams:
 Coded in Scala
 JAR bundle, use it from Scala or...
http://streamreasoning.org/events/rsp2016
Code examples
 Parse SPARQLStream
val query= “PREFIX sr4ld: <…>. SELECT ?a …”
v...
http://streamreasoning.org/events/rsp2016
Code examples
 Register and Pull
val queryid= adapter.registerQuery(query,mappi...
http://streamreasoning.org/events/rsp2016 74
Querying RSPs in Practice
http://streamreasoning.org/events/rsp2016 75
ExecContext context=new ExecContext(HOME, false);
String queryString =" SELEC...
http://streamreasoning.org/events/rsp2016 76
CQELS fed by a TripleWave WebSocket
val conf = ConfigFactory.load.getConfig("...
http://streamreasoning.org/events/rsp2016
Similar models,
similar (not equals!) query languages
77
SELECT ?sensor
FROM NAM...
http://streamreasoning.org/events/rsp2016 78
Query using SQL on Streams
Model
Continuous
execution
Union,Join,
Optional,
F...
http://streamreasoning.org/events/rsp2016 79
W3C RSP-CG: RSP-QL
PREFIX e: <http://somevocabulary.org/>
PREFIX s: <http://s...
http://streamreasoning.org/events/rsp2016 80
RSP Communication
http://streamreasoning.org/events/rsp2016 81
RDF Stream Processing
RSP
Engine
RDF
graphs
input RDF streams streams of resu...
http://streamreasoning.org/events/rsp2016 82
Reactive Systems
Event-Driven
Jonas Boner. Go Reactive: Event-Driven, Scalabl...
http://streamreasoning.org/events/rsp2016 83
Actor Model
Actor
1
Actor
2
m No shared mutable state
Avoid blocking operator...
http://streamreasoning.org/events/rsp2016 84
RDF Streams: Actors
val sys=ActorSystem.create("system")
val consumer=sys.act...
http://streamreasoning.org/events/rsp2016 85
RSP Producer & Consumer
Proce
ssor
faster producers >> slower processor/consu...
http://streamreasoning.org/events/rsp2016 87
Dynamic Push-Pull
Producer
Consumer
m
data flow
demand flow
Push when consume...
http://streamreasoning.org/events/rsp2016 88
Evaluation: throughput
Basic dynamic pull push
On top of CQELS
Limitations of...
http://streamreasoning.org/events/rsp2016 89
Reactive RSP workflows
Morph
Streams
CSPARQL
s
Etali
s
TrOWL
s
s CQELS
Dyn
am...
http://streamreasoning.org/events/rsp2016 90
RSPs and the Linked Data Principles
http://streamreasoning.org/events/rsp2016 91
URIs as Names of Things
http://mysensorweb.me/mytemperature/20151110Z10:00:
0...
http://streamreasoning.org/events/rsp2016 92
HTTP URIs
http://mysensorweb.me/mytemperature/latest
Internet of Things
How a...
http://streamreasoning.org/events/rsp2016 93
De-referenceable URIs
GET http://mysensorweb.me/mytemperature/latest
:Obs1 a ...
http://streamreasoning.org/events/rsp2016 94
Link to other URIs
• Broken links?
• Mix streaming and stored data
• Persist ...
http://streamreasoning.org/events/rsp2016 95
For Java/Scala developers
http://streamreasoning.org/events/rsp2016 96
Semantic Web Devs
Have fun
Love challenges
Need cool tools
RDF4J
http://streamreasoning.org/events/rsp2016 97
Scala: Functions and Objects
JVM language
Both object and functional oriented...
http://streamreasoning.org/events/rsp2016 98
RDF in Jena: in Scala
String personURI = "http://somewhere/JohnSmith";
Model ...
http://streamreasoning.org/events/rsp2016 99
Some more RDF
99
String personURI = "http://somewhere/JohnSmith";
String give...
http://streamreasoning.org/events/rsp2016 100
Some more RDF in Jena
implicit val m=createDefaultModel
val ex="http://examp...
http://streamreasoning.org/events/rsp2016 101
Exploring an RDF Graph
ArrayList<String> names=new ArrayList<String>();
Node...
http://streamreasoning.org/events/rsp2016 102
http://streamreasoning.org/events/rsp2016 103
Query with SPARQL
val queryStr = """select distinct ?Concept
where {[] a ?Co...
http://streamreasoning.org/events/rsp2016 104
Muchas Gracias!
Tutorial on RDF Stream
Processing 2016
M.I. Ali, J-P Calbimonte, D. Dell'Aglio,
E. Della Valle, and A. Mauri
http://stream...
Upcoming SlideShare
Loading in …5
×

RDF Stream Processing Tutorial: RSP implementations

1,254 views

Published on

RDF Stream Processing Tutorial: RSP implementations

Published in: Internet

RDF Stream Processing Tutorial: RSP implementations

  1. 1. Tutorial on RDF Stream Processing 2016 M.I. Ali, J-P Calbimonte, D. Dell'Aglio, E. Della Valle, and A. Mauri http://streamreasoning.org/events/rsp2016 RDF Stream Processing Implementations Jean-Paul Calbimonte jean-paul.calbimonte@hevs.ch http://jeanpi.org @jpcik
  2. 2. http://streamreasoning.org/events/rsp2016 Share, Remix, Reuse — Legally  This work is licensed under the Creative Commons Attribution 3.0 Unported License.  You are free: • to Share — to copy, distribute and transmit the work • to Remix — to adapt the work  Under the following conditions • Attribution — You must attribute the work by inserting – “[source http://streamreasoning.org/rsp2014]” at the end of each reused slide – a credits slide stating - These slides are partially based on “RDF Stream Processing 2014” by M. Balduini, J-P Calbimonte, O. Corcho, D. Dell'Aglio, E. Della Valle http://streamreasoning.org/rsp2014  To view a copy of this license, visit http://creativecommons.org/licenses/by/3.0/ 2
  3. 3. http://streamreasoning.org/events/rsp2016 RSP for developers • RDF Streams in practice • RSP Query Engines • Developing with an RSP Engine • Handling Results • RSP Services 3
  4. 4. http://streamreasoning.org/events/rsp2016 RDF Streams in Practice 4
  5. 5. http://streamreasoning.org/events/rsp2016 RSP: Keep the data moving 5 Process data in-stream Not required to store Active processing model input streams RSP queries/ rules output streams/events RDF Streams
  6. 6. http://streamreasoning.org/events/rsp2016 6 RDF Stream … Gi Gi+1 Gi+2 … Gi+n … unboundedsequence Gi {(s1,p1,o1), (s2,p2,o2),…} [ti] 1+ triples implicit/explicit timestamp/interval RDF streams in theory How do I code this? Use Web standards?
  7. 7. http://streamreasoning.org/events/rsp2016 7 Linked Data on the Web Web of Data Linked Data W3C Standards: RDF, SPARQL, etc.
  8. 8. http://streamreasoning.org/events/rsp2016 8 Linked Data principles for RDF streams? e.g. publish sensor data as RDF/Linked Data? URIs as names of things HTTP URIs useful information when URI is dereferenced Link to other URIs users application s WEB Use RDF model to continuously query real-time data streams? static vs. streams one-off vs. continuous
  9. 9. http://streamreasoning.org/events/rsp2016 9 (Sensor) Data Streams on the Web 9 http://mesowest.utah.edu/ http://earthquake.usgs.gov/earthquakes/feed/v1.0/ http://swiss-experiment.ch • Monitoring • Alerts • Notifications • Hourly/daily update • Myriad of Formats • Ad-hoc access points • Informal description • Convention-semantics • Uneven use of standards • Manual exploration
  10. 10. http://streamreasoning.org/events/rsp2016 10 RDF Streams before RDF Streams http://richard.cyganiak.de/2007/10/lod/ 2011 Linked Sensor Data MetOffice AEMET
  11. 11. http://streamreasoning.org/events/rsp2016 11 Sensor Data & Linked Data 11 Zip Files Number of Triples Example: Nevada dataset -7.86GB in n-triples format -248MB zipped An example: Linked Sensor Data http://wiki.knoesis.org/index.php/LinkedSensorData
  12. 12. http://streamreasoning.org/events/rsp2016 12 Sensor Data & Linked Data 12 <http://knoesis.wright.edu/ssw/MeasureData_Precipitation_4UT01_2003_3_31_5_10_00> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#MeasureData> . <http://knoesis.wright.edu/ssw/MeasureData_Precipitation_4UT01_2003_3_31_5_10_00> <http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#floatValue> "30.0"^^<http://www.w3.org/2001/XMLSchema#float> . <http://knoesis.wright.edu/ssw/MeasureData_Precipitation_4UT01_2003_3_31_5_10_00> <http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#uom> <http://knoesis.wright.edu/ssw/ont/weather.owl#centimeters> . <http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://knoesis.wright.edu/ssw/ont/weather.owl#PrecipitationObservation> . <http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00> <http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#observedProperty> <http://knoesis.wright.edu/ssw/ont/weather.owl#_Precipitation> . <http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00> <http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#procedure> <http://knoesis.wright.edu/ssw/System_4UT01> . <http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00> <http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#samplingTime> <http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00> . <http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00> <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> <http://www.w3.org/2006/time#Instant> . <http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00> <http://www.w3.org/2006/time#inXSDDateTime> "2003-03-31T05:10:00-07:00^^http://www.w3.org/2001/XMLSchema#dateTime" . What do we get in these datasets? Nice triples What is measured Measurement Unit Sensor When is it measured
  13. 13. http://streamreasoning.org/events/rsp2016 13 RDF Streams before RDF Streams i.e. just use RDF :observation1 rdf:type om-owl:Observation . :observation1 om-owl:observedProperty weather:_AirTemperature . :observation1 om-owl:procedure :sensor1 . :observation1 om-owl:result :obsresult1 . :observation1 om-owl:resultTime "2015-01-01T10:00:01" :obsresult1 om-owl:floatValue 35.4 . Plain triples Where is the timestamp? :observation2 rdf:type om-owl:Observation . :observation2 om-owl:observedProperty weather:_AirTemperature . :observation2 om-owl:procedure :sensor1 . :observation2 om-owl:result :obsresult2 . :observation2 om-owl:resultTime "2015-01-01T10:00:02" :obsresult2 om-owl:floatValue 36.4 . What is the order in the RDF graph? Appended to a file? Or to some RDF dataset? How to store it?
  14. 14. http://streamreasoning.org/events/rsp2016 14 Feed an RDF Stream to a RSP engine Ad-hoc Conversion to RDF Live Non-RDF Streams RDF RDF datasets RSP Add (internal) timestamp on insertion What is currently done in most RSPs Continuous additions RDF + timestamps
  15. 15. http://streamreasoning.org/events/rsp2016 15 Feed an RDF Stream to C-SPARQL public class SensorsStreamer extends RdfStream implements Runnable { public void run() { .. while(true){ ... RdfQuadruple q=new RdfQuadruple(subject,predicate,object, System.currentTimeMillis()); this.put(q); } } } something to run on a thread timestamped triple the stream is “observable” Data structure, execution and callbacks are mixed Observer pattern Tightly coupled listener Added timestamp
  16. 16. http://streamreasoning.org/events/rsp2016 16 Actor Model Actor 1 Actor 2 m No shared mutable state Avoid blocking operators Lightweight objects Loose coupling communicate through messages mailbox state behaviornon-blocking response send: fire-forget Implementations: e.g. Akka for Java/Scala
  17. 17. http://streamreasoning.org/events/rsp2016 17 RDF Stream object DemoStreams { ... def streamTriples={ Iterator.from(1) map{i=> ... new Triple(subject,predicate,object) } } Data structure Infinite triple iterator Execution val f=Future(DemoStreams.streamTriples) f.map{a=>a.foreach{triple=> //do something }} Asynchronou s iteration Message passing f.map{a=>a.foreach{triple=> someSink ! triple }} send triple to actor Immutable RDF stream  avoid shared mutable state  avoid concurrent writes  unbounded sequence Ideas using akka actors Futures  non blocking composition  concurrent computations  work with not-yet- computed results Actors  message-based  share-nothing async  distributable
  18. 18. http://streamreasoning.org/events/rsp2016 18 RDF Stream … other issues: Graph implementation? Timestamps: application vs system? Serialization?  Loose coupling  Immutable data streams  Asynchronous message passing  Well defined input/output
  19. 19. http://streamreasoning.org/events/rsp2016 19 Data stream characteristics 19 Data regularity • Raw data typically collected as time series • Very regular structure. • Patterns can be exploited E.g. mobile NO2 sensor readings 29-02-2016T16:41:24,47,369,46.52104,6.63579 29-02-2016T16:41:34,47,358,46.52344,6.63595 29-02-2016T16:41:44,47,354,46.52632,6.63634 29-02-2016T16:41:54,47,355,46.52684,6.63729 ... Data order • Order of data is crucial • Time is the key attribute for establishing an order among the data items. • Important for indexing • Enables efficient time-based selection, filtering and windowing Timestamp Sensor Observed Value Coordinates
  20. 20. http://streamreasoning.org/events/rsp2016 20 Feed an RDF Stream to a RSP engine Conversion to RDF Live Non-RDF Streams RDF RDF datasets RSP Add (internal) timestamp on insertion Adding mappings to the data flow Continuous additions RDF + timestamps
  21. 21. http://streamreasoning.org/events/rsp2016 21 R2RML Mappings 21 :ObsValueMap rr:subjectMap [ rr:template "http://opensense.epfl.ch/data/ObsResult_NO2_{sensor}_{time}"]; rr:predicateObjectMap [ rr:predicate qu:numericalValue; rr:objectMap [ rr:column "no2"; rr:datatype xsd:float; ]]; rr:predicateObjectMap [ rr:predicate obs:uom; rr:objectMap [ rr:parentTriplesMap :UnitMap; ]]. :ObservationMap rr:subjectMap [ rr:template "http://opensense.epfl.ch/data/Obs_NO2_{sensor}_{time}"]; rr:predicateObjectMap [ rr:predicate ssn:observedProperty; rr:objectMap [ rr:constant opensense:NO2]]; URI of subject URI of predicate Object: colum name Column names in a template Can be used for mapping both databases, CSVs, JSON, etc
  22. 22. http://streamreasoning.org/events/rsp2016 22 An example: TripleWave Running modes Sources
  23. 23. http://streamreasoning.org/events/rsp2016 23 RDF Streams in W3C RSP :g1 {:axel :isIn :RedRoom. :darko :isIn :RedRoom} {:g1, prov:generatedAtTime, "2015-06-18T12:00:00Z"^^xsd:dateTime} :g2 {:axel :isIn :BlueRoom. } {:g2, prov:generatedAtTime, "2015-06-18T12:00:35"^^xsd:dateTime} :g3 {:minh :isIn :RedRoom. } {:g3, prov:generatedAtTime, "2015-06-18T12:02:07"^^xsd:dateTime} ... https://www.w3.org/community/rsp/ http://streamreasoning.github.io/RSP-QL/RSP_Requirements_Design_Document/ Graph-based Flexible time property RDF-friendly Flexible metadata :g_1 :startsAt "2015-06-18T12:00:00"^^xsd:dateTime :g_1 :endsAt "2015-06-18T13:00:00"^^xsd:dateTime :g_2 :validBetween [:startsAt "2015-06-18T12:00:00"^^xsd:dateTime; :endsAt "2015-06-18T13:00:00"^^xsd:dateTime] Intervals
  24. 24. http://streamreasoning.org/events/rsp2016 24 RSP Engine Implementations
  25. 25. http://streamreasoning.org/events/rsp2016 Existing RSP systems (oversimplified!)  C-SPARQL: RDF Store + Stream processor • Combined architecture  CQELS: Implemented from scratch. Focus on performance • Native + adaptive joins for static-data and streaming data 25 RDF Store Stream processor C-SPARQL query continuous results Native RSPCQELS query continuous results translator
  26. 26. http://streamreasoning.org/events/rsp2016 Existing RSP systems (oversimplified!)  SPARQLstream: Ontology-based stream query answering • Virtual RDF views, using R2RML mappings • SPARQL stream queries over the original data streams.  EP-SPARQL: Complex-event detection • SEQ, EQUALS operators  Instans: RETE-based evaluation 26 DSMS/CEPSPARQLStream query continuous results rewriter R2RML mappings Prolog engine EP-SPARQL query continuous results translator
  27. 27. http://streamreasoning.org/events/rsp2016 Classification of existing systems Model Continuous execution Union,Join, Optional, Filter Aggregates Timewindow Triplewindow R2Soperator Sequence, Co-ocurrence TA- SPARQL TA-RDF ✗ ✔ Limited ✗ ✗ ✗ ✗ tSPARQL tRDF ✗ ✔ ✗ ✗ ✗ ✗ ✗ Streaming SPARQL RDF Stream ✔ ✔ ✗ ✔ ✔ ✗ ✗ C-SPARQL RDF Stream ✔ ✔ ✔ ✔ ✔ Rstream only time function CQELS RDF Stream ✔ ✔ ✔ ✔ ✔ Istream only ✗ SPARQLStr eam (Virtual) RDF Stream ✔ ✔ ✔ ✔ ✗ ✔ ✗ EP- SPARQL RDF Stream ✔ ✔ ✔ ✗ ✗ ✗ ✔ Instans RDF ✔ ✔ ✔ ✗ ✗ ✗ ✗ 27Disclaimer: other features may be missing
  28. 28. http://streamreasoning.org/events/rsp2016 28 C-SPARQL
  29. 29. http://streamreasoning.org/events/rsp2016 29 A Reminder of SPARQL
  30. 30. http://streamreasoning.org/events/rsp2016 30 Where C-SPARQL Extends SPARQL
  31. 31. http://streamreasoning.org/events/rsp2016 31 C-SPARQL Language Query and Stream Registration
  32. 32. http://streamreasoning.org/events/rsp2016 32 C-SPARQL Language Query and Stream Registration  All C-SPARQL queries over RDF streams are continuous • Registered through the REGISTER statement  The output of queries is in the form of • Instantaneous tables of variable bindings • Instantaneous RDF graphs • RDF stream  Only queries in the CONSTRUCT form can be registered as generators of RDF streams  Composability: • Query results registered as streams can feed other registered queries just like every other RDF stream 32
  33. 33. http://streamreasoning.org/events/rsp2016 C-SPARQL Language Query registration - Example  Using the social stream fb, Who is where? REGISTER QUERY QWhoIsWhereOnFb AS PREFIX : <http://…/sr4ld2014-onto#> SELECT ?room ?person FROM STREAM <http://…/fb> [RANGE 1m STEP 10s] WHERE { ?person1 :posts [ :who ?person ; :where ?room ] . }  The resulting variable bindings has to be interpreted as an instantaneous. It expires as soon as the query is recomputed 33
  34. 34. http://streamreasoning.org/events/rsp2016 C-SPARQL Language Stream registration - Example  Results of a C-SPARQL query can be stream out for down stream queries REGISTER STREAM SWhoIsWhereOnFb AS PREFIX : <http://…/sr4ld2014-onto#> CONSTRUCT { ?person :isIn ?room } FROM STREAM <http://…/fb> [RANGE 1m STEP 10s] WHERE { ?person1 :posts [ :who ?person ; :where ?room ] . }  The resulting RDF triples are streamed out on an RDF stream • More details in the C-SPARQL Engine hands-on session 34
  35. 35. http://streamreasoning.org/events/rsp2016 C-SPARQL Language Stream Registration - Notes  The output is constructed in the format of an RDF stream.  Every query execution may produce from a minimum of zero triples to a maximum of an entire graph.  The timestamp is always dependent on the query execution time only, and is not taken from the triples that match the patterns in the WHERE clause. 35
  36. 36. http://streamreasoning.org/events/rsp2016 36 C-SPARQL Language FROM STREAM Clause
  37. 37. http://streamreasoning.org/events/rsp2016 C-SPARQL Language FROM STREAM Clause  FROM STREAM clauses are similar to SPARQL datasets • They identify RDF stream data sources • They represent windows over a RDF stream  They define the RDF triples available for querying and filtering. 37
  38. 38. http://streamreasoning.org/events/rsp2016 C-SPARQL Language FROM STREAM Clause - windows  physical: a given number of triples  logical: a variable number of triples which occur during a given time interval (e.g., 1 hour) • Sliding: they are progressively advanced of a given STEP (e.g., 5 minutes) • Tumbling: they are advanced of exactly their time interval 38
  39. 39. http://streamreasoning.org/events/rsp2016 C-SPARQL Language FROM STREAM Clause - Example  Using the social stream fb, how many people are in the same room? Count on a window of 1 minute that slides every 10 seconds REGISTER QUERY HowManyPoepleAreInTheSameRoom AS PREFIX : <http://…/sr4ld2014-onto#> SELECT ?room (COUNT(DISTINCT ?s) as ?person) FROM STREAM <http://…/fb> [RANGE 1m STEP 10s] WHERE { ?person1 :posts [ :who ?person ; :where ?room ] . } GROUP BY ?room 40
  40. 40. http://streamreasoning.org/events/rsp2016 41 C-SPARQL Language C-SPARQL reports only snapshots t t+10 t+20 t+30 t+40 t+50 t+60 t+70 t+80 d1 d2 d3 d1 d1 d1 d1 d1 d2 d2 d2 d2 d3 d3 Incoming timestamped RDF triples Time window [RANGE 40s STEP 10s] Window contentt+40 d1 d1, d2 d1, d2 d1, d2, d3 d2, d3 t+50 t+60 t+70 t+80
  41. 41. http://streamreasoning.org/events/rsp2016 C-SPARQL Language Multiple FROM STREAM Clause - Example  Using the social stream fb and fs, how many people are in the same room? Count on a window of 1 minute that slides every 10 seconds REGISTER QUERY HowManyPoepleAreInTheSameRoom AS PREFIX : <http://…/rsp2014-onto#> SELECT ?room (COUNT(DISTINCT ?s) as ?person) FROM STREAM <http://…/fb> [RANGE 1m STEP 10s] FROM STREAM <http://…/fs> [RANGE 1m STEP 10s] WHERE { ?person1 :posts [ :who ?person ; :where ?room ] . } GROUP BY ?room 42
  42. 42. http://streamreasoning.org/events/rsp2016 C-SPARQL Language Query Chaining  A C-SPARQL query Q1 registered using the STREAM clause streams results on an RDF stream  A down stream C-SPARQL query Q2 can open a window on the RDF stream of Q1 using the FROM STREAM clause  E.g., 43 Is in on 4 query 4 Stream f Stream Is with on f query Is In across f and 4 query Stream Stream :Bob :posts [ :who :Bob ; :where :BlueRoom ] . :Carl :posts [ :who :Carl , :Bob ] . :Bob :isIn :BlueRoom . :Carl :isWith :Bob . :Carl :isIn :BlueRoom .
  43. 43. http://streamreasoning.org/events/rsp2016 44 C-SPARQL Language TimeStamp Function
  44. 44. http://streamreasoning.org/events/rsp2016 C-SPARQL Language TimeStamp Function – Syntax and Semantics  The timestamp of a triple can be bound to a variable using a timestamp() function  Syntax • timestamp(variable|IRI|bn, variable|IRI, variable|IRI|bn|literal)  Semantics 45 Triple Result of evalutaion It is not in the window Type Error It appears once in the window Timestamp of triple It appears multiple times in the window The timestamp of the most recent triple
  45. 45. http://streamreasoning.org/events/rsp2016 C-SPARQL Language TimeStamp Function - Example  Who is “following” whom? REGISTER QUERY FindFollowers AS PREFIX f: <http://larkc.eu/csparql/sparql/jena/ext#> PREFIX : <http://…/sr4ld2014-onto#> SELECT ?someOne ?someOneElse ?room FROM STREAM <http://…/isIn> [RANGE 1m STEP 10s] WHERE { ?someOne :isIn ?room . ?someOneElse :isIn ?room . FILTER(?someOne!=?someOneElse ) FILTER (f:timestamp(?someOne :isIn ?room) < f:timestamp(?someOneElse :isIn ?room) } 46
  46. 46. http://streamreasoning.org/events/rsp2016 C-SPARQL Language Accessing background Information  C-SPARQL allows for asking the engine to issue the query also against RDF graphs using the FROM clauses.  E.g., Where else can Alice go? REGISTER QUERY WhereElseCanAliceGo AS PREFIX : <http://…/sr4ld2014-onto#> SELECT ?room FROM STREAM <http://…/isIn> [RANGE 10m STEP 10m] FROM <http://…/bgInfo> WHERE { ?:Alice :isIn ?someRoom . ?someRoom :isConnectedTo ?room . } 47 IRI identifying the graph containing the background information
  47. 47. http://streamreasoning.org/events/rsp2016 C-SPARQL Language C-SPARQL queries and reasoning - example  Memo • posts is a sub property of observes  Data  Query under RDFS entailment regime REGISTER QUERY QueryUnderRDFSEntailmentRegime AS PREFIX : <http://…/sr4ld2014-onto#> SELECT ?x ?room ?person FROM STREAM <http://…/fs> [RANGE 1m STEP 10s] FROM STREAM <http://…/sensors> [RANGE 1m STEP 10s] WHERE { ?x :observes [ :who ?person ; :where ?room ] .}  Results at t2 + 10s 48 RDF graph Time-stamp Stream :RedSensor :observes [ :who :Alice; :where :RedRoom ] . t1 sensors :Bob :posts [ :who :Bob ; :where :RedRoom] . t2 fs ?x ?room ?person :RedSensor :RedRoom :Alice :Bob :RedRoom :Bob
  48. 48. http://streamreasoning.org/events/rsp2016 49 Introduction C-SPARQL Engine Architecture  Simple, modular architecture  It relies entirely on existing technologies  Integration of • DSMSs (Esper) and • SPARQL engines (Jena- ARQ)
  49. 49. http://streamreasoning.org/events/rsp2016 Introduction C-SPARQL Engine Features at a glance 1/3  In-memory RDF stream Processing • Continuous queries, filtering, aggregations, joins, sub-queries via C-SPARQL • Push based • Reactive  C-SPARQL Engine 0.9.5 supports • SPARQL 1.1 (tested with http://www.w3.org/wiki/SRBench) • query chaining • background RDF graph access and update (via SPARQL 1.1 Update) • naïve stream reasoning (via Jena Generic Rule Reasoner) • time aware matching via timestamp function  50
  50. 50. http://streamreasoning.org/events/rsp2016 Introduction C-SPARQL Engine Features at a glance 2/3  Extensible Middleware • Runtime management of – RDF streams – C-SPARQL query – Result listerners • API driven  Quick start available • C-SPARQL Engine – http://streamreasoning.org/download/csparqlreadytogopack  Source code are released open source under Apache 2.0 • C-SPARQL Engine – https://github.com/streamreasoning/CSPARQL-engine – https://github.com/streamreasoning/CSPARQL-ReadyToGoPack 51
  51. 51. http://streamreasoning.org/events/rsp2016 Introduction C-SPARQL Engine Features at a glance 3/3  Known limitations • large background data and timestamp function can spoil performance • no support for named graphs and named streams • no support for multiple windows on the same stream • triple based windows are buggy 52
  52. 52. http://streamreasoning.org/events/rsp2016 Introduction C-SPARQL Engine as general RSP  RSP-services proposes a unified interface for the RDF stream processors and offers Rest services to interact with them.  RSP-services-csparql represents the specific implementation of the RSP-services for the C-SPARQL engine (more detailed information in the hands-on session)  Quick start available • RDF Stream Processging RESTful Interface (RSP-service) for C-SPARQL Engine – http://streamreasoning.org/download/rsp-service4csparql  Source code are released open source under Apache 2.0 • RSP-services – https://github.com/streamreasoning/rsp-services-csparql – https://github.com/streamreasoning/rsp-services-api – https://github.com/streamreasoning/rsp-services-client-example 53
  53. 53. http://streamreasoning.org/events/rsp2016 Resources  Read out more • C-SPARQL semantics – Davide Francesco Barbieri, Daniele Braga, Stefano Ceri, Emanuele Della Valle, Michael Grossniklaus: C-SPARQL: a Continuous Query Language for RDF Data Streams. Int. J. Semantic Computing 4(1): 3-25 (2010) • Most recent syntax – D. F. Barbieri, D. Braga, S. Ceri, E. Della Valle, M. Grossniklaus, Querying RDF streams with C-SPARQL, SIGMOD Record 39 (1) (2010) 20–26. • RSP-services – M Balduini,E Della Valle: A Restful Interface for RDF Stream Processors. International Semantic Web Conference (Posters & Demos) 2013: 209-212  Downloads • http://streamreasoning.org/download/csparqlreadytogopack • http://streamreasoning.org/download/rsp-service4csparql  Contact points • marco.balduini@polimi.it • emanuele.dellavalle@polimi.it 54
  54. 54. http://streamreasoning.org/events/rsp2016 55 SPARQL Stream & Morph-streams
  55. 55. http://streamreasoning.org/events/rsp2016 Morph-streams: Overview 56 Query rewriting Query Processing Client SPARQLStream [tuples] [triples/bin dings] Algebra expression R2RML Mappings Morph-streams procesing SPARQLStream queries SELECT ?proximity FROM STREAM <http://streamreasoning.org/SensorReadings.srdf> [NOW–5 S] WHERE { ?obs a ssn:ObservationValue; qudt:numericalValue ?proximity; FILTER (?proximity>10) } SELECT prox FROM sens.win:time(5 sec) WHERE prox >10 π timed,prox ω σprox>10 5 Seconds sens Data translation SNEE Esper GSN Cosm pull/push https://github.com/jpcik/morph-streams Other
  56. 56. http://streamreasoning.org/events/rsp2016 SPARQLStream Language 57 FROM NAMED STREAM ISTREAM DSTREAM RSTREAM WINDOW Underlying data source restrictions
  57. 57. http://streamreasoning.org/events/rsp2016 SPARQLStream Language  NamedStream  ‘FROM’ [‘NAMED’] ‘STREAM’ StreamIRI ‘[’ Window ‘]’  Window  ‘NOW-’ Integer TimeUnit [UpperBound] [Slide]  UpperBound  ‘TO NOW-’ Integer TimeUnit  Slide  ‘SLIDE’ Integer TimeUnit  TimeUnit  ‘MS’ | ‘S’ | ‘MINUTES’| ‘HOURS’ | ‘DAY’  Select  ‘SELECT’ [Xstream] [Distinct | Reduced] …  Xstream  ‘RSTREAM’ | ‘ISTREAM’ | ‘DSTREAM’ 58 SELECT ISTREAM ?room FROM NAMED STREAM <http://www.streamreasoning.org/streams/socialsensor.srdf> [NOW-10 S] WHERE {…
  58. 58. http://streamreasoning.org/events/rsp2016 SPARQLStream: examples 59 PREFIX sr4ld: <http://www.streamreasoning.org/ontologies/socialsensor,owl#> SELECT ?room FROM NAMED STREAM <http://www.streamreasoning.org/streams/socialsensor.srdf> [NOW-10 S] WHERE { ?obs sr4ld:observedBy ?sensor. ?obs sr4ld:where ?room. } SPARQLStream All rooms where something was observed in the last 10s PREFIX sr4ld: <http://www.streamreasoning.org/ontologies/socialsensor,owl#> SELECT (COUNT(?person) AS ?nmb) ?room FROM NAMED STREAM <http://www.streamreasoning.org/streams/socialsensor.srdf> [NOW-10 S] WHERE { ?obs sr4ld:who ?pers. ?obs sr4ld:where ?room. } GROUP BY ?room Number of persons observed in each room in the last 10s
  59. 59. http://streamreasoning.org/events/rsp2016 Underlying Query Processors Esper • CEP/DSMS • EPL language SNEE • DSMS/Sensor Network Query Evaluator • Compile queries to sensor code GSN • Sensor middleware • REST API Cosm/Xively • Sensor middleware • Open platform • REST API 60 SELECT prox FROM sensors [FROM NOW-5 MINUTES TO NOW] WHERE prox >10 SELECT prox FROM sensors.win:time(5 minute) WHERE prox >10 http://montblanc.slf.ch:22001/multidata?vs[0]=sens ors& field[0]=proximity_field&c_min[0]=10& from=15/05/2012+05:00:00&to=15/05/2012+10:00: 00 http://api.cosm.com/v2/feeds/14321/datastreams/ 4?start=2012-05-15T05:00:00Z&end=2012-05- 15T10:00:00Z
  60. 60. http://streamreasoning.org/events/rsp2016 Morph-streams: Overview 61 Query rewriting Query Processing Client SPARQLStream [tuples] [triples/bin dings] Algebra expression R2RML Mappings Morph-streams procesing SPARQLStream queries SELECT ?proximity FROM STREAM <http://streamreasoning.org/SensorReadings.srdf> [NOW–5 S] WHERE { ?obs a ssn:ObservationValue; qudt:numericalValue ?proximity; FILTER (?proximity>10) } SELECT prox FROM sens.win:time(5 sec) WHERE prox >10 π timed,prox ω σprox>10 5 Seconds sens Data translation SNEE Esper GSN Cosm pull/push https://github.com/jpcik/morph-streams Other
  61. 61. http://streamreasoning.org/events/rsp2016 3rd: Mapping the two models 62 Observation Sensor Person Roomwhere who observes subClassOf (person, room,…) detections Define mappings
  62. 62. http://streamreasoning.org/events/rsp2016 R2RML – There is a recommendation! 63 We can use the W3C recommendation
  63. 63. http://streamreasoning.org/events/rsp2016 R2RML - Overview 64
  64. 64. http://streamreasoning.org/events/rsp2016 Encoding in R2RML 65 :triplesMap a rr:TriplesMap; rr:logicalTable [ rr:tableName ”sensors"; ] rr:subjectMap [ rr:template "http://streamreasoning.org/data/Observation/{person}{timed}"; rr:class sr4ld:Observation; rr:graph sr4ld:socialstream.srdf ]; rr:predicateObjectMap [ rr:predicate sr4ld:who ; rr:objectMap [ rr:template “http://streamreasoning.org/data/Person/{person}” ]];. the stream name subject URI triple predicate + object Mapping definition stream attributes the object (a URI in this case)
  65. 65. http://streamreasoning.org/events/rsp2016 Underlying Query Processors 66 SELECT ?proximity FROM STREAM <http://streamreasoning.org/SensorReadings.srdf> [NOW–5 S] WHERE { ?obs a ssn:ObservationValue; qudt:numericalValue ?proximity; FILTER (?proximity>10) } SELECT prox FROM sensors [FROM NOW-5 MINUTES TO NOW] WHERE prox >10 timed, prox π ω σprox>10 5 Seconds sensors SELECT prox FROM sensors.win:time(5 minute) WHERE prox >10 http://montblanc.slf.ch:22001/multidata?vs[0]=sensors&field[0]=proximi ty_field&c_min[0]=10& from=15/05/2012+05:00:00&to=15/05/2012+10:00:00 http://api.cosm.com/v2/feeds/14321/datastreams/4?start=2012-05- 15T05:00:00Z&end=2012-05-15T10:00:00Z Query rewriting R2RML SNEE (DSMS) Esper (CEP) GSN (middlwr) Cosm Xively SPARQLStream
  66. 66. http://streamreasoning.org/events/rsp2016 Underlying query processors Features Esper SNEE GSN Cosm/Xively Projection ✔ ✔ ✔ Fixed Proj expression ✔ ✔ ✖ ✖ Joins ✔ ✔✖ only window ✖ ✖ Union ✖ ✔✖ not windows ✔ ✖ Selection ✔ ✔ ✔ ✖✔ limited Aggregates ✔ ✔ ✔✖ ✖ Time window ✔ ✔ ✔ ✔ Tuple window ✔ ✔ ✔ ✖ R2S ✔ ✔ ✖ ✖ Conjunction, Disj ✔ ✖ ✖ ✖ Repetition pattern ✔ ✖ ✖ ✖ Sequence ✔ ✖ ✖ ✖ 67
  67. 67. http://streamreasoning.org/events/rsp2016 Morph-streams: With reasoning! 68 Query rewriting Query Processing Client SPARQLStream [tuples] [triples/bin dings] Algebra expression R2RML Mappings Morph-streams procesing SPARQLStream queries Data translation SNEE Esper GSN Cosm pull/push https://github.com/jpcik/morph-streams Other Ontology TBox Rewrite taking into account the ontology TBox
  68. 68. http://streamreasoning.org/events/rsp2016 Reasoning with Morph-streams  Rewriting the SPARQLStream queries: 69 SELECT ?x FROM NAMED STREAM <http://linkeddata.es/s/obs.srdf> [NOW - 5 MINUTES] WHERE { ?x ssn:observedBy ?y } SELECT ?x FROM NAMED STREAM <http://linkeddata.es/s/obs.srdf> [NOW - 5 MINUTES] WHERE { {?x ssn:observedBy ?y} UNION {?x a ssn:Observation} UNION {?x a aws:TemperatureObservation} UNION {?x a aws:HumidityObservation} }
  69. 69. http://streamreasoning.org/events/rsp2016 Morph-streams: With reasoning! 70 Query rewriting Query Processing Client SPARQLStream [tuples] [triples/bin dings] Algebra expression R2RML Mappings Data translation SNEE Esper GSN Cosm pull/push https://github.com/jpcik/morph-streams Other Ontology TBox Rewrite only happens once Query rewriting + translation Then continuous query is registered
  70. 70. http://streamreasoning.org/events/rsp2016 Now some code Morph-streams:  Coded in Scala  JAR bundle, use it from Scala or Java code  Maven, Sbt  Examples • One off query • Register continuous query • Pull data • Push • Basic REST  https://github.com/jpcik/morph-streams  https://github.com/jpcik/morph-web 71
  71. 71. http://streamreasoning.org/events/rsp2016 Code examples  Parse SPARQLStream val query= “PREFIX sr4ld: <…>. SELECT ?a …” val syntax= StreamQueryFactory.create(query);  Execute One-off query val query= “PREFIX sr4ld: <…>. SELECT ?a …” mapping=Mapping(new URI(mappings/social.ttl)) val adapter:QueryEvaluator=Application.adapter(system) val results= adapter.executeQuery(query,mapping) 72 Mapping Bindings
  72. 72. http://streamreasoning.org/events/rsp2016 Code examples  Register and Pull val queryid= adapter.registerQuery(query,mapping) val results1=adapter.pull(queryid) val results2=adapter.pull(queryid)  Register and Push class ExampleReceiver extends StreamReceiver{ override def receiveData(s:SparqlResults):Unit= Logger.debug("got: "+res) } val receiver=new ExampleReceiver val queryid= adapter.listenToQuery(query,mapping,receiver) 73 Query identifier Implement receiver For Java users: Exactly the same interface!
  73. 73. http://streamreasoning.org/events/rsp2016 74 Querying RSPs in Practice
  74. 74. http://streamreasoning.org/events/rsp2016 75 ExecContext context=new ExecContext(HOME, false); String queryString =" SELECT ?person ?loc … ContinuousSelect selQuery=context.registerSelect(queryString); selQuery.register(new ContinuousListener() { public void update(Mapping mapping){ String result=""; for(Iterator<Var> vars=mapping.vars();vars.hasNext();) result+=" "+ context.engine().decode(mapping.get(vars.next())); System.out.println(result); } }); RSP Querying Example with CQELS (code.google.com/p/cqels) CQELS continuous query: get result updates adding listener register query SELECT ?person ?loc WHERE { STREAM <http://deri.org/streams/rfid> [RANGE 3s] {?person :detectedAt ?loc} } Tightly coupled listeners Results delivery: push & pull?
  75. 75. http://streamreasoning.org/events/rsp2016 76 CQELS fed by a TripleWave WebSocket val conf = ConfigFactory.load.getConfig("experiments.rsp") val qq="""CONSTRUCT { ?s ?p ?o } WHERE { STREAM <ws://localhost:4040/primus> [RANGE 0ms] {?s ?p ?o} }""" val sys=new RspSystem("wstreams") val cqels=new CqelsEngine sys.startStream(Props( new WebSocketStream(cqels,"ws://localhost:4040/primus",conf))) cqels.registerQuery(qq, cqels.createListener(lissy)) def lissy(triples:TripleList):Unit={ println("tikki: "+triples) }
  76. 76. http://streamreasoning.org/events/rsp2016 Similar models, similar (not equals!) query languages 77 SELECT ?sensor FROM NAMED STREAM <http://www.cwi.nl/SRBench/observations> [NOW-3 HOURS SLIDE 10 MINUTES] WHERE { ?observation om-owl:procedure ?sensor ; om-owl:observedProperty weather:WindSpeed ; om-owl:result [ om-owl:floatValue ?value ] . } GROUP BY ?sensor HAVING ( AVG(?value) >= "74"^^xsd:float ) SELECT ?sensor WHERE { STREAM <http://www.cwi.nl/SRBench/observations> [RANGE 10800s SLIDE 600s] { ?observation om-owl:procedure ?sensor ; om-owl:observedProperty weather:WindSpeed ; om-owl:result [ om-owl:floatValue ?value ] .} } GROUP BY ?sensor HAVING ( AVG(?value) >= "74"^^xsd:float ) SELECT ?sensor FROM STREAM <http://www.cwi.nl/SRBench/observations> [RANGE 1h STEP 10m] WHERE { ?observation om-owl:procedure ?sensor ; om-owl:observedProperty weather:WindSpeed ; om-owl:result [ om-owl:floatValue ?value ] . } GROUP BY ?sensor HAVING ( AVG(?value) >= "74"^^xsd:float ) SPARQLStream CQELS C-SPARQL
  77. 77. http://streamreasoning.org/events/rsp2016 78 Query using SQL on Streams Model Continuous execution Union,Join, Optional, Filter Aggregates Timewindow Triple window R2Soperator Sequence, Co- ocurrence Time function TA-SPARQL TA-RDF ✗ ✔ Limite d ✗ ✗ ✗ ✗ ✗ tSPARQL tRDF ✗ ✔ ✗ ✗ ✗ ✗ ✗ ✗ Streaming SPARQL RDF Stream ✔ ✔ ✗ ✔ ✔ ✗ ✗ ✗ C-SPARQL RDF Stream ✔ ✔ ✔ ✔ ✔ ✗ ✗ ✔ CQELS RDF Stream ✔ ✔ ✔ ✔ ✔ ✗ ✗ ✗ SPARQLStrea m (Virtual) RDF Stream ✔ ✔ ✔ ✔ ✗ ✔ ✗ ✗ EP-SPARQL RDF Stream ✔ ✔ ✔ ✗ ✗ ✗ ✔ ✗ Instans RDF ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✗ W3C RSP  review features in existing systems  agree on fundamental operators  discuss on possible semantics https://www.w3.org/community/rsp/wiki/RSP_Query_Features RSP is not always/only SPARQL- like querying SPARQL protocol is not enough RSP RESTful interfaces? Powerful languages for continuous query processing
  78. 78. http://streamreasoning.org/events/rsp2016 79 W3C RSP-CG: RSP-QL PREFIX e: <http://somevocabulary.org/> PREFIX s: <http://someinvasivesensornetwork.org/streams#> PREFIX g: <http://somesocialnetwork.org/graphs#> PREFIX : <http://acrasycompany.org/rsp> REGISTER STREAM :GallehaultWasTheBar UNDER ENTAILMENT REGIME <http://www.w3.org/ns/entailment/RIF> AS CONSTRUCT ISTREAM { ?poi rdf:type :Gallehault ; :count ?howmanycouples ; :for (?somebody ?someoneelse) } FROM NAMED WINDOW :veryLongWindow ON s:1 [RANGE PT4H STEP PT1H] FROM NAMED WINDOW :longWindow ON s:1 [FROM NOW-PT35M TO NOW-PT5M STEP PT5M] FROM NAMED WINDOW :shortWindow ON s:1 [RANGE PT10M STEP PT5M] FROM NAMED GRAPH g:SocialGraph FROM GRAPH g:POIs WHERE { ?poi rdf:type e:bar . WINDOW :veryLongWindow { {?somebody e:enters ?poi} BEGIN AT ?t3 {?someoneelse e:enters ?poi} BEGIN AT ?t4 FILTER(?t3>?t4) } WINDOW :longWindow { { ?somebody e:isCloseTo ?someoneelse MINUS { ?somebody e:isCloseTo ?yetanotherone . FILTER (?yetanotherone != ?someoneelse) } } WITH DURATION ?duration FILTER (?duration>="PT30M"^^xsd:duration) } WINDOW :shortWindow { { ?somebody e:exits ?bar} BEGIN AT ?t1 { ?someoneelse e:exits ?bar } BEGIN AT ?t2 FILTER (abs(?t2-?t1)<"PT1M"^^xsd:duration ) } GRAPH g:SocialGraph { FILTER NOT EXIST { ?somebody e:knows ?someoneelse } } FILTER (?somebody != ?someoneelse) } AGGREGATE { GROUP BY ?poi COUNT(?somebody) AS ?howmanycouples } Continuously look for bars where people are falling in love (because o a book ) Register stream Time windows Sequencing Duration Stored Graphs Aggregates Access to time Reasoning They entered the same bar They are close to each other, with no-one else They get out together Didn’t know each other
  79. 79. http://streamreasoning.org/events/rsp2016 80 RSP Communication
  80. 80. http://streamreasoning.org/events/rsp2016 81 RDF Stream Processing RSP Engine RDF graphs input RDF streams streams of results background knowledge continuous queries streamproducers RSP Engine producer subscribe notify cont. query consumer push results subscribe streamconsumers continuous queries RSP Implementations
  81. 81. http://streamreasoning.org/events/rsp2016 82 Reactive Systems Event-Driven Jonas Boner. Go Reactive: Event-Driven, Scalable, Resilient & Responsive Systems. 2013. Events:reactto ScalableLoad: ResilientFailure: ResponsiveUsers:
  82. 82. http://streamreasoning.org/events/rsp2016 83 Actor Model Actor 1 Actor 2 m No shared mutable state Avoid blocking operators Lightweight objects Loose coupling communicate through messages mailbox state behavior non-blocking response send: fire-forget Implementations: e.g. Akka for Java/Sc Parent Actor 1 Supervisi on hierarchy Supervision Actor 2 Acto r4 X Actor 2 Actor 1 Actor 2 m Actor 3 Actor 4 m m Remoting
  83. 83. http://streamreasoning.org/events/rsp2016 84 RDF Streams: Actors val sys=ActorSystem.create("system") val consumer=sys.actorOf(Props[RdfConsumer]) class Streamer extends StreamRDF{ override def triple(triple:Triple){ consumer ! triple } } class RdfConsumer extends Actor{ def receive= { case t:Triple => if (t.predicateMatches(RDF.‘type‘)) println(s"received triple $t") } RDF consumer Actor receive method Implements behavior Message-passing model RDF producer Async message passing
  84. 84. http://streamreasoning.org/events/rsp2016 85 RSP Producer & Consumer Proce ssor faster producers >> slower processor/consumer Prod ucer Prod ucer Prod ucer Prod ucer RDF streams Cons umer Cons umer Cons umer unresponsive overload Overload of the processor/receiver Unresponsiveness in stream processor
  85. 85. http://streamreasoning.org/events/rsp2016 87 Dynamic Push-Pull Producer Consumer m data flow demand flow Push when consumer is faster Pull when producer is faster Dynamically switch modes Communication is dynamic depending on demand vs supply Producer Consumer m m m m m m m m m m push
  86. 86. http://streamreasoning.org/events/rsp2016 88 Evaluation: throughput Basic dynamic pull push On top of CQELS Limitations of Thread model Not yet fully async
  87. 87. http://streamreasoning.org/events/rsp2016 89 Reactive RSP workflows Morph Streams CSPARQL s Etali s TrOWL s s CQELS Dyn amit e s Minimal agreements: standards, serialization, interfaces Formal models for RSPs and reasoning Working prototypes/systems! Event-driven message pass Async communication Immutable streams Transparent Remoting Parallel and distributed Supervised Failure Handling Responsive processing Reactive RSPs
  88. 88. http://streamreasoning.org/events/rsp2016 90 RSPs and the Linked Data Principles
  89. 89. http://streamreasoning.org/events/rsp2016 91 URIs as Names of Things http://mysensorweb.me/mytemperature/20151110Z10:00: 00 Different concepts http://mysensorweb.me/mytemperature/latest http://mysensorweb.me/mytemperature/lastMinute http://mysensorweb.me/mytemperature/lastMonth Different granularities Different uses http://mysensorweb.me/mytemperature/avgLastMonth http://mysensorweb.me/mytemperature
  90. 90. http://streamreasoning.org/events/rsp2016 92 HTTP URIs http://mysensorweb.me/mytemperature/latest Internet of Things How about XMPP, CoAP, MQTT? Websockets?
  91. 91. http://streamreasoning.org/events/rsp2016 93 De-referenceable URIs GET http://mysensorweb.me/mytemperature/latest :Obs1 a my:TemperatureObservation; my:hasValue 33.5 ; my:hasUnit u:Celsius; my:atTime “20151110Z10:00:00”. GET http://mysensorweb.me/mytemperature Get the whole stream? GET http://mysensorweb.me/mytemperature/lastMonth Get continuous updates?
  92. 92. http://streamreasoning.org/events/rsp2016 94 Link to other URIs • Broken links? • Mix streaming and stored data • Persist or not persist? • Volatile links? http://mysensorweb.me/mytemperature/20151110Z10:00:00
  93. 93. http://streamreasoning.org/events/rsp2016 95 For Java/Scala developers
  94. 94. http://streamreasoning.org/events/rsp2016 96 Semantic Web Devs Have fun Love challenges Need cool tools RDF4J
  95. 95. http://streamreasoning.org/events/rsp2016 97 Scala: Functions and Objects JVM language Both object and functional oriented Easy Java-interop Reuse Java libraries Growing community
  96. 96. http://streamreasoning.org/events/rsp2016 98 RDF in Jena: in Scala String personURI = "http://somewhere/JohnSmith"; Model model = ModelFactory.createDefaultModel(); model.createResource(personURI).addProperty(VCARD.FN,"John Smith"); Type inference Not too useful ; and () Terser & compact code Type-safe DSL Compiler takes care val personURI = "http://somewhere/JohnSmith" val model = ModelFactory.createDefaultModel model.createResource(personURI).addProperty(VCARD.FN,"John Smith") sw:John Smith “John Smith” vcard:FN val personURI = "http://somewhere/JohnSmith" implicit val model = createDefaultModel add(personURI,VCARD.FN->"John Smith") boilerplate String converted to Resource
  97. 97. http://streamreasoning.org/events/rsp2016 99 Some more RDF 99 String personURI = "http://somewhere/JohnSmith"; String givenName = "John"; String familyName = "Smith"; String fullName = givenName + " " + familyName; Model model = ModelFactory.createDefaultModel(); model.createResource(personURI) .addProperty(VCARD.FN,fullName) .addProperty(VCARD.N,model.createResource() .addProperty(VCARD.Given,givenName) .addProperty(VCARD.Family,familyName)); val personURI = "http://somewhere/JohnSmith" val givenName = "John" val familyName = "Smith" val fullName = s"$givenName $familyName" implicit val model = createDefaultModel add(personURI,VCARD.FN->fullName, VCARD.N ->add(bnode,VCARD.Given -> givenName, VCARD.Family->familyName)) sw:John Smith “John Smith” vcard:FN _:n “John” “Smith”vcard:N vcard:Giv en vcard:Fam ily Blank node Scala DSLs customizable Predicate-objects are pairs
  98. 98. http://streamreasoning.org/events/rsp2016 100 Some more RDF in Jena implicit val m=createDefaultModel val ex="http://example.org/" val alice=iri(ex+"alice") val bob=iri(ex+"bob") val charlie=iri(ex+"charlie") alice+(RDF.`type`->FOAF.Person, FOAF.name->"Alice", FOAF.mbox->iri("mailto:alice@example.org"), FOAF.knows->bob, FOAF.knows->charlie, FOAF.knows->bnode) bob+ (FOAF.name->"Bob", FOAF.knows->charlie) charlie+(FOAF.name->"Charlie", FOAF.knows->alice) Still valid Jena RDF You can do it even nicer
  99. 99. http://streamreasoning.org/events/rsp2016 101 Exploring an RDF Graph ArrayList<String> names=new ArrayList<String>(); NodeIterator iter=model.listObjectsOfProperty(VCARD.N); while (iter.hasNext()){ RDFNode obj=iter.next(); if (obj.isResource()) names.add(obj.asResource() .getProperty(VCARD.Family).getObject().toString()); else if (obj.isLiteral()) names.add(obj.asLiteral().getString()); } val names=model.listObjectsOfProperty(VCARD.N).map{ case r:Resource=> r.getProperty(VCARD.Family).obj.toString case l:Literal=> l.getString } Imperative iteration of collections Type-based conditional execution Type casting Case type Map applied to operators
  100. 100. http://streamreasoning.org/events/rsp2016 102
  101. 101. http://streamreasoning.org/events/rsp2016 103 Query with SPARQL val queryStr = """select distinct ?Concept where {[] a ?Concept} LIMIT 10""" val query = sparql(queryStr) query.serviceSelect("http://dbpedia.org/sparql").foreach{implicit qs=> println(res("Concept").getURI) } val f=Future(query.serviceSelect("http://es.dbpedia.org/sparql")).fallbackTo( Future(query.serviceSelect("http://dbpedia.org/sparql"))) f.recover{ case e=> println("Error "+e.getMessage) } f.map(_.foreach{implicit qs=> println(res("Concept").getValue) }) Remote SPARQL endpoin Simplified access to Query solutions Futures: asnyc execution Non blocking code Fallback alternative execut
  102. 102. http://streamreasoning.org/events/rsp2016 104 Muchas Gracias!
  103. 103. Tutorial on RDF Stream Processing 2016 M.I. Ali, J-P Calbimonte, D. Dell'Aglio, E. Della Valle, and A. Mauri http://streamreasoning.org/events/rsp2016 RDF Stream Processing Implementations Jean-Paul Calbimonte jean-paul.calbimonte@hevs.ch http://jeanpi.org @jpcik

×