RDF Stream Processing Tutorial: RSP implementations

Tutorial on RDF Stream
Processing 2016
M.I. Ali, J-P Calbimonte, D. Dell'Aglio,
E. Della Valle, and A. Mauri
http://streamreasoning.org/events/rsp2016
RDF Stream Processing
Implementations
Jean-Paul Calbimonte
jean-paul.calbimonte@hevs.ch http://jeanpi.org @jpcik

Share, Remix, Reuse — Legally
 This work is licensed under the Creative Commons
Attribution 3.0 Unported License.
 You are free:
• to Share — to copy, distribute and transmit the work
• to Remix — to adapt the work
 Under the following conditions
• Attribution — You must attribute the work by inserting
– “[source http://streamreasoning.org/rsp2014]” at the end of
each reused slide
– a credits slide stating
- These slides are partially based on “RDF Stream Processing 2014”
by M. Balduini, J-P Calbimonte, O. Corcho, D. Dell'Aglio, E. Della
Valle http://streamreasoning.org/rsp2014
 To view a copy of this license, visit
http://creativecommons.org/licenses/by/3.0/
2

RSP for developers
• RDF Streams in practice
• RSP Query Engines
• Developing with an RSP Engine
• Handling Results
• RSP Services
3

RDF Streams in Practice
4

RSP: Keep the data moving
5
Process data in-stream
Not required to store
Active processing model
input streams
RSP
queries/
rules output streams/events
RDF Streams

http://streamreasoning.org/events/rsp2016 6
RDF Stream
…
Gi
Gi+1
Gi+2
…
Gi+n
…
unboundedsequence
Gi {(s1,p1,o1),
(s2,p2,o2),…} [ti]
1+ triples
implicit/explicit
timestamp/interval
RDF streams in theory
How do I code this?
Use Web standards?

Linked Data on the Web
Web of Data
Linked Data
W3C Standards: RDF, SPARQL, etc.

Linked Data principles for RDF streams?
e.g. publish sensor data as RDF/Linked Data?
URIs as names of
things
HTTP URIs
useful information when URI
is dereferenced
Link to other
URIs
users
application
s
WEB
Use RDF model to continuously query real-time
data streams?
static vs. streams
one-off vs.
continuous

(Sensor) Data Streams on the Web
9
http://mesowest.utah.edu/
http://earthquake.usgs.gov/earthquakes/feed/v1.0/
http://swiss-experiment.ch
• Monitoring
• Alerts
• Notifications
• Hourly/daily update
• Myriad of Formats
• Ad-hoc access points
• Informal description
• Convention-semantics
• Uneven use of standards
• Manual exploration

RDF Streams before RDF Streams
http://richard.cyganiak.de/2007/10/lod/
2011
Linked Sensor Data
MetOffice
AEMET

Sensor Data & Linked Data
11
Zip Files
Number of Triples
Example: Nevada dataset
-7.86GB in n-triples format
-248MB zipped
An example: Linked Sensor Data
http://wiki.knoesis.org/index.php/LinkedSensorData

Sensor Data & Linked Data
12
<http://knoesis.wright.edu/ssw/MeasureData_Precipitation_4UT01_2003_3_31_5_10_00>
<http://www.w3.org/1999/02/22-rdf-syntax-ns#type>
<http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#MeasureData> .
<http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#floatValue>
"30.0"^^<http://www.w3.org/2001/XMLSchema#float> .
<http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#uom>
<http://knoesis.wright.edu/ssw/ont/weather.owl#centimeters> .
<http://knoesis.wright.edu/ssw/Observation_Precipitation_4UT01_2003_3_31_5_10_00>
<http://knoesis.wright.edu/ssw/ont/weather.owl#PrecipitationObservation> .
<http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#observedProperty>
<http://knoesis.wright.edu/ssw/ont/weather.owl#_Precipitation> .
<http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#procedure>
<http://knoesis.wright.edu/ssw/System_4UT01> .
<http://knoesis.wright.edu/ssw/ont/sensor-observation.owl#samplingTime>
<http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00> .
<http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00>
<http://www.w3.org/2006/time#Instant> .
<http://knoesis.wright.edu/ssw/Instant_2003_3_31_5_10_00>
<http://www.w3.org/2006/time#inXSDDateTime>
"2003-03-31T05:10:00-07:00^^http://www.w3.org/2001/XMLSchema#dateTime" .
What do we get in these datasets?
Nice triples
What is measured
Measurement
Unit
Sensor
When is it measured

RDF Streams before RDF Streams
i.e. just use RDF
:observation1 rdf:type om-owl:Observation .
:observation1 om-owl:observedProperty weather:_AirTemperature .
:observation1 om-owl:procedure :sensor1 .
:observation1 om-owl:result :obsresult1 .
:observation1 om-owl:resultTime "2015-01-01T10:00:01"
:obsresult1 om-owl:floatValue 35.4 .
Plain triples
Where is the
timestamp?
:observation2 rdf:type om-owl:Observation .
:observation2 om-owl:observedProperty weather:_AirTemperature .
:observation2 om-owl:procedure :sensor1 .
:observation2 om-owl:result :obsresult2 .
:observation2 om-owl:resultTime "2015-01-01T10:00:02"
:obsresult2 om-owl:floatValue 36.4 .
What is the order
in the RDF graph?
Appended to a file?
Or to some RDF dataset?
How to store it?

Feed an RDF Stream to a RSP engine
Ad-hoc
Conversion to
RDF
Live Non-RDF Streams
RDF
RDF datasets
RSP
Add (internal)
timestamp
on insertion
What is currently done in most RSPs
Continuous
additions
RDF +
timestamps

Feed an RDF Stream to C-SPARQL
public class SensorsStreamer extends RdfStream implements Runnable {
public void run() {
..
while(true){
...
RdfQuadruple q=new RdfQuadruple(subject,predicate,object,
System.currentTimeMillis());
this.put(q);
}
}
}
something
to run on
a thread
timestamped
triple
the stream is
“observable”
Data structure, execution
and callbacks are mixed
Observer pattern
Tightly coupled listener
Added timestamp

Actor Model
Actor
1
Actor
2
m No shared mutable state
Avoid blocking operators
Lightweight objects
Loose coupling
communicate
through messages
mailbox
state
behaviornon-blocking response
send: fire-forget
Implementations: e.g. Akka for Java/Scala

RDF Stream
object DemoStreams {
...
def streamTriples={
Iterator.from(1) map{i=>
...
new Triple(subject,predicate,object)
}
}
Data structure
Infinite
triple
iterator
Execution
val f=Future(DemoStreams.streamTriples)
f.map{a=>a.foreach{triple=>
//do something
}}
Asynchronou
s iteration
Message passing
f.map{a=>a.foreach{triple=>
someSink ! triple
}}
send triple to
actor
Immutable RDF stream
 avoid shared mutable
state
 avoid concurrent writes
 unbounded sequence
Ideas using akka actors
Futures
 non blocking composition
 concurrent computations
 work with not-yet-
computed results
Actors
 message-based
 share-nothing async
 distributable

RDF Stream
… other issues:
Graph implementation?
Timestamps: application vs system?
Serialization?
 Loose coupling
 Immutable data streams
 Asynchronous message passing
 Well defined input/output

Data stream characteristics
19
Data regularity
• Raw data typically collected as time series
• Very regular structure.
• Patterns can be exploited
E.g. mobile NO2 sensor readings
29-02-2016T16:41:24,47,369,46.52104,6.63579
29-02-2016T16:41:34,47,358,46.52344,6.63595
29-02-2016T16:41:44,47,354,46.52632,6.63634
29-02-2016T16:41:54,47,355,46.52684,6.63729
...
Data order
• Order of data is crucial
• Time is the key attribute for establishing an order among the data items.
• Important for indexing
• Enables efficient time-based selection, filtering and windowing
Timestamp Sensor Observed
Value
Coordinates

Feed an RDF Stream to a RSP engine
Conversion to
RDF
Live Non-RDF Streams
RDF
RDF datasets
RSP
Add (internal)
timestamp
on insertion
Adding mappings to the data flow
Continuous
additions
RDF +
timestamps

R2RML Mappings
21
:ObsValueMap
rr:subjectMap [
rr:template "http://opensense.epfl.ch/data/ObsResult_NO2_{sensor}_{time}"];
rr:predicateObjectMap [
rr:predicate qu:numericalValue;
rr:objectMap [ rr:column "no2"; rr:datatype xsd:float; ]];
rr:predicate obs:uom;
rr:objectMap [ rr:parentTriplesMap :UnitMap; ]].
:ObservationMap
rr:subjectMap [
rr:template "http://opensense.epfl.ch/data/Obs_NO2_{sensor}_{time}"];
rr:predicate ssn:observedProperty;
rr:objectMap [ rr:constant opensense:NO2]];
URI of subject
URI of predicate
Object: colum name
Column names in a template
Can be used for mapping both databases, CSVs, JSON, etc

An example: TripleWave
Running modes
Sources

RDF Streams in W3C RSP
:g1 {:axel :isIn :RedRoom. :darko :isIn :RedRoom}
{:g1, prov:generatedAtTime, "2015-06-18T12:00:00Z"^^xsd:dateTime}
:g2 {:axel :isIn :BlueRoom. }
{:g2, prov:generatedAtTime, "2015-06-18T12:00:35"^^xsd:dateTime}
:g3 {:minh :isIn :RedRoom. }
{:g3, prov:generatedAtTime, "2015-06-18T12:02:07"^^xsd:dateTime}
...
https://www.w3.org/community/rsp/
http://streamreasoning.github.io/RSP-QL/RSP_Requirements_Design_Document/
Graph-based
Flexible time
property
RDF-friendly
Flexible
metadata
:g_1 :startsAt "2015-06-18T12:00:00"^^xsd:dateTime
:g_1 :endsAt "2015-06-18T13:00:00"^^xsd:dateTime
:g_2 :validBetween [:startsAt "2015-06-18T12:00:00"^^xsd:dateTime;
:endsAt "2015-06-18T13:00:00"^^xsd:dateTime]
Intervals

RSP Engine Implementations

Existing RSP systems (oversimplified!)
 C-SPARQL: RDF Store + Stream processor
• Combined architecture
 CQELS: Implemented from scratch. Focus on
performance
• Native + adaptive joins for static-data and streaming
data
25
RDF Store
Stream
processor
C-SPARQL
query
continuous
results
Native RSPCQELS
query
continuous
results
translator

Existing RSP systems (oversimplified!)
 SPARQLstream: Ontology-based stream query answering
• Virtual RDF views, using R2RML mappings
• SPARQL stream queries over the original data streams.
 EP-SPARQL: Complex-event detection
• SEQ, EQUALS operators
 Instans: RETE-based evaluation
26
DSMS/CEPSPARQLStream
query
continuous
results
rewriter
R2RML mappings
Prolog
engine
EP-SPARQL
query
continuous
results
translator

Classification of existing systems
Model
Continuous
execution
Union,Join,
Optional,
Filter
Aggregates
Timewindow
Triplewindow
R2Soperator
Sequence,
Co-ocurrence
TA-
SPARQL
TA-RDF ✗ ✔ Limited ✗ ✗ ✗ ✗
tSPARQL tRDF ✗ ✔ ✗ ✗ ✗ ✗ ✗
Streaming
SPARQL
RDF
Stream
✔ ✔ ✗ ✔ ✔ ✗ ✗
C-SPARQL
RDF
Stream
✔ ✔ ✔ ✔ ✔ Rstream
only
time
function
CQELS
RDF
Stream
✔ ✔ ✔ ✔ ✔ Istream
only
✗
SPARQLStr
eam
(Virtual)
RDF
Stream
✔ ✔ ✔ ✔ ✗ ✔ ✗
EP-
SPARQL
RDF
Stream
✔ ✔ ✔ ✗ ✗ ✗ ✔
Instans RDF ✔ ✔ ✔ ✗ ✗ ✗ ✗
27Disclaimer: other features may be missing

C-SPARQL

A Reminder of SPARQL

Where C-SPARQL Extends SPARQL

C-SPARQL Language
Query and Stream Registration

C-SPARQL Language
Query and Stream Registration
 All C-SPARQL queries over RDF streams are continuous
• Registered through the REGISTER statement
 The output of queries is in the form of
• Instantaneous tables of variable bindings
• Instantaneous RDF graphs
• RDF stream
 Only queries in the CONSTRUCT form can be registered as
generators of RDF streams
 Composability:
• Query results registered as streams can feed other registered
queries just like every other RDF stream
32

C-SPARQL Language
Query registration - Example
 Using the social stream fb, Who is where?
REGISTER QUERY QWhoIsWhereOnFb AS
PREFIX : <http://…/sr4ld2014-onto#>
SELECT ?room ?person
FROM STREAM <http://…/fb> [RANGE 1m STEP 10s]
WHERE {
?person1 :posts [ :who ?person ; :where ?room ] .
}
 The resulting variable bindings has to be interpreted as an
instantaneous. It expires as soon as the query is
recomputed
33

C-SPARQL Language
Stream registration - Example
 Results of a C-SPARQL query can be stream out for down
stream queries
REGISTER STREAM SWhoIsWhereOnFb AS
CONSTRUCT { ?person :isIn ?room }
WHERE {
}
 The resulting RDF triples are streamed out on an RDF
stream
• More details in the C-SPARQL Engine hands-on session
34

C-SPARQL Language
Stream Registration - Notes
 The output is constructed in the format of an RDF stream.
 Every query execution may produce from a minimum of
zero triples to a maximum of an entire graph.
 The timestamp is always dependent on the query
execution time only, and is not taken from the triples that
match the patterns in the WHERE clause.
35

C-SPARQL Language
FROM STREAM Clause

C-SPARQL Language
FROM STREAM Clause
 FROM STREAM clauses are similar to SPARQL datasets
• They identify RDF stream data sources
• They represent windows over a RDF stream
 They define the RDF triples available for querying and
filtering.
37

C-SPARQL Language
FROM STREAM Clause - windows
 physical: a given number of triples
 logical: a variable number of triples which occur during a
given time interval (e.g., 1 hour)
• Sliding: they are progressively advanced of
a given STEP (e.g., 5 minutes)
• Tumbling: they are advanced of exactly their time interval
38

C-SPARQL Language
FROM STREAM Clause - Example
 Using the social stream fb, how many people are in the
same room? Count on a window of 1 minute that slides
every 10 seconds
REGISTER QUERY HowManyPoepleAreInTheSameRoom AS
SELECT ?room (COUNT(DISTINCT ?s) as ?person)
WHERE {
}
GROUP BY ?room
40

C-SPARQL Language
C-SPARQL reports only snapshots
t
t+10
t+20
t+30
t+40
t+50
t+60
t+70
t+80
d1
d2
d3
d1 d1 d1 d1 d1
d2 d2 d2 d2
d3 d3
Incoming
timestamped
RDF triples
Time window [RANGE 40s STEP 10s] Window
contentt+40
d1
d1, d2
d1, d2
d1, d2, d3
d2, d3
t+50 t+60 t+70 t+80

C-SPARQL Language
Multiple FROM STREAM Clause - Example
 Using the social stream fb and fs, how many people are
in the same room? Count on a window of 1 minute that
slides every 10 seconds
REGISTER QUERY HowManyPoepleAreInTheSameRoom AS
PREFIX : <http://…/rsp2014-onto#>
SELECT ?room (COUNT(DISTINCT ?s) as ?person)
FROM STREAM <http://…/fs> [RANGE 1m STEP 10s]
WHERE {
}
GROUP BY ?room
42

C-SPARQL Language
Query Chaining
 A C-SPARQL query Q1 registered using the STREAM clause
streams results on an RDF stream
 A down stream C-SPARQL query Q2 can open a window on
the RDF stream of Q1 using the FROM STREAM clause
 E.g.,
43
Is in on 4
query
4
Stream
f
Stream
Is with on f
query
Is In across
f and 4
query
Stream
Stream
:Bob :posts [ :who :Bob ; :where :BlueRoom ] .
:Carl :posts [ :who :Carl , :Bob ] .
:Bob :isIn :BlueRoom .
:Carl :isWith :Bob .
:Carl :isIn :BlueRoom .

C-SPARQL Language
TimeStamp Function

C-SPARQL Language
TimeStamp Function – Syntax and Semantics
 The timestamp of a triple can be bound to a variable using
a timestamp() function
 Syntax
• timestamp(variable|IRI|bn, variable|IRI, variable|IRI|bn|literal)
 Semantics
45
Triple Result of evalutaion
It is not in the window Type Error
It appears once in the
window
Timestamp of triple
It appears multiple times in
the window
The timestamp of the most recent
triple

C-SPARQL Language
TimeStamp Function - Example
 Who is “following” whom?
REGISTER QUERY FindFollowers AS
PREFIX f: <http://larkc.eu/csparql/sparql/jena/ext#>
SELECT ?someOne ?someOneElse ?room
FROM STREAM <http://…/isIn> [RANGE 1m STEP 10s]
WHERE {
?someOne :isIn ?room .
?someOneElse :isIn ?room .
FILTER(?someOne!=?someOneElse )
FILTER (f:timestamp(?someOne :isIn ?room) <
f:timestamp(?someOneElse :isIn ?room)
}
46

C-SPARQL Language
Accessing background Information
 C-SPARQL allows for asking the engine to issue the query
also against RDF graphs using the FROM clauses.
 E.g., Where else can Alice go?
REGISTER QUERY WhereElseCanAliceGo AS
SELECT ?room
FROM STREAM <http://…/isIn> [RANGE 10m STEP 10m]
FROM <http://…/bgInfo>
WHERE {
?:Alice :isIn ?someRoom .
?someRoom :isConnectedTo ?room .
}
47
IRI identifying the graph
containing the
background information

C-SPARQL Language
C-SPARQL queries and reasoning - example
 Memo
• posts is a sub property of observes
 Data
 Query under RDFS entailment regime
REGISTER QUERY QueryUnderRDFSEntailmentRegime AS
SELECT ?x ?room ?person
FROM STREAM <http://…/fs> [RANGE 1m STEP 10s]
FROM STREAM <http://…/sensors> [RANGE 1m STEP 10s]
WHERE { ?x :observes [ :who ?person ; :where ?room ] .}
 Results at t2 + 10s
48
RDF graph Time-stamp Stream
:RedSensor :observes [ :who :Alice; :where :RedRoom ] . t1 sensors
:Bob :posts [ :who :Bob ; :where :RedRoom] . t2 fs
?x ?room ?person
:RedSensor :RedRoom :Alice
:Bob :RedRoom :Bob

Introduction
C-SPARQL Engine Architecture
 Simple, modular
architecture
 It relies entirely on
existing technologies
 Integration of
• DSMSs (Esper) and
• SPARQL engines (Jena-
ARQ)

Introduction
C-SPARQL Engine Features at a glance 1/3
 In-memory RDF stream Processing
• Continuous queries, filtering, aggregations, joins, sub-queries
via C-SPARQL
• Push based
• Reactive
 C-SPARQL Engine 0.9.5 supports
• SPARQL 1.1 (tested with http://www.w3.org/wiki/SRBench)
• query chaining
• background RDF graph access and update (via SPARQL 1.1
Update)
• naïve stream reasoning (via Jena Generic Rule Reasoner)
• time aware matching via timestamp function

50

Introduction
 Extensible Middleware
• Runtime management of
– RDF streams
– C-SPARQL query
– Result listerners
• API driven
 Quick start available
• C-SPARQL Engine
– http://streamreasoning.org/download/csparqlreadytogopack
 Source code are released open source under Apache 2.0
• C-SPARQL Engine
– https://github.com/streamreasoning/CSPARQL-engine
– https://github.com/streamreasoning/CSPARQL-ReadyToGoPack
51

Introduction
 Known limitations
• large background data and timestamp function can spoil
performance
• no support for named graphs and named streams
• no support for multiple windows on the same stream
• triple based windows are buggy
52

Introduction
C-SPARQL Engine as general RSP
 RSP-services proposes a unified interface for the RDF
stream processors and offers Rest services to interact with
them.
 RSP-services-csparql represents the specific
implementation of the RSP-services for the C-SPARQL
engine (more detailed information in the hands-on session)
 Quick start available
• RDF Stream Processging RESTful Interface (RSP-service) for
C-SPARQL Engine
– http://streamreasoning.org/download/rsp-service4csparql
 Source code are released open source under Apache 2.0
• RSP-services
– https://github.com/streamreasoning/rsp-services-csparql
– https://github.com/streamreasoning/rsp-services-api
– https://github.com/streamreasoning/rsp-services-client-example
53

Resources
 Read out more
• C-SPARQL semantics
– Davide Francesco Barbieri, Daniele Braga, Stefano Ceri, Emanuele
Della Valle, Michael Grossniklaus: C-SPARQL: a Continuous Query
Language for RDF Data Streams. Int. J. Semantic Computing 4(1):
3-25 (2010)
• Most recent syntax
– D. F. Barbieri, D. Braga, S. Ceri, E. Della Valle, M. Grossniklaus,
Querying RDF streams with C-SPARQL, SIGMOD Record 39 (1)
(2010) 20–26.
• RSP-services
– M Balduini,E Della Valle: A Restful Interface for RDF Stream
Processors. International Semantic Web Conference (Posters &
Demos) 2013: 209-212
 Downloads
• http://streamreasoning.org/download/csparqlreadytogopack
• http://streamreasoning.org/download/rsp-service4csparql
 Contact points
• marco.balduini@polimi.it
• emanuele.dellavalle@polimi.it
54

SPARQL Stream &
Morph-streams

Morph-streams: Overview
56
Query
rewriting
Query
Processing
Client
SPARQLStream
[tuples]
[triples/bin
dings]
Algebra
expression
R2RML
Mappings
Morph-streams procesing SPARQLStream queries
SELECT ?proximity
FROM STREAM
<http://streamreasoning.org/SensorReadings.srdf> [NOW–5 S]
WHERE {
?obs a ssn:ObservationValue;
qudt:numericalValue ?proximity;
FILTER (?proximity>10) }
SELECT prox
FROM sens.win:time(5 sec)
WHERE prox >10
π timed,prox
ω
σprox>10
5 Seconds
sens
Data
translation
SNEE
Esper
GSN
Cosm
pull/push
https://github.com/jpcik/morph-streams
Other

SPARQLStream Language
57
FROM NAMED STREAM
ISTREAM
DSTREAM
RSTREAM
WINDOW
Underlying
data source
restrictions

SPARQLStream Language
 NamedStream  ‘FROM’ [‘NAMED’] ‘STREAM’ StreamIRI ‘[’ Window ‘]’
 Window  ‘NOW-’ Integer TimeUnit [UpperBound] [Slide]
 UpperBound  ‘TO NOW-’ Integer TimeUnit
 Slide  ‘SLIDE’ Integer TimeUnit
 TimeUnit  ‘MS’ | ‘S’ | ‘MINUTES’| ‘HOURS’ | ‘DAY’
 Select  ‘SELECT’ [Xstream] [Distinct | Reduced] …
 Xstream  ‘RSTREAM’ | ‘ISTREAM’ | ‘DSTREAM’
58
SELECT ISTREAM ?room
FROM NAMED STREAM <http://www.streamreasoning.org/streams/socialsensor.srdf> [NOW-10 S]
WHERE {…

SPARQLStream: examples
59
PREFIX sr4ld: <http://www.streamreasoning.org/ontologies/socialsensor,owl#>
SELECT ?room
WHERE {
?obs sr4ld:observedBy ?sensor.
?obs sr4ld:where ?room.
}
SPARQLStream
All rooms where something was observed in the last 10s
PREFIX sr4ld: <http://www.streamreasoning.org/ontologies/socialsensor,owl#>
SELECT (COUNT(?person) AS ?nmb) ?room
WHERE {
?obs sr4ld:who ?pers.
?obs sr4ld:where ?room.
}
GROUP BY ?room
Number of persons observed in each room in the last 10s

Underlying Query Processors
Esper
• CEP/DSMS
• EPL language
SNEE
• DSMS/Sensor Network Query Evaluator
• Compile queries to sensor code
GSN
• Sensor middleware
• REST API
Cosm/Xively
• Sensor middleware
• Open platform
• REST API
60
SELECT prox FROM sensors [FROM NOW-5
MINUTES TO NOW]
WHERE prox >10
SELECT prox FROM sensors.win:time(5 minute)
WHERE prox >10
http://montblanc.slf.ch:22001/multidata?vs[0]=sens
ors&
field[0]=proximity_field&c_min[0]=10&
from=15/05/2012+05:00:00&to=15/05/2012+10:00:
00
http://api.cosm.com/v2/feeds/14321/datastreams/
4?start=2012-05-15T05:00:00Z&end=2012-05-
15T10:00:00Z

Morph-streams: Overview
61
Query
rewriting
Query
Processing
Client
SPARQLStream
[tuples]
[triples/bin
dings]
Algebra
expression
R2RML
Mappings
SELECT ?proximity
FROM STREAM
<http://streamreasoning.org/SensorReadings.srdf> [NOW–5
S]
WHERE {
SELECT prox
FROM sens.win:time(5 sec)
WHERE prox >10
π timed,prox
ω
σprox>10
5 Seconds
sens
Data
translation
SNEE
Esper
GSN
Cosm
pull/push
Other

3rd: Mapping the two models
62
Observation
Sensor
Person
Roomwhere
who
observes
subClassOf
(person, room,…)
detections
Define mappings

R2RML – There is a recommendation!
63
We can use the W3C
recommendation

R2RML - Overview
64

Encoding in R2RML
65
:triplesMap a rr:TriplesMap;
rr:logicalTable [ rr:tableName ”sensors"; ]
rr:subjectMap [
rr:template "http://streamreasoning.org/data/Observation/{person}{timed}";
rr:class sr4ld:Observation; rr:graph sr4ld:socialstream.srdf ];
rr:predicate sr4ld:who ;
rr:objectMap [ rr:template “http://streamreasoning.org/data/Person/{person}” ]];.
the stream
name
subject URI
triple predicate + object
Mapping definition
stream
attributes
the object (a URI in this case)

Underlying Query Processors
66
SELECT ?proximity
FROM STREAM <http://streamreasoning.org/SensorReadings.srdf>
[NOW–5 S]
WHERE {
SELECT prox FROM sensors [FROM NOW-5 MINUTES TO NOW]
WHERE prox >10
timed,
prox
π
ω
σprox>10
5 Seconds
sensors
SELECT prox FROM sensors.win:time(5 minute)
WHERE prox >10
http://montblanc.slf.ch:22001/multidata?vs[0]=sensors&field[0]=proximi
ty_field&c_min[0]=10&
from=15/05/2012+05:00:00&to=15/05/2012+10:00:00
http://api.cosm.com/v2/feeds/14321/datastreams/4?start=2012-05-
15T05:00:00Z&end=2012-05-15T10:00:00Z
Query
rewriting
R2RML
SNEE (DSMS)
Esper (CEP)
GSN (middlwr)
Cosm Xively
SPARQLStream

Underlying query processors
Features Esper SNEE GSN Cosm/Xively
Projection ✔ ✔ ✔ Fixed
Proj expression ✔ ✔ ✖ ✖
Joins ✔ ✔✖ only window ✖ ✖
Union ✖ ✔✖ not windows ✔ ✖
Selection ✔ ✔ ✔ ✖✔ limited
Aggregates ✔ ✔ ✔✖ ✖
Time window ✔ ✔ ✔ ✔
Tuple window ✔ ✔ ✔ ✖
R2S ✔ ✔ ✖ ✖
Conjunction, Disj ✔ ✖ ✖ ✖
Repetition pattern ✔ ✖ ✖ ✖
Sequence ✔ ✖ ✖ ✖
67

Morph-streams: With reasoning!
68
Query
rewriting
Query
Processing
Client
SPARQLStream
[tuples]
[triples/bin
dings]
Algebra
expression
R2RML
Mappings
Data
translation
SNEE
Esper
GSN
Cosm
pull/push
Other
Ontology
TBox
Rewrite taking into account
the ontology TBox

Reasoning with Morph-streams
 Rewriting the SPARQLStream queries:
69
SELECT ?x
FROM NAMED STREAM <http://linkeddata.es/s/obs.srdf> [NOW - 5 MINUTES]
WHERE {
?x ssn:observedBy ?y
}
SELECT ?x
FROM NAMED STREAM <http://linkeddata.es/s/obs.srdf> [NOW - 5 MINUTES]
WHERE {
{?x ssn:observedBy ?y}
UNION
{?x a ssn:Observation}
UNION
{?x a aws:TemperatureObservation}
UNION
{?x a aws:HumidityObservation}
}

Morph-streams: With reasoning!
70
Query
rewriting
Query
Processing
Client
SPARQLStream
[tuples]
[triples/bin
dings]
Algebra
expression
R2RML
Mappings
Data
translation
SNEE
Esper
GSN
Cosm
pull/push
Other
Ontology
TBox
Rewrite only happens once
Query
rewriting
+
translation
Then continuous query is registered

Now some code
Morph-streams:
 Coded in Scala
 JAR bundle, use it from Scala or Java code
 Maven, Sbt
 Examples
• One off query
• Register continuous query
• Pull data
• Push
• Basic REST
 https://github.com/jpcik/morph-streams
 https://github.com/jpcik/morph-web
71

Code examples
 Parse SPARQLStream
val query= “PREFIX sr4ld: <…>. SELECT ?a …”
val syntax= StreamQueryFactory.create(query);
 Execute One-off query
val query= “PREFIX sr4ld: <…>. SELECT ?a …”
mapping=Mapping(new URI(mappings/social.ttl))
val adapter:QueryEvaluator=Application.adapter(system)
val results= adapter.executeQuery(query,mapping)
72
Mapping
Bindings

Code examples
 Register and Pull
val queryid= adapter.registerQuery(query,mapping)
val results1=adapter.pull(queryid)
val results2=adapter.pull(queryid)
 Register and Push
class ExampleReceiver extends StreamReceiver{
override def receiveData(s:SparqlResults):Unit=
Logger.debug("got: "+res)
}
val receiver=new ExampleReceiver
val queryid= adapter.listenToQuery(query,mapping,receiver)
73
Query identifier
Implement receiver
For Java users: Exactly the same interface!

Querying RSPs in Practice

ExecContext context=new ExecContext(HOME, false);
String queryString =" SELECT ?person ?loc …
ContinuousSelect selQuery=context.registerSelect(queryString);
selQuery.register(new ContinuousListener()
{
public void update(Mapping mapping){
String result="";
for(Iterator<Var> vars=mapping.vars();vars.hasNext();)
result+=" "+ context.engine().decode(mapping.get(vars.next()));
System.out.println(result);
}
});
RSP Querying
Example with CQELS (code.google.com/p/cqels)
CQELS continuous query:
get result updates
adding
listener
register
query
SELECT ?person ?loc WHERE {
STREAM <http://deri.org/streams/rfid> [RANGE 3s]
{?person :detectedAt ?loc}
}
Tightly coupled listeners
Results delivery: push & pull?

CQELS fed by a TripleWave WebSocket
val conf = ConfigFactory.load.getConfig("experiments.rsp")
val qq="""CONSTRUCT { ?s ?p ?o }
WHERE {
STREAM <ws://localhost:4040/primus> [RANGE 0ms]
{?s ?p ?o}
}"""
val sys=new RspSystem("wstreams")
val cqels=new CqelsEngine
sys.startStream(Props(
new WebSocketStream(cqels,"ws://localhost:4040/primus",conf)))
cqels.registerQuery(qq, cqels.createListener(lissy))
def lissy(triples:TripleList):Unit={
println("tikki: "+triples)
}

Similar models,
similar (not equals!) query languages
77
SELECT ?sensor
FROM NAMED STREAM <http://www.cwi.nl/SRBench/observations> [NOW-3 HOURS SLIDE 10 MINUTES]
WHERE {
?observation om-owl:procedure ?sensor ;
om-owl:observedProperty weather:WindSpeed ;
om-owl:result [ om-owl:floatValue ?value ] . }
GROUP BY ?sensor HAVING ( AVG(?value) >= "74"^^xsd:float )
SELECT ?sensor
WHERE {
STREAM <http://www.cwi.nl/SRBench/observations> [RANGE 10800s SLIDE 600s] {
om-owl:result [ om-owl:floatValue ?value ] .} }
SELECT ?sensor
FROM STREAM <http://www.cwi.nl/SRBench/observations> [RANGE 1h STEP 10m]
WHERE {
om-owl:result [ om-owl:floatValue ?value ] . }
SPARQLStream
CQELS
C-SPARQL

Query using SQL on Streams
Model
Continuous
execution
Union,Join,
Optional,
Filter
Aggregates
Timewindow
Triple
window
R2Soperator
Sequence,
Co-
ocurrence
Time
function
TA-SPARQL TA-RDF ✗ ✔ Limite
d
✗ ✗ ✗ ✗ ✗
tSPARQL tRDF ✗ ✔ ✗ ✗ ✗ ✗ ✗ ✗
Streaming
SPARQL
RDF
Stream
✔ ✔ ✗ ✔ ✔ ✗ ✗ ✗
C-SPARQL RDF
Stream
✔ ✔ ✔ ✔ ✔ ✗ ✗ ✔
CQELS RDF
Stream
✔ ✔ ✔ ✔ ✔ ✗ ✗ ✗
SPARQLStrea
m
(Virtual)
RDF
Stream
✔ ✔ ✔ ✔ ✗ ✔ ✗ ✗
EP-SPARQL RDF
Stream
✔ ✔ ✔ ✗ ✗ ✗ ✔ ✗
Instans RDF ✔ ✔ ✔ ✗ ✗ ✗ ✗ ✗
W3C RSP
 review features in existing
systems
 agree on fundamental
operators
 discuss on possible
semantics
https://www.w3.org/community/rsp/wiki/RSP_Query_Features
RSP is not always/only SPARQL-
like querying
SPARQL protocol is not enough
RSP RESTful interfaces?
Powerful languages for
continuous query processing

W3C RSP-CG: RSP-QL
PREFIX e: <http://somevocabulary.org/>
PREFIX s: <http://someinvasivesensornetwork.org/streams#>
PREFIX g: <http://somesocialnetwork.org/graphs#>
PREFIX : <http://acrasycompany.org/rsp>
REGISTER STREAM :GallehaultWasTheBar
UNDER ENTAILMENT REGIME <http://www.w3.org/ns/entailment/RIF> AS CONSTRUCT ISTREAM {
?poi rdf:type :Gallehault ;
:count ?howmanycouples ;
:for (?somebody ?someoneelse) }
FROM NAMED WINDOW :veryLongWindow ON s:1 [RANGE PT4H STEP PT1H]
FROM NAMED WINDOW :longWindow ON s:1 [FROM NOW-PT35M TO NOW-PT5M STEP PT5M]
FROM NAMED WINDOW :shortWindow ON s:1 [RANGE PT10M STEP PT5M]
FROM NAMED GRAPH g:SocialGraph
FROM GRAPH g:POIs
WHERE {
?poi rdf:type e:bar .
WINDOW :veryLongWindow {
{?somebody e:enters ?poi} BEGIN AT ?t3
{?someoneelse e:enters ?poi} BEGIN AT ?t4
FILTER(?t3>?t4)
}
WINDOW :longWindow {
{ ?somebody e:isCloseTo ?someoneelse MINUS {
?somebody e:isCloseTo ?yetanotherone .
FILTER (?yetanotherone != ?someoneelse) }
} WITH DURATION ?duration
FILTER (?duration>="PT30M"^^xsd:duration)
}
WINDOW :shortWindow {
{ ?somebody e:exits ?bar} BEGIN AT ?t1
{ ?someoneelse e:exits ?bar } BEGIN AT ?t2
FILTER (abs(?t2-?t1)<"PT1M"^^xsd:duration )
}
GRAPH g:SocialGraph {
FILTER NOT EXIST { ?somebody e:knows ?someoneelse }
}
FILTER (?somebody != ?someoneelse)
}
AGGREGATE { GROUP BY ?poi COUNT(?somebody) AS ?howmanycouples }
Continuously look for
bars where people are
falling in love (because o
a book )
Register
stream
Time windows
Sequencing
Duration
Stored Graphs
Aggregates
Access to time
Reasoning
They entered the same bar
They are close to each other,
with no-one else
They get out together
Didn’t know each other

RSP Communication

RDF Stream Processing
RSP
Engine
RDF
graphs
input RDF streams streams of results
background
knowledge
continuous queries
streamproducers
RSP
Engine
producer
subscribe
notify
cont. query consumer
push results
subscribe
streamconsumers
continuous queries
RSP Implementations

Reactive Systems
Event-Driven
Jonas Boner. Go Reactive: Event-Driven, Scalable, Resilient & Responsive
Systems. 2013.
Events:reactto
ScalableLoad:
ResilientFailure:
ResponsiveUsers:

Actor Model
Actor
1
Actor
2
m No shared mutable state
Avoid blocking operators
Lightweight objects
Loose coupling
communicate
through messages
mailbox
state
behavior
non-blocking response
send: fire-forget
Implementations: e.g. Akka for Java/Sc
Parent
Actor
1
Supervisi
on
hierarchy
Supervision
Actor
2
Acto
r4
X
Actor
2
Actor
1
Actor
2
m
Actor
3
Actor
4
m
m
Remoting

RDF Streams: Actors
val sys=ActorSystem.create("system")
val consumer=sys.actorOf(Props[RdfConsumer])
class Streamer extends StreamRDF{
override def triple(triple:Triple){
consumer ! triple
}
}
class RdfConsumer extends Actor{
def receive= {
case t:Triple =>
if (t.predicateMatches(RDF.‘type‘))
println(s"received triple $t")
}
RDF consumer
Actor receive method
Implements behavior
Message-passing model
RDF producer
Async message passing

RSP Producer & Consumer
Proce
ssor
faster producers >> slower processor/consumer
Prod
ucer
Prod
ucer
Prod
ucer
Prod
ucer
RDF streams Cons
umer
Cons
umer
Cons
umer
unresponsive
overload
Overload of the processor/receiver
Unresponsiveness in stream processor

Dynamic Push-Pull
Producer
Consumer
m
data flow
demand flow
Push when consumer is faster
Pull when producer is faster
Dynamically switch modes
Communication is dynamic
depending on demand vs supply
Producer Consumer
m
m
m m
m
m
m
m
m
m
push

Evaluation: throughput
Basic dynamic pull push
On top of CQELS
Limitations of Thread model
Not yet fully async

Reactive RSP workflows
Morph
Streams
CSPARQL
s
Etali
s
TrOWL
s
s CQELS
Dyn
amit
e
s
Minimal agreements: standards, serialization, interfaces
Formal models for RSPs and reasoning
Working prototypes/systems!
Event-driven message pass
Async communication
Immutable streams
Transparent Remoting
Parallel and distributed
Supervised Failure Handling
Responsive processing
Reactive RSPs

RSPs and the Linked Data Principles

URIs as Names of Things
http://mysensorweb.me/mytemperature/20151110Z10:00:
00
Different concepts
http://mysensorweb.me/mytemperature/latest
http://mysensorweb.me/mytemperature/lastMinute
http://mysensorweb.me/mytemperature/lastMonth
Different granularities
Different uses
http://mysensorweb.me/mytemperature/avgLastMonth
http://mysensorweb.me/mytemperature

HTTP URIs
http://mysensorweb.me/mytemperature/latest
Internet of Things
How about XMPP, CoAP, MQTT?
Websockets?

De-referenceable URIs
GET http://mysensorweb.me/mytemperature/latest
:Obs1 a my:TemperatureObservation;
my:hasValue 33.5 ;
my:hasUnit u:Celsius;
my:atTime “20151110Z10:00:00”.
GET http://mysensorweb.me/mytemperature
Get the whole stream?
GET http://mysensorweb.me/mytemperature/lastMonth
Get continuous updates?

Link to other URIs
• Broken links?
• Mix streaming and stored data
• Persist or not persist?
• Volatile links?
http://mysensorweb.me/mytemperature/20151110Z10:00:00

For Java/Scala developers

Semantic Web Devs
Have fun
Love challenges
Need cool tools
RDF4J

Scala: Functions and Objects
JVM language
Both object and functional oriented
Easy Java-interop
Reuse Java libraries
Growing community

RDF in Jena: in Scala
String personURI = "http://somewhere/JohnSmith";
Model model = ModelFactory.createDefaultModel();
model.createResource(personURI).addProperty(VCARD.FN,"John Smith");
Type
inference
Not too useful ;
and ()
Terser & compact code
Type-safe DSL
Compiler takes care
val personURI = "http://somewhere/JohnSmith"
val model = ModelFactory.createDefaultModel
model.createResource(personURI).addProperty(VCARD.FN,"John Smith")
sw:John
Smith
“John Smith”
vcard:FN
implicit val model = createDefaultModel
add(personURI,VCARD.FN->"John Smith")
boilerplate
String
converted to
Resource

Some more RDF
99
String personURI = "http://somewhere/JohnSmith";
String givenName = "John";
String familyName = "Smith";
String fullName = givenName + " " + familyName;
Model model = ModelFactory.createDefaultModel();
model.createResource(personURI)
.addProperty(VCARD.FN,fullName)
.addProperty(VCARD.N,model.createResource()
.addProperty(VCARD.Given,givenName)
.addProperty(VCARD.Family,familyName));
val givenName = "John"
val familyName = "Smith"
val fullName = s"$givenName $familyName"
implicit val model = createDefaultModel
add(personURI,VCARD.FN->fullName,
VCARD.N ->add(bnode,VCARD.Given -> givenName,
VCARD.Family->familyName))
sw:John
Smith
“John Smith”
vcard:FN
_:n
“John”
“Smith”vcard:N
vcard:Giv
en
vcard:Fam
ily
Blank node
Scala DSLs customizable
Predicate-objects are pairs

Some more RDF in Jena
implicit val m=createDefaultModel
val ex="http://example.org/"
val alice=iri(ex+"alice")
val bob=iri(ex+"bob")
val charlie=iri(ex+"charlie")
alice+(RDF.`type`->FOAF.Person,
FOAF.name->"Alice",
FOAF.mbox->iri("mailto:alice@example.org"),
FOAF.knows->bob,
FOAF.knows->charlie, FOAF.knows->bnode)
bob+ (FOAF.name->"Bob",
FOAF.knows->charlie)
charlie+(FOAF.name->"Charlie",
FOAF.knows->alice)
Still valid Jena RDF
You can do it even nicer

Exploring an RDF Graph
ArrayList<String> names=new ArrayList<String>();
NodeIterator iter=model.listObjectsOfProperty(VCARD.N);
while (iter.hasNext()){
RDFNode obj=iter.next();
if (obj.isResource())
names.add(obj.asResource()
.getProperty(VCARD.Family).getObject().toString());
else if (obj.isLiteral())
names.add(obj.asLiteral().getString());
}
val names=model.listObjectsOfProperty(VCARD.N).map{
case r:Resource=>
r.getProperty(VCARD.Family).obj.toString
case l:Literal=>
l.getString
}
Imperative iteration of collections
Type-based conditional execution
Type casting
Case type
Map applied to operators

Query with SPARQL
val queryStr = """select distinct ?Concept
where {[] a ?Concept} LIMIT 10"""
val query = sparql(queryStr)
query.serviceSelect("http://dbpedia.org/sparql").foreach{implicit qs=>
println(res("Concept").getURI)
}
val f=Future(query.serviceSelect("http://es.dbpedia.org/sparql")).fallbackTo(
Future(query.serviceSelect("http://dbpedia.org/sparql")))
f.recover{
case e=> println("Error "+e.getMessage)
}
f.map(_.foreach{implicit qs=>
println(res("Concept").getValue)
})
Remote SPARQL endpoin
Simplified access to
Query solutions
Futures: asnyc execution
Non blocking code
Fallback alternative execut

Muchas Gracias!

RDF Stream Processing Tutorial: RSP implementations

RDF Stream Processing Tutorial: RSP implementations

Recommended

More Related Content

What's hot

What's hot(20)

Similar to RDF Stream Processing Tutorial: RSP implementations

Similar to RDF Stream Processing Tutorial: RSP implementations(20)

More from Jean-Paul Calbimonte

More from Jean-Paul Calbimonte(20)

Recently uploaded

Recently uploaded(20)

RDF Stream Processing Tutorial: RSP implementations