On correctness in RDF stream processor
benchmarking
Daniele Dell’Aglio, Jean-Paul Calbimonte, Marco Balduini,
Oscar Corcho...
The correctness problem (1)
:alice :isIn :hall
:alice :isIn :kitchen
:bob :isIn :hall
:bob :isIn :kitchen

S

S1

S2

S3

...
The correctness problem (2)
:alice :isIn :hall
:alice :isIn :kitchen
:bob :isIn :hall
:bob :isIn :kitchen

S

S1

S2

S3

...
Different results for the same query? Yes!
Given the same data and
the same query, we notice
that a system can provide
mul...
Our contribution



A common model for the RDF stream processor
operational semantics
We propose CSR-bench, an extension...
The CSR Model (by CQL)
Input stream


S2R
operator





R2R
operator





R2S
operator


Converts the infinite strea...
The window operator (through SECRET)
REPORT: When is the window content
made available to the R2R operator?
Non-empty cont...
Classification of existing systems




The report and the tick policies are related to the
RSPs implementation and on ho...
RSP output correctness
:alice :isIn :hall
:alice :isIn :kitchen
:bob :isIn :hall
:bob :isIn :kitchen

S

S1

S2

S3

S4

1...
RSP output correctness
:alice :isIn :hall
:alice :isIn :kitchen
:bob :isIn :hall
:bob :isIn :kitchen

S

S1

S2

S3

S4

1...
The oracle
Online
RSP

q

S
Stream
importer

Query
transformer

Query
executor

Result
matcher

M

Correctness
assessment
...
Design of the tests


As data set, we consider the LinkedSensorData data
set




We designed the query set taking into ...
Results








Initialization
Slide parameter
Window contents

timestamps of the triples


13/16

Internal timestam...
(Removable) constraints
Reasonin
g

From single to
multi stream

S2R
S2R

R2R

R2S

S2R

From single
to multi
window
q2

1...
Conclusions


A model that describes in a more accurate way the
RSPs’ operational semantics helps their
improvement:


...
Thank you! Questions?

On correctness in RDF stream processor
benchmarking
Daniele Dell’Aglio (DEIB, Politecnico di Milano...
Upcoming SlideShare
Loading in …5
×

On correctness in RDF stream processor benchmarking

454 views
351 views

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
454
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

On correctness in RDF stream processor benchmarking

  1. 1. On correctness in RDF stream processor benchmarking Daniele Dell’Aglio, Jean-Paul Calbimonte, Marco Balduini, Oscar Corcho and Emanuele Della Valle
  2. 2. The correctness problem (1) :alice :isIn :hall :alice :isIn :kitchen :bob :isIn :hall :bob :isIn :kitchen S S1 S2 S3 S4 1 3 6 9 t Executio 1° answer 2° answer Where are Alice and n Bob, when they are 1 :hall [6] :kitchen [11] together? 2 :hall [5] :kitchen [10]  Let’s consider a 3 :hall [6] :kitchen [11] tumbling window 4 - [7] - [12] slide Which is the correct answer? W(ω=β=5) All of them!  Let’s execute the width experiment 4 times processor benchmarking 2/16 On correctness in RDF stream ISWC, 24 October 2013 
  3. 3. The correctness problem (2) :alice :isIn :hall :alice :isIn :kitchen :bob :isIn :hall :bob :isIn :kitchen S S1 S2 S3 S4 1 3 6 9 t System 1 System 2 Executio n 1° answer 2° answer Executio n 1° answer 2° answer 1 :hall [6] :kitchen [11] 1 :hall [3] :kitchen [9] 2 :hall [5] :kitchen [10] 2 3 :hall [6] :kitchen [11] 3 4 - [7] - [12] 3/16 No answers :hall [3] :kitchen [9] 4 No answers Which system behaves in the Both! correct way? On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  4. 4. Different results for the same query? Yes! Given the same data and the same query, we notice that a system can provide multiple correct answers    The operational semantics of the systems often do not explain this behaviour  Two systems can provide different correct answers  Even if the system implements similar operational semantics, they behave in different ways (that the model does not explain) Why is it important to understand those behaviours?   4/16 To assess the correct implementation of the systems To improve the comprehension of the benchmarking On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  5. 5. Our contribution   A common model for the RDF stream processor operational semantics We propose CSR-bench, an extension of the SRbench benchmark that focuses on correctness    An oracle (an automatic correctness validator) A test suite We considered the window-based RDF stream processors    5/16 CQELS C-SPARQL SPARQLstream On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  6. 6. The CSR Model (by CQL) Input stream  S2R operator   R2R operator   R2S operator  Converts the infinite stream of RDF elements in a finite set of mappings The window operators: time-based, tuple-based, … Transform a set of mappings in another set of mappings SPARQL 1.0/1.1 queries Each set of mapping produced by the R2R operator is transformed and appended to the output stream Operators: RStream, DStream, IStream Output stream 6/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  7. 7. The window operator (through SECRET) REPORT: When is the window content made available to the R2R operator? Non-empty content, Content-change, Window-close, Periodic t0: When does the window start? (internal window param) R2R operator ω β TICK: When are data stream elements added to the window? Triple-based vs graph-based S W(ω,β) S1 S3 S2 S4 S6 S5 S7 S9 S11 S8 S10 S12 t 7/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  8. 8. Classification of existing systems   The report and the tick policies are related to the RSPs implementation and on how they implement the window operators Analysing the RSPs and their related documentation (papers, technical reports, etc.), we classified the systems: CQELS Report Content-change Tick Tuple-driven Empty relation No notification 8/16 C-SPARQL SPARQLstream Window-close Non-empty content Window-close Non-empty content Tuple-driven Tuple-driven Yes No On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  9. 9. RSP output correctness :alice :isIn :hall :alice :isIn :kitchen :bob :isIn :hall :bob :isIn :kitchen S S1 S2 S3 S4 1 3 6 9 t t0=0 t0=1 t0=2 Execution 1° answer 2° answer 1 :hall [6] :kitchen [11] 2 :hall [5] :kitchen [10] 3 :hall [6] :kitchen [11] 4 - [7] - [12] 9/16 Window 1° answer 2° answer t0=0 t0=1 t0=2 :hall [5] :kitchen [10] :hall [6] :kitchen [11] - [7] - [12] On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  10. 10. RSP output correctness :alice :isIn :hall :alice :isIn :kitchen :bob :isIn :hall :bob :isIn :kitchen S S1 S2 S3 S4 1 3 6 9 t System 1 System 2 Executio n 1° answer 2° answer Executio n 1° answer 2° answer 1 :hall [6] :kitchen [11] 1 :hall [3] :kitchen [9] 2 :hall [5] :kitchen [10] 2 3 :hall [6] :kitchen [11] 3 4 - [7] - [12] 4 Window-close vs content-change report policy 10/16 No answers :hall [3] :kitchen [9] No answers Empty relation notification (yes|no) On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  11. 11. The oracle Online RSP q S Stream importer Query transformer Query executor Result matcher M Correctness assessment Offline Available at: https://github.com/dellaglio/csrbench-ora (Apache 2.0 licence) 11/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  12. 12. Design of the tests  As data set, we consider the LinkedSensorData data set   We designed the query set taking into account      Data stream describing blizzards and hurricanes in the US Window size and slide parameters Presence of aggregation operators Joins of timestamped triples We collected a set of seven parametrized queries The list of the queries and their explanation is available at: http://www.w3.org/wiki/CSRBench 12/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  13. 13. Results      Initialization Slide parameter Window contents timestamps of the triples  13/16 Internal timestamp management On correctness in RDF stream processor benchmarking SPARQLstrea the window operator C-SPARQL  CQELS  All the three systems that we considered in our experiments showed wrong behaviours The defects we identified are related to: Q1 Q2 Q3 Q4 Q5 Q6 Q7 ISWC, 24 October 2013
  14. 14. (Removable) constraints Reasonin g From single to multi stream S2R S2R R2R R2S S2R From single to multi window q2 14/16 Multiple queries On correctness in RDF stream processor benchmarking Static knowledge ISWC, 24 October 2013
  15. 15. Conclusions  A model that describes in a more accurate way the RSPs’ operational semantics helps their improvement:    A common and shared test environment helps developers of both existing and upcoming RSPs   Better design of the system Prediction of the expected behaviours It becomes easier to set up experiments to detect defects and to correct them Possible improvements of the existing benchmarks (e.g., SRbench and LSBench)   Design of new tests Better interpretation of the experiment results 15/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
  16. 16. Thank you! Questions? On correctness in RDF stream processor benchmarking Daniele Dell’Aglio (DEIB, Politecnico di Milano) Jean-Paul Calbimonte (OEG, Universidad Politécnica de Madrid) Marco Balduini (DEIB, Politecnico di Milano) Oscar Corcho (OEG, Universidad Politécnica de Madrid) Emanuele Della Valle (DEIB, Politecnico di Milano) wiki: http://www.w3.org/wiki/CSRBench software: https://github.com/dellaglio/csrbench-oracle 16/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013

×