Successfully reported this slideshow.
Upcoming SlideShare
×

# On correctness in RDF stream processor benchmarking

662 views

Published on

• Full Name
Comment goes here.

Are you sure you want to Yes No
• Be the first to comment

• Be the first to like this

### On correctness in RDF stream processor benchmarking

1. 1. On correctness in RDF stream processor benchmarking Daniele Dell’Aglio, Jean-Paul Calbimonte, Marco Balduini, Oscar Corcho and Emanuele Della Valle
2. 2. The correctness problem (1) :alice :isIn :hall :alice :isIn :kitchen :bob :isIn :hall :bob :isIn :kitchen S S1 S2 S3 S4 1 3 6 9 t Executio 1° answer 2° answer Where are Alice and n Bob, when they are 1 :hall [6] :kitchen [11] together? 2 :hall [5] :kitchen [10]  Let’s consider a 3 :hall [6] :kitchen [11] tumbling window 4 - [7] - [12] slide Which is the correct answer? W(ω=β=5) All of them!  Let’s execute the width experiment 4 times processor benchmarking 2/16 On correctness in RDF stream ISWC, 24 October 2013 
3. 3. The correctness problem (2) :alice :isIn :hall :alice :isIn :kitchen :bob :isIn :hall :bob :isIn :kitchen S S1 S2 S3 S4 1 3 6 9 t System 1 System 2 Executio n 1° answer 2° answer Executio n 1° answer 2° answer 1 :hall [6] :kitchen [11] 1 :hall [3] :kitchen [9] 2 :hall [5] :kitchen [10] 2 3 :hall [6] :kitchen [11] 3 4 - [7] - [12] 3/16 No answers :hall [3] :kitchen [9] 4 No answers Which system behaves in the Both! correct way? On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
4. 4. Different results for the same query? Yes! Given the same data and the same query, we notice that a system can provide multiple correct answers    The operational semantics of the systems often do not explain this behaviour  Two systems can provide different correct answers  Even if the system implements similar operational semantics, they behave in different ways (that the model does not explain) Why is it important to understand those behaviours?   4/16 To assess the correct implementation of the systems To improve the comprehension of the benchmarking On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
5. 5. Our contribution   A common model for the RDF stream processor operational semantics We propose CSR-bench, an extension of the SRbench benchmark that focuses on correctness    An oracle (an automatic correctness validator) A test suite We considered the window-based RDF stream processors    5/16 CQELS C-SPARQL SPARQLstream On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
6. 6. The CSR Model (by CQL) Input stream  S2R operator   R2R operator   R2S operator  Converts the infinite stream of RDF elements in a finite set of mappings The window operators: time-based, tuple-based, … Transform a set of mappings in another set of mappings SPARQL 1.0/1.1 queries Each set of mapping produced by the R2R operator is transformed and appended to the output stream Operators: RStream, DStream, IStream Output stream 6/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
7. 7. The window operator (through SECRET) REPORT: When is the window content made available to the R2R operator? Non-empty content, Content-change, Window-close, Periodic t0: When does the window start? (internal window param) R2R operator ω β TICK: When are data stream elements added to the window? Triple-based vs graph-based S W(ω,β) S1 S3 S2 S4 S6 S5 S7 S9 S11 S8 S10 S12 t 7/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
8. 8. Classification of existing systems   The report and the tick policies are related to the RSPs implementation and on how they implement the window operators Analysing the RSPs and their related documentation (papers, technical reports, etc.), we classified the systems: CQELS Report Content-change Tick Tuple-driven Empty relation No notification 8/16 C-SPARQL SPARQLstream Window-close Non-empty content Window-close Non-empty content Tuple-driven Tuple-driven Yes No On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
9. 9. RSP output correctness :alice :isIn :hall :alice :isIn :kitchen :bob :isIn :hall :bob :isIn :kitchen S S1 S2 S3 S4 1 3 6 9 t t0=0 t0=1 t0=2 Execution 1° answer 2° answer 1 :hall [6] :kitchen [11] 2 :hall [5] :kitchen [10] 3 :hall [6] :kitchen [11] 4 - [7] - [12] 9/16 Window 1° answer 2° answer t0=0 t0=1 t0=2 :hall [5] :kitchen [10] :hall [6] :kitchen [11] - [7] - [12] On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
10. 10. RSP output correctness :alice :isIn :hall :alice :isIn :kitchen :bob :isIn :hall :bob :isIn :kitchen S S1 S2 S3 S4 1 3 6 9 t System 1 System 2 Executio n 1° answer 2° answer Executio n 1° answer 2° answer 1 :hall [6] :kitchen [11] 1 :hall [3] :kitchen [9] 2 :hall [5] :kitchen [10] 2 3 :hall [6] :kitchen [11] 3 4 - [7] - [12] 4 Window-close vs content-change report policy 10/16 No answers :hall [3] :kitchen [9] No answers Empty relation notification (yes|no) On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
11. 11. The oracle Online RSP q S Stream importer Query transformer Query executor Result matcher M Correctness assessment Offline Available at: https://github.com/dellaglio/csrbench-ora (Apache 2.0 licence) 11/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
12. 12. Design of the tests  As data set, we consider the LinkedSensorData data set   We designed the query set taking into account      Data stream describing blizzards and hurricanes in the US Window size and slide parameters Presence of aggregation operators Joins of timestamped triples We collected a set of seven parametrized queries The list of the queries and their explanation is available at: http://www.w3.org/wiki/CSRBench 12/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
13. 13. Results      Initialization Slide parameter Window contents timestamps of the triples  13/16 Internal timestamp management On correctness in RDF stream processor benchmarking SPARQLstrea the window operator C-SPARQL  CQELS  All the three systems that we considered in our experiments showed wrong behaviours The defects we identified are related to: Q1 Q2 Q3 Q4 Q5 Q6 Q7 ISWC, 24 October 2013
14. 14. (Removable) constraints Reasonin g From single to multi stream S2R S2R R2R R2S S2R From single to multi window q2 14/16 Multiple queries On correctness in RDF stream processor benchmarking Static knowledge ISWC, 24 October 2013
15. 15. Conclusions  A model that describes in a more accurate way the RSPs’ operational semantics helps their improvement:    A common and shared test environment helps developers of both existing and upcoming RSPs   Better design of the system Prediction of the expected behaviours It becomes easier to set up experiments to detect defects and to correct them Possible improvements of the existing benchmarks (e.g., SRbench and LSBench)   Design of new tests Better interpretation of the experiment results 15/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013
16. 16. Thank you! Questions? On correctness in RDF stream processor benchmarking Daniele Dell’Aglio (DEIB, Politecnico di Milano) Jean-Paul Calbimonte (OEG, Universidad Politécnica de Madrid) Marco Balduini (DEIB, Politecnico di Milano) Oscar Corcho (OEG, Universidad Politécnica de Madrid) Emanuele Della Valle (DEIB, Politecnico di Milano) wiki: http://www.w3.org/wiki/CSRBench software: https://github.com/dellaglio/csrbench-oracle 16/16 On correctness in RDF stream processor benchmarking ISWC, 24 October 2013