The document summarizes benchmarking tests conducted on several semantic repositories using the Press Association dataset and University Ontology Benchmark dataset. BigOWLIM demonstrated the best average query response times and answered the most queries, but was slower for loading and modification tests. Sesame, Jena, Virtuoso and Oracle also had sub-second response times for most queries. Allegrograph answered more queries than those four but had the highest average response times. Further benchmarking on larger datasets and additional parameters was recommended.

1. A Pragmatic Approach to Semantic Repositories Benchmarking Dhaval Thakker , Taha Osman, Shakti Gohil, Phil Lakin © Dhaval Thakker, Press Association , Nottingham Trent University

2. Outline Introduction to the Semantic Technology Project at PA Images Semantic Repository benchmarking Parameters Datasets Results and Analysis Loading and querying results Modification tests Conclusions

3. http://www.pressassociation.com Press Association & its operations UK’s leading multimedia news & information provider Core News Agency operation Content and Editorial services: Sports data, entertainment guides, weather forecasting, photo syndication Semantic Web project Benchmarking Results Conclusions

4. Press Association Images Semantic Web project Benchmarking Results Conclusions

5. Current Search Engine Semantic Web project Benchmarking Results Conclusions

6. Browsing Engine Images of Sports, Entertainment, News domain entities: people, events, locations etc. Lacks a meta-data rich browsing engine functionality that can utilize these entities for a greater browsing experience. To help the searchers to browse through images based on these entities and their relationships Semantic web based browsing engine Semantic Web project Benchmarking Results Conclusions

7. Semantic Repository Benchmarking “ a tool, which combines the functionality of an RDF-based DBMS and an inference engine and can store data and evaluate queries, regarding the semantics of ontologies and metadata schemata.” * Criteria for selection: The analytical parameters, such as expected level of reasoning and query language support Selected semantic repositories AllegroGraph, BigOWLIM, Oracle, Sesame,TDB Jena, Virtuoso Semantic Web project Benchmarking Results Conclusions * Kiryakov, A, Measurable Targets for Scalable Reasoning., Ontotext Technology White Paper, Nov 2007.

8. PA Dataset Ontology – OWL-lite to OWL-DL Classification, subproperties, inverse properties and hasValue for automatic classification 147 classes, 60 object properties and 30 data properties Knowledge base – Entities 6.6 M triples Approx 1.2M entities Disk space: 1.23 GB Image annotations - Each Image: 2-4 triples. 8M triples Approx 5 M images Disk space: 1.57 GB Semantic Web project Benchmarking Results Conclusions

9. Published Benchmarks & datasets The Lehigh University Benchmark (LUBM) first standard platform to benchmark OWL systems but it gradually fell behind with the increasing expressivity of OWL reasoning The University Ontology Benchmark (UOBM) benchmark improve the reasoning coverage of LUBM OWL-DL and OWL-Lite inferencing Berlin SPARQL benchmark (BSBM) BSBM focuses provides comprehensive evaluation for SPARQL query features. Semantic Web project Benchmarking Results Conclusions

10. Benchmarking Parameters Classification of semantic stores in Native, Memory-based or Database-based (A) Forward-chaining or backward-chaining (A) Load Time (P) Query Response time (P) Query results analysis (P) RDF store update tests (P) Study different serialisation and impact on performance Scalability (A) Reasoner Integration (A) Query Language supported (A) Clustering supported (A) Programming Languages support (A) Platform support (A) RDFview support (support for non-rdf data) (A) Semantic Web project Benchmarking Results Conclusions

11. UOBM: Load time Semantic Web project Benchmarking Results Conclusions

12. Load Time: PA Dataset Semantic Web project Benchmarking Results Conclusions

13. Dataset Queries Measuring query execution speed SPARQL queries From JAVA based client Measured execution speed based on three runs PA Dataset: 13 queries to test the expressiveness supported Subsumption, Inverse properties (Q6,Q12, Q15), Automatic classification UOBM Dataset: 15 queries - 12 queries fall under OWL-Lite and 3 queries are of OWL-DL expressivity Q5 and Q7 involves transitive ( owl:TransitiveProperty ) Q6 relies on semantic repositories to support ( owl:inverseOf) Q10 requires symmetric Semantic Web project Benchmarking Results Conclusions

14. UOBM: Query execution Partially Answered N Query was not answered by this tool Semantic Web project Benchmarking Results Conclusions N N N N N N Q15 0.016 N N N N(infinite) N Q14 N N N N N N Q13 0.016 N N N 476.507 N Q12 0.062 N(infinite) 0.094(P) 0.001(P) N(infinite) N Q11 0.016 0.001 0.001 0.001(P) 0.25 0 Q10 0.031(P) N N N N N Q9 0.031 N N N 6.843(P) N Q8 0.001 N N N 300.12 N Q7 0.047 N N N 1153.025 N Q6 0.047 N N N 1.281 N Q5 0.063 120 0.14 N N(infinite) N Q4 0.062 0.016 0.109 N 651.237 N Q3 0.062 0.001(P) 0.001(P) N 8.906(P) N Q2 0.047 0.031 0.203 0.141 21.921 6.766 (P) Q1 BigOWLIM Jena TDB Sesame Oracle Allegrograph Virtuoso No. Execution Timings (seconds)

15. PA Dataset: Query execution Semantic Web project Benchmarking Results Conclusions Partially Answered N Query was not answered by this tool 0.031 N N 1.688 N Q15 0.641 0.001 0.016(P) 1.812 5.563(P) Q13 0.079 N N 16.14 N Q12 0.062 0.001 0.11 1.734 N Q11 0 N 0.047 1.734 0.001 Q10 0.016 N 0.171 1.782 0.156 Q9 0.062 0.001 0.11 3.39 0.047 Q8 0.093 N 0.203 28.688 84.469 Q7 0.45 0.001 N 3.765 N Q6 0.078 N 0.141 1.719 0.172 Q5 0.047 N N N N Q4 0.063 N N N N Q2 0.219 0.047 0.469(P) 26.422 2.234 (P) Q1 BigOWLIM Jena TDB Sesame Allegrograph Virtuoso Query No.

16. Two complete stores: BigOWLIM v/s Allegrograph Semantic Web project Benchmarking Results Conclusions

17. Two fast stores: BigOWLIM v/s Sesame Semantic Web project Benchmarking Results Conclusions

18. Modification Tests: Insert Semantic Web project Benchmarking Results Conclusions

19. Modification Tests: Update & Delete Semantic Web project Benchmarking Results Conclusions

20. Conclusions PA Dataset benchmarking Essential and desirable requirements of our application into a set of functional (practical) and non-functional (analytical) parameters To consolidate our findings we use UOBM, a public benchmark that satisfies the requirements of our target system Analysis All the repositories are sound ..however not complete BigOWLIM provides the best average query response time and answers maximum number of queries for both the datasets. But Slower in loading, modification tests. Sesame, Jena, Virtuoso and Oracle offered sub-second query response time for the majority of queries they answer. Allegrograph answers more queries than the former four repositories hence offers better coverage of OWL properties. Average query response time for Allegrograph was the highest for both the dataset Further work Expanding this benchmark exercise to billion triples More repositories Adding extra benchmarking parameters such as the performance impact of concurrent users and transaction-related operations Semantic Web project Benchmarking Results Conclusions