Presentation at ACM Conference - Semantics2017, September 11--14, 2017, Amsterdam, Netherlands
This work was supported by grants from the EU H2020 Framework Programme provided for the project HOBBIT (GA no. 688227).
Benchmarking Faceted Browsing Capabilities of Triple Stores
1. Benchmarking Faceted Browsing Capabilities
of Triple Stores
Horizon 2020
GA No 688227
01/12/2015 – 30/11/2018
Henning Petzka, Claus Stadler, Georgios Katsimpras, Bastian Haarmann, Jens Lehmann
13.09.2017
SEMANTiCS Amsterdam 2017
2. HOllistic Benchmarking of Big lInked daTa
Rationale:
A community-driven unified benchmarking platform for the community
• Focus on Big Linked Data
• Provide benchmarks and baselines
• Provide reference implementation of KPIs
• Extensible and referenceable
• Result analysis
• Open Source
http://project-hobbit.eu
7. • Benchmarks I: Generation & Acquisition
measures performance of SPARQL query processing systems when faced with streams of
data in terms of efficiency and completeness
• Benchmarks II: Analysis & Processing
test performance on instance matching tools for Linked Data and performance on machine
learning methods for data analytics
• Benchmarks III: Storage & Curation
has its focus on storage components and versioning systems to efficiently manage evolving
linked datasets
• Benchmarks IV: Visualization & Services
has its focus on benchmarks regarding question answering and faceted browsing.
8. Faceted Browsing
stands for a session-based and state-dependent
interactive method for query formulation over a multi-
dimensional information space.
A browsing scenario consists of applying (or removing) filter restrictions defined by
object-valued properties or of changing the range of a property value of various data
types.
11. Choke Points
! In a browsing scenario it is the efficient transition
from one state to next one that determines the user
experience !
Three basic types of transition
1. Class-based transition
2. Property- or property path-based transition
3. Entity type switch
14. Scenarios
• make sense in a real-world browing scenario and
• cover all types of transitions as specified by the choke points
15. Key Performance Indicators
• Instance retrieval:
• Query-per-second score
• Precision
• Recall
• F1-Score
• Facet counts:
• Query-per-second score
• Several metrics for accuracy
Over all queries and for each choke point
individually
16. MOCHA Challenge at ESWC 2017
Benchmark on Faceted Browsing was part of the
Mighty Storage Challenge at the ESWC 2017
Two participants vs. baseline system
• QUAD by Ontos
• Virtuoso 8.0 Commercial Edition (beta release)
vs. Virtuoso 7.2 Open-Source Edition
No results for QUAD due to time out.