The HOBBIT platform is an open-source benchmarking framework for big linked data systems, supported by the EU's H2020 programme. It allows users to configure and execute benchmarks, compare results, and add new systems for evaluation. The platform facilitates high-performance testing, providing both local and online instances for public challenges.

1. http://project-hobbit.eu
@hobbit_project
HOLISTIC
BENCHMARKING
of Big Linked Data
This work was supported by a grant from the EU H2020 Framework
Programme provided for the project HOBBIT (GA no. 688227).
HOBBIT Link Discovery Benchmarks at OM 2017
M. Röder2,3, T. Saveta1, I. Fundulaki1, and A.-C. Ngonga Ngomo2,3
michael.roeder@uni-paderborn.de, {jsaveta, fundul}@ics.forth.gr, axel.ngonga@upb.de
#1 Institute of Computer Science-FORTH Greece
#2 Institute for Applied Informatics, Germany
#3 Paderborn University, Germany
HOBBIT PLATFORM
The HOBBIT Platform serves as a framework for benchmarking Big Linked Data systems. Benchmarks that focus on the
evaluation of the quality of a system using consecutive requests as well as benchmarks aiming at efficiency can run on the
platform. The platform supports the handling of Big Linked Data to make sure that even for high-performance systems a
maximum load can be generated.
USE CASES
• Benchmark a System: the user selects the benchmark to
test his system with. The platform loads the appropriate
configuration parameters for the benchmark, as well as the
list of available systems for this benchmark. The user
configures the benchmark and selects one of the available
systems to benchmark.
• Show and Compare Benchmark Results: the user can view
the results of a single or multiple benchmarks.
• Add a System: the user adds the system to be benchmarked
in the platform by providing a Docker Image of the system
and a system adapter which serves as a proxy between the
benchmark and the system.
4) The Task Generator sends
the Target Dataset to the
Benchmarked System and
forwards the Gold Standard
to the Evaluation Storage.
5) When the Benchmarked
System finishes its task, it
sends the results to the
Evaluation Storage.
6) The Evaluation Module
receives the results of the
Benchmarked System and the
Gold Standard and reports the
Key Performance Indicators for
the experiment.
1) The Platform Controller makes sure
that the benchmark can be started
and ensures that all nodes of the
cluster are available. It communicates
with Benchmarked System, ensures
that it is working properly and
generates the Benchmark Controller.
2) The Benchmark Controller
is responsible for producing
the Data Generator, the Task
Generator and the Evaluation
Storage.
3) The Data Generator produces the
Source Dataset that is sent to the
Benchmarked System, the Target
Dataset and the Gold Standard that
are sent to the Task Generator.
GOALS
• Offers an open-source evaluation platform that can be
downloaded and executed locally.
• Offers an online instance of the platform for:
a) running public challenges
b) making sure that even people without the
required infrastructure are able to run the
benchmarks