A presentation of the HOBBIT Survey Results (This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 688227.)

1. HOBBIT
Survey results
Ruben Taelman
Horizon 2020
GA No 688227
01/12/2016–30/11/2018
ESWC 2016
Crete, Greece
June 22, 2016
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 1 / 1

2. Goal
In what areas are organizations active?
What do people expect from benchmarks?
How are benchmarks being used?
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 2 / 1

3. Participant profiles
50 participants
Profile Count
Solution provider 39
Technology user 39
Scientific community 40
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 3 / 1

4. Linked Data areas
Evenly distributed:
Storage / Querying
Interlinking
Classification / Enrichment
Discovery
Extraction
Reasoning
Solution providers mainly active in storage and querying.
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 4 / 1

5. Linked Data areas
Other areas:
Reporting
Visualization
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 5 / 1

6. Key performance indicators
Correctness
Accuracy (precision, recall, F-measure, mean reciprocal rank)
Runtime / Speed
Scalability
Memory usage
CPU usage
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 6 / 1

7. Benchmarks
48% already evaluates software using a benchmark
78% of benchmarking done for storage and querying, much less for other
domains
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 7 / 1

8. Datasets
Types of datasets:
22% synthetic data
46% real data
32% both
66% is or can be made public!
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 8 / 1

9. Dataset sizes
In
number of triples.
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 9 / 1

10. Survey still open!
http://goo.gl/forms/1iRIoG4Xpb
Ruben Taelman (iMinds) HOBBITSurvey results June 22, 2016 10 / 1