A short talk at the Dagsthul Seminar "Data, Responsibly" on benchmarking systems regarding their data responsibility. (This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 688227.)

1. Assessing your algorithms, tools, systems:
benchmark, benchmark, benchmark!
Irini Fundulaki
Institute of Computer Science – FORTH, Greece
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 1

2. Status of the Linked Open Data Cloud, 2014
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 2
Media
Government
Geographic
Publications
User-generated
Life sciences
Cross-domain
More than 31B triples in LOD
Links (external): 500M

3. Proliferation of Big Data Stores
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 3

4. The Question(s)
• Which are the problems that I wish to solve?
• Which are the relevant key performance indicators?
• Which is the behavior of the existing tools w.r.t. the key
performance indicators?
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 4
Which are the tool(s) that I should
use for my data and
for my use case?

5. The Answer: Benchmark your tools!
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 5
• Benchmark comprises of
– datasets (synthetic or real)
– set of software tools
• Synthetic data generators
• Test case generators
– Key performance indicators, and
– set of clear execution rules
• Standardized application scenario(s) that serve as a basis for
testing systems
• Must include a clear set of factors to be measured and the
conditions under which the systems should be measured

6. Benchmarks for the Linked Data Value Chain
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 6
What about data responsibility?
We need benchmarks that test
whether the tools for each step the
Big Linked Data value chain
are data responsible!

7. Are we data responsible?
• Can we identify use cases of interest?
– Emergency Response Priorities
– Targeted Ads
– Loans
– Criminal Justice
– (Universal) Healthcare, Precision Medicine
– Evaluation of employee performance (e.g., teachers, public
servants)
– Pricing Schemes
– H2020 and NSF Project Funding
– …
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 7
©Input from the Use Case & Benchmarks
Breakout Group@Dagstuhl Seminar 16291

8. Are we data responsible?
• Can we identify Key Performance Indicators?
– Discrimination/Fairness
– Neutrality
– Transparency
– Diversity
– …
• Can we identify tools whose responsibility we must
measure?
– Search Engines?
– Service Providers (e.g., Amazon, eBay)?
– Social Network Providers (e.g., FaceBook, Twitter)?
– Decision Makers?
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 8

9. Are we data responsible?
• Can we find datasets for our use cases?
– Open (public/private) datasets
– User-provided datasets (through crowdsourcing?)
– …
• What are the tests that we need to set up to benchmark
the tools?
– Differ according to the use case
7/21/16 Dagstuhl Seminar 16291: Data, Responsibly 9
• We need a principled, non biased set of guidelines for
benchmark definition!
• We must consider guidelines, legislation (national and
international) when defining benchmarks!
• Multidisciplinary work required!