This document describes a two-fold quality assurance approach for dynamic knowledge bases that was used for the 3cixty knowledge base project. The approach uses exploratory testing with the Loupe tool to find unexpected errors and scripted fine-grained analysis with SPARQL queries tested by the SPARQL Interceptor tool. Both techniques were complementary and helped identify issues like inconsistencies, outliers, missing properties, and constraint violations during knowledge base generation. The approach aims to adapt lessons from software engineering for quality assurance of knowledge bases.

1. A Two-Fold Quality Assurance Approach
for Dynamic Knowledge Bases:
The 3cixty Use Case
31st of May, 2016
1st International Workshop on Completing and Debugging the Semantic Web
at the 13th Extended Semantic Web Conference
Nandana Mihindukulasooriya1, Giuseppe Rizzo2 , Raphaël Troncy3 ,
Oscar Corcho1, and Raúl Garcı́a-Castro1
1Ontology Engineering Group, UPM, Spain.
2ISMB, Italy.
3EURECOM, France.
Acknowledgments:
FPI grant (BES-2014-068449), Innovation activity 3cixty (14523) of EIT Digital,
and 4V (TIN2013-46238-C4-2-R), Juan Carlos Ballesteros (Localidata)

2. Outline
2Ontology Engineering Group, Universidad Politécnica de Madrid
• 3cixty use case
• Motivation
• Techniques and tools
• Results

3. 3cixty knowledge base
3Ontology Engineering Group, Universidad Politécnica de Madrid
A semantic web platform that enables to build real-world and
comprehensive knowledge bases in the domain of culture and tourism
for cities using the public the information about places and events.

4. The 3cixty architecture
4Ontology Engineering Group, Universidad Politécnica de Madrid

5. Motivation
5Ontology Engineering Group, Universidad Politécnica de Madrid
:
• Data with 4Vs
• Volume, Variety, Velocity, Veracity
• Evolving schema
• Plenty of tools involved in the process
• Multiple geographically dispersed teams
• Dependent applications
Many chances for potential errors
The need for a good quality assurance approach

6. 6Ontology Engineering Group, Universidad Politécnica de Madrid
Can we adapt some
lessons learnt from
Software Engineering for
knowledge base
generation?

7. Continuous Integration is essential
7Ontology Engineering Group, Universidad Politécnica de Madrid

8. Cost of defects Vs. Time
8Ontology Engineering Group, Universidad Politécnica de Madrid
Time
Cost

9. Agile testing quadrants
9Ontology Engineering Group, Universidad Politécnica de Madrid
check for
expected
outputs
analyze
undefined,
unknown,
& unexpected

10. A Two-Fold Quality Assurance Approach
• Two techniques
• Scripted fine-grained analysis
• checking for expected results
• Exploratory testing
• analyzing the unexpected results
• Two techniques are complementary
• Exploratory testing can provide heuristics for fine-grained
analysis
• Supported by two tools
• SPARQL Interceptor
• Loupe
10Ontology Engineering Group, Universidad Politécnica de Madrid

11. Exploratory Testing
11Ontology Engineering Group, Universidad Politécnica de Madrid
simultaneous learning, test
design and test execution
minimal planning and
maximum text execution

12. Loupe – Linked Data Inspector
• Web application for exploring and inspecting datasets
• Class explorer
• Property explorer
• Triple pattern explorer
• Named graph explorer
• Starts from high-levels statistics and allows to “zoom
in” several levels of details
• Analysis of different datatypes
• most common and least common values
• numeric - min, max, mode, std. dev
• string – string length, uri like strings
• Avoid the need for boiler-plate SPARQL queries
• Ability to view the relevant data directly
12Ontology Engineering Group, Universidad Politécnica de Madrid

13. Loupe Architecture
13Ontology Engineering Group, Universidad Politécnica de Madrid
http://loupe.linkeddata.es/

14. Loupe UI
14Ontology Engineering Group, Universidad Politécnica de Madrid

15. Fine-grained analysis
15Ontology Engineering Group, Universidad Politécnica de Madrid
• a set of user-defined SPARQL queries (as unit tests)
• Knowledge-based specific
Test
SPARQL
Queries
System
Requirements
Schema
Constraints
Conventions
and other
restrictions
Inputs from
Exploratory
Testing

16. SPARQL Interceptor
• seamless integration with Jenkins continuous
integration system
• executes automatically for each build
• provides
• summary reports
• configurable email notifications
• for each failed test
• the reason for the failure
• a description of the query
• a link to failed data using an SPARQL endpoint
16Ontology Engineering Group, Universidad Politécnica de Madrid

17. SPARQL Interceptor
17Ontology Engineering Group, Universidad Politécnica de Madrid
Designed and implemented by Localidata.

18. Defects found in exploratory testing
18Ontology Engineering Group, Universidad Politécnica de Madrid
• Inconsistencies in using vocabularies
• locn:hasAddress Vs schema:streetAddress
• http://xmlns.com/foaf/0.1/ and http://xmlns.com/foaf/spec/
• URIs as strings
• ¨http://.....¨
• Outliers
• Typos
• class names with small letters
• Inconsistencies with the schema
• domain, range
• Value patterns
• codes with 5 letters, URIs with given prefix
• Date time format inconsistencies

19. Defects found in fine-grained analysis
19Ontology Engineering Group, Universidad Politécnica de Madrid
• property cardinalities related issues
• missing of properties
• Each dul:Place or lode:Event must have a title
• presence of duplicated properties
• dul:Place or lode:Event must have exactly one geo
location
• missing language labels
• one label per each language
• Out of bound values for a fixed upper and lower limits
• Neighboring cells in a grid (3 to 8)
• Datatype syntax errors
• numeric types
• Datetime types

20. Defects found in fine-grained analysis
20Ontology Engineering Group, Universidad Politécnica de Madrid
• Constraints on value ranges
• geo:lat and geo:long must be in a within the city’s bounding
box area
• triples not associated with producer graphs
• each triple belongs to a producer graph
• presence of unsolicited instances
• home locations are removed from the knowledge base

21. Conclusions and future work
21Ontology Engineering Group, Universidad Politécnica de Madrid
• Dynamic knowledge bases require good quality
assurance approaches
• Knowledge-base publishers can learn from / adapt
practices from software engineering
• Supporting tools improve quality assurance
• In the future,
• Integration with outlier detection algorithms
• Generation of constraints in Loupe
• Integration of SPARQL Interceptor with W3C SHACL