Last year we introduced the importance of Data Quality for the implementation of high quality software applications and algorithms and we developed some scripts that analyzed the completeness, validity and accuracy of the Open Data Hub datasets. The journey in the data quality world did not end there. Unknowingly, that day was the beginning of a new journey that has one destination: make data quality testing accessible to anyone. The by-product of this path is a released open source software that gives the possibility of seeing the quality of the dataset based on initially proposed tests and criteria, verify what are the suggested warnings, modify the tests and automate the tests so that they are repeated periodically. During the talk we will present our journey, what we have learned from it and the first version of the data quality testing software.

1. Data Quality Testing
at the service
of Open Data

3. we love data

4. we love high quality data

5. us 😊

6. Data quality testing concept
ODH day 2022

7. Data quality rules
Data quality rules
validity accuracy
completeness
integrity consistency
timeless

8. Data quality testing concept
ODH day 2022
Development of codeless data quality testing
2023

9. Today is the big day

10. Having a tool is not
sufficient to have
high data quality

11. Data quality is a
journey made of
analysis, exploration
and goals

12. Outline of presentation
- The journey behind data quality testing
- CatchSolve and Open Data Hub

13. The journey behind
data quality testing

14. Analysis Exploration Goals
3 stages of the journey

15. Analysis Exploration Goals
3 stages of the journey

16. Analysis
Dataset and domain
Requirements and
constraints
Risk assessment

17. Severity
likelihood

18. Dataset and domain
Requirements and
constraints
Risk assessment
We obtain the information about what to test, why we test it
and the key requirements for choosing how to test it

19. Analysis Exploration Goals
3 stages of the journey

20. Exploration
Tools Processes

21. Analysis Exploration Goals
3 stages of the journey

22. Goals
Data quality KPIs Meta-KPIs

23. Improvement
requires
continuous work

24. CatchSolve and
Open Data Hub

25. Our core
technology
Customization #1
Open Data Hub
Customization #...
We adapt our core data
testing technology to be
compatible with companies’
requirements and needs
In this way, the
adoption of data quality
testing is eased

27. Tourism’s dataset example

28. Mobility’s dataset example

31. Analysis
Dataset and domain
Requirements and
constraints
Risk assessment
Mobility and Tourism Low number of KPIs
No configuration by
user
Integration Open Data
Hub system
Mobility: sensor might
not send data
Tourism: fields might be
empty or missing
Open and closed data
Mobility: sensor might
send inaccurate data

32. Exploration
Tools Processes
CatchSolve’s core
technology
Openapi
Who sees the results
and of what
Plan of automation of
the tests
tests configuration
processes

33. Goals
Data quality KPIs Meta-KPIs
Differentiated by
domain

34. Data quality KPIs
Tourism
Mobility
validity
completeness
accuracy
timeliness
Empty (null) values
Values = 0
Missing compulsory fields
Not changing sensor values
Old sensor data ( > 1h)
Negative values

35. Tourism
Mobility
Empty (null) values
Values = 0
Missing compulsory fields
Not changing sensor values
Old sensor data ( > 1h)
Negative values

36. Don’t fear errors.
Fear not knowing them.

37. Only by knowing them,
you can improve your data

38. CatchSolve info@catch-solve.tech
www.catch-solve.tech