Presentation for the workshop on the Combination of surveys and administrative data, Vienna, Austria.

1. A quality framework
for the evaluation of
administrative and survey data
Piet J.H. Daas, Judit Arends-Tóth,
Barry Schouten and Léander Kuivenhoven
Statistics Netherlands

2. Overview
 Reason for work
 View on Quality
 Starting point
 Literature study results
 Overview of the framework
 Application
 Future work

3. Reason for work
 Statistics Netherlands increases the use of
data (sources) collected and maintained by
others
• To decrease response burden and costs
 As a result:
• More dependent on external data sources
• Must be able to monitor the quality of external
data sources
– Develop a way to monitor the quality of external
data sources

4. View on quality
 Statistics Netherlands definition of the quality of
external data sources:
“Usability for the production of statistics”
 Differs from quality as used by the data source
maintainer

5. Starting point
 Previous work at Statistics Netherlands
• Most recent: Work of Daas and Fonville
– Determination of register quality
– Described in a paper for the register seminar in Finland
(2007)
• Improvement is possible
– Predominantly business register data
– International experience could be included more
 New project started
• Develop quality framework for the evaluation of
external data sources (focus on registers and
administrative data)

6. Literature study (1)
 Performed and extensive literature study
• Publically available papers/books that studied
the quality of administrative data sources and
registers → Dutch and English
• A lot of research focuses on quality of survey
collected → these were excluded
• Ended up with quite a limited lists of important
papers:
– Book of the Wallgren’s (S)
– Daas and Fonville (NL)
– Eurostat paper on Quality of administrative Data
– Work performed at ONS and by Thomas (UK)
– UNECE paper of Nordic countries

7. Literature study (2)
 Conclusions:
• A general level of mutuality
– The papers identified many similar quality aspects
(quality indicators)
• None of the ‘views’ on quality were exactly alike
• How to combine all these views?
• Something higher than a dimension was needed
• Karr et al. 2006 used the term Hyperdimension to
distinguish different views on quality
• Combine all quality aspects identified in all studies
and new aspects in a single framework !!

8. Quality framework
 Framework has 4 hyperdimensions
• Four views on the quality of the external data source
 The hyperdimensions identified are:
• Source → Data source as a whole
• Metadata → Conceptual metadata of data in source
• Data → Facts (values) in data source
• Process → Processing related quality aspects

9. Quality framework levels
 Levels distinguished:
HYPERDIMENSION
n>1
DIMENSION
n >= 1
QUALITY INDICATOR
1:n
Measurement method

10. 1) Source hyperdimension
 Here the data source is viewed upon as a file
delivered by the data source maintainer to
the NSI
 Dimensions (5):
• Supplier, Relevance, Privacy and security,
Delivery, and Procedures

11. Source hyperdimension
Hyper- Dimension Indicator Measurement method
dimension
Source 1. Supplier
Supplier Contact Name, Contact information
1.1 Contact - Name of the data source
- Data source contact info
Relevance Adm. burden Effect of use on adm. burden
- NSI a contact person
of NSI
1.2 Purpose (time and money)
- Reason for use of the data
source by NSI
Privacy and Legal provision Check if Personal Data
2. Relevance
security2.1 Usefulness - Importance data source for NSI
Protection act applies
Delivery2.2 Envisaged use
Costs - Potentialuse for NSI use of data
Costs of statistical
source
2.3 Information - Does the data source satisfy

12. 2) Metadata hyperdimension
 Focuses on the conceptual metadata quality
aspects of the data source.
 Other metadata aspects (such as process
meta) are not included
 Dimensions (4):
• Clarity, Comparability, Unique keys, and Data
treatment by data source maintainer

13. Metadata hyperdimension
Hyper- Dimension Indicator Measurement method
dimension
Metadata Clarity
1. Clarity Population Description of the population
1.1 Population definition - Clarity scoredata source
definition used in of the definition
Unique Definition of variables
1.2 keys Identification - Clarity scoreunique keys
Presence of of the definition
(and categories)
keys present (which)
1.3 Time dimensions - Clarity score of the definition
Data 1.4 Geographic demarcation - Clarity score of the definition
treatment Checks Variable value checks
by data source performed
1.5 Definition changes
maintainer - Familiarity with occurred
changes
2. Comparability Modifications Familiarity with data
modifications
2.1 Population definition - Comparability with NSI
comparison definition

14. 3) Data hyperdimension
 Aspects related to data in the data source
• All aspects are accuracy related
 Actively being discussed at our office
• Future changes may be possible
 Dimensions (9)
• Over coverage, Under coverage, Linkability, Unit
non-response, Item non-response, Measurement,
Processing, Precision, and Sensitivity
Remark: Precision was mainly included for (externally collected) survey data

15. Data hyperdimension
Hyper- Dimension Indicator Measurement method
dimension
Data Over coverage Non-pop. units
1. Over coverage Percentage of units not 1.1
Non-population units - Percentage of units to population of NSI
belonging not
belonging to population
Linkability Linkable units Percentage if units linked
2. Under coverage
2.1 Missing units - Percentage of missing
Measurement Incompatible population units with violated
Fraction of fields
records edit rules
2.2 Selectivity - R-index for population
Processing Adjustment composition
Fraction of fields adjusted
2.3 Effect on core variables - Maximum bias of average for
Imputation Fraction of fields imputed
core variable
- Maximum RMSE of average
R-index: Representative index; RMSE: Root mean square Error; MSE: Mean Square Error

16. 4) Process hyperdimension
 Focuses on the processing of the data source
• by the data source maintainer
• by the NSI
 Not discussed here, future work
 Framework was developed without specifically
focusing at process related quality aspects
• main focus was product related

17. Framework and external data sources
 Developed for administrative data
• Registers a.o.
 Why not use it for surveys?
• In the case were the data is collected by an
organization other than Statistics Netherlands
• Experiences in the past resulted in many quality
related discussions
– transparency of data collection process?
 Resulted in some minor adjustments of the
framework
• Some terms were adjusted
• Review with sample approach in mind

18. Application of the framework
 Should be applied to externally collected data
sources
• Administrative data, registers, non-NSI surveys
 How to apply?
• Source and Metadata hyperdimension
– Checklists have been developed
• Data hyperdimension
– Methods of calculation have been proposed
– Currently looking at a practical means to apply these
• Process hyperdimension
– Under investigation

19. Application of the framework (2)
For each data source and use of data source
1) Evaluate Source with checklist
– 2 ways: a quick an a complete scan
– When no problems occur continue
2) Evaluate Metadata with checklist
– 3 ways: replace, additional or new
– When no problems occur continue
3) Evaluate Data
– In a standardized way (scripts or computer program)
– Probably requires some very specific test at the end
– Framework should be generally applicable
– User mostly has a specific use in mind

20. Future work
 Evaluate registers and external survey data
• Is the framework generally applicable for all
sources?
 Thoroughly test Source and Metadata checklists
• Feed-back on usability by users
 Calculation methods for Data
• A single way of determining every Quality indicator
 Study how to efficiently evaluate Data
• E.g. Scripts or computer program
 Determine the quality aspects in the Process
hyperdimension

21. Questions?