Event Slides

1. Improving quality of
admin data
Chair: Adam Douglas, Admin Data Division,
ONS

2. Quality Assurance of
Administrative Data (QAAD)
The context for communicating quality
Marina Wright - Admin Data Division

3. Content
• History and the need for a framework
• The toolkit
 Risk/Profile matrix
 4 practice areas associated with data quality
• Using the framework across ONS and GSS
 UK Trade
 Population statistics
 Census and QAAD
• Development of helpful tools and documents

4. Benefits of the toolkit
Adaptable, Pragmatic, Proportionate
A tool to understand and improve quality
A basis for a shared understanding

5. sampling
survey
non-sampling error
sampling error
Administrative data

6. Context of Admin Data
• Admin data used for statistics for over 150 years
in the UK
• New technology enabling greater use of admin
data
• General assumptions:
‘admin data can be
relied upon with little
challenge’
‘unlike survey data,
admin data are not subject
to any uncertainties’

7. Meeting the regulatory standard
National Statistics status
means that official statistics
meet the highest standards of
• trustworthiness
• quality
• public value
Code of Practice for Official Statistics

8. History:
Police Recorded Crime Statistics
In January 2014 the independent UK Statistics
Authority removed the “National Statistics”
designation from Police Recorded Crime Statistics
• Concern over quality of
underlying data
• Concern over compliance with
recording standards
• Lack of information on quality

9. Admin Data Quality Toolkit
• UK Statistics Authority Developed
Administrative Data Quality Assurance Toolkit
• Enable benefits of administrative data
• And recognise that statistics derived from
administrative data are subject to:
• a range of potential biases
• incompleteness
• errors

10. Essential questions
• How do you know the data are
sufficiently reliable and suitable
to be used to produce official
statistics?
• What do users need to know
about their quality to use them
appropriately?

11. Quality Assurance Toolkit
• Assessing the
assurance level
• Providing evidence
to support rationale
• Collating evidence
of the actions taken
to comply
• Presenting evidence
of embedded
practices for
keeping QA
arrangements under
review

12. Level of risk of quality concerns /
public interest profile
Pragmatic and proportionate
1. Consider the likelihood of quality issues arising in
the data that may affect the quality of the statistics
2. Consider the nature of the public interest served
by the statistics
3. Judgment about the suitability of the administrative
data for use in producing official statistics should be
pragmatic and proportionate
12

13. Risk/profile matrix
Level of risk of
quality concerns
Public interest profile
Lower Medium Higher
Low Statistics of lower
quality concern and
lower public interest
[A1]
Statistics of low
quality concern and
medium public
interest [A1/A2]
Statistics of low
quality concern and
higher public interest
[A1/A2]
Medium Statistics of medium
quality concern and
lower public interest
[A1/A2]
Statistics of medium
quality concern and
medium public
interest [A2]
Statistics of medium
quality concern and
higher public interest
[A2/A3]
High Statistics of higher
quality concern and
lower public interest
[A1/A2/A3]
Statistics of higher
quality concern and
medium public
interest [A3]
Statistics of higher
quality concern and
higher public interest
[A3]

14. Levels of assurance
14
A1: Basic assurance
Statistical producer has reviewed and published a
summary of the administrative data QA arrangements
A2: Enhanced assurance
Statistical producer has evaluated the administrative data
QA arrangements and published a fuller description of the
assurance
A3: Comprehensive assurance
Statistical producer has investigated the administrative
data QA arrangements, identified the results of
independent audit, and published detailed documentation
about the assurance and audit

15. Quality Assurance Matrix
Assurance
level
Operational
context
Communication Data suppliers’
QA
Producers’
QA
A0 NO ASSURANCE
A1 BASIC
A2 ENHANCED
A3 COMPREHENSIVE

16. Four practice areas associated
with data quality
Operational context &
admin data collection
Communication with
data supply partners
QA principles,
standards and checks
by data suppliers
Producers' QA
investigations &
documentation

17. • environment and processes for
compiling the administrative data
• factors which affect data quality and
cause bias
• safeguards which minimise the risks
• role of performance measurements
and targets; potential for distortive
effects
Operational context & admin data collection

18. Communication with data supply partners
• build relationships with:
- data collectors
- suppliers
- IT specialists
- policy and operational
colleagues
• formal agreements detailing
arrangements
• regular engagement with
collectors, suppliers and users

19. QA principles, standards and checks
by data suppliers
• data assurance arrangements in
data collection and supply
• quality information about the data
from suppliers
• role of operational inspection and
internal/external audit in data
assurance process

20. Producers' QA investigations & documentation
• QA checks carried out by
statistics producer
• quality indicators for input data
and output statistics
• strengths and limitations of the
data in relation to use
• explanation for users about the
data quality and impact on the
statistics

21. Quality management actions:
Investigate: Manage: Communicate
Communicate
Manage
Investigate
Investigate – such as:
 Data suppliers’ own QA arrangements
 Results of external audit of the admin
data
 Areas of uncertainty and bias
 Distortive effects of targets and
performance management regimes
Communicate – such as:
 Description of data collection process
 Regular dialogue with suppliers and providers
 Document quality guidelines for each set of statistics
 Description of errors and biases and their effects on the statistics
 Communicate with users
Manage – such as:
 Cooperative relationship
with suppliers, IT and
operational, and policy
officials
 Guidance information on
data requirements
 QA checks and
corroboration against
other sources

22. Using the framework across
ONS and GSS

23. Using the framework across ONS

24. UK Trade Background
• Data are supplied from over 30
feeder sources, including a
variety of administrative data
sources, the main one being
HM Revenue and Customs
(HMRC).
• UK Trade statistics compiled by
ONS has been one of the countries
key economic indicators
• Aim : reinstatement of official
statistics badge and improvement
of the quality assurance reporting

25. • Introduction of the QAAD
workshop
• Mapping data sources
• Determining risk to quality and
public interest
• Data suppliers information day
• Video conference meetings
• Presenting quality information
to users
UK Trade work on QAAD

26. Development of helpful tools and
documents
• Literature review
• Risk/Profile Matrix Template
• Admin data Supplier Questionnaire template
• Guidance to apply QAAD
• QAAD – FAQs for Statistical Producers
• QAAD Questions - what do I need to ask
• QAAD – Case examples

27. Admin data sources -
Data Supplier Questionnaire
Data Supplier Questionnaire Template
This questionnaire is part of an ongoing assessment of ... (insert output).
It is a part of a requirement set by United Kingdom Statistics Authority to assess the quality
assurance of the administrative data being provided and published as Official Statistical by the
Office for National Statistics.
We would be very grateful if you complete and return to us the following series of questions:
1. Contact Name(s):
2. Contact Telephone Number(s):
3. Contact Email Address(es):
4. Organisation/Department/Business:
5. Data description (please give brief description of the data supplied to ONS)
6. What is primeval purpose of the data?
7. How are the data provided to ONS sourced e.g. internal administrative system? A number of
administrative sources reporting to one department (e.g. number of GPs send information to
Clinical Commissioning groups and then to the Department of Health)?
8. How are the data originally collected (e.g. collected by individual GP surgeries across the
county using standard self completion form, information manually put on to the system)?

28. Process map
Statistics on Police
Recorded Crime
Statistics in Northern
Ireland
Extract from User
Guide illustrating crime
recording process,
potential sources of risk
and risk mitigation

29. Results so far – UK Trade
• List all admin data sources and apply the toolkit to
them
• Get better understanding of their admin data
sources
• Improve communication with data suppliers
• Produce detailed process map
• All Standard Level Agreements (SLAs) have been
re-visited
• Actions in place to address difficult to reach data
suppliers
• Final report is being finalised for UKSA
assessment

30. QAAD use in Population Statistics
• Work began in late 2015 to support
Population Statistics bid for National
Statistics accreditation on key
releases
• Established admin data use across
Population Statistics
• Surprisingly large: Population
Estimates alone have 19 sources
• Lots of re-use of admin data within
Population Statistics

31. QAAD use in Population Statistics –
next steps
• Programme of work agreed with
UK Statistics Authority (to be
completed in 2016)
• Risk-Profile scores and assurance
level for a source vary within
Population Statistics depending
on use in statistical estimation
process
• Highest assurance level used as
Population Statistics assurance
level, not the highest risk score
and the highest profile score

32. QAAD use in Population Statistics -
challenges
• Admin data sources not always straight
forward
• Included in peer review of all reports are:
UK Statistics Authority
Producers
Suppliers
Welsh Government
NISRA
National Records of Scotland

33. QAAD and Admin Data Census
• Data sources for Census
being used for research
• Working to put QAAD in
place
• Ready for Official
Statistics publication
 Preparing for the start

34. Benefits of the toolkit
Adaptable, Pragmatic, Proportionate
A tool to understand and improve quality
A basis for a shared understanding

35. Contact:
Marina Wright – Admin Data Division
Marina.Wright@ons.gov.uk

36. Collaboration with admin
data suppliers
Louise O’Leary

37. Impact and goals
 Shorter term: Better understanding,
knowledge sharing, evidence for Admin
Data Toolkit – communicating quality
 Longer term: Methodological
improvement, improvements in accuracy

38. How Census is working with Admin Data Suppliers
Different circumstances with different suppliers:
• Pursuing new data sources
• ‘Feasibility’ data –
can we use it?
• Current data – working to understand the
features of statistical quality with suppliers

39. How Census is working with Admin Data Suppliers
Review
requirements
Revise
acquisition
plan
Acquire and
feasibility
research
Decision:
ongoing/
revisions/ not
meet needs
Establish
Feedback
loop
Quality
Data
Suppliers
Group
Secondments
and loans
Statistical data
quality working
groups

40. Statistical Data Quality Working Group

41. Health data supplier model
Established statistical data quality
working group
Identified contacts at working level
Identified mutual benefits
Workshop to understand data collection

42. Outcomes
 Learning from the working level contacts
 Understanding data collection in action:
• Complexities
• Ambiguities
• Pitfalls
 Sharing the purpose of our questions
 Ability to build collaborative assumptions
Next step: to roll out to other data suppliers
depending on requirements – meeting the
Admin Data Toolkit requirements

43. Contact:
Louise O’Leary – Admin Data Census,
Census Transformation Programme
Louise.O’Leary@ons.gov.uk

44. Data Quality: a supplier
perspective
The NHS Personal Demographics Service
44Published: 22/06/16 – v2

45. What can you get from the PDS?
Nominated Pharmacy
Electronic Prescription Service
Shared Secret
Birth Notification
Service
Call back Consent
eReferrals service
Clinical Birth Details
Consent
Consent to NHS Care
Record Sharing
Mr Samuel Smith 29 / 02 / 1954 M999 999 9999 / /
Corner Cottage
73 School Lane
Southside
Town
North County
CC88 9ZZ
03291 111222
s.smith@uk.com
Y12345 Southside Surgery Mrs Sandra Smith
NHS Number Full Name DOD Gender
Usual
Address
Registered GP Practice
Phone / mobile / email
Carer / Next of Kin
DOB
Mansion Towers
5 The Street
Northside
City
Another County
DL11 9LL
Temporary
Address
also held or processed for other services

46. Who uses and updates the PDS?
PERSONAL
DEMOGRAPHICS
SERVICE (PDS)
Live since June 2004
Available 24/7 365
800,000+ smartcard users
70m + patients
Maternity
Units
Child Health
Departments
New Born
Screening
NHS
Trusts
new patients;
demographic updates;
informal deaths
births
births
births
GRO /
ONS
Pharmacies
Social
Care
Dentists
Home
Countries
tracing,
nominated
pharmacy tracing
tracing
births, updates
GP registration, tracing
births,
deaths (formal)
Researchers Commissioners
secondary uses
extracts
Home
Office
PDS
NBO
immigration
health
surcharge
data quality;
specialist
processes
new patients; GP registration;
updates; consent; informal
deaths; eRS preferences
GP
Practices
Primary
Care
Supportupdates
registration

47. Systems that create & access PDS records
Spine
PDS
NHS Trusts,
GP practices,
local
authorities,
independent
sector …
DSA PDS NBO,
specialist users
Spine Mini
Service
Demographics
Batch Service
SCRa
PDS-
compliant
system
Local
systems
GP/
NHAIS
GP registrations
Maternity
Birth notifications
NHS Trust
New allocation
NSTS
Initial load 2004
Visitors & MigrantsHome
Office

48. PDS Update events and triggers
PDS
birth deathregistering
with a GP
name
change
change of
address
payment of
immigrant
health
surcharge
adding
a new
patient
changing
contact
details
nominate
pharmacy
adoption
gender
change
restricting
access
incorrect
death
status
confusions
incorrect
birth info
duplicates
Health and Care Processes
PDS NBO Specialist Processes

49. Data Quality
• “Datasets” vs. “databases”
• ‘Data quality’ means different things:
– Completeness, timeliness, conformance
– Recording what’s needed for the workflow
• Mixing the two can lead to unintended
consequences
• ‘Errors’ aren’t always obvious …
• … and neither are the causes

50. Consequences of success
• ‘Use the NHS Number’
• Staff (and systems) know this and act on it
• Many can now ‘allocate’ NHS Numbers
• But imperative can lead to poor practice:
– The 200-year old patients
– “Unknown Eritrean”
– Duplicate/multiple records for same patient

51. PDS National Back Office activities
PDS data quality resolution and
process support
Monthly
Volume
Duplicates 1,169
Confusions 702
Immigration Health Surcharge 12,000
New Birth related 224
Unmatched Civil Registrations (Death) 13,820
Unmatched Civil Registrations (Birth) 260
Allocations Service 128
Data Quality Queries 546
Adoptions 524
Gender Reassignment 113

52. Local vs. National
• Much local EPR-PDS interaction is seamless:
– So the data is largely kept consistent
– But user awareness of local/national not always clear
– And local practices can have an impact
• Hence the rare but real examples of discrepancies:
– “Baby”
– Walking the dog/Bridge on Thursday afternoons
– Key safe codes
– ‘Ping-pong’
• From user perspective makes sense …
• … even if it’s obviously ‘wrong’ usage

53. Synchronisation

54. The odd and the mundane
• From woman in Wales to man in Bradford
• I thought he was dead
• We don’t always know where you live:
– Different address types
– ZZZ addresses
– ‘SAME’ address
– The mailman always delivers (so don’t worry
about the postcode)

55. Final thoughts
• ‘Aggregate data from operational data’ isn’t as simple as
it sounds
• Care about assumptions re national databases:
– Users aren’t consciously ‘collecting data’
– Context, provenance and operational use all need to be
understood
– For NHS: tens of thousands of users, thousands of
organisations, dozens of service/business uses
• Conversely, caution about adding data items (and never
ask about “a flag on PDS”)
• On scale of national surveys, unlikely to have massive
impact
• But linkage of datasets may be affected

56. Any questions?

58. Questions
• What challenges have you faced with using admin data (in local
authorities etc)?
• What are your main concerns about the statistical quality of admin
data?
• What trade-offs on quality do you make?
• What trade-offs on quality would you be willing to accept
• How can we best / what other ways can we collaborate and engage
with suppliers of admin data?
• As suppliers, what challenges might you face with providing the type
of information the toolkit requires? How best can we work with you
to help identify/understand these?