Stephen Lean School of Engineering and Advanced Technology Massey University (Thursday, 2.00pm, Sigma Room)

1. A Conceptual Framework for
Determining Health Information
Quality Indicators

2. 1. Motivation
2. Issues
3. Objective/Method
4. Results/Conclusions
5. Ongoing/Future work

3. Motivation
• More/better use of health information.
• Safe sharing.
– Out of context use
– Aggregated data, disparate sources
– At point of care
• Judgement on whether to use/rely on
information.

4. Issues
• Wanting to make use of this information.
• Information collected and stored will have
some level of quality issues e.g. accuracy.
• Good quality information is required for
effective use.
• What is good quality information?

5. Issues
“Data are of high quality if they are fit for their
intended uses in operations, decision
making, and planning. Data are fit for use if
they are free of defects and possess desired
features”
• In the absence of any indication, a judgment
as to the information's fitness for use must be
made.

6. Issues
• Is there enough information, or enough
desired information, to allow for a judgement.
• Ensuring patient safety is not compromised.
• Out of context use, where determinations of
quality are especially challenging.
• What do information users want to know
about the information that aids them in the
judgment.

7. Objectives/Method
• Needed to identify:
– How the information is currently used and relied
on?
– What aspects of information, currently
available, influence a user’s perception of the
quality of that information?
– What additional data could be made available that
would further assist users in their assessment of
quality?

8. Objectives/Method
• Semi-structured interviews using vignettes.
• Three areas of health information used:
– Adverse events
– Medical history
– Medicines list
• Ratings exercise.

9. Results
• Transcribed interviews
• Thematic analysis of transcripts using NVivo
• Some statistical analysis of ratings exercise.

10. Results
Quality Criteria Interviewees
Dr.A Dr.B Dr. C Dr.D Dr.E Dr.F
1 : Indication of contradictory entries 1 2 1 1 0 3
2 : Knowing information was produced at an organisation known for its expertise in that field 1 3 1 3 0 2
3 : Knowing information was produced at an organisation that has a good reputation 2 4 0 1 0 6
4 : Attestation of entries recorded 0 0 0 1 0 0
5 : Internal consistancy or cohesion of record 2 3 1 2 0 1
6 : Positive or negative personal experience with provider 1 13 1 5 0 9
7 : Qualifications, experience and position of person attesting to information 1 0 0 2 0 0
8 : Contextual narrative for event for entry 6 3 1 13 0 9
9 : Indication of where to go for further information 0 2 0 4 0 5
10 : Knowing the qualifications of the clinician involved in healthcare event 2 2 3 8 0 1
11 : Years of experience of the clinician 0 4 0 6 0 1
12 : Age of the data 0 0 0 6 0 1
13 : Reason for change of entry 1 0 0 4 0 3
14 : Where the information was captured 0 0 0 1 0 0
15 : Position held by the clinician 1 1 0 3 0 1
16 : Knowing testing methodology for test results 0 0 0 0 0 0
17 : Knowing the length of time from information captured to information recorded 0 0 0 0 0 0
18 : Knowing the qualifications of the recorder of the information 0 0 0 0 0 0
19 : Who has been responsible for storing the information 0 0 0 0 0 0
20 : Years of experience of the recorder of the information 0 0 0 0 0 0
21 : Position held by the recorder of the information 0 0 0 0 0 0

11. Results Interviewees Count
Quality Criteria Average Total
Dr.A Dr.B Dr.C Dr.D Dr.E Dr.F 1 2 3 4
C1 Indication of contradictory entries 4 4 4 4 4 4 4 24 6
C2 Knowing information was produced at an organisation 3 4 4 4 4 2
known for its expertise in that field 3.5 21 1 1 4
C3 Knowing the information was produced at an 2 4 4 4 4 3
organisation that has a good reputation 3.5 21 1 1 4
C4 Attestation of entries recorded 2 4 3 4 4 4 3.5 21 1 1 4
C5 Internal consistency/cohesion of record 4 3 3 3 4 4 3.5 21 3 3
C6 Positive/negative personal experience with provider 4 4 3 4 2 3 3.33 20 1 2 3
C7 Qualifications, experience and position of person 2 4 4 4 2 4 2
attesting to information 3.33 20 4
C8 Contextual narrative for event for entry 4 2 3 4 3 4 3.33 20 1 2 3
C9 Indication of where to go for further information 4 1 3 4 4 4 3.33 20 1 1 4
C10 Knowing the qualifications of the clinician involved in 4 1 4 3 4 3
healthcare event 3.17 19 1 2 3
C11 The years of experience of the clinician 3 3 3 4 3 3 3.17 19 5 1
C12 Age of the data 4 2 2 4 3 4 3.17 19 2 1 3
C13 Reason for change of entry e.g. change of diagnosis or 4 2 2 4 3 4 2
change in prescribed medication 3.17 19 1 3
C14 Where the information was captured 2 4 2 4 4 2 3 18 3 3
C15 The position held by the clinician 1 1 4 4 4 3 2.83 17 2 1 3
C16 Knowing testing methodology for test results 4 3 1 2 4 1 2.5 15 2 1 1 2
C17 Knowing length of time from information captured to 2 4 1 2 4 2 1 3
information recorded 2.5 15 2
C18 Knowing the qualifications of the recorder of the 1 1 1 4 2 4 3
information 2.17 13 1 2
C19 Who has been responsible for storing the information 1 1 2 3 2 1 1.67 10 3 2 1
C20 The years of experience of the recorder of the 1 3 1 1 1 3 4 2
information 1.67 10
C21 The position held by the recorder of the information 1 1 1 2 1 3 1.5 9 4 1 1

12. Results
• C3: “In one site I would have some considerable
confidence that in fact the information presented
to me was true and valid. In another site I would
have to go off into a routine of actually justifying
that information because I don't have the
confidence that it's true.”
• C6: “I mean if you produce generally rubbish
notes, then do I believe the answers you put
there? Ah, no.”

13. Results
• C1: “What's the bottom line here. If the
bottom line doesn't make any sense to the top
line, that's when you start reading.”
• C9: “I suppose in that particular case I raised
one of the difficulties was finding where the
information was. So perhaps some indication
of "if you need further information about
this, this is where you could go.”

14. Conceptual Framework

15. Conclusions
• For Organisational and Personal
criteria, appear to rely on tacit knowledge and
community knowledge.
• Past experience counted highly when
assessing fitness for use (quality of data).
• For Intrinsic and Contextual, appear to use
professional experience, built up over time.
• Much of the process is an ad hoc procedure
based on tacit knowledge and experience.

16. Conclusions
• Gap in providing GPs with accompanying, or
enough, information for determination of
fitness for use.
• Interviewees felt there were criteria, if made
available, that would positively assist in the
process.
• Interviewees felt quality indicators for
information, at point of care, could be
beneficial.

17. Ongoing/Future Work
• Quality Criteria used as inputs to generate
Quality Indicators (QI).
• Modelling Quality Criteria (QC).
• Information Model to operate over.
– will extend Continuity of Care Record standard
• Implement models and processing algorithm
within a system.
– show that QC can be implemented and processed
over the information model to produce QIs