We evaluated the use of a previously described crowdsourcing methodology to generate a problem-laboratory test knowledge base, identifying appropriately linked problem-laboratory test pairs by clinicians during e-ordering. Existing evaluation metrics, including patient frequency and link ratio, were not correlated with appropriateness for 600 links manually validated. Further research is necessary to better evaluate these associations.

1. Use of the Crowdsourcing Methodology to
Generate a Problem-Laboratory Test Knowledge Base
Allison B. McCoy, PhD1, Adam Wright, PhD2, Jacob A. McCoy, MS3, Dean F. Sittig, PhD1
1The University of Texas Health Science Center at Houston – School of Biomedical Informatics
2Brigham and Women’s Hospital, Harvard Medical School
Objective Methods Results Discussion
To determine whether crowdsourcing is We applied crowdsourcing to create a Clinicians asserted 17,555 links (4,067 distinct Crowdsourcing has previously been
effective at generating a knowledge base of knowledge base of problem-laboratory test problem-laboratory test pairs) during the study demonstrated to be effective at identifying
related clinical problems and laboratory test pairs. We retrieved all problem-laboratory period. Of the 600 links evaluated, 429 appropriate problem-medication pairs. Despite
results using links asserted by clinicians tests pairs linked by clinicians at a large, multi- (71.5%) were determined to be appropriate. similar processes used by clinicians in
during e-ordering. specialty, ambulatory academic practice Of these, 2,715 (66.8%) were asserted for ordering medications and laboratory tests, it
between June 1, 2010 and May 31, 2011. only one patient and 224 (5.5%) were did not generalize well when applied to
We calculated the frequency of patients with asserted for 10 or more patients. generating a problem-laboratory test
Background knowledge base.
each link and the proportion of patients with a The table depicts the appropriateness for each
Knowledge bases that link clinical problems to
co-occurring laboratory test and problem for threshold group. No threshold groups One potential cause of the poor performance
medications, laboratory tests, and other
whom a link had been manually asserted (i.e., achieved appropriateness greater than or is the use of laboratory test results for
clinical data can be applied to improve patient
link ratio). We stratified links into threshold equal to 95%, and we did not observe an diagnostic purposes; because the clinician
safety and meet Meaningful Use criteria in a
groups according to each measure. Blinded to increasing trend in appropriateness as the has not determined whether a patient has a
variety of ways, including generating patient
the threshold values, we reviewed 25 patient frequency or link ratio increased. given disease, the problem has not been
summaries, facilitating medication
randomly selected pairs from each group to added to the patient’s list and is therefore not
reconciliation, and delivering clinical decision
determine whether the pair was appropriate. linked during ordering. In this case, clinicians
support. While many problem-medication
may arbitrarily select a problem to link or link
knowledge bases are available, few problem-
Appropriateness Thresholds for Problem-Laboratory Test Link Inference to a health maintenance code, which we
laboratory test knowledge bases exist. Current
excluded from our analysis.
methods for generating such knowledge 0 Link Ratio
bases, such as association rule mining, are 0.1 0.1-0.19 0.2-0.29 0.3-0.49 >=0.5 Total This study is limited in that it was performed in
limited or computationally complex. 1 72% 68% 68% 80% 72% 72% a single setting with a small subset of linked
problem-laboratory test pairs. Additional
Patient Link
2 76% 64% 72% 76% 64% 76%
Frequency
3-4 84% 76% 84% 80% 88% 84% research is necessary to determine whether
Crowdsourcing more data may improve the findings, and
5-9 76% 64% 72% 80% 68.8% 76%
Crowdsourcing, defined as outsourcing a task whether the methods are generalizable.
>=10 56% 68% 56% 48% 77.8% 56%
to a group of people, represents a novel
Total 72.8% 68% 70.4% 72.8% 74% 72.8%
method for generating such a knowledge
base. The approach takes advantage of Summary of Conclusions
manually linked laboratory tests to clinical Example Links Use of the crowdsourcing methodology with
problems by clinicians during standard e- the initially proposed evaluation metrics did
ordering, a task required by many institutions Appropriate Problem-Laboratory Test Pairs Inappropriate Problem-Laboratory Test Pairs not adequately identify appropriate problem-
for billing purposes. Laboratory Laboratory laboratory test links. Further research may
Problem Frequency Ratio Problem Frequency Ratio
Test Test better evaluate these associations.
Abdominal Pain Fecal 4 0.11 Chronic Renal TSH, 3rd 1 0.03
Sample Screen for Linking to a Problem Leukocyte Stain Insufficiency Generation
Abdominal Pain CBC 7 0.01 Aortic TSH, 3rd 2 0.12 Acknowledgements
Acute Streptococcus 6 0.60 Aneurysm Generation This project was supported by Contract No.
Pharyngitis Test Rapid Aortic Hemoglobin A1c 3 0.23 10510592 for Patient-Centered Cognitive Support
Aneurysm under the SHARP Program from the Office of the
Atrial Fibrillation Prothrombin 54 0.26
National Coordinator for Health Information
Time with INR Aortic Stenosis Hepatic 4 0.57
Function Panel Technology and NCATS grant UL1 TR000371.
Abdominal Pain Calprotectin, 13 0.30
Stool Hyperlipidemia PSA Total+ 13 0.54 Please contact the first author via email:
% Free allison.b.mccoy@uth.tmc.edu