A Methodology for Quantitative Measurement of Quality and Comprehensiveness of a Research Data Repository   (IDR Snapshot)...
Acknowledgement <ul><li>Wisconsin Institute for discovery </li></ul><ul><ul><li>Private branch: Morgridge Institute  </li>...
Agenda <ul><li>Introduction </li></ul><ul><li>Methods </li></ul><ul><ul><li>Design requirements </li></ul></ul><ul><ul><li...
<ul><li>Clinical Informatics </li></ul><ul><ul><li>Subfield of Biomedical Informatics </li></ul></ul>Background marshfield...
<ul><li>Clinical Informatics </li></ul><ul><ul><li>Subfield of Biomedical Informatics </li></ul></ul>Background marshfield...
Introduction <ul><li>Advances </li></ul><ul><ul><li>Federated research data warehouse </li></ul></ul><ul><ul><li>Future da...
SHRINE: Query federation example <ul><li>Weber at al. (2009)  </li></ul><ul><ul><li>Application of Information Technology:...
Motivation <ul><li>Motivation </li></ul><ul><ul><li>10 institutions with IDRs; budget allows only 3; How do you choose? </...
Measure <ul><li>Measures </li></ul><ul><ul><li>Qualitative description </li></ul></ul><ul><ul><li>Quantitative: Simple cou...
Data Structures (VDW) linkedin.com/in/vojtechhuser
Data Structures (i2b2) linkedin.com/in/vojtechhuser
Data Structures (HealthFlow) linkedin.com/in/vojtechhuser healthcareworkflow.wordpress.com
Data structures (data schemas) <ul><li>VDW </li></ul><ul><ul><li>Several tables </li></ul></ul><ul><ul><li>Data domain spe...
Event table (IDR Snapshot v1) code.google.com/p/IDRSnapshot (Event-DOB) + ‘3000-01-01’
Results <ul><li>Events table  (ev_measure01) </li></ul><ul><li>Generation took 2.5 hours </li></ul><ul><li>Size: 43 GB (in...
Code code.google.com/p/IDRSnapshot
Code code.google.com/p/IDRSnapshot
Code code.google.com/p/IDRSnapshot
Cross institution comparison code.google.com/p/IDRSnapshot
Monitoring code.google.com/p/IDRSnapshot
What’s in it for me? <ul><li>Use it </li></ul><ul><ul><li>Download our SQL code </li></ul></ul><ul><ul><li>Use it to evalu...
Future work <ul><li>Learn from IDRs of other institutions </li></ul><ul><li>IDR Snapshot version 2 </li></ul><ul><ul><li>E...
Thank you <ul><li>[email_address] </li></ul><ul><li>http:// code.google.com/p/idrsnapshot </li></ul><ul><li>Slides:  http:...
Upcoming SlideShare
Loading in …5
×

Vojtech huser-data-warehouse-evaluation-2010-04-idr-snapshot014c

619 views
531 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
619
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • open Tset
  • Completeness. Modeling all relevant performance factors to provide a holistic measurement of the concept. Concision. A calculation that is as simple and straightfoward as possible, making it understandable and logical to users. Measurability. Using direct performance data rather than relying too heavily on proxies or subjective measures. And from a practical perspective, if you can’t reliably gather valid data, the exercise is futile. Independence. The components of the measure need to be independent so that variation in one component doesn’t directly drive another.
  • Completeness. Modeling all relevant performance factors to provide a holistic measurement of the concept. Concision. A calculation that is as simple and straightfoward as possible, making it understandable and logical to users. Measurability. Using direct performance data rather than relying too heavily on proxies or subjective measures. And from a practical perspective, if you can’t reliably gather valid data, the exercise is futile. Independence. The components of the measure need to be independent so that variation in one component doesn’t directly drive another.
  • Completeness. Modeling all relevant performance factors to provide a holistic measurement of the concept. Concision. A calculation that is as simple and straightfoward as possible, making it understandable and logical to users. Measurability. Using direct performance data rather than relying too heavily on proxies or subjective measures. And from a practical perspective, if you can’t reliably gather valid data, the exercise is futile. Independence. The components of the measure need to be independent so that variation in one component doesn’t directly drive another.
  • Completeness. Modeling all relevant performance factors to provide a holistic measurement of the concept. Concision. A calculation that is as simple and straightfoward as possible, making it understandable and logical to users. Measurability. Using direct performance data rather than relying too heavily on proxies or subjective measures. And from a practical perspective, if you can’t reliably gather valid data, the exercise is futile. Independence. The components of the measure need to be independent so that variation in one component doesn’t directly drive another.
  • Completeness. Modeling all relevant performance factors to provide a holistic measurement of the concept. Concision. A calculation that is as simple and straightfoward as possible, making it understandable and logical to users. Measurability. Using direct performance data rather than relying too heavily on proxies or subjective measures. And from a practical perspective, if you can’t reliably gather valid data, the exercise is futile. Independence. The components of the measure need to be independent so that variation in one component doesn’t directly drive another.
  • Vojtech huser-data-warehouse-evaluation-2010-04-idr-snapshot014c

    1. 1. A Methodology for Quantitative Measurement of Quality and Comprehensiveness of a Research Data Repository (IDR Snapshot) Vojtech Huser MD PhD
    2. 2. Acknowledgement <ul><li>Wisconsin Institute for discovery </li></ul><ul><ul><li>Private branch: Morgridge Institute </li></ul></ul><ul><li>CTSA grant (NIH (NCRR) 1UL1RR025011) </li></ul><ul><ul><li>U of Wisconsin-Madison + Marshfield Clinic </li></ul></ul><ul><li>Marshfield Clinic Research foundation </li></ul>marshfieldclinic.org/birc
    3. 3. Agenda <ul><li>Introduction </li></ul><ul><li>Methods </li></ul><ul><ul><li>Design requirements </li></ul></ul><ul><ul><li>Measure components </li></ul></ul><ul><li>Results </li></ul><ul><ul><li>Marshfield Clinic </li></ul></ul><ul><li>Discussion/Conclusion </li></ul>marshfieldclinic.org/birc
    4. 4. <ul><li>Clinical Informatics </li></ul><ul><ul><li>Subfield of Biomedical Informatics </li></ul></ul>Background marshfieldclinic.org/birc Background Hersh BMC Medical Informatics and Decision Making (2009) 9:24 doi:10.1186/1472-6947-9-24
    5. 5. <ul><li>Clinical Informatics </li></ul><ul><ul><li>Subfield of Biomedical Informatics </li></ul></ul>Background marshfieldclinic.org/birc Background Hersh BMC Medical Informatics and Decision Making (2009) 9:24 doi:10.1186/1472-6947-9-24
    6. 6. Introduction <ul><li>Advances </li></ul><ul><ul><li>Federated research data warehouse </li></ul></ul><ul><ul><li>Future data growth </li></ul></ul><ul><li>Conduct of research </li></ul><ul><ul><li>RCT vs. cohort and case-control studies </li></ul></ul><ul><li>IDR = Integrated Data Repository </li></ul><ul><li>Choice of multiple institutions </li></ul><ul><li>Quick evaluation of an IDR </li></ul>marshfieldclinic.org/birc Frueh FW. Back to the future: why randomized controlled trials cannot be the answer to pharmacogenomics and personalized medicine. Pharmacogenomics 2009;10(7):1077-81. TRIAD
    7. 7. SHRINE: Query federation example <ul><li>Weber at al. (2009) </li></ul><ul><ul><li>Application of Information Technology: The Shared Health Research Information Network (SHRINE): A Prototype Federated Query Tool for Clinical Data Repositories </li></ul></ul><ul><ul><ul><li>JAMIA 2009;16:624-63 </li></ul></ul></ul><ul><li>Fusaro et al. </li></ul><ul><ul><li>Electronic Medical Record Analysis Using Cloud Computing </li></ul></ul><ul><ul><ul><li>AMIA CRI summit, 2010 </li></ul></ul></ul>marshfieldclinic.org/birc
    8. 8. Motivation <ul><li>Motivation </li></ul><ul><ul><li>10 institutions with IDRs; budget allows only 3; How do you choose? </li></ul></ul><ul><ul><li>Tracking improvement within an organization </li></ul></ul><ul><li>Assumptions (limitations) </li></ul><ul><ul><li>Goal: lifetime and complete EHR (+genetics, +behavior, +environment) </li></ul></ul><ul><ul><li>Improvement limits </li></ul></ul><ul><ul><li>General measures (rather then project specific) </li></ul></ul><ul><ul><li>Simple evaluation, pragmatic approach </li></ul></ul><ul><ul><li>Dichotomous approach to data sources (not profiling them) </li></ul></ul><ul><ul><li>Early stage (community interest solicitation) </li></ul></ul>linkedin.com/in/vojtechhuser
    9. 9. Measure <ul><li>Measures </li></ul><ul><ul><li>Qualitative description </li></ul></ul><ul><ul><li>Quantitative: Simple counts, Composite measure </li></ul></ul><ul><li>Composite scores: APGAR, FICO, Google PageRank </li></ul><ul><ul><li>Score: A short-hand way to communicate about a complex concept </li></ul></ul><ul><li>Data warehouse quality </li></ul><ul><ul><li>Size (1M vs. 10M), quality (billing data vs. structured EHR data) </li></ul></ul><ul><li>Possible requirements </li></ul><ul><ul><li>Design issues </li></ul></ul><ul><ul><ul><li>Completeness, Concision (simple), Measurability (objective, reliable), Independence </li></ul></ul></ul><ul><li>Our requirements </li></ul><ul><ul><li>Intuitive to interpret </li></ul></ul><ul><ul><li>Facilitates improvement </li></ul></ul><ul><ul><li>Fairness </li></ul></ul>linkedin.com/in/vojtechhuser
    10. 10. Data Structures (VDW) linkedin.com/in/vojtechhuser
    11. 11. Data Structures (i2b2) linkedin.com/in/vojtechhuser
    12. 12. Data Structures (HealthFlow) linkedin.com/in/vojtechhuser healthcareworkflow.wordpress.com
    13. 13. Data structures (data schemas) <ul><li>VDW </li></ul><ul><ul><li>Several tables </li></ul></ul><ul><ul><li>Data domain specific structures </li></ul></ul><ul><ul><li>Optimized for queries, users </li></ul></ul><ul><li>i2b2, HealthFlow </li></ul><ul><ul><li>One event table (+ attributes) </li></ul></ul><ul><ul><li>Generic data structures </li></ul></ul><ul><ul><li>Flexibility, extensibility </li></ul></ul>
    14. 14. Event table (IDR Snapshot v1) code.google.com/p/IDRSnapshot (Event-DOB) + ‘3000-01-01’
    15. 15. Results <ul><li>Events table (ev_measure01) </li></ul><ul><li>Generation took 2.5 hours </li></ul><ul><li>Size: 43 GB (includes additional info) </li></ul>code.google.com/p/IDRSnapshot
    16. 16. Code code.google.com/p/IDRSnapshot
    17. 17. Code code.google.com/p/IDRSnapshot
    18. 18. Code code.google.com/p/IDRSnapshot
    19. 19. Cross institution comparison code.google.com/p/IDRSnapshot
    20. 20. Monitoring code.google.com/p/IDRSnapshot
    21. 21. What’s in it for me? <ul><li>Use it </li></ul><ul><ul><li>Download our SQL code </li></ul></ul><ul><ul><li>Use it to evaluate your IDR </li></ul></ul><ul><li>Collaborate with us </li></ul><ul><ul><li>Paper in writing (3 institutions involved so far) </li></ul></ul><ul><ul><li>IDR Snapshot version 2 </li></ul></ul>code.google.com/p/IDRSnapshot IRB note: aggregate numbers
    22. 22. Future work <ul><li>Learn from IDRs of other institutions </li></ul><ul><li>IDR Snapshot version 2 </li></ul><ul><ul><li>Event table structure </li></ul></ul><ul><ul><li>HL7 vMR (lite, SQL format) </li></ul></ul><ul><li>Delphi method to extend measures </li></ul><ul><ul><li>Qualitative measures (transfers, insurance, PHR, family history) </li></ul></ul><ul><ul><li>Institutional context </li></ul></ul><ul><ul><li>Quantitative measures (general level, more complex “average patient“ pattern) </li></ul></ul><ul><li>Structured data </li></ul><ul><ul><li>Beyond claims data (e.g. nursing homes, decision support audit trails) </li></ul></ul><ul><li>Specialty snapshot </li></ul><ul><ul><ul><li>Prioritizing CDS interventions </li></ul></ul></ul><ul><ul><ul><li>CKD example (dialysis data) </li></ul></ul></ul>code.google.com/p/IDRSnapshot
    23. 23. Thank you <ul><li>[email_address] </li></ul><ul><li>http:// code.google.com/p/idrsnapshot </li></ul><ul><li>Slides: http:// www.linkedin.com/in/vojtechhuser </li></ul><ul><li>http://marshfieldclinic.org/birc </li></ul><ul><li>Questions? </li></ul>

    ×