Sam Stewart - Knowledge Linages

473 views
419 views

Published on

Knowledge Linkages: Augmenting Online Clinical Care Discussions with Published Literature

Published in: Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
473
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Sam Stewart - Knowledge Linages

  1. 1. Knowledge Linkages Augmenting Online Clinical Care Discussions with Published Literature Sam Stewart, Syed Sibte Raza Abidi NICHE Research Group Faculty of Computer Science Dalhousie University, Halifax, Canada November 30, 2010 Sam Stewart (Dal) Knowledge Linkages November 30, 2010 1 / 35
  2. 2. Outline Introduction Problem Description Knowledge Linkage Framework Preliminary Results Conclusion Sam Stewart (Dal) Knowledge Linkages November 30, 2010 2 / 35
  3. 3. Introduction Introduction Pediatric pain management is a complex subject children lack the cognitive ability to properly express their pain, which can lead to incorrect interventions. Lack of specialized knowledge or training in pediatric pain management. Because of the temporal and physical restrictions that clinicians face, traditional educational systems are not a plausible solution Web 2.0 technologies provide alternate knowledge dissemination mediums for clinicians to converge and share their knowledge. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 3 / 35
  4. 4. Introduction Rationale for Knowledge Linkages Pediatric Pain Mailing List (PPML) Brings together over 700 pediatric pain practitioners from around the world to share their clinical experiences and seek advice The knowledge shared on the PPML is practice-based rather than evidence based It is important to augment the practice-based (tacit) knowledge on the PPML with explicit knowledge The goal of this project is to establish knowledge linkages between discussions on the PPML and published literature on Pubmed. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 4 / 35
  5. 5. Introduction Project Objectives The objective of knowledge linkage is to reaffirm the practice-related recommendations on the PPML with evidence-based literature from Pubmed The outcome of the project will allow users to search through PPML archives to find topics of interest Retrieve research articles related to the topics from Pubmed, through a “single-click” evidence retrieval strategy. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 5 / 35
  6. 6. Introduction Knowledge Linkage Framework PPML Archives Online Disucssion Forum Message Parsing Filtered Messages Threading Algorithm Threads Mapping To MeSH Thread MeSHThread MeSHThread Papers Information Retrieval Sam Stewart (Dal) Knowledge Linkages November 30, 2010 6 / 35
  7. 7. Project Framework Step 1: Processing the Archives The archives are stored in simple ASCII text files, organized by month, starting June 1993 and ending December 2008 The messages are processed to extract the sender, date, subject line and content of the messages The messages are filtered to remove non-substantive content PPML Archives Online Message Parsing Filtered Messages Threading Algorithm Threads Mapping To MeSH Thread MeSHThread MeSHThread Sam Stewart (Dal) Knowledge Linkages November 30, 2010 7 / 35
  8. 8. Project Framework Step 2: Threading A thread is a series of messages centred around a common subject. They are the embodiment of experiential knowledge on the PPML. The messages are assigned to threads using their subject lines. PPML Archives Online Message Parsing Filtered Messages Threading Algorithm Threads Mapping To MeSH Thread MeSHThread MeSHThread Sam Stewart (Dal) Knowledge Linkages November 30, 2010 8 / 35
  9. 9. Project Framework Step 3: Mapping to MeSH Threads Mapping To MeSH Thread MeSH The threads are parsed and connected to formal MeSH terms, using the Metamap program Metamap creates a mapping score, which is a measure of the strength of the connection Each mesh-based thread is used to query PubMed Sam Stewart (Dal) Knowledge Linkages November 30, 2010 9 / 35
  10. 10. Project Framework Metamap: Mapping free text to MeSH Metamap is a program developed by Dr. Alan Aronson at the NLM that maps biomedical text to the MeSH lexicon Each mapping is assigned a score that is a measure of the strength of the mapping. 1000×(Centrality+Variation+2×Coverage+2×Cohesiveness)/6 The scores provide a baseline measure of how well the mapped MeSH term represents the original term in the thread Sam Stewart (Dal) Knowledge Linkages November 30, 2010 10 / 35
  11. 11. Project Framework Example Sample Statement ‘‘The report stated that when music therapy is used, the babies required less pain medication. Does anyone know of any published reports of empirical research demonstrating the effect?’’ Source MeSH Term Score music therapy Music Therapy 1000 the babies Infant 966 less pain medication Pain 660 less pain medication Pharmaceutical Preparations 827 published reports Publishing 694 empirical research Empirical Research 1000 Sam Stewart (Dal) Knowledge Linkages November 30, 2010 11 / 35
  12. 12. Project Framework Step 4: Literature Search StrategyPPML Archives Online Disucssion Forum Message Parsing Threading Algorithm Mapping To MeSH Thread MeSHThread MeSHThread Papers Information Retrieval Passively links the threads to published medical literature Naive approach: Retrieve all papers that contain every MeSH term in the thread. If no papers exist the algorithm would drop the lowest scoring terms and reiterate Sam Stewart (Dal) Knowledge Linkages November 30, 2010 12 / 35
  13. 13. Project Framework Naive Approach The naive approach has several problems It doesn’t provide any kind of ordering on the resulting papers It doesn’t fully utilize the MeSH scores It doesn’t take into account the possibility of incorrect mappings One of the challenges of mapping free text with Metamap is its inaccuracy. The presence of a false MeSH term with a high MeSH score will prevent the retrieval of useful papers Sam Stewart (Dal) Knowledge Linkages November 30, 2010 13 / 35
  14. 14. Project Framework Improved Search Strategy Our improved search strategy makes full use of the Metamap scores It also addresses the problem of incorrect mappings It is based on the Extended Boolean Information Retrieval (eBIR) algorithm Customizes the algorithm to deal with pediatric pain by adding a specialized filter Let (Mi , mi ) be MeSH term i and the associated Metamap score. Q = [Infant OR Child OR Adolescent] AND [(M1, m1) ORP (M2, m2) ORP . . . (Mn, mn)] (1) Sam Stewart (Dal) Knowledge Linkages November 30, 2010 14 / 35
  15. 15. Project Framework Step 5: Discussion Forum Archives Online Disucssion Forum Mapping To MeSH Thread MeSHThread MeSHThread Papers Information Retrieval An online forum is being developed that allows practitioners to interact with the PPML discussions and review the research articles for a specific discussion thread. The forum will be navigated by a standard search function, or by a search function based on MeSH terms As well the threads will be organized into a hierarchy based on their MeSH terms Sam Stewart (Dal) Knowledge Linkages November 30, 2010 15 / 35
  16. 16. Project Framework Discussion Forum Sam Stewart (Dal) Knowledge Linkages November 30, 2010 16 / 35
  17. 17. Project Framework Example Thread Sam Stewart (Dal) Knowledge Linkages November 30, 2010 17 / 35
  18. 18. Project Framework Linked Papers Sam Stewart (Dal) Knowledge Linkages November 30, 2010 18 / 35
  19. 19. Results Example This first example is the first thread ever transmitted on the PPML It is on the subject of Music Therapy The following slides show the discussion, then the list of MeSH terms mapped, then a sampling of the papers returned Sam Stewart (Dal) Knowledge Linkages November 30, 2010 19 / 35
  20. 20. Results Music Therapy I Sender: 1 Subject: Music Therapy Date: Mon Jun 28 21:19:36 ADT 1993 Thread: 3, false The last several days, the local NBC station aired a ”medical report” about the use of music therapy. The report was from Miami and included a short report on the use of music therapy in a NICU. The report stated that when music therapy was used, the babies required less pain medication. Does anyone know of any published reports of empirical research demonstrating this effect? Sam Stewart (Dal) Knowledge Linkages November 30, 2010 20 / 35
  21. 21. Results Music Therapy II Message Sender:2 Subject: Music Therapy Date: Tue Jun 29 08:25:12 ADT 1993 Thread: 3, true I would suggest that you might contact **** ******** in Pediatrics at Washington University Medical School. Her research is on neonatal pain and she might know where the local station picked up the report. I haven’t seen any data on the topic. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 21 / 35
  22. 22. Results Music Therapy III Message Sender: 3 Subject: Music Therapy Date: Tue Jun 29 10:20:41 ADT 1993 Thread: 3, true I’m not aware of specific studies conducted using music therapy to reduce the need for pain medication (i.e., music therapy to manage pain). However, several cognitive interventions have been used quite effectively to manage pain. Donald Meichenbaum developed a technique in the early 1970s called stress inoculation training which combines aspects of self-instruction training and relaxation training. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 22 / 35
  23. 23. Results Music Therapy MeSH Terms MeSH Score Papers Music Therapy -4802 1621 Pain -4215 237890 Research -2000 254813 Pharmaceutical Preparations -1688 438290 Infant, Newborn -1660 422011 Education -1654 492705 Teaching -1320 52938 Vaccination -1320 44092 Intensive Care Units, Neonatal -1000 6847 Pediatrics -1000 34612 Empirical Research -1000 8897 Relaxation Therapy -1000 5677 Vision, Ocular -966 18516 Behavior -966 879624 Infant -966 795209 Air -966 16342 Awareness -861 9711 Schools, Medical -861 17436 Biomedical Research -827 28157 Publishing -694 27222 Cognition -589 76048 Sam Stewart (Dal) Knowledge Linkages November 30, 2010 23 / 35
  24. 24. Results Music Therapy Papers Returned Bo LK, Callaghan P. Soothing pain-elicited distress in Chinese neonates. Pediatrics:2000,105(4). 10742370. Cignacco E, Hamers JP, Stoffel L, van Lingen RA, Gessler P, McDougall J, Nelle M. The efficacy of non-pharmacological interventions in the management of procedural pain in preterm and term neonates. A systematic literature review. European journal of pain (London, England):2006,11(2). 16580851. Kemper KJ, Danhauer SC. Music as therapy. Southern medical journal:2005,98(3). 15813154. Tagore T. Why music matters in childbirth. Midwifery today with international midwife:2009,(89). 19397157. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 24 / 35
  25. 25. Results Pilot Study A pilot study was conducted on all messages from 2007 and 2008 100 threads were reviewed to determine 1 the accuracy of the message parsing 2 the accuracy of the thread assignment 3 The accuracy of the papers returned The message parsing was successful on 74% of the messages The threading was successful on 92% of the messages Sam Stewart (Dal) Knowledge Linkages November 30, 2010 25 / 35
  26. 26. Results Recall, Precision, Utility Precision and relative recall were compared between the modified search strategy, the eBIR model, and a traditional VSM. Relative recall is for comparing search strategies of unannotated databases Precision = Number of relevant papers returned by the search Total number of papers returned Recall = Number of relevant papers returned by the search Number of relevant papers returned by all searches Sam Stewart (Dal) Knowledge Linkages November 30, 2010 26 / 35
  27. 27. Results Precision 2 4 6 8 10 12 14 0.000.050.100.150.20 Top k papers Precision q q q q q q q q q q q q q q q q Custom VSM eBIR Sam Stewart (Dal) Knowledge Linkages November 30, 2010 27 / 35
  28. 28. Results Recall 2 4 6 8 10 12 14 0.00.20.40.60.8 Top k papers RelativeRecall q q q q q q q q q q q q q q q q Custom VSM eBIR Sam Stewart (Dal) Knowledge Linkages November 30, 2010 28 / 35
  29. 29. Results Precision-Recall 0.2 0.4 0.6 0.8 0.080.100.120.140.160.18 Relative Recall Precision q q q q q q q q q q q q q q qq Custom VSM eBIR Sam Stewart (Dal) Knowledge Linkages November 30, 2010 29 / 35
  30. 30. Results Search Strategy Results The precision of the modified algorithm is significantly higher than the other two algorithms at k = 15 (p-values of 0.013 and 0.003 respectively) The recall, however, is only significantly different between the modified and ebir models (p < 0.0001) and not with the VSM algorithm (p = 0.351) Ultimately, a search is “good” if it returns at least one pertinent result Utility at level k is an indicator of whether the search returns a relevant paper in the first k results. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 30 / 35
  31. 31. Results Utility vs. k 2 4 6 8 10 12 14 0.00.10.20.30.40.50.6 Top k papers Utility q q q q q q q q q q q q q q q q Custom VSM eBIR Sam Stewart (Dal) Knowledge Linkages November 30, 2010 31 / 35
  32. 32. Conclusion Conclusion The mapping of experiential to explicit clinical knowledge is critical, given the rapid changes in medical knowledge and its application in specialized domains Clinical experiences should be supported by clinical evidence, and this has been achieved through our Knowledge Linkage Framework Presented a method of leveraging web 2.0 techniques by incorporation medical information retrieval strategies to improve the overall medical knowledge base Automatic query generation, using clinical terms and contexts, is a unique aspect of the research Sam Stewart (Dal) Knowledge Linkages November 30, 2010 32 / 35
  33. 33. Conclusion Future Work The next step is to provide open access to a wide number of users and get their feedback More time should be spent looking into the variables within the eBIR algorithm and the modified algorithm Q = [Infant ORp1 Child ORp1 Adolescent] ANDp2 [M1 ORp3 M2 ORp3 . . . ORp3 Mn] Tweaking the Metamap scores, either within the Metamap system or through post-processing, should also be explored Sam Stewart (Dal) Knowledge Linkages November 30, 2010 33 / 35
  34. 34. Conclusion Acknowledgement This work is carried out with the aid of a grant from the International Development Research Centre (IDRC), Ottawa, Canada Sam Stewart (Dal) Knowledge Linkages November 30, 2010 34 / 35
  35. 35. Conclusion Thank you Sam Stewart (Dal) Knowledge Linkages November 30, 2010 35 / 35
  36. 36. Appendix Appendix Sam Stewart (Dal) Knowledge Linkages November 30, 2010 1 / 11
  37. 37. Appendix Metamap Metamap Algorithms There are three general types of matches: Simple match a direct connection between the recognized noun and the UMLS term Complex match when a noun phrase can be mapped directly to a combination of UMLS semantic types Partial match when part of the noun/noun-phrase does not map to UMLS The general mapping strategy is, for each term SPECIALIST recognizes: generate all variants of the noun-phrase, form the candidate set of all the UMLS strings that contain 1 of the variants, sort the candidate set by the strength of mapping, combine candidates for disjoint parts of the noun-phrase, then select the mapping with the best score. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 2 / 11
  38. 38. Appendix Metamap Variants The variants are all composite parts of a noun-phrase, along with all acronyms, abbreviations and synonyms of those terms, all variants of those variants, etc . . . For the term ocular the following figure depicts the generation of the variants Sam Stewart (Dal) Knowledge Linkages November 30, 2010 3 / 11
  39. 39. Appendix Metamap Metamap Scores The scores range from [-1000, 0], with lower scores being better The score is based on 4 metrics: centrality, variation, coverage and cohesiveness. The final score is calculated as: −1000×(Centrality +Variation+2×Coverage +2×Cohesiveness)/6 Sam Stewart (Dal) Knowledge Linkages November 30, 2010 4 / 11
  40. 40. Appendix Metamap Metamap Scores I Centrality a 1/0 indicating whether the match is to the head of the phrase Variation A measure of the distance the matched term is from the root word. The distance, D, is a sum of the following variations. The score is calculated as 4 D+4 .: spelling: 0 inflectional: 1 synonym/acronym/abbreviation: 2 derivational: 3. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 5 / 11
  41. 41. Appendix Metamap Metamap Scores II Coverage How much of both the UMLS string and the phrase are involved in the match. The number of words in each phrase are computed, as well as the spans of each term, i.e., the length of the matching terms, ignoring non-matching terms. The score is calculated as 2 3 Span UMLS Length + 1 3 Span term length Cohesiveness Like coverage, but focusing on connected terms. It calculates the length of connected components (the maximal sequence of connected words in both terms), and takes a weighted mean again, this time of the sum of squares. 2 3 SS UMLS con comps UMLS length2 + 1 3 SS phrase con comps phrase length2 Sam Stewart (Dal) Knowledge Linkages November 30, 2010 6 / 11
  42. 42. Appendix Metamap Sample Metamap Scores From PPARCH.199603 Noun UMLS Cent. Var. Cov. Coh. aired Air 1 D=1;4/5 1 1 of music therapy Music Therapy 1 0;1 1 1 a NICU ICU, Neonatal 1 0;1 1 1 the babies Infant 1 D=1; 4/5 1 1 less pain medication Pain 0 0;1 2 3 1 1 + 1 3 1 3 2 3 1 1 + 1 3 1 9 Pharm Prep 1 0;1 2 3 2 2 + 1 3 1 3 2 3 2 2 + 1 3 1 9 of any published reports Publishing 0 0;1 2 3 1 1 + 1 3 1 2 2 3 1 1 + 1 3 1 4 empirical research Empirical Research 1 0;1 1 1 Noun UMLS TOTAL aired Air −1000 × (1 + 4/5 + 2(1) + 2(1))/6 = −966 of music therapy Music Therapy −1000 a NICU ICU, Neonatal −1000 the babies Infant −1000 × (1 + 4/5 + 2(1) + 2(1))/6 = −966 less pain medication Pain −1000 × (0 + 1 + 2( 7 9 ) + 2( 19 27 ))/6 = −660 Pharm Prep −1000 × (1 + 1 + 2( 7 9 ) + 2( 19 27 )/6 = − − 827 −660−827 2 = −743.5 of any published reports Publishing −1000(0 + 1 + 2( 10 12 ) + 2( 36 48 ))/6 = −694 empirical research Empirical Research −1000 Sam Stewart (Dal) Knowledge Linkages November 30, 2010 7 / 11
  43. 43. Appendix Metamap Extended Boolean Information Retrieval (eBIR) The eBIR system incorporates query weights into the traditional BIR model Let the set of query terms be A = {(A1, s1), . . . , (An, sn)}, where Ai is the ith query term, and si is the associated score Let the OR and AND queries be QOR(p) = {(A1, s1) ORp . . . ORp (An, sn)} QAND(p) = {(A1, s1) ANDp . . . ANDp (An, sn)} The selection of p effects the influence of high-scoring terms on the returned query scores. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 8 / 11
  44. 44. Appendix Metamap Modified IR Algorithm The problem with applying the eBIR algorithm to this project is that it doesn’t address the issue of specialized domains MeSH keywords such as Pediatrics or Pain could be implicitly representative of all conversations on the list Our algorithm modified the eBIR algorithm by adding a specialized filter adding an AND operator to the query Sam Stewart (Dal) Knowledge Linkages November 30, 2010 9 / 11
  45. 45. Appendix Metamap Modified IR Algorithm The new query would modify the search query by adding Infant, Child or Adolescent to the set of MeSH terms, as demonstrated in equation (2) Let (Mi , mi ) be MeSH term i and the associated Metamap score. Q = [Infant ORp Child ORP Adolescent] ANDP [(M1, m1) ORP (M2, m2) ORP . . . (Mn, mn)] (2) Using the eBIR algorithm the next step would be to apply query weights to the terms in the specialized filter and then find a suitable value for p. Sam Stewart (Dal) Knowledge Linkages November 30, 2010 10 / 11
  46. 46. Appendix Metamap Modified IR Algorithm In order to accommodate the filter the eBIR algorithm was modified, making the AND operator a strict Boolean operator, and leaving the query weights on the OR operator The decision was also made to set p = 1. Q = [Infant OR Child OR Adolescent] AND [(M1, m1) ORP (M2, m2) ORP . . . (Mn, mn)] (3) The result is a search strategy customized to the pediatric pain domain, that makes full use of the Metamap scores to return a pertinent set of papers Sam Stewart (Dal) Knowledge Linkages November 30, 2010 11 / 11

×