Assessing an adaptive, profile-based PubMed search tool

1,785 views

Published on

Presentation at MLA annual conference on MiSearch, MLA ID 08-pap-479-mla

Published in: Technology, Travel

Assessing an adaptive, profile-based PubMed search tool

  1. 1. Assessing an adaptive, profile-based PubMed search tool Marisa Conte meese@umich.edu NLM Associate Fellow University of Michigan Health Sciences Libraries
  2. 2. MiSearch  Adaptive biomedical literature search tool  Utilizes implicit relevance feedback  Builds statistical profile to predict which results will be most relevant to user
  3. 3. Objective  To compare the performance of a locally- developed third-party PubMed search tool to PubMed in terms of user satisfaction, search success and efficiency
  4. 4. Collaboration
  5. 5. Methodology  N = 11 librarians (12 recruited)  Tasks derived from mediated search files  Each subject completed 2 tasks with each search tool  Each task completed by 3 subjects with each search tool  Pre-training conducted by tool developer
  6. 6. Data collection  Web-based surveys  Demographic survey  Task-specific survey  PubMed search histories  MiSearch data logfiles
  7. 7. Sample questions  In patients with diabetes mellitus, does increasing hemoglobin A1c reduce the risk of amputation and kidney failure?  What are the attitudes, beliefs and socio- cultural factors that influence cervical cancer screening in the US? What about specifically among immigrant groups? And I’m only interested in research from 1990 – present.
  8. 8. Limitations  Convenience sample  Unclear survey questions  Study tasks too complicated  Problems with data collection
  9. 9. Sample results – Query 2 Average satisfaction MiSearch PubMed Search activity 1 1.7 Tool 1.3 1.7 Speed 1.3 1.7 Accuracy 1.3 2 Completeness 1.7 2 1 = low
  10. 10. Overall satisfaction with searches 60 50 % responses 40 30 PubMed 20 Misearch 10 0 High Fair Low User satisfaction
  11. 11. Overall satisfaction with tools 70 60 % responses 50 40 30 PubMed 20 MiSearch 10 0 High Fair Low User satisfaction
  12. 12. Suggestions  Improve database response time  Add more filters  Add a “reset” function  Make search details more transparent
  13. 13. Next steps  Librarians suggested changes which:  were easy to implement  dramatically improved the tool’s functionality  First usability test – helped establish formal usability protocols and heuristics  New roles for librarians
  14. 14. New roles for librarians  Usability experts  Heuristic evaluation of tools  Portal development  Formal usability studies  Weblog analyses  Education, promotion, dissemination  Research collaborations
  15. 15. Resources  MiSearch: http://misearch.ncibi.org/  NCIBI: https://portal.ncibi.org/gateway/  States, D.J., Ade, A.S., Wright, Z.C., Bookvich, A.V. & Athey, B.D. (2008) Misearch Adaptive PubMed Search Tool. Bioinformatics. PMID 18326507
  16. 16. Acknowledgments  Jean Song, Nirit Glazer, Barbara Mirel, David States  Librarians from UM’s Health Sciences and Shapiro Science Libraries  National Center for Integrative Biomedical Informatics (NIH grant #U54DA021519)  National Library of Medicine This research was supported in part by an appointment to the NLM Associate Fellowship Program sponsored by the National Library of Medicine and administered by the Oak Ridge Institute for Science and Education.

×