MEDLINE search skill retention


Published on

Published in: Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

MEDLINE search skill retention

  1. 1. Measuring MEDLINE Searching Skill Retention in Medical Students: A Curriculum Integrated Instruction Follow-up Study Melissa L. Just
  2. 2. Introduction <ul><li>MEDLINE classes part of medical student orientation for years </li></ul><ul><li>Medical school transitioned to case-based learning in 2002 </li></ul><ul><li>Library skills became more crucial </li></ul><ul><ul><li>Moved from one-shot class to curriculum-integrated instruction </li></ul></ul>
  3. 3. Curriculum-Integrated Model <ul><li>MS1s get three hours of instruction during orientation </li></ul><ul><ul><li>1.5h MEDLINE </li></ul></ul><ul><ul><li>1.5h Finding Information for Cases </li></ul></ul><ul><li>Three search assignments follow </li></ul><ul><ul><li>1: Given clinical scenario and question; perform search </li></ul></ul><ul><ul><li>2: Given scenario; formulate question, perform search </li></ul></ul><ul><ul><li>3: Develop own scenario and question, perform search </li></ul></ul><ul><li>Each assignment is graded by the librarians using a locally-developed grading tool </li></ul><ul><li>Students are graded and receive narrative feedback on each assignment </li></ul>
  4. 4. But… <ul><li>No formalized searching required after first year </li></ul><ul><ul><li>Some clerkship directors ask students to “bring in an article on this patient/case” </li></ul></ul><ul><li>No measurement of skill retention or decay over time </li></ul>
  5. 5. Assignment <ul><li>Student pick one of seven scenarios </li></ul><ul><li>Search Ovid MEDLINE </li></ul><ul><li>Submit search strategy and five relevant articles to library for grading </li></ul><ul><li>Graded on </li></ul><ul><ul><li>Search process </li></ul></ul><ul><ul><li>Article selection </li></ul></ul>
  6. 6. Follow-up Study <ul><li>Fourth year students complete assignment in required clerkship </li></ul><ul><li>Assigned one of three scenarios </li></ul><ul><li>Conduct Ovid MEDLINE search </li></ul><ul><li>Submit strategy and selected articles to library for grading </li></ul><ul><li>Students also complete resource usage questionnaire </li></ul>
  7. 7. Research Question <ul><li>What is the retention rate of medical students’ searching skills from year one to year four? </li></ul>
  8. 8. Research Methodology <ul><li>Participants </li></ul><ul><ul><li>Class of 2007 </li></ul></ul><ul><ul><li>102 paired sets of grades (years 1 and 4) </li></ul></ul><ul><li>Measures </li></ul><ul><ul><li>LiSSA: 100 point scale and rubric for grading searches </li></ul></ul><ul><ul><li>Questionnaire: gathers data about students’ online resource usage </li></ul></ul>
  9. 9. Research Methodology <ul><li>Procedures </li></ul><ul><ul><li>Students completed assignment in year one (2003) </li></ul></ul><ul><ul><li>Students completed assignment in year four (2006/7) </li></ul></ul><ul><ul><li>For each student: scores were compared using paired t -test </li></ul></ul><ul><ul><li>Recall, precision, and F -measure were also compared </li></ul></ul>
  10. 10. Recall and Precision <ul><li>Recall: percentage of relevant articles retrieved in a search </li></ul><ul><ul><li>How many of the relevant articles in the database were retrieved? </li></ul></ul><ul><ul><li>Sensitivity </li></ul></ul><ul><li>Precision: percentage of retrieved articles that are also relevant </li></ul><ul><ul><li>How many of the retrieved articles were relevant? </li></ul></ul><ul><ul><li>Specificity </li></ul></ul>
  11. 11. F -measure <ul><li>Recall and precision are inversely related </li></ul><ul><li>F -measure harmonizes recall and precision – provides an “average” of the two scores </li></ul><ul><li>Equally rewards (or punishes) both sensitive and specific searchers </li></ul>
  12. 12. Results – Grades <ul><li>Mean of Assignment Grades </li></ul><ul><ul><li>Year 1: 91.69 </li></ul></ul><ul><ul><li>Year 4: 93.76 </li></ul></ul><ul><li>Appears skills were retained (and improved) over time </li></ul><ul><li>But, is there really a difference? </li></ul>
  13. 13. Significant Difference in Scores? <ul><li>t- test </li></ul><ul><ul><li>t (101) = -1.38, p =.177 </li></ul></ul><ul><li>Correlation </li></ul><ul><ul><li>r = .114, p = .255 </li></ul></ul><ul><ul><li>r 2 = .01 </li></ul></ul><ul><li>No significant change from years 1 to 4 </li></ul>
  14. 14. Results – Recall, Precision <ul><li>Recall </li></ul><ul><ul><li>Year 1: .42 </li></ul></ul><ul><ul><li>Year 4: .27 </li></ul></ul><ul><li>Precision </li></ul><ul><ul><li>Year 1: .48 </li></ul></ul><ul><ul><li>Year 4: .32 </li></ul></ul><ul><li>F -measure </li></ul><ul><ul><li>Year 1: .40 </li></ul></ul><ul><ul><li>Year 4: .24 </li></ul></ul>
  15. 15. Significant Difference in Scores? <ul><li>t -test declines were significant ( p < .0001) level for recall, precision, and F </li></ul><ul><li>Correlations </li></ul><ul><ul><li>Years 1 & 4 recall rates not correlated </li></ul></ul><ul><ul><li>Years 1 & 4 precision rates not correlated </li></ul></ul><ul><ul><li>F -measure rates moderately correlate ( r = .239) & were significant ( p < .05) </li></ul></ul>
  16. 16. Discussion <ul><li>Searching skills did not significantly decrease or increase over time </li></ul><ul><li>Students search output results significantly declined over time </li></ul><ul><li>What’s more important: process or output? </li></ul><ul><ul><li>Are we teaching the right thing? </li></ul></ul>
  17. 17. Implications <ul><li>For practice </li></ul><ul><ul><li>Teach outcomes focused searching </li></ul></ul><ul><ul><li>Grade students’ output instead of process </li></ul></ul><ul><li>For research </li></ul><ul><ul><li>Measure retention of skills of new teaching model </li></ul></ul><ul><ul><li>Measure impact on future practice </li></ul></ul>
  18. 18. Questions? Melissa L. Just [email_address]