Collecting User Preferences for Web-Based MEDLINE Training: A ...


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Collecting User Preferences for Web-Based MEDLINE Training: A ...

  1. 1. Collecting User Preferences for Web-Based MEDLINE Training: A Capitol Idea Kelly Near Andrea Horne Karen Grandage Claude Moore Health Sciences Library
  2. 2. Overview <ul><li>Proliferation of electronic resources </li></ul><ul><li>Busy health professionals and students </li></ul><ul><li>Need for continual end-user training and support, available any time & any place </li></ul>
  3. 3. Tutorial Study Objectives: <ul><li>Compare user preferences </li></ul><ul><li>Gain experience with new technologies for delivering online training </li></ul><ul><li>Evaluate the educational effectiveness of each tutorial format </li></ul><ul><li>Evaluate the cost/benefit ratio for each tutorial format </li></ul>
  4. 4. Background <ul><li>Advancements in technology </li></ul><ul><ul><li>Web frames </li></ul></ul><ul><ul><li>Streaming audio & video </li></ul></ul><ul><ul><li>Flash </li></ul></ul><ul><ul><li>Java Script Applets </li></ul></ul>
  5. 5. Background – MEDLINE Training <ul><li>Still our “bread and butter” resource </li></ul><ul><li>Medical and Nursing School curriculum training sessions </li></ul><ul><li>Monthly classes </li></ul><ul><li>Individual consultations </li></ul>
  6. 6. Planning <ul><li>Always allow more time than you think you’ll need! </li></ul><ul><li>We began with the assumption that online tutorials were an important part of educational delivery </li></ul>
  7. 7. Planning <ul><li>Tutorial formats were chosen based on - </li></ul><ul><ul><li>learning style theory </li></ul></ul><ul><ul><li>our skill & comfort level with the necessary technology </li></ul></ul>
  8. 8. Tutorial A <ul><li>Interactive </li></ul><ul><li>Created in Web frames </li></ul><ul><li>Search instructions in the left frame </li></ul><ul><li>“ Live” version of the MEDLINE database active in the right frame </li></ul>Tutorial A
  9. 9. Tutorial B <ul><li>Short animated screen capture illustrations of commonly performed database tasks </li></ul><ul><li>Audio narrative </li></ul>Tutorial B
  10. 10. Tutorial C <ul><li>Classroom format </li></ul><ul><li>Streaming video and audio </li></ul><ul><li>Clips of an instructor interspersed with screen shots of the database </li></ul>Tutorial C
  11. 11. Study Design <ul><li>Wanted our study participants to be a cross-section of our usual MEDLINE class students </li></ul><ul><li>10 participants were randomly assigned to each tutorial </li></ul><ul><li>We did not determine their MEDLINE searching abilities prior to the testing </li></ul><ul><li>None of the tutorials were meant to be as detailed as an actual class </li></ul>
  12. 12. <ul><li>Debate: </li></ul><ul><li>- should they answer specific search questions, such as “what is the symbol used to truncate in Ovid MEDLINE” </li></ul><ul><li>or </li></ul><ul><li>- should they conduct a complete search on a given topic, and have it measured against a “gold standard”? </li></ul>
  13. 13. Search Question <ul><li>Use MEDLINE </li></ul><ul><li>1996 – present </li></ul><ul><li>Perform a comprehensive search on the question: </li></ul><ul><li>Is there a connection between head injuries and depression in children? </li></ul>
  14. 14. Six Steps to a Good Search <ul><li>Rated each search on a 6 point scale </li></ul><ul><ul><li>1 point for selecting the MeSH term “craniocerebral trauma” </li></ul></ul><ul><ul><li>1 point for exploding it </li></ul></ul><ul><ul><li>1 point for selecting the MeSH term “depression” </li></ul></ul><ul><ul><li>1 point for NOT exploding it! </li></ul></ul><ul><ul><li>1 point for combining the terms with AND </li></ul></ul><ul><ul><li>1 point for applying the age limit </li></ul></ul>
  15. 15. Gold Standard Explode !
  16. 16. Developing Tutorial A <ul><li>Key features: </li></ul><ul><ul><li>HTML frames </li></ul></ul><ul><ul><li>Ovid Jumpstarts </li></ul></ul><ul><li>Products used: </li></ul><ul><ul><li>HTML Assistant </li></ul></ul><ul><ul><li>Notepad </li></ul></ul><ul><li>Cost = 0 </li></ul><ul><li>Challenges: </li></ul><ul><ul><li>Condensing a full class down to key learning points and brief descriptions </li></ul></ul><ul><li> </li></ul>Key:  =development time $ =money spent
  17. 17. Developing Tutorial B <ul><li>Key features: </li></ul><ul><ul><li>Short video & audio clips with text </li></ul></ul><ul><li>Products used: </li></ul><ul><ul><li>SnagIt! </li></ul></ul><ul><ul><li>Microphone and headset </li></ul></ul><ul><li>Cost = </li></ul><ul><ul><li>$39.95 for single user copy </li></ul></ul><ul><li> $ </li></ul>Key:  =development time $ =money spent
  18. 18. Developing Tutorial B <ul><li>Challenges: </li></ul><ul><ul><li>Time required to write a concise script </li></ul></ul><ul><ul><li>Time for the multiple takes needed to do a clean audio track … </li></ul></ul><ul><ul><ul><ul><li>especially using non-union help! </li></ul></ul></ul></ul>Take 5!
  19. 20. Starting a Subject Search This example will demonstrate a search for articles about oral cancers in teenage boys who use chewing tobacco. Begin by separating your search topic into individual concepts. Type your first search concept into the box labeled Enter Keyword or Phrase. For our search, we begin by typing in the keywords &quot;oral cancer&quot;. Be sure that the checkmark is in the box by Map Term to Subject Heading Next Topic Return to Tutorial Home
  20. 21. Developing Tutorial C <ul><li>Key features: </li></ul><ul><ul><li>Used our classroom as the “stage” </li></ul></ul><ul><ul><li>Hired a “professional actress” so fewer retakes were necessary </li></ul></ul><ul><li>Products used: </li></ul><ul><ul><li>Adobe Premiere </li></ul></ul><ul><li>Cost = $399 </li></ul><ul><li> $$$$ </li></ul>Key:  =development time $ =money spent
  21. 22. Developing Tutorial C <ul><li>Challenges: </li></ul><ul><ul><li>iMovie and Final Cut Pro didn’t allow us to intersperse SnagIt video clips with instructor video </li></ul></ul><ul><ul><li>Next tried videotaping the screen in the classroom as the MEDLINE search was being done </li></ul></ul><ul><ul><li>Adobe Premiere allowed us to combine the video and the SnagIt clips into one product </li></ul></ul>
  22. 24. Tutorial C <ul><li>Note – can we add a movie clip to show Tutorial C here? </li></ul>
  23. 25. Recruitment <ul><li>Sent email to medical students in all 4 years inviting them to participate in the study </li></ul><ul><li>Posted recruitment flyers around the Library, Medical School and neighboring Research Buildings </li></ul><ul><li>Word-of-mouth seemed to help </li></ul><ul><li>So did the FREE lunch! </li></ul>
  24. 26. Pre-Testing <ul><li>Pre-tested the process with 3 library staff </li></ul><ul><li>Identified problems with inconsistent instructions </li></ul><ul><li>Refined test questions and tutorials based on this input </li></ul>
  25. 27. Testing <ul><li>Randomly assigned 10 participants to each tutorial </li></ul><ul><li>The tutorials were taken in a library classroom </li></ul><ul><li>After completing the tutorial, they were each given the same search to complete in Ovid MEDLINE </li></ul>
  26. 28. Interviews <ul><li>Participants were asked12 questions </li></ul><ul><ul><li>Examples </li></ul></ul><ul><ul><ul><li>Have you ever taken a MEDLINE class before? </li></ul></ul></ul><ul><ul><ul><li>Did you have any technical problems with the tutorial? </li></ul></ul></ul><ul><ul><ul><li>Did you refer back to the tutorial to complete the steps for the test search? </li></ul></ul></ul>
  27. 29. Free Lunch <ul><li>Testing over the middle of the day worked very well as the lure of a </li></ul><ul><li>FREE LUNCH </li></ul><ul><li>was irresistible to all! </li></ul>
  28. 30. Results <ul><li>Question 1: </li></ul><ul><ul><li>Tested 30 people, divided into 3 groups of 10 (A, B, C). Each group watched a different tutorial, but then took the same test </li></ul></ul><ul><ul><li>Group A scored average (mean) 4.5 (out of 6 possible points) </li></ul></ul><ul><ul><li>Group B scored 5.1 </li></ul></ul><ul><ul><li>Group C scored 4.9 </li></ul></ul><ul><li>Is there a statistical difference in the test scores among the 3 different groups? </li></ul>
  29. 31. Test Scores Number of questions Number correct
  30. 32. ANOVA Test Results P value needs to be below 5% - this is 45% .446 .832 .933 1.122 2 27 29 1.867 30.300 32.167 Between Groups Within Groups Total Sig. F Mean Square df Sum of Squares
  31. 33. Results <ul><li>Question 2: </li></ul><ul><li>For each group we looked at one key question and whether it was answered correctly (exploding the MeSH term depression) </li></ul><ul><ul><li>Group A 6/10 answered correctly </li></ul></ul><ul><ul><li>Group B 10/10 answered correctly </li></ul></ul><ul><ul><li>Group C 7/10 answered correctly </li></ul></ul>
  32. 34. Explode Number of Participants Number Exploding
  33. 35. Chi Square Analysis <ul><li>Chi Square analysis included 3 separate tests </li></ul><ul><li>Only 1 showed a significant difference </li></ul><ul><li>No significant statistical differences were found overall </li></ul>
  34. 36. Tutorial Preferences <ul><li>Overall, interview results showed positive acceptance for all three tutorial types </li></ul>A B C
  35. 37. Results pie chart <ul><li>Majority of test takers (22/30) ranked “taking an online tutorial” as their most preferred method of learning, over </li></ul><ul><ul><li>“ taking a class in a classroom” (4/30) </li></ul></ul><ul><ul><li>“ reading a printed handout” (3/30) </li></ul></ul><ul><li>Many commented that they could envision using the online tutorial as a supplement to the MEDLINE classes they had already taken </li></ul>
  36. 38. Reported Learning Preferences Online Tutorial Class Printed 22 4 3
  37. 39. Lessons Learned … <ul><li>Only 8 users had not used MEDLINE before </li></ul><ul><li>We looked at this data to determine if there was a difference in the scores between new and experienced MEDLINE users </li></ul><ul><li>Did not find any difference between the 2 groups </li></ul>
  38. 40. Lessons Learned … <ul><li>May need a larger sample size </li></ul><ul><li>AVI files (from SnagIt!) are too large </li></ul><ul><li>Try Snapz Pro (Macintosh only) for video captures </li></ul>
  39. 41. One Caveat … <ul><li>Ten days before we were to start testing - we found we needed approval from the University’s Human Investigation! </li></ul>
  40. 42. Conclusions <ul><li>Scores = no difference </li></ul><ul><li>Next, we looked at production costs and time and found … </li></ul><ul><li> MAJOR DIFFERENCES </li></ul>
  41. 43. … and the Winner is -- Tutorial A!
  42. 44. Future Directions
  43. 45. Tutorial A = the one to trust!
  44. 46. Questions ? <ul><li>Collecting User Preferences for </li></ul><ul><li>Web-Based MEDLINE Training: A Capitol Idea </li></ul><ul><ul><li>Kelly Near, Andrea Horne, Karen Grandage </li></ul></ul>