Assessment of knowledge and skills in Nursing and Allied Health student information literacy instruction
•Download as PPTX, PDF•
4 likes•776 views
Presentation given at Canadian Health Libraries Association conference in Vancouver, BC. (June 22, 2015)
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Assessment of knowledge and skills in Nursing and Allied Health student information literacy instruction
Similar to Assessment of knowledge and skills in Nursing and Allied Health student information literacy instruction (20)
Recently uploaded
Recently uploaded (20)
Assessment of knowledge and skills in Nursing and Allied Health student information literacy instruction
- 1. ASSESSMENTOFKNOWLEDGEANDSKILLS INNURSINGANDALLIEDHEALTHSTUDENT INFORMATIONLITERACYINSTRUCTION A systematic review of the literature Presented by Pamela Harrison on behalf of Jill Boruff & Pamela Harrison June 22, 2015
- 2. Background • Primary researcher: Jill Boruff • Offering course-integrated instruction in PT/OT since 2008 • Boruff &Thomas, 2011 • Currently, assessment in PT/OT takes the following forms: • Clicker-based assessment • Self-assessment (confidence scale, short answer) • Assignment: include clinical question, search strategy and selected resources @jillyb3 #ILassess
- 3. Whatothersaresaying:Assessment • Oakleaf, 2008: overview of IL assessment methods • Oakleaf, 2009: Information Literacy Instruction Assessment Cycle (English 101) • Schilling & Applegate, 2012: assessment methods used by librarians (multidisciplinary) • Brettle, 2007: systematic review of measures for information skills training (health settings) #ILassess
- 4. Whatothersaresaying:Instruction • Maggio & Kung, 2014: information retrieval instruction (UGME); includes information on librarian involvement in teaching/assessment cycle • Just, 2012: information retrieval instruction and assessment (medical students and residents); data on intervention, assessment tool & target • Koufogiannakis & Wiebe, 2006: systematic review of IL instruction (undergraduates) #ILassess
- 5. Whatwehopetodo • Provide a better understanding of how information literacy instruction is being assessed in nursing and allied health • Provide librarians and other educators with suggestions on how to improve assessment within IL instruction #ILassess
- 6. Researchquestion How is assessment used in IL instruction in Nursing and Allied Health? Forms of assessment Target of assessment Instructional context #ILassess
- 7. • Tests of knowledge or skill • Examples: Multiple choice tests, short-answer tests • Self-assessment or attitudinal surveys • Examples:Quizzes, focus groups • Course assignments or products • Examples: citation analyses, bibliographies • Is the assessment formative or summative? Formsofassessment #ILassess
- 8. Targetofassessment 4 assessment areas identified (Khan & Coomarasamy, 2006; Schilling & Applegate, 2012): Knowledge • Recall and understand concepts taught Skills • Ability to apply knowledge in context Attitudes • Recognize the need to use acquired knowledge and skills in scenarios, and intend to use those skills in practice Behaviour • Use acquired knowledge and skills in practice #ILassess
- 9. Instructionalcontext • Course-integrated, librarian-led workshops • One-shot librarian-led workshops • Instructor-led instruction • Other forms of instruction (Sources: Khan & Coomarasamy, 2006; Maggio & Kung, 2014) #ILassess
- 10. Methodology:Searchdetails • Search: 1990 to January 16, 2015 • Ovid Medline • EMBASE • CINAHL (EBSCO) • ERIC (EBSCO) • LISA • LISTA • ProQuestTheses & Dissertations • Documentation: PRISMA flow chart (Source: Moher, D., Liberati,A.,Tetzlaff, J., Altman, D.G., & PRISMA Group. (2009)) #ILassess
- 11. Searchconcepts Nursing or Allied Health Educational program IL Instruction or Assessment #ILassess
- 12. Methodology:Inclusioncriteria ✔Nursing or Allied Health program ✔Degree-seeking students ✔Assessment of information literacy skills ✔Question formulation ✔Searching/resource selection ✔Critical appraisal (if a librarian is involved) #ILassess
- 13. Exclusioncriteria ✗Exclusively about medical students, medical residents, or non-degree programs ✗Continuing education programs or workplace/practitioner training programs ✗Exclusively about accreditation standards or competencies ✗Exclusively about faculty attitude toward, or use of, resources ✗Commentaries: position or policy statements; editorials; product reviews ✗Lists #ILassess
- 14. PRISMA Records identified through database searching (n = 4366 ) Records after duplicates removed (n = 2965) Records published from 1990 onwards, and screened (n = 2724) Additional records identified through other sources (n = TBD) Records excluded (n = 2143) #ILassess
- 15. PRISMAcontinued… Records screened (n = 2724) Records excluded (n = 2143) (from previous slide) Full-text articles assessed for eligibility (n = 605) Studies included in qualitative synthesis (n= 130) Full-text articles excluded (n = 475) • No librarian and no IL skills (n = 166) • No assessment described (n = 134) • No program described (n = 84) • Reasons from abstract level (n = 34) • Review articles (n = 4) • Article not available (n = 11) • Language (n = 33) • Duplicates (n = 6) • Other (n = 3) #ILassess
- 16. Preliminaryresults:(n =130) #ofincludedpapers,bydiscipline 0 10 20 30 40 50 60 70 80 90 100 PT / OT SLP Nursing Mix Nurs/AH #ILassess
- 17. Preliminaryresults:(n=22) PT/OT/SLPILassessmentsnapshot Assessment # Validated ( ✔ ) Assignment(s) 10 Adapted FresnoTest 3 ✔ Dutch Modified FresnoTest 1 ✔ Evidence-Based Practice Profile Questionnaire 1 ✔ Research Readiness Assessment 1 ✔ Vignette Skill Exercise 1 Survey/test: Self-efficacy 2 Survey: Satisfaction 3 Pre-/Post test: Knowledge 2 Survey/test: Behaviour and Attitudes 4 #ILassess
- 18. Nextsteps • Additional material: • For screening and data extraction: studies cited in review articles • For data extraction only: assessment tools or additional data (will contact authors) • Data extraction for accepted items • Analysis: • What are the most common forms of assessment?Are any of them validated? • Which assessments are being used for which EBP skills (i.e. question formulation or searching)? #ILassess
- 19. Observationsandlessonslearned • Full text screening for qualitative data takes a long time • Multi-purpose terms result in many false drops • Learning assessment vs. usability testing • Variation in depth of descriptions suggests that the literature would benefit from guidelines on reporting assessment methods #ILassess
- 20. Questions? Jill Boruff, Primary Investigator McGill University jill.boruff@mcgill.ca Pamela Harrison, Co-Investigator McGill University pamela.harrison@mcgill.ca @jillyb3 @tweet_nothings_
- 21. References • Boruff, J.T., &Thomas,A. (2011). Integrating evidence-based practice and information literacy skills in teaching physical and occupational therapy students. Health information and libraries journal, 28(4), 264. • Brettle, A. (2007). Evaluating information skills training in health libraries: a systematic review. Health Information & LibrariesJournal, 24(s1), 18. • Just, M. L. (2012). Is literature search training for medical students and residents effective? a literature review. Journal of the Medical Library Association:JMLA, 100(4), 270. • Khan, K. S., & Coomarasamy,A. (2006).A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Medical Education, 6(1), 59. doi:10.1186/1472-6920-6-59 • Koufogiannakis, D,. &Wiebe, N. (2006). Effective methods for teaching information literacy skills to undergraduate students: A systematic review and meta-analysis. Evidence Based Library and Information Practice, (1)3: 3. • Continued…
- 22. References(continued) • Maggio, L. A., & Kung, J.Y. (2014). How are medical students trained to locate biomedical information to practice evidence-based medicine? a review of the 2007–2012 literature. Journal of the Medical Library Association:JMLA, 102(3), 184. • Moher, D., Liberati,A.,Tetzlaff, J., Altman, D. G., & PRISMA Group. (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses:The PRISMA Statement. PLOS Medicine, 6(7), e1000097. • Oakleaf, M. (2008). Dangers and opportunities: a conceptual map of information literacy assessment approaches. portal: Libraries and the Academy, 8(3), 233. • Oakleaf, M. (2009).The information literacy instruction assessment cycle: A guide for increasing student learning and improving librarian instructional skills. Journal of Documentation, (65)4, 539. • Schilling, K., & Applegate, R. (2012). Best methods for evaluating educational impact:A comparison of the efficacy of commonly used measures of library instruction. Journal of the Medical Library Association: JMLA, 100(4), 258.
Editor's Notes
- motivation, discussion of assessment Talk about self-assessment
- The primary researcher for this project is Jill Boruff, who is McGill’s Liaison Librarian for the School of Physical and Occupational Therapy. In 2011, co-authored paper with SPOT faculty member outlining their collaboration in developing and implementing IL instruction for first-year undergrads (Bach. of rehab sciences) Includes description of instruction (lecture and hands-on workshop) and assessment (searching assignment, graded by the librarian) My role: I came in after the methodology had been decided upon, literature scoped, to help with the screening 2008: began teaching course-integrated IL to Physical and Occupational Therapy students 2011: Co-authored a paper outlining teaching; assignment that does not currently feed into assessment but could (Boruff & Thomas, 2011) Before moving on, let’s talk about where this project is: - SR
- Oakleaf – case study Schilling & Applegate – library instruction
- Goal: make an informed decision on what assessment to use Depending on what types of assessment methods we find, we hope to provide librarians and other educators with duggestions on how to improve assessment within IL instruction
- RQ informs our data collection In postsecondary education, how is assessment used in IL instruction in N & AH? What are the different forms of assessment being used? (What tools are we using? Tests? Surveys? Projects?) Does the assessment examine knowledge, skills, or attitudes? In what instructional contexts is it being used? Who is teaching, how long do they have with the students, and where is this happening?
- MC & short answer are easy to mark and administer, Bibliog analysis: often bibliog is for a larger assignment; variation in depth of analysis (resources type & quality); others may be more in depth, using checklist of characteristics that make up quality resources Essay: similar to bibliog analysis, but the essay can in some cases be a report of steps taken to acquire evidence Formative: assessment during instruction, allows for the instructor to respond to needs as they arise Summative: final test or assignment, captures an end-product
- Moving from theoretical to practice-based KSB can be assessed directly (direct/indirect: Schilling & Applegate, 2012) ; more option for miscommunication with attitudinal feedback Longitudinal: look retention by measuring knowledge, skills, attitude and behavioural changes over time (pre/post, end of degree, x years post-degree) (literature by Just, 2012) Behaviors difficult to assess at student level—(Need to differentiate between assessment in clinical practice, and assessment in a school context; the two are not necessarily equivalent) - decision to apply the skills – behaviour - concrete MeSH example Types of tests that go along with each Tests used, if come across papers which come across which tests are better? Which have you used? Which have you learned?
- Narrowed it down b/c of target population Context determines type of training: while formative evaluation gives feedback on students as they progress, limited time and one-shot opportunities for IL instruction favor summative evaluation only Classrooms Online Clinical setting Hybrid
- Talk about methods first: You will notice that this project deviates from traditional SR methods on several points; we have had this discussion at McGill Show of hands for argument re: calling it a systematic review: Search: multidisciplinary, moving across Health, Education & LIS literature PRISMA: Preferred reporting items for systematic reviews and meta-analyses (we will not be using the checklist) List to ensure transparency of reporting: tracking numbers through 4 phases : (1) identification; (2) screening (ti/ab); (3) eligibility (full-text and quality assessment); (4) inclusion
- Search run in Ovid Medline 1946 – , peer reviewed, based on original scoping search AH: OT/PT; AH occupations; nursing Specific tests: - Fresno & Berlin are target tests – look at 1st 4 stages of EBP (Ask, Acquire, Appraise, Apply, (assess)) Fresno test for IL (longer to mark) Berlin questionnaire for critical appraisal (multiple choice) – according to Shaneyfelt T, Baum KD, Bell D, et al. Instruments for Evaluating Education in Evidence-Based Practice: A Systematic Review. JAMA. 2006;296(9):1116-1127. doi:10.1001/jama.296.9.1116. Core search followed by publication date limit (1990 - ) – as you will see in the PRISMA, not much on this topic pre-1990 Interest in what the pre/post EBM distribution of literature would be
- Literature must address a nursing program or allied health program IL skills: Degree-seeking, so not continuing education, certificate or diploma programs Question: clinical/searchable question, PICO, PICOT, PESICO Some assessment of search strategy, database selection, or end-product (i.e. citation analysis, literature review), or self-report Critical appraisal: not always taught by librarian Note – do not limit by study design, and so more fi
- This list was developed iteratively We started with the first 2 criteria The others were formulated as we were comparing the results of our first screen, trying to reach consensus, and placing our trouble items up against the research question Remember Koufogiannakis & Wiebe’s inclusion criteria – LoE to determine effectiveness of an intervention Preliminary scope: no articles with compared assessment methodologies Limited validated methodologies Prevalence of assignments: while marking rubrics can be validated, the assignments are not
- Takeaway point: we come up against very high numbers here. Pros of keeping everything: systematic search spanning 2 bodies of literature Trace development and conversation around assessment since beginning of EBM My feeling is you need to know when to change course: We did not include validation studies in our study, but came up against a few; after the preliminary scope, it might have been good to take these few validation studies, and work outward, exploring the literature around them Need to be comprehensive Records published 1990 onwards, duplicates removed, look at brackets
- The full text screen really slowed us down—getting the full text and then screening 605 articles takes time! Why not screen for quality? Low number of validated studies (will see this) Prevalence of pre-experimental design (no control group) or quasi-experimental design (where group membership is not randomized, so allows for other factors to influence outcome; e.g. computer-based assessment vs. paper – students volunteering for computer-based group may be more comfortable with computers, skew results) abstract level records excluded N = 22 Put twitter feed Switch to corbel Ku, Y., Sheu, S., & Kuo, S.-M. (2007). Efficacy of integrating information literacy education into a women's health course on information literacy for RN-BSN students. Journal of Nursing Research, 15(1), 67-77. Kim, S., Brown, C. E., Fields, W., & Stichler, J. F. (2009). Evidence-based practice-focused interactive teaching strategy: a controlled study. Journal of Advanced Nursing, 65(6), 1218-1227. Liou, S., Cheng, C.-Y., Tsai, H.-M., & Chang, C.-H. (2013). Innovative strategies for teaching nursing research in Taiwan. Nursing Research, 62(5), 335-343. doi: No librarian and no IL skills--no librarian mentioned, no IL mentioned at all. Librarian but no assessment described--Librarian involved in IL instruction, but no assessment described (changed from mentioned) No assessment described--IL instruction happening, no librarian mentioned, no assessment described No program described--might mention IL skills, assessment, competencies, but not related to an educational program. Language other than English and French – included in search to check whether there were 1 or 2 languages holding a lot of relevant articles – would try to get funding for translation if this were possible Duplicates: including Thesis -> Article The "other category" has one for date, one for a strange interview/assessment hybrid, and one for an assessment that is more fully described in another article (this is only if someone asks!)
- Physical and Occupational Therapy: 18 Speech Language Pathology: 4 Nursing: 90 Mixed Allied Health: 18 (This could mean papers that mix nursing/rehab or nursing/midwives or pharmacy etc.)
- Of the 22 PT/OT/SLP articles, here is the rough break down of types of assessments (the number will be more than 22 because some articles talk about more than one assessment) Vignette skill - (This is from the Thomas paper--they are given a search vignette and have to go through the EBP cycle, but it wasn't an assignment)
- Data extraction: Note: quality assessment not as rigorous as in a traditional systematic review, because rigorous methodology is harder to find in IL instruction literature In health, can use study design; prevalent study design here is pre-experimental or quasi-experimental Lib lit will not be possible to filter for experimental studies - If you want to do this yourself, scope the lit, think about the numbers If want to limit to validated study, not as easy in lib lit Can’t really use a screening tool that looks at study design because so much LIS literature uses a pre-experimental study design Analysis—answering these questions in addition to the questions posed earlier in the presentation.
- Multi-purpose term e.g. informatics, used for: ehealth: use of information and communication technology in healthcare (e.g. discussions of learning how to work with electronic patient records) MHealth: mobile communications-based health care (e.g. use of PDAs – personal digital assistants – to access resources e.g. installed reference materials) – when the use of PDAs was the focus, we didn’t include the item Problematic term: health literacy - including the ability to support patients in information-seeking; increasing cultural sensitivity through research (nothing to do with evidence-based practice) Another sticking point is that our search picked up literature on assessing video instruction; this fell into 2 camps – if about IL assessment, kept (1); if about usability, rejected Many papers use online instruction/embedded video as a means of instruction, and then performed assessment; 1 paper measured skills following the instruction(included); the rest focused on UX – cut