Assessing the impact of evidence summaries in library and information studies: A mixed methods approach

Lorie Kloda, MLIS, PhD(c), AHIP, McGill UniversityDenise Koufogiannakis, MA, MLIS, PhD(c), University of AlbertaAlison Brettle, MLIS, PhD, University of SalfordMedical Library Association Annual Meeting, May 2012, Seattle

Gap between research and practice in library andinformation studies (LIS)(Booth, 2003; Crowley, 2005; Genoni, Haddow, & Ritchie, 2004; Turner, 2002)Only method likely to improve communication is“inclusion of research reports in (…) publicationsfrequently read by practitioners.”(Haddow&Klobas, 2004)Evidence Based Library and Information Practicejournal, 2006->200 evidence summaries

Structured abstract objective – design – setting – subjects – method – main results – conclusionCommentary 300-400 words appraisal of validity, reliability, applicability significance, implications for practice 3

To investigate the impact of evidencesummaries on library and informationprofessionals and their practiceKnowledgePracticeUsers

Phase 1 Development and face-validation of toolPhase 2 Survey questionnaire to readers (QUANT)Phase 3 Interviews (QUAL)

Development of a tool to assess impact

Findings (Phase 1)Development ofImpactAssessmentMethodGrad, R., Pluye, P., &Beauchamp, M.-E.(2007). Validation ofa method to assessthe clinical impact ofelectronicknowledgeresources, e-ServiceJournal, 5(2), 113-135. http://iam2009.pbworks.com/

Survey questionnaire to readers of evidence summaries

Findings (Phase 2)Survey 175 153 emailed 101Respondents recruited IAM survey respondents 1 unusable 15 3 bounced email incomplete 86 21 duplicates 49 no reply completed IAM survey 153 remaining 101 total respondents n=62

Findings (Phase 2) USACountry Canada UK Australia USA Ireland Australia Finland Malaysia UK Saudi Arabia Iran Puerto Rico Spain Canada Brazil Hong Kong

Findings (Phase 2) 50+ 25+Number of 15Evidence 14 13Summaries 12 11Read in Past 10 9Year 8 7 6 5 4 3 2 1 0 0 2 4 6 8 10 No. of respondents

 Decline in Reference Transactions with Few Questions Referred to Librarian when the Reference Desk is Staffed by a Paraprofessional (8) The Presence of Web 2.0 Applications Is Associated with the Overall Service Quality of Library Websites (6) Google Scholar Out-Performs Many Subscription Databases when Keyword Searching (4) Statistical Measures Alone Cannot Determine Which Database (BNI, CINAHL, MEDLINE, or EMBASE) Is the Most Useful for Searching Undergraduate Nursing Topic (4) A Graduate Degree in Library or Information Science Is Required, but not Sufficient, to Enter the Profession (3)

Findings (Phase 2)Reason for Freq. %Reading For general interest or curiosity 15 24%EvidenceSummary For personal continuing 18 29% professional educationn=62 To answer a specific question 21 34% or address a specific issue in my practice Other 8 13%

Findings (Phase 2)When the > 1 yearEvidence 6-12 monthsSummary was 3-6 monthsRead 1-3 months 1 week-1 month 1-7 days Today 0 5 10 15 20 25

Findings (Phase 2) Freq. %Cognitive My practice was (will be) improved 11 13%Impact I learned something new 36 42% I recalled something I already knew 14 16% It prompted me to investigate more 23 27% It confirmed I did (I am doing) the right thing 17 20% I was reassured 13 15% I was dissatisfied: There is a problem with the 1 1% presentation of this evidence summary I was dissatisfied: I disagree with the content 0 0% of this evidence summary It is potentially harmful 0 0% Other 9 10%

Findings (Phase 2_Cognitive freq. %Impact: Too much information 0 0%PresentationProblem Not enough information 1 1%“You reported: I was Information is poorly written 0 0%dissatisfied; There wasa problem with thepresentation of this Information is too technical 0 0%Evidence Summary.Which of the following Other 0 0%problems did youencounter?”

Findings (Phase 2) Freq. %Cognitive The structured abstract did not 0 0%Impact: adequately explain the original studyDisagree with The writer of the evidence summary 0 0%Content presented incorrect information“You reported: I was The commentary was overly negative 0 0%dissatisfied; I disagreewith the content of this The commentary was not critical 0 0%Evidence enoughSummary. Which of the The writer of the evidence summary did 0 0%following content not place this study in contextelements did youdisagree with?” Other 0 0%

Findings (Phase 2)Practice Freq. %Impact Change my service approach 5 6%“You reported: Change my approach to collections 1 1%My practice was(will be) Change my management approach 4 5%improved. What Change my approach to teaching 4 5%did you (will you)do differently Change my professional approach 4 5%after reading theEvidence Other 3 3%Summary?”

Findings (Phase 2)Community  NoneImpact  Hypothesized future/potential“If reading thisEvidence Summary impacts on usersresulted in some  Reinforced cognitive or practicechange to yourindividual practice, doyou think it led to an impacts, not user outcomesimpact on anyone  5 reported actual impact at thiswithin the communityyou serve or level:environment inwhich you work?  Change in teaching LIS studentsPlease explain in thecomment box.”  Observed (anecdotal) changes

Interviews with subsample of survey respondents

Findings (Phase 3)13 participants Subsample from survey respondent Various settings, education levels, countries, and professional positions

Proposed Impacts Suggested Confirmed Phase 1 Phase 2 Phase 3C1 Practice was improved ? ✗C2 Learned something new ✔ ✔C3 Recalled something ✔ ✓C4 Prompted to investigate ✔ ✔C5 Confirmed ✔ ✓C6 Reassured ✔ ?C7 Dissatisfied – presentation ? 0C8 Dissatisfied - content 0 0C9 Potentially harmful 0 0P1 Service ✔ ✔P2 Collections ✔ ✓P3 Management ✔ ✓P4 Teaching ✔ ✓P5 Professional practice ✔ ✓U User Community ? ?

Findings (Phase 3)Potential Discovery  New researchImpacts  Interesting topics  MethodsUncovered  Keeping current Sharing  With colleagues, managers  Report writing  Recommended reading Assistance  Research  Writing, presentations  Teaching (for professors)

Findings (Phase 3)“Likes” Format of evidence summaries  Descriptive  Accuracy of abstract  Concise Time saving Database of Evidence Summaries

 One evidence summary assessed per respondent Cognitive impact comparable to findings in Grad, Pluye, et al. (2006) Practice impact – two-tiered Low community impact

 Tool validation Cognitive impact Practice impact  Individual practice  Workplace practice Difficult to assess impact on community/users

Revise Impact Assessment Tool  Include new or revised impact items  Improve wording of itemsConduct survey of a larger sample of evidencesummary readersAdapt tool for other forms of researchdissemination

Canadian Association of Research LibrariesResearch in Librarianship GrantRoland Grad and Pierre Pluye, McGill University,for their feedbackAll of our survey respondents and interviewparticipants

lorie.kloda@mcgill.ca@loriekloda@eblipslideshare.net/lkloda

Assessing the impact of evidence summaries in library and information studies: A mixed methods approach

by Lorie Kloda on Jun 07, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Assessing the impact of evidence summaries in library and information studies: A mixed methods approach Presentation Transcript