Getting Evidence into Practice


Published on

Quebec Library Association Annual Meeting, Montreal, May 9, 2013
Closing Keynote

Published in: Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • I have much more experience in EBLIP, but am new to assessment. These are my preliminary thoughts on the two, and how they might overlap, or fit together, or contradict one another.Trying to make sense of of EBLIP in my job as Assessment Librarian. Reserve the right to change my mind on all of this!Make sure to stop and give an outline of the talk:My understanding of assessment work and Why I should assess.An overview of the EBLIP movement and how it benefits librarianshipHow I have come to think about these two things: What does the EBL movement/framework have to offer assessment? And vice versaSome examples of integrating eblip (research evidence) into practiceSome examples of doing assessment (using local evidence) in practice
  • Pam Ryan’s article on the two solitudes: “Beginning in 2005, individuals from one group have been showing up at the others’ conferences and events to discuss their methods, frameworks and processes. Are these separate movements within librarianship forming theoretical bridges? Is some sort of merger, fusion or takeover in the future?” 7 years later, I’m going to try to answer this question.
  • Question: How many of you do assessment as part of your work, formally or informally?]
  • in the Association of Research Libraries’ SPEC Kit 303: Library Assessment, Stephanie Wright and Lynda White state that “to assess, in general, is to determine the importance, size, or value of; to evaluate. Library staff assess operations by collecting, interpreting, and using data to make decisions and to improve customer service. They study internal processes, levels and quality of service, and library impact on institutional goals.”
  • The definition of academic library assessment by Martha Kyrillidou and Pam Ryan and on the ARL supported library assessment blog.
  • The goal of assessment is to inform decision-making for making changes/improvements (i.e., quality improvement – meeting the needs of users OR to demonstrate value/impact. “In assessment work, evidence can only be local. The nature of assessment work is that it circles around what users are experiencing, so the appropriate study design is frequently a qualitative, user‐engaging method.” Pam Ryan
  • But what about what you can’t (or don’t) assess? Obstacles to assessment: time, data, resources (funding, support), experience, skills (from Kouf & Crumley, 2006)
  •  So what is evidence?In the cases where one is not doing assessment (for whatever reason), one can use evidence: credible, transferable findings from research to inform DM.
  • Questions:A) How many of you are familiar with the term “EBLIP…”? Have heard it, know what it means
  • Purpose of EBL: to create a framework or approach in which decision making can take place.
  • 5 steps of EBLIP (the acronym for evidence-based library and information practice)Explain the steps
  • But what does it mean to use research evidence?
  • Changed the format to a linear process, just for illustrative purposes.
  • I find that the steps have overlaps with other frameworks or processes that librarians are familiar with:
  • IL standards – these are from ACRLThe Standards have a logical hierarchyEach standard is divided into several performance indicators.Standard OneThe information literate student determines the nature and extent of the information needed.(go on to explain the first 4 standards)
  • Talk about assessment steps (research steps)I would think that EBL should be second nature to librarians already, it is similar to the IL process, which we advocate. Similarly, assessment is research, typically on a local level. Steps in assessment/research process are extensions of EBL/IL steps from a knowledge creator’s (rather than knowledge user’s) perspective.
  • The eblip framework says that we should acquire published findings, and assessment research instructs us to gather local data to create findings for our setting.EBL is knowledge use, Assessment is knowledge creation (for use). Each is informed by the other. 2 pieces of a puzzle?Ideal is both together – “published” evidence that is credible and transferable + local evidence (data + interpretation = findings) = recommendations for action/changes or APPLICATION
  • Give brief overview of datesFrom Pam Ryan: “The EBL and library assessment movements grew up separately but at approximately the same time, with EBL first appearing in the literature in 1995 in an editorial by Margaret Haines and the initial calls for a “culture of assessment” in academic libraries appeared in articles by Stoffle in 1996 and Lakos in 1999.”Also mention timing of LibQUAL+ development by Cook et al!
  • Focus over last 15 years has been on scientific research, and coming out of the sciences/health model, emphasis was placed on study designs from clinical epidemiologist’s point of view – positivistic and therefore quantitative designs which favours controlled studies. These study designs are useful for assessing the effectiveness or success of specific interventions, but much research in LIS is not about that.Current state of EBL framework/movementDenise’s article on practice-based evidence, and Given’s article on qual research emphasize that not only different types of research, but not only “research” should be considered evidence for EBL. This evolution of the definition of EBL creates space for the type of evidence traditionally used for assessment – whether it be “small scale” evaluation or larger scale assessment of an entire department or library. EBL, then, includes assessment
  • From Denise’s 2010 paper on practice-based evidence:Getting back to the definition of evidence:While research evidence is of high importance to our profession and knowledge, LIS practitioners need to first of all consider local evidence. Local evidence is found in our working environment and is specific to the context in which we carry out our work. It includes such things as our experience with patrons in particular contexts, and what we observe to work in our interactions, assessment of programs, feedback from our users, project evaluations, and accumulated experiences over the course of careers. These things are not easily shared and often do not find a place in publications because they are too local. But data that comes from a local context is in fact often the most important evidence source that a LIS professional can consult because it gives us information that is directly applicable to, and about our users. For example, usage stats on e-journals, feedback and comments about our services, usability testing on our website, titles from our interlibrary loan requests; these are just a few examples of local evidence that is invaluable to our decision making. This local data doesn‟t often mean much to others outside of our organization, but it is of utmost importance to our situated knowledge. The trick is to figure out what local information to systematically collect, and how to use it.
  • Again from Denise:“They are all forms of evidence that come much more naturally to most LIS practitioners than research evidence.”
  • Bringing evidence source together
  • Includes needs assessment,which determines they what (and how much/relative importance) of the needs of users – and different user groups. Librarians are skilled at one type of user needs assessment – the reference interview. The reference interaction can be considered a micro-example of assessment: assessing needs, providing services/collections, assessing that needs have been met and the user is satisfied/more than satisfied. Librarian gains experience and knowledge which then feeds into the next transaction, and the cycle continues.[Follow up question: Does anyone want to revise their answer on whether or not they do assessment in their job?] 
  • Seating sweeps method – part of spacial analysis methods
  • Readscale.orgThere is also the LEAD scale for library effort assessment data. It can be used for any library function, not just reference.
  • In an article published in 2009, Megan Oakleaf describes the seven stages of the In- formation Literacy Instruction Assessment Cycle (ILIAC) and uses an assignment as an example to show how ILIAC provides a process for improving librarian teaching skills and student information literacy skills. The seven stages are: reviewing learning goals, identifying learning outcomes, creating learning activities, enacting learning activities, gathering data to check learning, interpreting data, and enacting decisions.
  • Getting evidence into practiceGetting practice into ebl – Denise’s workGetting assessment into ebl –Pam Ryan, and now me.
  • A revised process:Articulate – come to an understanding of the problem and articulate it.Assemble – assemble evidence from multiple sources that are most appropriate to the problem at hand.Assess(Appraise)– place the evidence against all components of the wider overarching problem. Assess the evidence for its quantity and quality.Agree – determine the best way forward and if working with a group, try to achieve consensus based on the evidence and organisational goals.Adapt –revisit goals and needs. Reflect on the success of the implementation. This cycle works for assessment as well.
  • But instead of thinking of it as two types of work:One is assessment and brings local evidence (“Creation”)And one is EBLIP and makes use of published evidence (“use”)…..
  • Both these types of evidence, and really, a broader definition of evidence to include professional knowledge
  • When I read this definition, I think it means that we need research, analysis of local data, and facts/planning which comes from professional knowledge.So a culture of assessment is one that Is evidence-based.
  • Getting Evidence into Practice

    1. 1. Getting Evidence into PracticeLorie KlodaMcGill UniversityQuebec Library AssociationMay 9, 2013
    2. 2. 1Assessment
    3. 3. Two Solitudes?Are these separate movementswithin librarianship formingtheoretical bridges? Is some sort ofmerger, fusion or takeoverin the future?Ryan, P. (2006). EBL and library assessment: Twosolitudes? Evidence Based Library and InformationPractice, 1(4), 77–80.“”2
    4. 4. What is assessment?
    5. 5. to assess, in general, is to determine theimportance, size, or value of; to evaluate. Librarystaff assess operations bycollecting, interpreting, and using data to makedecisions and to improve customer service. Theystudy internal processes, levels and quality ofservice, and library impact on institutional goals.Wright, S., & White, L. (2007). Library Assessment, SPEC Kit303 (Washington, DC: Association of ResearchLibraries, Office of Leadership and Management Services).“”4
    6. 6. includes any activities that seek to measure thelibrary’s impact on teaching, learning andresearch as well as initiatives that seek toidentify user needs or gauge user satisfaction orperceptions with the overall goal being thedata‐based and user‐centred continuousimprovement of our collections and services.Ryan, P. (2006). EBL and library assessment: Twosolitudes? Evidence Based Library and InformationPractice, 1(4), 77–80.“”5
    7. 7. Goals of Assessment6
    8. 8. 7
    9. 9. 8
    10. 10. What isevidence-based libraryand information practice?
    11. 11. an approach to information science thatpromotes the collection, interpretation andintegration of valid, important and applicableuser-reported, librarian observed, and research-derived evidence. The best availableevidence, moderated by user needs andpreferences, is applied to improve the quality ofprofessional judgements.Booth, A. (2000). Librarian heal thyself: Evidence basedlibrarianship, useful, practical, desirable? Proceedings from the 8thInternational Congress on Medical Librarianship. London, UK, July 2-5, 2000.“”10
    12. 12. EBLIP ProcessFormulatingtheQuestionSearchingtheEvidenceAppraisingtheEvidenceApplyingthe ResultsEvaluatingthe Process11
    13. 13. Getting researchevidence into practice
    14. 14. http://lis-systematic-reviews.wikispaces.com
    15. 15. 14
    16. 16. Evidence Summaries Based Library and Information Practicejournal, 2006->250 evidence summaries
    17. 17. 17
    18. 18. 18
    19. 19. 19
    20. 20. 20
    21. 21. 21
    22. 22. EBLIPFormulateSearchAppraiseApplyEvaluate22
    23. 23. 23
    24. 24. EBLIPFormulateSearchAppraiseApplyEvaluate24
    25. 25. EBLIPFormulateSearchAppraiseApplyEvaluateInformationLiteracyKnowAccessEvaluateUseEthical/Legal25American LibraryAssociation. (2006). Thestandards: Step-by-stepURL:
    26. 26. EBLIPFormulateSearchAppraiseApplyEvaluateInformationLiteracyKnowAccessEvaluateUseEthical/LegalAssessmentQuestionGatherAnalyzeConclude/DecideDiscuss/Apply26
    27. 27. 27PublishedevidenceLocalevidence
    28. 28. Evolution of EBLIP Movement1992 EBM coined at McMaster1994 HainesHILJ editorial1997 EldredgeHypothesis (MLA) article2000 Eldredge EBL framework series2001 First EBL (now EBLIP) conference2004 Booth& Brice book2006 EBLIP journal launches2013 Marshall MLA keynote28
    29. 29. 29
    30. 30. Local EvidenceUser feedbackLibrarian observationDiscussion/interactions with colleaguesUsage dataOrganizational realities30
    31. 31. Professional KnowledgeFormal and informal learningOn the job trainingTacit knowledgeReflective knowledge31
    32. 32. 32Koufogiannakis, D. (2010). Considering the place of practice-basedevidence within evidence based library and information practice(EBLIP). Library and Information Research, 35(111), 41–58.
    33. 33. How does evidence influencelibrarianship and librarians?
    34. 34. Impact of Published Evidence34Kloda, L., Koufogiannakis, D., &Brettle, A. Assessing the impact of evidencesummaries in library and information studies: A mixed methods approach.Speaker. Health Libraries Group Conference, Glasgow, Scotland, July 11-12, 2012.
    35. 35. Getting assessment evidenceinto practice
    36. 36. 36
    37. 37. 37
    38. 38. 38
    39. 39. 39
    40. 40. 40Gerlich, B. K., &Berard, G. L. (2010). Testing the viability ofthe READ scale (Reference Effort Assessment Data):Qualitative statistics for academic reference services.College and Research Libraries, 71(2), 116–137.
    41. 41. Information Literacy Instruction Assessment Cycle41Oakleaf, M. (2009). The Information Literacy Instruction AssessmentCycle: A guide for increasing student learning and improving librarianinstructional skills,” Journal of Documentation, 65(4) 539–560.
    42. 42. How does EBLIP fit intoassessment?Or is it the other way around?
    43. 43. Two Solitudes?Are these separate movementswithin librarianship formingtheoretical bridges? Is some sort ofmerger, fusion or takeoverin the future?Ryan, P. (2006). EBL and library assessment: Twosolitudes? Evidence Based Library and InformationPractice, 1(4), 77–80.“”43
    44. 44. Widening the EBLIP ModelArticulatetheproblemAssembleevidenceAppraisefor quality& quantityAgree onthe wayforwardAdapt44
    45. 45. 45PublishedevidenceLocalevidence
    46. 46. 46PublishedevidenceLocalevidenceArticulateAssembleevidenceAppraiseAgreeAdapt
    47. 47. 47ArticulateAssembleevidenceAppraiseAgreeAdapt
    48. 48. A culture of assessment is an organizationalenvironment in which decisions are based onfacts, research, and analysis, and where servicesare planned and delivered in ways thatmaximize positive outcomes and impacts forcustomers and stakeholders.Lakos, A., Phipps, S., & Wilson, B. Defining a ‘Culture ofAssessment.’ (1998–2002), Assessdef3-new.doc“”48
    49. 49. Questions1249
    50. 50. QuestionsHow can we foster EBLIP in our work or inour workplace?250
    51. 51. QuestionsHow can we foster EBLIP in our work or inour workplace?What are some ways to overcomeobstacles to to using evidence and doingassessment?51
    52. 52.