Successfully reported this slideshow.
Your SlideShare is downloading. ×

Library Evaluation in 3 Parts - Presented by Dr. Bill Irwin, Computers in Libraries, 2015


Check these out next

1 of 41 Ad

More Related Content

Slideshows for you (20)

Viewers also liked (10)


Similar to Library Evaluation in 3 Parts - Presented by Dr. Bill Irwin, Computers in Libraries, 2015 (20)

More from Hamilton Public Library (20)


Recently uploaded (20)

Library Evaluation in 3 Parts - Presented by Dr. Bill Irwin, Computers in Libraries, 2015

  1. 1. Ontario Library Association Super Conference, January 29 Dr. Bill Irwin, Huron University College Current research in library evaluation: A review in 3 parts
  2. 2. Today’s Presentation (in 3 parts) Current Research in Library Evaluation2 1. 2014 research on the introduction of new evaluation practice in public libraries and the impact of organizational culture Irwin. B. & St. Pierre, P. (2014) Creating a culture of meaningful evaluation in public libraries: Moving beyond qualitative metrics, Sage On-Line: 2. Early findings from 2014 survey to Ontario Municipal councillors on their expectations and considerations on the state of public library evaluation and reporting: “Making informed public decisions for public libraries” Irwin. B. & St. Pierre, P. In design and delivery partnership with Counting Opinions 3. Introduction of a future research initiative: Building an L-Value Index for Canadian Public Libraries: Cavanagh, M., Silk, K., & Irwin, B.
  3. 3. Project’s Goal Current Research in Library Evaluation3 Outcome models assist policy makers, both inside and outside of organizations, to address fundamental questions of resource allocation, delivery methods, and agency design and purpose.
  4. 4. Project’s Goal Current Research in Library Evaluation4 The key goal of our research is to examine existing evaluation systems of public libraries, in an effort to replace the current reliance on a predominantly output based model of performance measure, with new models of evaluation based on capturing outcomes.
  5. 5. Defining Outcome Evaluation Current Research in Library Evaluation5
  6. 6. Project Objectives Current Research in Library Evaluation6  exploring the limitations and challenges inherent in the current qualitative evaluation system  creating a better understanding of the cultural impacts and policy implications of a sociological performance framework, on library organizations, their constituencies, and their stakeholders in the establishment  capture changes in the Practice-Program-Policy continuum and the downstream implications for how libraries are viewed and valued upon the introduction of a new system of performance measurement
  7. 7. OPLA Child and Youth Services Committee Teen Services Benchmarks and Statistical Report 2013 Current Research in Library Evaluation7 Does your library measure outcomes or impacts of teen programming? % Yes 42.9% All 46.7% 100,000+ 38.1% 50,000<100,000 40.0% 15,000<50,000 46.4% 5,000<15,000 75.0% <5,000 42.9% Less than 50% of the libraries reported measuring outcomes or impacts of teen programming, although several libraries did note that they are planning to implement outcome- based measurement in the future.
  8. 8. Q: Are you satisfied with the methods used by your library to evaluate teen programs? Current Research in Library Evaluation8 • It would be helpful to have a clearer idea of what a youth program should be i.e. what is the purpose, what is the desire outcome. • It is not consistent • We do not evaluate anything past attendance. • I would like to find out what the participant got out of program • Difficult to measure anecdotal reports • Evaluation is based only on circulation and attendance performance indicators
  9. 9. Q: Are outcomes/impacts measured in other parts of your library? Current Research in Library Evaluation9 • Lack of staff capacity • Do not have anyone trained in outcome/impact measurement • Probably the best answer for why it hasn't happened yet is "too time consuming." • Lack of time. • Not currently considered a priority by our governing body • Assessing outcomes falls short of our priority list. • No formal criteria which is implemented system-wide
  10. 10. Q: You indicated that you do not assess outcomes/impacts of teen programs. Why not? Check all that apply. Current Research in Library Evaluation10
  11. 11. Q: Do the agencies that fund your teen programs require that you report outcome/impact measures? Current Research in Library Evaluation11
  12. 12. Organizational Culture and Impact Current Research in Library Evaluation12 Sub-cultures are currently the greatest inculcator of values (Schein, 1992) regarding: 1. the worth and purpose of evaluations, 2. postulation of negative influence, 3. evaluation seen as “busy work”
  13. 13. Sub-Culture as Inhibitor Current Research in Library Evaluation13 Common Responses from Follow- up Interviews  Seen as “busy-work”  Rely on informal feedback  Some staff value the process, some question it, seen as additional work  Against staff comfort level  Comments are filtered –each manager chooses  Staff wouldn’t want it (new system) to be more onerous on them  Staff question why, “Are you evaluating me?”
  14. 14. Inhibitors Current Research in Library Evaluation14  Lack of education, not able to produce meaningful results and change  Lack of inclusion: “Big picture” relevance  Required training in best practices  Librarians feel skill set is inadequate to the task, so they are reluctant to engage – (Similar circumstances to the introduction of technology in mid to late 1990s in public libraries (Author’s observation))  Technological impasse  Consider the law of unintended consequences (how can/does it apply here?)
  15. 15. Sustainable change requires cultural focusing (1) Current Research in Library Evaluation15  "Cultural change is one of the most important factors to consider but one of the hardest to implement" (Preston, 2004).  “In general, we found a number of library staff skeptical of quantitative or qualitative data from customers, preferring instead to rely on their own assumptions and past practices to make decisions.” (Hiller et al, 2008)  “Due to the ‘everyone does the same thing’ culture and operational model it was impossible to make a change to workflow in one area without it directly impacting other areas - therefore systemic change was necessary.” (Nussbaumer & Merkley, 2012)
  16. 16. Sustainable change requires cultural focusing (2) Current Research in Library Evaluation16
  17. 17. Integration of Culture & Evaluation Model (1) Current Research in Library Evaluation17  The following model was developed to demonstrate the steps an organization goes through in regards to moving from an input based to an outcome based evaluation system.  It highlights the operational and cultural transformation process.  This model can be used to determine where an organization is situated (at what stage) in terms of its use of outcome evaluation.  The model is designed to serve as a “roadmap”, to assist organizations in their successful movement through a series of steps, by infusing a culture of outcome evaluation.  Once an organization has achieved stage 4, evaluation then serves as a tool for: realizing organizational priorities, educating stakeholders and funders of the range of quantitative and qualitative program and service impacts,
  18. 18. Integration of Culture & Evaluation Model (2) 18 Dimension Stage 0 Stage 1 Stage 2 Stage 3 Stage 4 Purpose Complacency Justification Self Awareness Alignment Actualization Motivation Inertia Fear & Survival Toward sense of self-efficacy Enlightenment Success, Internalization Organization al Impact Tactical / Strategic Resistance Evaluation is ‘busy work’ Tactical: Short-term Strategic: Long-term Buy-in Shift from systems to individual to patron/staff focus Habituation Inculcation Organization al Impact Internal / External Closed system Internal (staff) Toward an open system Broad stakeholder inclusion Infusion (beware of gaming) Inhibitors & Enablers Lack of education, lack of meaningful results, inertia Lack of personal involvement. Buy-in not enough Cautious engagement Enthusiasm; Coordinating many voices Unintended consequences (positive or negative) Implications Diminishing budgets, failure Awakening Organizational learning Shared leadership Assessment is continual, naturally occurring Creating an outcome evaluation culture Current Research in Library Evaluation
  19. 19. Integration of Culture & Evaluation Model (3) Current Research in Library Evaluation19  Kotter (2008) argues that getting buy-in is not enough because it only engages the head, not the heart.  Moving through the stages is akin to Kotter’s step 6, creating short term wins, building staff confidence, an incremental approach  Somewhere in the transition between step 1 to 2 need to create and articulate a vision/mission statement. Why do we evaluate? What is it all about?  Building cultural assessment seen as a strategic priority – otherwise over shadowed by more “important work”  “When faculty and staff perceive that the administrators are only motivated to create a culture of assessment for accreditation, cynicism and low-motivation will likely result” (Lakos and Phipps, 2004).
  20. 20. Next Steps Current Research in Library Evaluation20  Formalize the Integration of Culture & Evaluation Model  Create an organizational evaluation audit to:  Assess the stage of the model an organization is at  Develop strategies to assist moving it forward  Create a sustainability approach  Design and deliver the next phase of the research
  21. 21. Next Steps Q: Do the agencies that fund your teen programs require that you report outcome/impact measures? Current Research in Library Evaluation21 Ontario-wide survey to elected municipal officials focused on understanding what information they consider when in their deliberations on financial support to libraries’ budgets. In part this research will provide insight for libraries in how to structure, and communication the results of, their institutional evaluations.
  22. 22. 49 45 32 10 15 13 1 1 2,501 to 5,000 5,001 to 15,000 15,001 to 30,000 30,001 to 50,000 50,001 to 100,000 100,001 to 250,000 Over 250,000 NR Number of Responses Responses by Community Population 6.9% 27.3% 23.4% 5.7% 4.5% 6.2% 0.4% 2,501 to 5,000 5,001 to 15,000 15,001 to 30,000 30,001 to 50,000 50,001 to 100,000 100,001 to 250,000 Over 250,000 % Response by Population Response Rate Total responses: n=166, 8.4%
  23. 23. Building an L-Value Index for Canadian Public Libraries Current Research in Library Evaluation36 This project will develop a new, theoretically sound and comprehensive framework for library evaluation. The project will be approached via a three-stage strategy: 1.) Identify the limitations inherent in existing library evaluation systems (the present situation); 2.) Develop a multi-faceted theoretical framework for evaluation - the "L-Index" -- which analyses outcomes and impact along four dimensions: economic, social, cultural, and lifelong learning. These constitute both qualitative and quantitative outcomes (an analytical tool); 3.) Assist libraries in deploying the L-Index, through training workshops, conference presentation, and publications, thereby helping them strategically focus their mission and vision, resources, operations,and service cultures (facilitate change).
  24. 24. Building an L-Value Index for Canadian Public Libraries Current Research in Library Evaluation37  A key goal of this research is to generate an inclusive, reliable, outcome-based performance value framework (PVF) that communities can adapt to local conditions, and that can also be used at the sector level.  The L-Index will include performance measures that facilitate the creation of benchmarks and quality indicators.
  25. 25. Building an L-Value Index for Canadian Public Libraries Current Research in Library Evaluation38 This research proposes to investigate the potential for establishing model evaluation systems for public libraries that are based on these sociological frames of reference; a model which will examine long term outcomes and impacts (sometimes referred to as “quality of life issues”) for public libraries – in a similar fashion that qualitative measures have been developed and implemented for public health or recreation programs and services.
  26. 26. Building an L-Value Index for Canadian Public Libraries – Research Methods Current Research in Library Evaluation39 One: Identify outcome/impact measures in use in libraries, other industries and organisational types, through a thorough review of the organizational assessment, evaluation, and public administration literature. Of particular interest are qualitative measures in use, such as "social return on investment.“ Two: Conduct a national survey of public libraries. A survey will include identification of current evaluation methodologies, their particular contexts, and the utility (real or perceived) of their results, including both advantages and limitations.
  27. 27. Building an L-Value Index for Canadian Public Libraries – Research Methods Current Research in Library Evaluation40 Three: Using results from the first two project phases, we will develop a provisional L-Index, a multi-dimensional framework that will measure the economic, social, cultural, and educational impact a library has on its community. Four: Test the validity, reliability, and utility of the L- index with partners in Alberta and Ontario, including First Nations communities, small, medium, and large urban and rural systems. Five: Promote the L-index via a toolkit to be made freely available nationally for adoption by Canadian libraries.
  28. 28. Building an L-Value Index for Canadian Public Libraries – Partners Current Research in Library Evaluation41  Administrators of Medium Sized Public Libraries of Ontario  Edmonton Public Library  Calgary Public Library  Calgary Public Library Foundation  The Federation of Ontario Public Libraries  Library Association of Alberta  Ontario Library Association  The Ontario Public Library Association