Defining New Metrics For Libraries Success
iSchool, University ofToronto Symposium
April 23, 2014
Dr. Bill Irwin, Huron Un...
Project’s Goal
Organizational Culture & Evaluation2
The key goal of our research is to examine existing
evaluation systems...
Project’s Goal
Organizational Culture & Evaluation3
Outcome models assist policy makers, both inside and
outside of organi...
Defining Outcome Evaluation
Organizational Culture & Evaluation4
Project Objectives
Organizational Culture & Evaluation5
 exploring the limitations and challenges inherent in the
current...
OPLA Child and Youth Services Committee Teen
Services Benchmarks and Statistical Report 2013
Organizational Culture & Eval...
Q: Are you satisfied with the methods used by your library
to evaluate teen programs?
Organizational Culture & Evaluation7...
Q: Are outcomes/impacts measured in other parts of your
library?
Organizational Culture & Evaluation8
• Lack of staff capa...
Q: You indicated that you do not assess outcomes/impacts of
teen programs. Why not? Check all that apply.
Organizational C...
Q: Do the agencies that fund your teen programs require
that you report outcome/impact measures?
Organizational Culture & ...
Organizational Culture and Impact
Organizational Culture & Evaluation11
Sub-cultures are currently the
greatest inculcator...
Sub-Culture as Inhibitor
Organizational Culture & Evaluation12
Common Responses from Follow-up
Interviews
 Seen as “busy-...
Inhibitors
Organizational Culture & Evaluation13
 Lack of education, not able to produce meaningful results and
change
 ...
Sustainable change requires cultural
focusing (1)
Organizational Culture & Evaluation14
 "Cultural change is one of the m...
Sustainable change requires cultural
focusing (2)
Organizational Culture & Evaluation15
Integration of Culture & Evaluation Model (1)
Organizational Culture & Evaluation16
 The following model was developed to...
Integration of Culture & Evaluation Model (2)
17
Dimension
Stage 0 Stage 1 Stage 2 Stage 3 Stage 4
Purpose Complacency Jus...
Integration of Culture & Evaluation Model (3)
Organizational Culture & Evaluation18
 Kotter (2008) argues that getting bu...
Next Steps
Organizational Culture & Evaluation19
 Formalize the Integration of Culture & Evaluation Model
 Create an org...
Next Steps
Q: Do the agencies that fund your teen programs require that
you report outcome/impact measures?
Organizational...
Upcoming SlideShare
Loading in …5
×

I school evaluation symposium, irwin, april 2014 pauledit

393 views

Published on

Published in: Business, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
393
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

I school evaluation symposium, irwin, april 2014 pauledit

  1. 1. Defining New Metrics For Libraries Success iSchool, University ofToronto Symposium April 23, 2014 Dr. Bill Irwin, Huron University College Paul St-Pierre, PHD Candidate, FIMS, UWO In Insights: Organizational cultures in public libraries: Gatekeepers to effective outcome evaluation?
  2. 2. Project’s Goal Organizational Culture & Evaluation2 The key goal of our research is to examine existing evaluation systems of public libraries, in an effort to replace the current reliance on a predominantly output based model of performance measure, with new models of evaluation based on capturing outcomes.
  3. 3. Project’s Goal Organizational Culture & Evaluation3 Outcome models assist policy makers, both inside and outside of organizations, to address fundamental questions of resource allocation, delivery methods, and agency design and purpose.
  4. 4. Defining Outcome Evaluation Organizational Culture & Evaluation4
  5. 5. Project Objectives Organizational Culture & Evaluation5  exploring the limitations and challenges inherent in the current qualitative evaluation system  creating a better understanding of the cultural impacts and policy implications of a sociological performance framework, on library organizations, their constituencies, and their stakeholders in the establishment  capture changes in the Practice-Program-Policy continuum and the downstream implications for how libraries are viewed and valued upon the introduction of a new system of performance measurement
  6. 6. OPLA Child and Youth Services Committee Teen Services Benchmarks and Statistical Report 2013 Organizational Culture & Evaluation6 Does your library measure outcomes or impacts of teen programming? %Yes 42.9% All 46.7% 100,000+ 38.1% 50,000<100,000 40.0% 15,000<50,000 46.4% 5,000<15,000 75.0% <5,000 42.9% Less than 50% of the libraries reported measuring outcomes or impacts of teen programming, although several libraries did note that they are planning to implement outcome-based measurement in the future.
  7. 7. Q: Are you satisfied with the methods used by your library to evaluate teen programs? Organizational Culture & Evaluation7 • It would be helpful to have a clearer idea of what a youth program should be i.e. what is the purpose, what is the desire outcome. • It is not consistent • We do not evaluate anything past attendance. • I would like to find out what the participant got out of program • Difficult to measure anecdotal reports • Evaluation is based only on circulation and attendance performance indicators
  8. 8. Q: Are outcomes/impacts measured in other parts of your library? Organizational Culture & Evaluation8 • Lack of staff capacity • Do not have anyone trained in outcome/impact measurement • Probably the best answer for why it hasn't happened yet is "too time consuming." • Lack of time. • Not currently considered a priority by our governing body • Assessing outcomes falls short of our priority list. • No formal criteria which is implemented system-wide
  9. 9. Q: You indicated that you do not assess outcomes/impacts of teen programs. Why not? Check all that apply. Organizational Culture & Evaluation9
  10. 10. Q: Do the agencies that fund your teen programs require that you report outcome/impact measures? Organizational Culture & Evaluation10
  11. 11. Organizational Culture and Impact Organizational Culture & Evaluation11 Sub-cultures are currently the greatest inculcator of values (Schein, 1992) regarding: 1. the worth and purpose of evaluations, 2. postulation of negative influence, 3. evaluation seen as “busy work”
  12. 12. Sub-Culture as Inhibitor Organizational Culture & Evaluation12 Common Responses from Follow-up Interviews  Seen as “busy-work”  Rely on informal feedback  Some staff value the process, some question it, seen as additional work  Against staff comfort level  Comments are filtered –each manager chooses  Staff wouldn’t want it (new system) to be more onerous on them  Staff question why,“Are you evaluating me?”
  13. 13. Inhibitors Organizational Culture & Evaluation13  Lack of education, not able to produce meaningful results and change  Lack of inclusion:“Big picture” relevance  Required training in best practices  Librarians feel skill set is inadequate to the task, so they are reluctant to engage – (Similar circumstances to the introduction of technology in mid to late 1990s in public libraries (Author’s observation))  Technological impasse  Consider the law of unintended consequences (how can/does it apply here?)
  14. 14. Sustainable change requires cultural focusing (1) Organizational Culture & Evaluation14  "Cultural change is one of the most important factors to consider but one of the hardest to implement" (Preston, 2004).  “In general, we found a number of library staff skeptical of quantitative or qualitative data from customers, preferring instead to rely on their own assumptions and past practices to make decisions.” (Hiller et al, 2008)  “Due to the‘everyone does the same thing’ culture and operational model it was impossible to make a change to workflow in one area without it directly impacting other areas - therefore systemic change was necessary.” (Nussbaumer & Merkley, 2012)
  15. 15. Sustainable change requires cultural focusing (2) Organizational Culture & Evaluation15
  16. 16. Integration of Culture & Evaluation Model (1) Organizational Culture & Evaluation16  The following model was developed to demonstrate the steps an organization goes through in regards to moving from an input based to an outcome based evaluation system.  It highlights the operational and cultural transformation process.  This model can be used to determine where an organization is situated (at what stage) in terms of its use of outcome evaluation.  The model is designed to serve as a “roadmap”, to assist organizations in their successful movement through a series of steps, by infusing a culture of outcome evaluation.  Once an organization has achieved stage 4, evaluation then serves as a tool for: realizing organizational priorities, educating stakeholders and funders of the range of quantitative and qualitative program and service impacts, and identifying operational issues and challenges.
  17. 17. Integration of Culture & Evaluation Model (2) 17 Dimension Stage 0 Stage 1 Stage 2 Stage 3 Stage 4 Purpose Complacency Justification SelfAwareness Alignment Actualization Motivation Inertia Fear & Survival Toward sense of self-efficacy Enlightenment Success, Internalization Organizationa l Impact Tactical / Strategic Resistance Evaluation is‘busy work’ Tactical: Short-term Strategic: Long-term Buy-in Shift from systems to individual to patron/staff focus Habituation Inculcation Organizationa l Impact Internal / External Closed system Internal (staff) Toward an open system Broad stakeholder inclusion Infusion (beware of gaming) Inhibitors & Enablers Lack of education, lack of meaningful results, inertia Lack of personal involvement. Buy- in not enough Cautious engagement Enthusiasm; Coordinating many voices Unintended consequences (positive or negative) Implications Diminishing budgets, failure Awakening Organizational learning Shared leadership Assessment is continual, naturally occurring Creating an outcome evaluation culture
  18. 18. Integration of Culture & Evaluation Model (3) Organizational Culture & Evaluation18  Kotter (2008) argues that getting buy-in is not enough because it only engages the head, not the heart.  Moving through the stages is akin to Kotter’s step 6, creating short term wins, building staff confidence, an incremental approach  Somewhere in the transition between step 1 to 2 need to create and articulate a vision/mission statement.Why do we evaluate?What is it all about?  Building cultural assessment seen as a strategic priority – otherwise over shadowed by more “important work”  “When faculty and staff perceive that the administrators are only motivated to create a culture of assessment for accreditation, cynicism and low- motivation will likely result” (Lakos and Phipps, 2004).
  19. 19. Next Steps Organizational Culture & Evaluation19  Formalize the Integration of Culture & Evaluation Model  Create an organizational evaluation audit to:  Assess the stage of the model an organization is at  Develop strategies to assist moving it forward  Create a sustainability approach  Design and deliver the next phase of the research
  20. 20. Next Steps Q: Do the agencies that fund your teen programs require that you report outcome/impact measures? Organizational Culture & Evaluation20 Ontario-wide survey to elected municipal officials focused on understanding what information they consider when in their deliberations on financial support to libraries’ budgets. In part this research will provide insight for libraries in how to structure, and communication the results of, their institutional evaluations.

×