Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Meaningful Measures: Why size (and volume) doesn't matter


Published on

CLA Webinar presentation from November 2013

Published in: Education, Technology

Meaningful Measures: Why size (and volume) doesn't matter

  1. 1. Rebecca Jones, MLS Rebecca Jones, MLS
  2. 2. Underlying Assumptions • Few libraries have a “culture of assessment”  Difficult and complex  Most measures indicate past performance  No cause-and-effect relationship between measures  Performance measures quantitative, but library outcomes are largely qualitative Measuring for Results: The Dimensions of Public Library Effectiveness Joe Matthews, 2004
  3. 3. Underlying Assumptions • Identifying & illustrating value depends on conversations with your stakeholders – The first conversation shouldn’t be when measures are presented • “More” isn’t necessarily better – In fact, “more” clouds the issue and the message “Not everything that counts can be counted & not everything that can be counted counts.” - Einstein
  4. 4. Three types of measures Performance or Value Operational Satisfaction
  5. 5. Measures…  are, by definition, based on a “beginning” or monitor results against an agreed-to objective or value
  6. 6. Questions driving measures • What’s the library or information centre doing? • How much is it doing? • How well is it doing it? • Who is it doing it for? • What positive impact is the service having for them? What action will you take as a result of knowing this?
  7. 7. Usage Statistics • We’ve long tracked “how much” • We also need to look at differences: • Peaks? • Dips? • Switches? • What are these statistics really telling us? – And not? • Are they highlighting where we should dig deeper?
  8. 8. Customer Satisfaction • Tends to focus on existing products and/or services • Usually measured through surveys
  9. 9. The performance question
  10. 10. Framework Page 1 What measures do you keep?
  11. 11. Meaningful performance measures • Matter to you & to your stakeholders • Are conveyed in the stakeholders’ language • Demonstrate that the service makes a difference • Focus attention on what is most important for the organization • Are critical for managing, planning & decision-making • Are organization-dependent • Connect strategic directives to employees
  12. 12. • No one magic measure • Successful organizations: • clarity of purpose • understand their culture • performance measurement system that fits that culture • “Value” is dynamic, economic , psychological & relative to alternatives Learning from others
  13. 13. Framework Page 2 Understand the Context Align Strategies & Objectives Communicate Results Translate Data into Outcomes & Impacts Identify Services & Programs Manage Measurement Data Define Measures
  14. 14. Understand the Context
  15. 15. Know the context and contextors How is value defined, measu red, and communicated? Name your stakeholders How do they measure their progress?
  16. 16.  Healthy living advocate Councilor Good Cause  Endorses physical activities  Spend for bike lanes, pools, rinks, etc  The library is just a community centre anyway  Says more people are impacted by his stuff than by the library Board Member: Ms. Expert  Does reconnaissance work on other library systems and reports back to Board  May not have a good grasp on what libraries do  Feels money is no object when it comes to something she sees as a trend  Wants to compete in the library industry
  17. 17. The Provost of Doom  Libraries days are numbered  Libraries aren’t relevant  Everyone has computers and internet  need information? Just google it  Starbucks has wireless  With eBooks and research online believes facilities can be made smaller Dean of Humanities Dr. Kee Paper  Influential in university & broader academic circles  Focus on research  Has deep seated beliefs in traditional library services  Quiet  No technology  Libraries are institutions for reading & research
  18. 18. Framework Page 3 Start with Stakeholders 1. Critical that you understand the context in which the measures are considered. In your organization, how is value defined, measured and conveyed? 2. Who are the stakeholders who must understand the value of your services? Name Position How do they Goals & measure objectives their progress? What do we know about them? What type of stakeholder are they? What do they know about our services?
  19. 19. Logic Model • Popular in non-profit sector & increasingly in libraries and culture • Based on “logical links” cause & effect • Logical links: – IF teens believe that the Library sees them as valuable, equal community members & takes their needs seriously THEN teens are more likely to engage in discussions & programs in the Library that will positively impact their studies & their lives – IF the information professional ensures current, relevant content is easily available on employees’ desktops THEN the organization will save $X in lost “hunt” time
  20. 20. Framework Page 4 It’s logical that INPUT OUTPUT OUTCOME IMPACT resource perspective operational perspective user perspective stakeholder perspective
  21. 21. INPUT Resource Perspective OUTPUT Operational Perspective Equipment Space Supplies Funding Staff Training module Program Report # of Program Attendees # of Programs Held
  22. 22. OUTCOME User Perspective Increased skill Know-how or knowthat Change in status Change in behaviour or attitude IMPACT Stakeholder Perspective Lower dropout rates Increased employment levels Increased recovery rates
  23. 23. Start with the end in mind Refocus from the activity to the impact
  24. 24. Align Strategies & Objectives
  25. 25. Align Strategies and Objectives Stakeholder’s Goals & Objectives Your Goals & Objectives: How they Contribute Provide excellent clinical care Support the development of procedures and policies relating to clinical care Increased research grants Support the research process Newcomers contributing to the community economically Equip newcomers with links to language, literacy, employment and social supports
  26. 26. Framework Page 5 Top Align objectives • Articulate & document your goals & objectives What are you trying to accomplish? • Clarify how these contribute towards your stakeholders’ objectives & desired outcomes
  27. 27. Identify Services & Programs
  28. 28. Framework Page 5 Bottom Identify Services  What services, programs or products contribute to your realizing an objective? Stakeholder Goals & Objectives Your Goals & Objectives Your Services
  29. 29. Academic Example Input One liaison librarian Output Faculty research support for a Grant Application Outcome Completed application submitted on time Impact Grant dollars received by faculty. Faculty member is able to hire 2 grad students to conduct research: increased university reputation, a number of articles are submitted and accepted by academic journals, increased citations for faculty member, university increases visibility.
  30. 30. Public Library Example Input -Funding -Staffing -Supplies =$10,000 to develop learning place programs Output 10 new programs each offered 5 times per year Outcome Number of program registrations increased by 25% year to year. 1,600 students ages 6-10 completed learning program Impact Children attending programs are able to maintain the skills taught in school verified by parent follow-up. Library recognized for helping children develop and maintain literacy skills. Board of Education partnership & recognition based on improvements in standardized testing results.
  31. 31. Government Example Input $100K resource budget Output 2500 resources utilized Outcome Resources used by 75% of employees in policy development Impact Usage enabled policy development to: 1. Eliminate department-specific resource purchases saving the organization $35K 2. Reduce policy duplication or “blind-sides” by at least 25% (as reported during policy debriefings)
  32. 32. Goal Objectives Objectives Activities or services Activities or services Outputs Logical flow Outputs Immediate Outcomes Long-term Outcomes Immediate Outcomes Objectives
  33. 33. Define Measures
  34. 34. Define Success  What will success look like?  How will you track progress towards meeting goals?  What indicators or measures will you use?  Qualitative? Quantitative?  What data needs to be collected? How?  Keep it key  Keep it simple  Be practical  Don’t get mired – or overwork frontline staff
  35. 35. Framework Page 6 Left Service Impact Inputs Person power, money, content, etc. Activity What you do Outputs What the activity produces Outcomes How the client uses the output Impact What changes for the client Outcome Measures Impact Measures
  36. 36. Define Outcomes and Impact  What will success look like?  And how will you “get” there? INPUT OUTPUT OUTCOME IMPACT
  37. 37. Define Outcomes and Impact OUTCOME User Perspective Increased skill Know-how or know-that Change in status Change in behaviour or attitude IMPACT Stakeholder Perspective Lower dropout rates Increased employment levels Increased recovery rates
  38. 38. Define measures by success being sought Inputs Person power, money, content, etc. Activity What you do Outcomes How the client uses the output Outputs What the activity produces Impact What changes for the client
  39. 39. Framework Page 6 Bottom So – what data will you need to collect? What data or measures do you What data or measures do you to collect to collect about Outcomes? about Impact?
  40. 40. Manage Measurement Data
  41. 41. Framework Page 7 Manage the data • Do it • Determine a collection schedule • Work out agreements with other departments that have needed data • Are you currently collecting data or indicators that are no longer relevant? • How long do you need to keep data? • Who is responsible?
  42. 42. Translate Data into Outcomes & Impacts
  43. 43. Framework Page 8 Bottom Interpret data  So... What?  What does the data say? What doesn’t it say?  Examine it from various angles  What is the progress towards the goals & objectives?
  44. 44. Communicate Results
  45. 45. Framework Page 8 Communicate results • Actually begins when you are understanding the context • Your message: “we have contributed towards your goals by……” • If your goals & objectives are meaningful for your planning & decision- making, your measures will be indispensable
  46. 46.
  47. 47. The real question How have we impacted your project, work or decisions today?
  48. 48. Thanks! Rebecca Jones With special thanks to... Moe Hosseini-Ara @ Markham Public Library