This is a presentation riddled with questions, very few of which have concrete answers.
The public libraries I work with usually use combinations of these
But I’ve been getting more and more of this, lately
Many public Libraries depend on circ stats almost exclusively. In a world where these same liraries see over 50% of their circs come from DVDs and CDs, these libraries are in trouble.
We are doing ourselves a disservice by relying so heavily on Circ stats. We’re not telling our whole story. We’re not making our case effectively. We’re not arming ourselves with enough data to make truly informed decisions. Where do we go from here?
Start with the questions we using metrics to answer.
Actual question I got.
Not important to the competition at hand.When we rely on the same old data to measure new initiatives, it can lead us astray. We need to formulate questions that reflect on the goals we want to accomplish in a meaningful way.
When looking at what we want metrics to tell us, ask yourself: Is this a question that we can ANSWER? – scope, framingWill the answer lead to new KNOWLEDGE? - verifiableDoes this knowledge help us assess our PROGRESS? - Can this assessment be used to suggest ACTION? - Will the actions help lead us closer to our GOALS?
Strategic planning. Mission, Goals should drive everything. If goals are clearly identified, we can start asking questions that matter and determining new ways of answering those questions.
When considering the goals, we need to think about BOTH outputs and Outcomes.Outputs What you’ve done (Make 100 Cupcakes)Outcomes Impact of the Outputs (100 Happy, sugared up kids) – libraries usually focus on Outputs. It’s easier, funders ask for it, etc.Outcomes are harder to get at, but more in line with progressing towards our goals.
Outcomes measurement – see this excellent book for practical examplesEvaluation of both process and results. Results are harder to measure, but more important. ROI often focuses on Input/Output equation, which is valuable. However, it often fails to get at the true impact of the initiative. Resume writing assistance: Inputs: 100 hours staff time, specialized staff training Outputs: 35 resumes for jobless library membersOutcomes – interviews, jobs attained; customer satisfaction; community engagement and awareness
Come back to these questions. This leads to…
Deciding what to measure in order to do our ouput and Outcomes analyses?What can we get?What would indicate a successful outcome?
One way of organizing your data collection is to define KPIs in line with the strategic goals.Much documentation on this process. My own requirements…
Specific Circulation StatisticsComputer terminal, facilities, equipment usage Ratio of professional staff (FTE) per capitaWorkflow efficiency ratesCustomer satisfaction survey feedback
KPIs should be framed as specific goals with a defined timeframe. Assessment of progress can be measured against the degree of success in reaching these KPIs
Factors to consider in determining Data collection methodology:Qualitative measuresDon’t forget qualitative measures!How do we quantify the intangibles of our successes? Constant vigilance – don’t sacrifice quality. It is at the core of your mission.
Are there standards within the industry or community that can be applied?
What kind of data is available?Open data resources are multiplying!
Impacts what you decide to measure>ILS Tools – use and collection statisticsIndependent analysis products – GIS, demographic comparisonsObservational methods – “field work”
Hi Tech makes new measures possible: Movement Sensors (Measuring Behaviors: Attention, Movement, Location, Response) GeotrackingHow long did someone browse that section before making a selection? Issues of compiling and storing data – track length of time between visits, borrowing habits, behaviors – privacy issues
Effective measurement should not impact the data itself – hard with qualitative or subjective dataWhat are we doing already, or what can be adapted from current procedures>Look at models from inside and outside the industry!The ever-present issue of qualitative measurement
Love this data collection tool. Simple, easy, at point of service. Fallable, yes, but reasonably straightforward. We need to make sure that the tools we use are appropriate to the questio, data, and situation.
Data access and security, protecting patron privacy. Anonymization of data, security measures, privacy policies, staff training
Once the data has been collected, we need to do something with it. Often I see library directors and staff take a pile f statistics, survey results, etc. and then stop the process. They stare at the numbers, usually looking for a straightforward gain or decline. But what can this data really tell us?
Determination of value, implications for each measure (How much is a “Circ” worth in 2014?)Identification of trendsWhat skills are needed for this? What do we, as a profession, need to work on?
Popularity vs. Importance – More people attended the concert on the lawn than the college search seminar. Causation vs. Correlation
Gotta read the numbers right. Someone sent me a chart much like this one once, telling me that they had an exponential growth in new members. Nope. Reasonably steady new membership, growing overall membership. Time to cull out the inactive cards.
Use data responsibly. Guard privacy. Use to further favorable goals, not personal agendas.
Photo: Phil Bradley
Presentation of data connecting with stakeholdersMaking strategic decisionsMotivating staffHow can we better use statistics and measures to tell our stories and make our case? What can we do to make the data more meaningful to our stakeholders?
Bigger = MoreNot just a list of numbers
I know how big a soda can is. This is a measurement I can relate to. I can connect with this data because it means something real to me.
Compare with Library card data. Where should we focus more early literacy campaigns?Compare with results of recent Affordable Care Act info initiative. How successful were we in reaching target demographics?
Providing context.Social media = connections (I’ve talked little about social and Web analytics)
New York City's daily carbon dioxide emissions as one-ton spheresIllustrating your point to stakeholders. Envision something like this for circ stats instead of a number.
Make a bold statement to get results. We recently needed to make a point to our executive board about the complexity f our ILS setup. We made a chart of system elements that covered an entire wall… point taken. Action approved. How can you make an impression on funders, legislators, community members by translating metrics into something that they can’t ignore?
Telling that story. How can we show (not tell!) the impact of our services to those we need to sell it to. What is it that makes our library important to the community we serve?
More storytelling – What can we tell about the mission of an organization by looking at how they represent themselves via data?
Bottom line: Turning the outcome measurements into something that can make an impact. Using them to further our goals.
We’ve asked a lot of questions about library metrics, and brought up some concerns. So where do we go from here?
MORE QUESTIONS!To consider:
If not, were do we get them?
If not, what tools need to be developed? Can we engage in partnerships?
How can we use it to Change the World?
Library Metrics and Measurement: Counting What Counts & Making it Matter
COUNTING WHAT COUNTS &
MAKING IT MATTER
Wild Wisconsin Winter Web Conference 2014
Web site visits
The Panicked Call…
We’ve been doing
so well! How can
our stats all be
Depending on Our Numbers
Complying with mandates
Gaining support from
Demonstrating the value of
Allocation of our internal
Asking The Right Questions
The Wrong Question
We bought a 3-D printer
and it seems to get a lot
of use, but circulation
The Wrong Question
That long jumper seems
like a really strong
athlete, but she doesn’t
run very fast!
Asking New Questions
Is this a question that we can
Will the answer lead to new
Does this knowledge help us assess
Can this assessment be used to
Go Back to Your Goals
Outputs vs. Outcomes
Results of Process
Results of the User’s
Adapted from: Rubin, Rhea Joyce. 2006. Demonstrating Results: Using Outcome Measurement in Your
Library. PLA Results Series. Chicago: American Library Association.
What Defines Success?
What is it that we’re
actually trying to
How will we know if
What Data Do We Need?
Key Performance Indicators
Indicates progress towards goals
Measured frequently and consistently
May be objective or subjective
Library KPIs Can Take Many
KPIs as Strategic Goals
By 2015, 80% of users surveyed will indicate that
they waited no longer than three minutes for staff
assistance at a public service desk.
By 2015, at least 85% of all purchased items will
be invoiced, cataloged, processed, ad sent for
shelving within two weeks of receipt.
By 2016, the number of children aged 6-11 with
library cards will increase from X to Y.
Each year, a minimum of 80% of adults and teens
who attend a training session on how to use
technology to create content will indicate that the
training session was very good or excellent.
Questions for Data Collection
Are we counting effectively?
What structures do libraries have in place for
gathering performance data?
What kind of models can we emulate for
improving our data collection strategies?
How do we get closer to gathering qualitative
measures as well?
Use of Tools
Accessing and Storing Data
Analysis and Interpretation
Now That We Have The Data…
Questions for Libraries
What areas of measurement might we get into in
response to changing organizational goals?
How can we present the data we gather and the
conclusions we draw from it to more effectively
demonstrate success, promote the library and its
goals, boost productivity, make more informed
decisions, tell the library’s story to stakeholders?
What skills do we, as Information
Pros, Managers, and Advocates need to develop
in order to effectively make analytics the powerful
tool we need it to be?
Do We Have the Skills?
Do We Have The Tools?
Can We Make it Matter?
Brophy, Peter. 2006. Measuring Library Performance: Principles and Techniques. London: Facet.
Hernon, Peter, Robert E. Dugan, and Joseph R. Matthews. 2014. Getting Started with Evaluation.
Chicago: American Library Association.
Ishak, Das. 2014. “Discovering the Right Key Performance Indicators in Libraries: A Review of
Literatures.” Accessed January 14.
Larkin, Richard. 2013. “Using Outcomes to Measure Nonprofit Success.” NPQ - Nonprofit Quarterly.
July 2. http://www.nonprofitquarterly.org/management/22549-using-outcomes-to-measure-nonprofitsuccess.html.
Matthews, Joseph R. 2007. The Evaluation and Measurement of Library Services. Westport, Conn:
Rubin, Rhea Joyce. 2006. Demonstrating Results: Using Outcome Measurement in Your Library. PLA
Results Series. Chicago: American Library Association.
Webbmedia Group. 20:12:02 UTC. “Key Performance Indicators For Libraries”. Education.
Wolf, Lesile. 2010. “What Gets Measured Gets Done: Key Performance Indicators.” California Digital
Zappone, Marisa. 2013. “Why It Makes Sense for Nonprofits To Focus on Contribution, Not Causation.”
Greenlights for Nonprofit Success. http://www.greenlights.org/blog/2013/09/16/why-it-makes-sense-for-