This document discusses outcome-based evaluation and provides examples from Vancouver Public Library programs. Outcome-based evaluation assesses the effectiveness of programs by tracking qualitative and quantitative changes in the community. It is most useful for evaluating a program's goals, effectiveness, and impact. The document summarizes evaluations of the Alligator Pie early literacy program, Aboriginal Storyteller in Residence program, and Vancouver Inspiration Pass program. Evaluations found that Alligator Pie and the Storyteller increased literacy and social skills, while the Pass increased cultural participation and community connectedness especially for low-income residents. The toolkit for outcome-based evaluation is available online.

1. Impact Studies:
Outcome-based Evaluation
Daphne Wood
Director, Planning & Organizational Development
Vancouver Public Library

2. Fooled by Experience
Approach with caution any list that purports to
reveal, for instance,
10 common traits of successful companies!
whether it is punctuated with an exclamation
point or comes with the seriousness of a
legitimate study.

3. What is outcome-based evaluation?
• Outcome-based evaluation (also known as
outcome measurement, results- based
evaluation, outcome assessment) is focused
on assessing the quality and effectiveness of
programs. This is accomplished by project staff
tracking both the qualitative and quantitative
changes created in the community through
their program.

4. When should it be used?
The outcome-based evaluation approach is most
useful when the purpose of the evaluation is to
assess a program’s results.
– To what extent is the program meeting its goals?
– How does the program compare against accepted
standards?
– Is the program effective?
– How does the program compare with competitive
programs?
– Should the program be continued?
– Should the program be expanded?

5. Alligator Pie: Reaching Children Who Can’t
Always Reach the Library

6. Alligator Pie
• Literacy based, professionally developed, and
needs‐appropriate library storytimes to
children in childcare programs who do not
have the opportunity to attend and benefit
from library storytimes
• Special emphasis on reaching Vancouver’s
most vulnerable children in at‐risk
neighbourhoods

7. Alligator Pie
• Three-year pilot program funded through the
Vancouver Public Library Foundation by an anonymous
donor and the Allan MacDougall Legacy Fund
• Desired outcomes:
– 50% of the children attending the Alligator Pie program
each year of the three-year pilot will demonstrate an
increase in:
• Emergent literacy skills, for example increased vocabulary (learned
new words), eyes following text from left to right, and appropriate
vocalization of sounds associated with specific letters.
• Developing social interactions, for example peer to peer eye
contact, interaction with the librarians, and social vocalization
(laughing, giggling).

8. Methodology

9. Results (2014)
• Based on formal and informal observation and
assessment, all participants who attended for the
duration of the program achieved increases in
emergent literacy skills and improvements in
positive social interaction.
• Program includes daycares in the Downtown
Eastside, Strathcona, and Mount Pleasant
neighbourhoods – caregivers report an increase
in skills; parents report an increase in confidence

10. Aboriginal Storyteller in Residence
Sharon Shorty (2015)
Photo: Mark Rutledge
• A pilot program from
2009-2011 funded by the
Vancouver Foundation
and the Vancouver Public
Library Foundation.
• In 2012, the VPL Board
established the Aboriginal
Storyteller in Residence
as a core VPL program, re-
commencing in 2013.

11. A unique program every year

12. Potential Outcomes

13. Potential Outcomes

14. Survey results (2014):

15. In the words of library staff
Through our staff training meeting that Columpa
attended I learned more practical tips to help me
plan and develop programs; Columpa clarified a lot
of questions I had regarding protocol: the
importance of "inviting people into your house,”
"seeking balance,” going where the communities
are, relationship building and longevity versus one-
off programs, why singing is important and the role
of different songs, the significance of the land, the
different role of storytellers in Coast Salish cultures,
tips for bridging the oral and written world, and
more!

16. Vancouver Inspiration Pass

17. SPARC BC evaluation (2014)
• Evaluation period of November 2012 through
September 2013
• Evaluate the following outcomes:
– Increase accessibility to Vancouver cultural,
recreational, and in-formal learning venues for all
Vancouver residents;
– Showcase Vancouver’s cultural, heritage, and
recreation venues;
– Increase participation in cultural, recreational, and in-
formal learning activities by Vancouver residents who
have little or no current participation; and
– Increase the sense of connectedness to community.

18. Methodology
• Surveys of participants (pass users)
• Focus groups of participants (pass users)
• Interviews with VPL staff and representatives
of a number of Program partners were
conducted to identify challenges and positive
outcomes.

19. Result (participation):
• Almost half of respondents who agreed to
report on their income (48%) had low or
‘modest’ household incomes, if modest is
defined as less than $60,000. Vancouver was
frequently noted in the survey feedback as an
expensive city in which to live with many
individuals and households experiencing
significant challenges in finding sufficient
discretionary income to allow them to attend
many fee based venues.

20. Result (connectedness):
• Two in three respondents (67%) reported that,
since using the Pass, they know more about what
happens in their community, while 86% felt that
they have a greater appreciation of the City and
its cultural and community life.
• Nine in 10 respondents (90%) either agreed or
strongly agreed that the Pass encouraged them to
get out and enjoy Vancouver while approximately
1 in 3 (32%) felt that the Pass had changed their
view of living in Vancouver.

21. Of particular note:
• Popularity of the program extended the waitlist
to 2+ years
• Strong desire among program partners to reach
those most socially excluded and economically
disadvantaged – current program meeting 48%
with household incomes of $60K or less
• VPL implemented change to Pass: admit 4 rather
than 6 people; create 40 passes to distribute
among community partners to reach most
vulnerable individuals
• Evaluation underway!

22. Access the toolkit
VPL/SPARC Outcome based evaluation toolkit
• http://www.vpl.ca/images/uploads/file/pdf/
Outcomes_Based_Evaluation_Toolkit.pdf
Contact: Daphne Wood
daphne.wood@vpl.ca
604 331 4006