2. Fooled by Experience
Approach with caution any list that purports to
reveal, for instance,
10 common traits of successful companies!
whether it is punctuated with an exclamation
point or comes with the seriousness of a
legitimate study.
3. What is outcome-based evaluation?
• Outcome-based evaluation (also known as
outcome measurement, results- based
evaluation, outcome assessment) is focused
on assessing the quality and effectiveness of
programs. This is accomplished by project staff
tracking both the qualitative and quantitative
changes created in the community through
their program.
4. When should it be used?
The outcome-based evaluation approach is most
useful when the purpose of the evaluation is to
assess a program’s results.
– To what extent is the program meeting its goals?
– How does the program compare against accepted
standards?
– Is the program effective?
– How does the program compare with competitive
programs?
– Should the program be continued?
– Should the program be expanded?
6. Alligator Pie
• Literacy based, professionally developed, and
needs‐appropriate library storytimes to
children in childcare programs who do not
have the opportunity to attend and benefit
from library storytimes
• Special emphasis on reaching Vancouver’s
most vulnerable children in at‐risk
neighbourhoods
7. Alligator Pie
• Three-year pilot program funded through the
Vancouver Public Library Foundation by an anonymous
donor and the Allan MacDougall Legacy Fund
• Desired outcomes:
– 50% of the children attending the Alligator Pie program
each year of the three-year pilot will demonstrate an
increase in:
• Emergent literacy skills, for example increased vocabulary (learned
new words), eyes following text from left to right, and appropriate
vocalization of sounds associated with specific letters.
• Developing social interactions, for example peer to peer eye
contact, interaction with the librarians, and social vocalization
(laughing, giggling).
9. Results (2014)
• Based on formal and informal observation and
assessment, all participants who attended for the
duration of the program achieved increases in
emergent literacy skills and improvements in
positive social interaction.
• Program includes daycares in the Downtown
Eastside, Strathcona, and Mount Pleasant
neighbourhoods – caregivers report an increase
in skills; parents report an increase in confidence
10. Aboriginal Storyteller in Residence
Sharon Shorty (2015)
Photo: Mark Rutledge
• A pilot program from
2009-2011 funded by the
Vancouver Foundation
and the Vancouver Public
Library Foundation.
• In 2012, the VPL Board
established the Aboriginal
Storyteller in Residence
as a core VPL program, re-
commencing in 2013.
15. In the words of library staff
Through our staff training meeting that Columpa
attended I learned more practical tips to help me
plan and develop programs; Columpa clarified a lot
of questions I had regarding protocol: the
importance of "inviting people into your house,”
"seeking balance,” going where the communities
are, relationship building and longevity versus one-
off programs, why singing is important and the role
of different songs, the significance of the land, the
different role of storytellers in Coast Salish cultures,
tips for bridging the oral and written world, and
more!
17. SPARC BC evaluation (2014)
• Evaluation period of November 2012 through
September 2013
• Evaluate the following outcomes:
– Increase accessibility to Vancouver cultural,
recreational, and in-formal learning venues for all
Vancouver residents;
– Showcase Vancouver’s cultural, heritage, and
recreation venues;
– Increase participation in cultural, recreational, and in-
formal learning activities by Vancouver residents who
have little or no current participation; and
– Increase the sense of connectedness to community.
18. Methodology
• Surveys of participants (pass users)
• Focus groups of participants (pass users)
• Interviews with VPL staff and representatives
of a number of Program partners were
conducted to identify challenges and positive
outcomes.
19. Result (participation):
• Almost half of respondents who agreed to
report on their income (48%) had low or
‘modest’ household incomes, if modest is
defined as less than $60,000. Vancouver was
frequently noted in the survey feedback as an
expensive city in which to live with many
individuals and households experiencing
significant challenges in finding sufficient
discretionary income to allow them to attend
many fee based venues.
20. Result (connectedness):
• Two in three respondents (67%) reported that,
since using the Pass, they know more about what
happens in their community, while 86% felt that
they have a greater appreciation of the City and
its cultural and community life.
• Nine in 10 respondents (90%) either agreed or
strongly agreed that the Pass encouraged them to
get out and enjoy Vancouver while approximately
1 in 3 (32%) felt that the Pass had changed their
view of living in Vancouver.
21. Of particular note:
• Popularity of the program extended the waitlist
to 2+ years
• Strong desire among program partners to reach
those most socially excluded and economically
disadvantaged – current program meeting 48%
with household incomes of $60K or less
• VPL implemented change to Pass: admit 4 rather
than 6 people; create 40 passes to distribute
among community partners to reach most
vulnerable individuals
• Evaluation underway!
22. Access the toolkit
VPL/SPARC Outcome based evaluation toolkit
• http://www.vpl.ca/images/uploads/file/pdf/
Outcomes_Based_Evaluation_Toolkit.pdf
Contact: Daphne Wood
daphne.wood@vpl.ca
604 331 4006
Editor's Notes
VPL shares their research about the value and impact of a number of their programs: Aboriginal Storyteller in Residence program, Alligator Pie program
for children and the Inspiration Pass program which provides free access to 18+ cultural and recreational venues. Get lots of tips and insights!
By Emre Soyer and Robin M. Hogarth, May 2015 HBR
Here are some key questions that are typically answered in an outcome-based evaluation
As you can tell, evaluating a pilot is ideal…
The City of Vancouver Group Care Child Care report from August 2009 indicates that Vancouver has 581 childcare facilities. Despite their desire to do so, children’s librarians are unable to bring programs to this number of childcare facilities.
A community‐led approach would be used to collaborate with agencies such as daycare providers, family places and Community Centres to ensure that community needs are identified and met. These community agencies have indicated a desire to work in a collaborative model to serve children and families.
19 children regularly attended Alligator Pie in Year 1; 37 children regularly attended Alligator Pie in Year 2; and 38 children regularly attended Alligator Pie in Year 3.
– focused on two daycares in the Downtown Eastside of Vancouver – serving vulnerable young children measured by the Early Development Instrument
In addition to use of the formal observation tool, changes in behaviours related to emergent literacy and social skills were noted during informal discussions and interviews with parents and caregivers. The Alligator Pie librarian’s direct observations of the children’s responses and behaviours provided another key evaluation component.
Some participants achieved greater increases than others. Children with developmental delays experienced smaller increases in emergent literacy and social skills, but these increases are nonetheless impressive given their individual situations and challenges.
Sharon Shorty is the 6th Aboriginal Storyteller in Residence
The Aboriginal Storyteller in Residence program, formerly known as the First Nations Storyteller in Residence program, was a pilot program from 2009-2011 funded by the Vancouver Foundation and the Vancouver Public Library Foundation. In 2012, the VPL Board established the Aboriginal Storyteller in Residence as a core VPL program, re-commencing in 2013.
The Aboriginal Storyteller is to spend part of the residency developing personal storytelling project, and the other part on public programming. The goal is to promote story as a way of sharing culture and building memory.
Each year the program will be informed by the previous and current Storytellers in Residence (for example, the name of the program changed in 2013 from First Nations Storyteller in Residence)
Number of responses: 117 participants
Figure 1 shows that 78 respondents (66%) either agreed or strongly agreed with the statement, 22 (19%) were neutral, 9 (7%) disagreed or strongly disagreed, and 8 (7%) were either unsure or had no answer.
Columpa Bobb was the 2013 Aboriginal Storyteller in Residence.
Launched October 24, 2012
Good for two adults and four children at twenty-two cultural and recreational venues. Each pass had an approximate value of $1850 (assuming all sites visited by all eligible participants)
120 passes available every 2-weeks