2. The idea behind the project
• To assist the Community Food Bank of
Southern Arizona in developing an overall
evaluation framework that will enable them to
track the outcomes of their feed the line
programs
• This enables the Community Food Bank of
Southern Arizona to identify successful
programs and target resources more
effectively
3. Project Scope
• Create logic models for each feed the line
program.
– Tools for completion
• Define Food Delivery to Food Justice in The
Community Food Bank context
– Define the steps required for short term change
• Present Evaluation and Process models for
long term change.
4. Implementation Steps
• Fill in the gaps in the logic models.
– Answer the key Logic model questions for each
program
• Define a new mission statement for the entire
agency
– Create an environment of change
– Ensure Employee willingness to change
– Implement short term changes in the feed the line
programs
• Choose one of the Process and Evaluation Models
to implement for long term change
6. Logic Model Construction
Completed within outline of programs To be completed by individual programs
Food Justice: articulate an overarching goal for the agency
7. Table of contents
Logic Model Questions:
Going forward, in order to establish linkages and
outcomes, the following questions must be answered:
1. Each program has to define long-term outcomes
2. Each program has to define how they interact with
other programs
3. How does each program’s outcome contribute to the
goal of food justice for the entire agency?
8. Agency Market Logic Model
Inputs
collections from
grocers
distributors
brokers
Feeding
America
Strategies
The Agency
Market provide
s dry and
perishable food
and non-food
products packed
in retail and
institutional
size packaging
for the use of
non-profit
501(c)(3)
agencies with
on-site feeding
programs.
Outputs
A high
percentage of
donated
products are
acquired
through
Feeding
America, a
nationwide
network of food
banks.
Outcomes
The Agency
Market distribut
es 5,938,103
pounds of
donated items
each year.
The Agency
Market is
currently
serving over
140 agencies
with 400 agency
feeding sites.
Impact
Obscure or
nonexistent
measures
Rationales/Assumptions: Non-profit agencies such as the
Community Food Bank benefit from food contributors of
agency markets to provide their facilities with extra produce
for their clients.
Situation: No Applicable Measures
10. Food Plus (CSFP) Logic Model
Inputs
Funded by
USDA
Seniors (Tucson,
Marana, Green
Valley, Sahuarita,
Amado, Nogales)
Strategies
Provides eligible
participants
with a food
package
Outputs
Recipients of
CSFP-Food
Plus receive a
one year
refillable
prescription for
USDA
commodities.
Outcomes
As of June
2014, 8 sites:
5,093
households
served
Nogales
Produce:
25,000 lbs
Store
Donations:
45,000 lbs
USDA:
209,190 lbs
Impact
Obscure or
nonexistent
measures
Rationales/Assumptions: No Applicable Measures
Situation: Low income elderly persons requiring
additional food assistance
11. Grocery Rescue Logic Model
Inputs
Agency Market
On-Site Pantry
Food Bank
clients
Food Bank
staff
Strategies
Salvage food
nearing the
expiration date
or no selected
by customers
Outputs
Meat, dairy,
prepared and
perishable
items from
Grocery
Rescue in
Tucson are
mainly
distributed
through our
Agency Market
program.
Outcomes
1,861,068 lbs.
of food
distributed as
of February
2015.
Currently
seeing a 5%
increase in
store donations
Impact
Obscure or
nonexistent
measures
Rationales/Assumptions: Food nearing the expiration date
can be salvaged and distributed to food agencies, providing
nutritious foods to those in need.
Situation: Food nearing the expiration date or
unappealing to customers gets thrown out
12. Supplemental Nutrition Assistance
Program (SNAP) Logic Model
Inputs
Federal
Government
Low-income
individuals and
families
Strategies
Participants
receive a debit
card with a
specific
monthly
amount that
can be used to
buy food in
most grocery
stores.
Outputs
In July 2012,
the average
individual on
SNAP in
Arizona
received
$29.14 in
SNAP benefits
per week
Outcomes
Meant to
supplement a
household’s food
budget to help
people put
nutritious food
on the table.
Impact
Obscure or
nonexistent
measures
Rationales/Assumptions: Providing extra nutritious
foods and benefits to those in need can increase
quality of life for those in need.
Situation: Low income individuals and families require
additional food assistance and health benefits to contend
against hunger.
2014 Arizona Food Bank Conference
http://www.fns.usda.gov/snap/supplement
al-nutrition-assistance-program-snap
13. Disclaimer
• These program logic models were created
based on published materials and external
sources.
• To prevent ambiguities or incorrect data,
please have each program meet to discuss
their logic models. This will lead to a more
accurate representation of each program.
16. Change Theory for the Feed the Line
Programs
• The Feed the Line Programs were created with
the culture of a food distribution center in
mind not Food Justice.
• Currently the Feed the Line Programs only
measure Outputs
• While these output measures can lead to
correlations they can not prove causality.
17. Organizational
Change Theory
and overcoming
Resistance to
Change
Conditions for Change
•New Mission Statement
Cultural Willingness
•Reinforcement of New Mission
Statement though detailing
who current work flow is not
completely adequate to meet
the new mission
Follow Through
•Lowest Entry Cost
Organizational Change Theory: Implications for Health Promotion
Practice
24. (TEFAP) Logic Model with Applied Evaluative
Frameworks
Inputs
Low-income
senior citizens,
families
Volunteers
Donated and
purchased food
and items
Strategies
Provides
low-income
individuals
and
households
with surplus
commodities
Outputs
Dining
Room/Soup
Kitchen
TEFAP
Emergency
Box
Home Meals
Mobile
Pantry
Snack/Meal
Program
Outcomes/Impact
RBA
Client
Centered
Re-Aim
Articulation of community/agency goals.
• % of food distributed by agency.
• % of food received by clients.
• Continuous data reporting
Reach: community population
Effectiveness: Impact of providing
emergency food assistance
Adoption: representativeness of an
interventions and obtaining support
Implementation: use of resources and
strategies
Maintenance: appropriate frameworks and
programs become standard.
Allows the client to choose the services
necessary for their needs in terms of service
delivery. Removal of a “one size fits all”
approach.
28. RE-AIM: AN OVERVIEW
RE-AIM is an acronym for Reach, Effectiveness, Adoption,
Implementation, and Maintenance.
• RE-AIM begins with a series of research questions designed to
move beyond the current reductionist approach to assess
interventions in order to isolate effective programs or activities.
• RE-AIM hypothesizes that the overall social-change impact of an
intervention is a function of all five RE-AIM dimensions not
simply client-based outcomes. That is all five dimensions are
important and equally in need of evaluation.
• Importantly, Foodbank RE-AIM evaluation allows summary
indices (Success Score) to be developed for the use in
determining overall impact of individual programs as well as
initiative areas.
31. Pro’s & Con’s
• Emphasizes cohesion
• Well-researched
• Nuance
• Clear implementation
• Includes a timeline
• Labor intensive process
• No software
• No specific measurement
of social change, although
all factors measured
facilitate change
In 5 years? 10 years?
32. Overview: Results Based Accountability
• Start with ends, work backward to means. What do we
want? How will we recognize it? What will it take to get
there?
• Keep accountability for populations separate from
accountability for programs and agencies.
• Customer or client results are the responsibility of program
managers.
• Use data (indicators and performance measures) to gauge
success or failure against a baseline.
• Use data to drive a disciplined business-like decision
making process.
• Involve a broad set of partners.
• Get from planning to action as quickly as possible.
37. Pro’s & Con’s
• Data driven
measurement of social
change
• Evaluation is distributed
among many partners
• Programs become
clearly accountable for
food justice results
• Clear process with an
implementation
timeline
• The tremendous
amount of data
collection might be
tedious
• Cost of software: $50
per user, monthly
• This is a large change
in process, it will be
challenging for the
organizational culture
to adapt
In 5 years? 10 Years?
38. Which framework to use?
RE-AIM
RBA
•Academic inception
•Deliberate decision making
•Institutional knowledge
confined to one user
•Industry oriented
•Rapid deployment
•Accountability spread
39. Moving Forward
Evaluate the proposed logic models for
each feed the line program
Create a change environment to
food justice in the food bank
Adopt and customize an evaluation
framework such as RBA or RE-AIM
Link data between feed the line
and shorten the line programs
40. Annotated Bibliography
• Cohen, Barbara. USDA Community Food Security Assessment Toolkit. IQ Solutions, Inc.,
(July, 2002). Food Assistance & Nutrition Research Program. Efan.
http://www.ers.usda.gov/media/327699/efan02013_1_.pdf
The USDA Community Food Security Assessment Toolkit report analyzes the issue of
food security and provides guidelines and assessment tools, which food banks can adopt to
deal with the growing concerns of food access, food security, and other topics. Data
collecting tools, surveys and other helpful materials are included for food banks to utilize.
• Supplemental Nutrition Assistance Program (SNAP). United States Department of
Agriculture. http://www.fns.usda.gov/snap/supplemental-nutrition-assistance-program-snap
SNAP (formerly known as food stamps) provides low-income individuals and families
with a debit card to purchase nutritious foods. The website provides more information on
eligibility and purpose.
• The Emergency Food Assistance Program (TEFAP). United States Department of Agriculture.
http://www.fns.usda.gov/tefap/emergency-food-assistance-program-tefap
TEFAP provides an emergency supply of nutritious food to low-income families and
individuals. More information is provided on the website.
41. Annotated Bibliography continued…
• Vancouver Coastal Health Community Food Action Initiative
http://www.smartfund.ca/current_cfai.htm Evaluation report:
http://www.smartfund.ca/docs/eval_vch_cfai_2011_full.pdf
The initiative website provides useful information including evaluation reports,
management plans, and other useful materials. The frameworks and models give valuable
insight in dealing with the issue of food security.
• Le Groupe‐conseil baastel ltée. Baastel: Creating a Macro Results Framework for the Middle
East and North Africa Transition Fund “Feasibility Assessment” Final Report. June, 2014.
http://www.menatransitionfund.org/sites/mena_trans_fund/files/documents/Baastel_Macro_R
esults_Framework_Assessment_Appendix_Final.pdf
This feasibility assessment pertains to world monetary banks and focuses on creating a
macro framework for the Middle East and North Africa (MENA) Transition Fund. While the
report does not discuss food banks, it does provide logic models and theories of change
frameworks and evaluation tools.
42. Annotated Bibliography continued…
• Person Centered Planning Preparation and Procedure Guide. 2nd ed. Augusta, Me.: Maine
Dept. of Behavioral and Developmental Services, 2003. Print.
• "Organizational Change Theory: Implications for Health Promotion Practice." Health
Promotion International (2014). Print
• RE-AIM: Virginia Tech College of Agriculture and Life Sciences. http://www.re-
aim.hnfe.vt.edu/index.html
This site provides an explanation of and resources for those wanting to apply the RE-
AIM framework. The RE-AIM framework is designed to enhance the quality, speed, and
public health impact of efforts to translate research into practice. The Key Features of RE-
AIM website are: Tools and resources to facilitate implementation, a comprehensive list of
RE-AIM publications and presentations organized alphabetically by year
• Results Based Accountability (RBA). http://raguide.org/
Website, guidebook, evaluation tools and software for Results Based Accountability.
Results-Based Accountability™ (RBA), also known as Outcomes-Based Accountability™
(OBA), is a disciplined way of thinking and taking action that communities can use to
improve the lives of children, youth, families, adults and the community as a whole. RBA is
also used by organizations to improve the performance of their programs or services.
Editor's Notes
The institutional evaluation framework will assess the “feed-the-line” hunger-relief line-of-business, as well as its “shorten-the-line self-sufficiency services/program. Special emphasis is placed on the “feed-the-line,” as this historically has been an output-measured endeavor rather than an outcome-based one.
The inputs, outputs and strategies section is filled out for each of the core feed-the-line programs. The right side of the model (outcomes and impact measures) were vague, ambiguous or non existent.
The first step: complete the logic models for each program by answering the three key logic model questions. Doing this, will give your agency a complete picture of everything you do and how the programs interact.
The green arrow indicates a correlation between outcome measures and impact measures.
Disclaimer: Data may exist already but not able to find in the public documents (which we were working off from). Hence, there were many inconsistencies and obscurities in the data, especially with the outcome/impact measures.
Change theory essentially states that organizations are created with a limited purpose in mind, but as they grow they realize that this limited focus will not solve the issue so they have to take on additional task. In your case simply handing out food does not end hunger in Tucson we have to tackle the sources of food shortages like education or access to social services. Thus you created the shorten the line programs to deal with these issues. And your morphed yourselves from a simple distribution network to a food justice organization.
The Current feed the line Programs were developed when the community food bank was only concerned with being a food distribution center, and that is reflected in the metrics that are used to measure its success. The tonnage of food that is handed out does this really well.
However you had indicated that you wished there way a way from the feed the line programs to have a measurable outcome. The problem is the measurements that you are using (lbs of food distributed) are not sufficient. You can look at long term trends between the food distributed and proxies for self sufficiency and food justice like SNAP enrollment in the area but the trends are really shaky and can only prove correlation not causality.
To remedy this situation and bring the Feed the Line programs into line with the community food bank becoming a food justice organization there needs to be discrete data collected on all clients who use the programs so that their progress can be tracked over time.
And that is where change theory for the Feed the line programs come into play You have the good fortune that clients come to you for help and you have a good that gets them in the door to begin with, that doesn't exist in a lot of nonprofit work. So you can leverage this advantage by adding a the mission of data collection to the feed the line programs. This way you can track the outcomes of the clients who receive the benefits of these services by comparing it to long term trends and you can better serve your notion of a food justice organization by helping every client (or at least pointing them in the direction of) services that address the underlying causes of their food insecurities.
The Idea of change theory may be awesome but there will be organizational resistance to any new tasks being implemented.
The three major ways that it can be combated are though:
Creating appropriate conditions for change
Changing the culture
And having explicit follow though.
Appropriate Conditions For Change:
If people think that they are ok in their current setting even talking about a change initiate will require immense amounts of political capitol from a director. The employees have to take it on faith that their boss is right, and that is big ask if things are ok as they are. At the community Food bank this would involve changing
Solution: A new mission statement has to be created for the feed the line programs in which the change from a delivery service to a food justice service which is client centered is explicitly stated. Doing this sets the a new bar to be met, and staff/volunteers can see that the current system is no longer adequate.
Appropriate Conditions For Change:
But simply creating a new mission statement for staff/volunteers to read once will not do the trick. It needs to be reinforced over and over and over again.
This has to be a change in the culture of the organization its almost a full internal rebranding. Sings in the break room or department slogans may seems silly but given enough time they really do get in your head. With enough emphasis eventually people will be willing to accept the change. This is not a costly process it is just long term.
During this time period there has to be a culture of learning that is established. Every employee has to understand why new procedural changes are required to meet this new goal in the feed the line programs. And more importantly they have to understand their role in this new mission, how they contribute.
Appropriate Conditions For Change:
The enthusiasm for new projects and missions can quickly die out if the new tasks are pile on too quickly. To make sure cultural change is permanent and not a fleeting thing there needs to be explicit follow though and evaluation of even the most minute details to make sure they work and that they are being implemented with the lowest entry cost possible
The way that I would envision that a food justice mission would be implemented though a client centered model in the beginning is though a simple survey that clients fill out when they receive goods or services. It would track how often they are coming, relevant data like do they have SNAP? and if they accessed any services other than the food programs. This is already done sometimes on an individual basis but there is no programmatic wide way of tracking long term outcomes for the feed the line programs.
This simple survey is easy to implement and allows the impacts of the feed the line programs as a “data entry point” to be easily seen.
My portion of this project involved contacting food banks across the nation to see what tools they use for program evaluation.
It is important to note that Chris touched on a major industry trend, the shift from food delivery to food justice. New evaluation methods play a significant role in this shift. Food justice is much more difficult to define and measure than the # of meals served to a set # of people.
The movement toward social change begins with asking whether or not services are client centered, but food banks who are further along in the process are using evaluation frameworks such as “Results-Based Accountability” or “RE-AIM” to add structure and definition to the concept of food security.
Vermont Food Bank is using the Results Based Accountability evaluation framework.
Santa Cruz and Santa Barbara are using the RE-AIM evaluation frameworks.
Oregon Food Bank and Ohio’s Second Harvest Food Bank are both building their own “recipient based” frameworks.
This seems to be similar to the current undertaking at the Community Food Bank of Southern Arizona. Client centered logic models have been created. Needs assessments are performed. Client impact-based outcomes are reimagined.
Northern Alabama and NYFoodlink appear to be using Feeding America’s Performance Framework Dashboard. This framework fits the food delivery model more than other options.
This is an example of the three evaluative frameworks (Results Based Accountability, Client Centered or Recipient Based, RE-AIM) applied to the TEFAP program. The following slides provide more depth into each framework.
Results Based Accountability: data-driven decision making to solve problems. Programs and staff are held accountable for each action and process that occurs.
Client Centered: Clients have flexibility in choosing a food assistance service that fits their needs. Uses logic models and Needs Assessments
Re-Aim: Moves beyond client centered models and asks a series of research questions to analyze social change.
We have spoken a little bit about “client centered” models. What do these evaluation methods look like? What is different?
Impact measures are recipient-based instead of commodity-based. This means measuring the coordination of family food sources such as WIC, SNAP, School lunch programs and markets), community inclusion and participation, behavioral changes that increase health (Meal planning, buying in bulk, cooking at home, using left-overs, reducing fat and sodium, increasing fresh fruit and vegetables). It means stopping the focus on the pounds of food, meals or people served.
Logic Models which are client centered (next slide)
Needs assessments, also client centered
These are steps that the Community Food Bank of Southern Arizona has already taken.
This is a sample of Oregon’s client centered logic model for the Child Hunger Programs. It is included in our annotated bibliography. It is important to note the distinction between intermediate and long term impacts, and the separate description of a program theory.
Needs Assessment, which was also provided by Oregon. The Needs Assessment seems to be an adaptation of Mazlow’s Hierarchy of Needs. It doesn’t specifically connect to food security.
This is a general description of RE-AIM, which was the evaluation framework adopted by Santa Cruz and Santa Barbara. RE-AIM was originally developed by the Department of Human Nutrition at Virginia Tech. Like RBA, this framework begins with research, and from there, hopes to shift evaluative measurements toward social change.
This is a visual representation of the RE-AIM framework. In this vision, all 5 elements should work in tandem.
This is an example of an evaluation matrix for RE-AIM. Each RE-AIM element has a specific set of evaluative questions attached. After asking these questions, you receive a success rate and an impact rate. These rates are calculated based on RE-AIM calculation tools, which are available in our annotated bibliography. RE-AIM also offers a free online module which trains you on the evaluation techniques.
Well researched:
RE-AIM had an academic inception. Since the original paper in 1999, there have been approximately 100 publications on RE-AIM by a variety of authors in diverse public-health related fields
Aside from balanced and cohesive services, a clear vision of the next 5 or 10 years isn’t immediately apparent. A dedicated staff member would have that answer after an in-depth research project. RE-AIM research guidelines are very clear, but the research process doesn’t come in the form of a simple survey. There are no multiple choice answers. In this regard, RE-AIM is quite true to its academic roots. An ideal situation would be to have a graduate student take this on as a dissertation. The process would be quite clear, but the research will certainly take more than one semester.
Results Based Accountability is an open source evaluation framework. The implementation guide is available online for downloading. A link to this guide book will be included in our annotated bibliography. Evaluation software and other tools are available for purchase on the same website. This slide describes the general premise of RBA.
Like RE-AIM, RBA begins with specific research questions. RE-AIM begins with a deep and nuanced analysis. RBA’s research questions are much more pointed. They measure the accountability of programs to recipients. This is a sample of questions which are asked.
Before selecting an evaluation framework, we would recommend a thorough reading of both RBA and RE-AIM’s research questions. Which questions are most important to the core client population? Which questions fit the goals, mission and culture of the food bank?
RBA specifically measures social change. After determining a baseline trend, RBA performance indicators illustrate whether or not change is being achieved.
Several performance measurement tools are included in on the RBA website.
This is Oregon’s example of an RBA performance measure in action. This sample is specifically connected measuring effectiveness of their Backpack Program.
