Linda Koser, Community Foundation of Harford County, Maryland
Martin O’Malley Anthony G. Brown
Governor Lt. Governor
Eric Brenner, Director
Governor’s Grants Office
21 State Circle, Annapolis, MD 21401
June 15, 2007
Limerick to a Logic Model A model I wanted to be But the runway was not for me So logic I tried And to my surprise It fit me to a T
What is a Logic Model? A logic model is a systematic and visual way to present and share your understanding of the relationships among the resources you have to operate your program, the activities you plan to do, and the changes or results you hope to achieve. W.K. Kellogg Foundation Logic Model Development Guide
THE WHY OF LOGIC MODELS
A simple, common sense picture of a program
A way to show a series of if-then relationships
show what the program is to accomplish
Resources to be invested
Activities to take place
Benefits or results
Key element for program planning, management, evaluation, and reporting
A way of thinking
Through a flow chart, shows a chain of events linking what we start with (raw inputs) through desired results
In a linear fashion, helps to “detail” general goals
Summarizes the main elements of a program
Shows the difference between activities and outcomes
Shows what to evaluate
A picture is worth 1,000 words, so let’s draw a picture Inputs Outputs Outcomes / Impacts What is invested What is done What results Core of a logic model Activities What is produced
Inputs - Resources dedicated to or consumed by the program that are available to be used in order that a program can do its work. Inputs may include constraints on the program such as laws, regulations, and requirements (a one year grant).
Inputs, sometimes called resources, can include: staff volunteers money facilities, equipment supplies, other organizational and community resources
Activities – What the program does with the resources it has at its disposal in order to carry out the program. Activities are processes, actions, techniques and events that are a part of the program implementation. By undertaking these activities, presumably changes or results are brought about. Developing curriculum or feeding the homeless are activities.
Outputs – Direct products of program activities. Outputs usually have a number associated with them and measure the volume of work accomplished . Examples would be the number of clients served, the number of volunteers trained, the number of classes held.
Inputs Outputs Outcomes / Impacts What is invested What is done What results Activities Who we reach Note: Some logic models will define outputs as what is done (activities) and who is reached (participants). In such cases, outputs are shown as below. Participation
Very often the words outcome and impact are used interchangeably. By-in-large, that’s what we will do today. For the curious, however, we can broadly distinguish between the two terms as follows:
Outcomes – C hanges or benefits for individuals, groups, organizations expressed as changes in behavior, knowledge, skills, status,attitudes, level of functioning, etc. Outcomes are what participants know, think, or can do. Outcomes answer the very important question, “So What?”
Impact – The ultimate result , the long-term outcome . It is a program’s long-term goal . Impact is the fundamental intended (or unintended) change occurring in organizations, communities, or systems as a result of program activities. According to the Kellogg Foundation, impacts occur within 7 – 10 years, often well beyond the scope of a grant program.
mid-term (action) - behavior, practice, decision-making, policies, and social action); and
long-term (conditions) – social economic, civic, and environmental aka impact
Each outcome along the 3 stage continuum is more complex, and outcomes may take years to achieve. The more “complex” the outcome, the longer achievement may take, the more outside influences, and the less control and influence the program will have.
You may be used to writing outcome objectives for grant proposals. Outcome objectives in grant proposals tend to be more descriptive and specific than those written in the logic models. Outcome objective : 45% of the 50 youth participating in the tutoring program will improve their reading skills by the end of the program. Outcome : youth improve reading skills
A quick summary:
Inputs - What we use
Activities – What we do
Outputs – The amount we do (produce)
Outcomes – What difference we make
Inputs Outputs Outcomes / Impacts Activities Re-sources are needed to operate a program If you have access to them, then you can use them to accomp-lish your planned activities If you accomplish your planned activities, then you hopefully will deliver the amount of product and/or services you intended If you accomplish your planned activities to the extent intended, then participants will benefit in certain ways. If these benefits are achieved, then certain changes to organizations, communities, or systems might be expected to occur. How to Read a Logic Model: U of Wisconsin Extension
LOGIC MODEL for a HEADACHE Situation: Your head is pounding like a steel drum. Pills and water Take Pills Feel Better Resources Actions Results
LOGIC MODEL for a PARENT Situation: While shopping with her health conscious parent, a toddler begins crying and begging for every sweet in the grocery store. healthful distraction for child Give child juice box Child is happy; parent headache is averted Resources Actions Results
LOGIC MODEL for a TRIP Situation: Planning a vacation Flight schedule Money Annual leave Submit leave request Book a flight Arrange for hotel Rent a car Continued good will among co- workers Nervous breakdown averted Resources Activities Outcomes Outputs Signed leave slip Airline tickets Hotel Re-servation Car rental confirma-tion
LOGIC MODEL for an EMPLOYEE Situation: Attending logic model training Friend Car Notepad and pencil Pick up friend for moral support Drive to training Listen attentively Participate in exercises Understand logic model concept Practice examples with friend on way home Include stunning logic model in next grant proposal Inputs Activities Outcomes Outputs 3 exercises completed Volunteer twice 5 ques-tions answered
Inputs Outputs Outcomes / Impacts What are these boxes and what do they show?
Logical relationships between the program’s resources, the activities undertaken, and the anticipated changes or benefits all represented in graphic form.
Logical sequence of actions and events believed to bring about results over a period of time.
Logical reasoning linking investments to results.
Logical way of showing connection of inter-dependent parts that make up the whole.
A hint for “building” a logic model: as you are constructing a logic model, complete each statement for inputs, activities, outputs and outcomes found on the next slide.
In order to accom-plish our set of activities, we will need the following resources: In order to address our problem, we will need to conduct the following activities: We expect if com-pleted or ongoing, these activities will lead to the following changes in 1-3, 4-6 years, and 7 – 10 years: Inputs Activities Outcomes/Impact Outputs We expect once completed or underway, these activities will produce the following evidence of service delivery:
Inputs Outputs Outcomes/Impacts Activities A quick review of definitions: Inputs are the resources dedicated or used by the program such as money, staff, volunteers, facilities, equipment, and supplies. Activities are what the program does with the inputs, in other words, the strategies and techniques that make up the program’s service methodology. Examples for a homeless program might include sheltering, feeding, training, and counseling homeless persons. It is hoped that by undertaking activities, changes or results will be brought about. Outputs are the direct product of program activities, usually measured by number.
Outcomes are changes or benefits for participants, whether individuals, groups, organizations, or communities. Outcomes may be expressed as changes in behavior, knowledge, skills, or level of functioning. Outcomes answer the very important question, “So What?”. And if you care to distinguish, Impact is the ultimate result, the long-term outcome. It is a program’s long-term goal. Impact is a change in organizations, communities, or systems resulting from program activities.
The Game NAME THAT LOGIC MODEL COMPONENT
Try your hand at identifying the following Logic Model components for an Employment Program for Displaced Workers Aged 55 and Older
Participants increase skill in conducting job interviews
Workshop on dressing for success is held
45 job interviews were completed
Changes or benefit such as gain in skills are (definition)
Number of employers participating in the program increases
Program resources are (definition)
Decrease in participant fears about not obtaining employment
25 participants enrolled in program
Presentation on resume preparation given
Participants remain employed 6 months after completing program
10 volunteers were recruited as job coaches
Visit to job services
Direct products of program activities often shown as a number are (definition)
What the program does with resources (definition)
Informational interviews are conducted week 4
Mock job interviews held
Identify the following as an outcome or an impact:
National attitudes of employers toward hiring older workers are changed
10 half day workshops were held
Increased skill in conducting and analyzing interviews
Outcomes are often described as short term, _____, and _____.
Midterm and long term (or impact)
Participants increase skill in preparing resumes
Guest lecturer discusses creative job seeking
What is produced (the product) describes what term
LOGIC MODEL OBSERVATIONS Logic models usually are depicted as a flowchart with boxes and arrows. HOWEVER, logic models may show components and linkages in other ways such as a concept map. They may go top to bottom, bottom to top, or in a circle. Regardless, a logic model should be shown in a single image and in such a way that the end users understand it.
Adding an Interesting Layer of Complexity
EVEN SIMPLE LOGIC MODELS GET A LITTLE MORE COMPLICATED
The Complete Logic Model P R I O R I T I E S S I T U A T I O n Inputs What We invest Activities What we do Outcomes / Impacts Short medium long-term Staff Money Volunteers Time Partners Materials Conduct Deliver Develop Train Counsel # classes # sessions # clients # volunteers Learning Awareness Skills Knowledge Attitudes Opinions Actions Behavior Decision- Making Policies Practice Conditions Social Economic Civic Environ-mental Assumptions External Factors Outputs Products of service delivery
The heart of logic model development
What the program will address
A part of many other factors (sociopolitical, environmental, and economic)
If you incorrectly understand the situation and misdiagnose the problem, everything that follows is likely to be wrong.
What is the problem?
Why is this a problem and what causes the problem?
For whom does this problem exist (a group or subgroup, the community, society)?
Who cares whether the problem is solved?
What do we know about the problem and the people involved?
What research and/or experience do we have and what does it tell us?
The State of Euphoria is increasingly concerned about the unhappy work environment for individuals working for it. A study conducted by the Department of Happiness & Rapture (DHR) showed that employees working for the state 10 or more years are twice as likely to lose their sense of wonder and delight as the general population. Across all State of Euphoria departments, only 10% of agencies allow employees to take enjoyment breaks. The DHR study, backed up by National Institute of Happiness data, suggests that a statewide contentment program will save 1,000 lives per year for each year it is in effect.
Model County Tobacco-Free Coalition is increasingly concerned about the unhealthy work environments for county youth. A recent Chamber of Commerce study showed 75% of county youth with part-time and summer jobs work in the service industry, mainly in restaurants where youth workers are exposed to cigarette smoke. Ten percent of the county's restaurants (non-bars) and 75% of fast-food establishments are voluntarily smoke-free. Research suggests that smoking bans and restrictions in public places not only reduce environmental tobacco smoke exposure but also are associated with lower youth smoking rates and delayed onset of smoking Example from U of WIS Extension
What is the problem?
What causes the problem (why)?
For whom does this problem exist (a group or subgroup, the community, society)?
Who cares whether the problem is solved?
* What do we know about the problem and the people involved, what research and/or experience do we have and what does it tell us?
Mission of the agency,
What others are doing in relation to the problem
Unless resources are unlimited, setting priorities becomes necessary. Priorities are set once the situation is fully analyzed. Think about
INPUTS are The resources and contributions made to the program. Inputs may include time, staff, volunteers, money, materials, equipment, partnerships, and technology among other things. What we have to work with allows the program to create outputs.
Actions we take (what we do). We can conduct meetings, hold events, develop resources, counsel participants, train volunteers, and so forth.
What is done can take the form of activities, services, events, and products.
These efforts must reach people who participate in the program. People may be individuals, groups, agencies, the community, etc.
The accomplishment or product of the activity. Examples might be 50 volunteers recruited; 3 meetings held 7 focus groups conducted OUTPUTS are
The direct results or benefits for individuals, families, groups, communities, organizations, or systems.
Typically, described as short term, medium term, or longer-term.
Outcomes can be positive (increase) or negative (decrease) or neutral. While we would like them to be intended, they may also be unintended.
Examples include changes in knowledge, skill development, changes in behavior, capacities or decision-making, policy development.
as we are using it in this training refers to the ultimate consequence or effects of the program (the long-term outcome farthest to the right on a logic model).
Impact refers to ultimate, longer-term changes in social, economic, civic, or environmental conditions.
Examples are increased life expectancy for former smokers, improved air quality in Maryland, etc.
In common usage impact and outcomes are often used interchangeably.
ASSUMPTIONS are The beliefs we have about the program and the people involved and the way we think the program will work. This is the "theory" that we are able to validate with research and experience. Assumptions underlie and influence the program decisions we make.
Assumptions Assumptions are principles, beliefs, ideas about: The problem or situation. The resources and staff. The way the program will operate. What the program expects to achieve. The knowledge base. The external environment. The internal environment The participants: how they learn, their behavior, motivations, etc.
Assumptions In developing a logic model, we want to make explicit all the implicit assumptions we are making. They may not all be portrayed in the one-page graphic, but we do want to explore and discuss them. Often, inaccurate or overlooked assumptions are the basis for failure or less than expected results.
Juice box (healthful distraction) Give child juice box Child is happy; adult headache / embarrassment is averted Resources Actions Results You are with a toddler in a grocery store when she begins crying and begging for every sweet she sees. Experience says that the child needs to be distracted. So, the logic model shows that first you need to find something that will be healthful such as a juice box. Then, you give the toddler the juice. The end result is that the child is happy and you do not get a headache or feel embarrassed.
Assumptions rooted in the logic model: It assumes that you can find a juice box. It assumes that the toddler will want and will drink the juice. It assumes that the result will be similar to your previous experience when the child has been crying and begging in the grocery store. It assumes that the distraction of the juice box will last until you leave the store.
EXTERNAL FACTORS are The environment in which the program exists that can influence the program's success. External factors include the surroundings in which people live and which influence them, the climate, economic structure, housing patterns, culture, demographic patterns, political environment, background and experiences of program participants, media influence, changing policies and priorities, and so forth. These external factors may have a major influence on the achievement of outcomes.
External Factors External Factors may affect a variety of things including the following: Program implementation Participants and recipients The speed and degree to which change occurs Staffing patterns and resources available. External factors interact with programs. They not only influence the initiative but are influenced by the initiative. A program does not sit in isolation - somehow "outside" or "apart" from its surrounding environment. A program is affected by and affects these external factors.
External Factors Affecting a Program to Keep Low-Income Homeowners in their Homes
The culture of participants regarding accepting assistance (male clients, female staff)
The war in Iraq
Changes in bank foreclosure policies
Challenge yourself to identify whether each of the following 21 phrases is an Input Activity Output Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome) Situation Assumption External Factor Answers are at the end of the presentation.
# 2 – Because of an unusually large amount of inexpensive heroin coming into the area, heroin induced deaths have increased. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
# 3 - Within the community, social attitudes change and smoking is viewed unfavorably by 95% of youth ages 10 – 18. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
# 4 - Graduates of a training program apply the skills they learned on the job. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
# 5 - There is a decrease in traffic accidents at an at-risk intersection. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
# 7 – Children without a male role model do not form relationships as successfully as children that do. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
# 17 – Severe head traumas are increasing because children are not wearing helmets while bicycling. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
# 18 – After advertising, an executive director is hired. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
# 19 – Parents in the parenting program spend more time with their children. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
# 21 – Parents will value their children more than drugs. Input Situation Activity Assumption Output External Factor Short-term/Medium-term outcome Impact (Long-term [ultimate benefit/Impact] outcome)
Looking at logic models developed by others may help in developing your own. The following examples of logic models were selected to illustrate the diversity that can be found in the world of logic models.
Dental Program Sunflower Foundation: Health Care for Kansas For this and more logic model examples from the Sunflower Foundation, go to http://www.sunflowerfoundation.org/html/pdf/07_101_Logic_Model_Examples.doc Change in patient attitude about need for dental care 90% reduction in the # of children and adults with untreated caries 50% increase in medical referrals to dental clinic 90% patient satisfaction for 3 years # of dental operatories # of Dental Staff hired # of patients enrolled in the clinic/year # of patient visits/year Locate/refurbish dental operatory (due date) Hire dental staff (due date) Purchase and test new equipment (due date) Implement new practice (due date) Report program development results at end of the project (due date) Executive Director 3 FTE Dental Staff Dental Equipment Sunflower Foundation grant Other funding Begin a new dental practice Outcomes Outputs Activities Inputs Focus Areas
To view this logic model online, go to http://www.cyfernet.org/training/jr_logic.pdf
To view this and other logic models, go to http://outreach.msu.edu/CapableCommunities/examples.html (Michigan State University)
To view this logic model online, go to http://sanctuaries.noaa.gov/education/evaluation/logic.pdf
This logic model may be found online at http://marathon.uwex.edu/ag/documents/IncLogicModelFinal.pdf
For this and other logic model examples from RUSH (Research Utilization Support and Help), go to http://www.researchutilization.org/logicmodel/examples.html
This logic model is part of Documenting Outcomes in Tobacco Control Programs , University of Wisconsin Extension, and is found at http://www.uwex.edu/ces/pdande/evaluation/pdf/tobaccomanual.pdf
Logic Models are helpful in evaluation. We will briefly discuss two key components:
Asking the right questions and
The first step in evaluation is to develop appropriate questions. Since you can’t evaluate everything about the program, answering selected questions well is better than answering a lot of questions poorly. Good questions help us not to let evaluation get out of control so that too much time is spent collecting data we will never have time to analyze.
University of Wisconsin Extension Parent Child Abuse and Neglect Education Program
Another element to consider when building a logic model is indicators . Indicators are an important part of evaluation. While indicators are associated with all parts of a logic model, we will focus on outcome indicators .
Indicators are items of data tracked to measure how well a program is achieving an outcome. They are linked to the evaluation questions we want answered. In evaluation we must ask, “ What do we want to know?” Indicators explain how we will know it.
In order to evaluate a program, we must have at least one indicator for each outcome. The indicator will help those involved in implementing the program know whether the outcome has been achieved.
An Indicator must
Represent the achievement of the outcome and must be a characteristic or change that is measurable and observable and
Generally will have a number and percent attached to it that summarizes the level of achievement.
Though we will focus on indicators with numbers and percentages (quantitative), know that sometimes indicators will be qualitative. In a drug treatment program, an outcome might be increased self-esteem. Indicators might be 1) Number and percent of positive self-esteem statements checked on XYZ test (quantitative) 2) Self-reported increase in self-esteem (qualitative)
When developing indicators, ask yourself the following questions:
How will I know the outcome has happened?
Can I observe and measure this?
Does the indicator tell me if the outcome has been reached?
Outcome: Teachers are knowledgeable about the signs of child abuse. Indicator Ex: Number and % of teachers who can name 3 signs of abuse. Outcome: The quality of life for disabled veterans increases. Indicator Ex: Number and % of veterans reporting ways their quality of life has improved after completing the program.
Outcome: Binge drinking decreases among teenage youth. Indicator Ex: Number and percent of teens reporting fewer incidents of binge drinking. Outcome: Community members sign up to clean up the stream Indicator Ex: Number of residents on sign up sheets.
Can you identify whether the following examples are indicators or outcomes? The answers are at the end of the presentation.
Drug treatment program Participants stop using illegal drugs Number and percent of participants who are clean at release Number and percent who have not relapsed 6 months after program completion Participants do not relapse
After School Program Participants learn conflict resolution skills Attendance improves Number and percent of who miss less than 5 days during program Number and percent of youth using 1 or more conflict resolution skills
Teen Pregnancy Prevention Program Teens delay pregnancy Number and percent of teens delaying pregnancy until after high school graduation Number and percent of teens delaying parenthood until marriage. Number and percent of teen mothers who do not have another child while in the program
Hospice Program Patients are not in pain Number and percent of patients reporting they are pain free Family members exhibit less stress Number and percent of family members reporting ways in which stress has been reduced Number and percent patients reporting they are at peace with death
To learn more about Logic Models
University of Wisconsin Extension http://www.uwex.edu/ces/lmcourse/#
United Way http://national.unitedway.org/outcomes/
W. K. Kellogg Foundation http://www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf
Answers to THE CHALLENGE Input # 1 - A $10,000 corporation contribution External Factor # 2 - Because of an unusually large amount of inexpensive heroin coming into the area, heroin induced deaths have increased. Impact #3 - Within the community, social mores change and smoking is viewed unfavorably by 95% of youth ages 10 – 18. Outcome #4 - Graduates of a training program apply the skills they learned on the job. Outcome #5 - There is a decrease in traffic accidents at a high-risk intersection. Output # 6 - Twenty prenatal classes are taught. Situation # 7 - Children without a male role model do not form relationships as successfully as children that do. Activity # 8 - A needs assessment is conducted. Input # 9 – A dollar for dollar match Assumption # 10 - Rehabilitation can work. Activity # 11 – Letters are sent to potential clients. Outcome # 12 – Children’s interest in books increases. Impact # 13 – The rate of global warming slows. External Factor # 14 - Change in political party winning an election Input # 15 – Volunteers Output #16 – Forty counseling sessions were held. Situation # 17 - Severe head traumas are increasing because children are not wearing helmets while bicycling. Activity #18 – After interviews, an executive director is hired. Outcome #19 – Parents in the parenting program spend more time with their children. Output #20 – Sixty participants are recruited. Assumption # 21 - Parents will value their children more than drugs.
Drug treatment program Participants stop using illegal drugs - outcome Number and percent of participants who are clean at release - indicator Number and percent who have not relapsed 6 months after program completion - indicator Participants do not relapse – outcome After School Program Participants learn conflict resolution skills - outcome Attendance improves - outcome Number and percent of who miss less than 5 days during program - indicator Number and percent of youth using 1 or more conflict resolution skills - indicator Answers to Indicators vs. Outcomes
Teen Pregnancy Prevention Program Teens delay pregnancy - outcome Number and percent of teens delaying pregnancy until after high school graduation - indicator Number and percent of teens delaying parenthood until marriage - indicator Number and percent of teen mothers who do not have another child while in the program - indicator Hospice Program Patients are not in pain - outcome Number and percent of patients reporting they are pain free - indicator Family members exhibit less stress - outcome Number and percent of family members reporting ways in which stress has been reduced - indicator Number and percent patients reporting they are at peace with death - indicator