School of Management Program Evaluation MPA 513 Week 3 School of Management Policy in the NewsReview Needs Assessment / StakeholdersProcess EvaluationsExercise:Performance MonitoringExercise: City Stat exampleQuestions and Conclusions Class Overview and Objectives * School of Management In the News Public Administration Evaluation in the News * School of Management Logic Models Stakeholders Review * School of Management Involving StakeholdersGain broader perspective, avoid blind spots, try to ensure utilization of resultsKey stakeholders:Those served or affected by activityThose involved in program operationsThose in a position to make decisions about the activityFor a manageable process, the list of stakeholders must be narrowed to primary intended users School of Management Evaluating Internal Processes “Now that this is the law of the land, let’s hope we can get our government to carry it out.” John F. Kennedy School of Management What is a Process Evaluation? Process (formative) evaluations are aimed at enhancing your program by understanding it more fully, and whether it is functioning as intended. Process evaluations study what is being done, and for whom these services are provided * Evaluators often distinguish between process/implementation/formative vs. outcome/impact/summative evaluations. School of Management Process vs. Outcome Evaluation Process (Formative) – program managers, front-line staff, program designers, evaluation professionals and other internal and external entities focused on wanting to know why the program (or class of programs) is or is not working and what sort of program adaptations are appropriate. Outcome (Summative) measures – legislators, accounting entities, interest groups, other levels of government, and other external entities focused on accountability or accreditation. Evaluators often distinguish between: process or implementation (formative) vs. outcome or impact (summative): * School of Management Illustration of Process Evaluation Formative Evaluation Research Examines Inside The Program At “The Process” (1)Jablonski, J.R. Total Quality Management. Technical Management Consortium Albuquerque, NM. * School of Management Organization Change and Process EvaluationProcess evaluation supports a program administrator’s desire to correct program deficiencies.Problem-solving orientation is different from evaluations that are more outcome-focused.Process intervention model provides a framework for planned organizational change. “We are interested not so much in whether X causes Y as in the question , if Y is not happening, what is wrong with X.” –Sylvia et al, p.70 * Sylvia, Sylvia, and Gunn. 1997. Program Planning and Evaluation for the Public Manager. Waveland Press. * School of Management Activities Inputs Outputs Intermediate Effects/ Outcomes Short-term Effects/ Outcomes Long-term Effects/ Outcomes Context Assumptions Stage of Develo.

1. School of Management
Program Evaluation
MPA 513
Week 3
School of Management
Policy in the NewsReview Needs Assessment /
StakeholdersProcess EvaluationsExercise:Performance
MonitoringExercise: City Stat exampleQuestions and
Conclusions
Class Overview and Objectives
*
School of Management
In the News
Public Administration
Evaluation in the News
*
School of Management
Logic Models

2. Stakeholders
Review
*
School of Management
Involving StakeholdersGain broader perspective, avoid blind
spots, try to ensure utilization of resultsKey stakeholders:Those
served or affected by activityThose involved in program
operationsThose in a position to make decisions about the
activityFor a manageable process, the list of stakeholders must
be narrowed to primary intended users
School of Management
Evaluating Internal Processes
“Now that this is the law of the land, let’s hope we can get our
government to carry it out.” John F. Kennedy
School of Management
What is a Process Evaluation?
Process (formative) evaluations are aimed at enhancing your
program by understanding it more fully, and whether it is
functioning as intended.
Process evaluations study what is being done, and for whom
these services are provided

3. *
Evaluators often distinguish between
process/implementation/formative vs.
outcome/impact/summative evaluations.
School of Management
Process vs. Outcome Evaluation
Process (Formative) – program managers, front-line staff,
program designers, evaluation professionals and other internal
and external entities focused on wanting to know why the
program (or class of programs) is or is not working and what
sort of program adaptations are appropriate.
Outcome (Summative) measures – legislators, accounting
entities, interest groups, other levels of government, and other
external entities focused on accountability or accreditation.
Evaluators often distinguish between: process or
implementation (formative) vs. outcome or impact (summative):
*
School of Management
Illustration of Process Evaluation
Formative
Evaluation
Research
Examines
Inside The
Program

4. At “The
Process”
(1)Jablonski, J.R. Total Quality Management. Technical
Management Consortium Albuquerque, NM.
*
School of Management
Organization Change and Process EvaluationProcess evaluation
supports a program administrator’s desire to correct program
deficiencies.Problem-solving orientation is different from
evaluations that are more outcome-focused.Process intervention
model provides a framework for planned organizational change.
“We are interested not so much in whether X causes Y as in the
question , if Y is not happening, what is wrong with X.” –Sylvia
et al, p.70
* Sylvia, Sylvia, and Gunn. 1997. Program Planning and
Evaluation for the Public Manager. Waveland Press.
*
School of Management
Activities
Inputs
Outputs
Intermediate Effects/ Outcomes

5. Short-term Effects/ Outcomes
Long-term Effects/
Outcomes
Context
Assumptions
Stage of Development
Illustration Using the Logic Model
(1)CDC Presentation on Process Evaluation:
www.cdc.gov/vaccines/programs/progeval
*Lots of this will have been decided when you “staked the
claim” but worth considering again.How far out the chain is
determined by several considerations. These are not a formula
that will produce a result, but rather separate conversations
which may lead to contradictory conclusions. In that case, then
a need to reconcile where to put the resources.Key questions
include:Ripple effect or reasonableness: This one should have
been answered in “staking the claim”. The question is how far
out the chain of causation is it reasonable to expect the
intervention to have an impact. Through the process of
refinement in staking the claim, the program should have made
decisions about synchronizing the level and mix of activities
and the desired “claim.”Accountability: This is what the major
stakeholders are expecting of you. As mentioned, they might be

6. “wrong” and err in either a too ambitious or not ambitious
enough direction. Still, in designing an evaluation, the
information of most importance to the stakeholder is a major
consideration.Measurability: As mentioned, this should never
be the reason to scale back an evaluation, except as an
admission that NOW we are unable to measure this effect.
School of Management
Activities
Inputs
Outputs
Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/
Outcomes
Context
Assumptions
Stage of Development
What the program and its staff actually do
Results of activities: who, what will change

7. *
School of Management
Activities
Inputs
Outputs
Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/
Outcomes
Context
Assumptions
Stage of Development
Sphere of control
Sphere of influence

8. *
School of Management
Activities
Inputs
Outputs
Intermediate Effects/ Outcomes
Short-term Effects/ Outcomes
Long-term Effects/
Outcomes
Context
Assumptions
Stage of Development
Outcome evaluation
Process
evaluation
*

9. School of Management
Why Study Implementation?Implementation is the most over-
looked phase of the process approach to program change
Change is frequently resisted and nearly always disruptive
Critical components of planned change:
Clear agenda and chain of command;
Proper specificity for assigned responsibilities; and
System for managing and monitoring implementation
*
School of Management
When Can Process Evaluation be Useful?When expected
outcomes are not observed, process evaluation can suggest
reasons.
When outcomes are positive, process evaluation may help
understand which components are especially important.
When an activity is newly implemented and it is too early to
expect changes in outcomes.
When an outcome evaluation is not feasible due to resource
constraints.
School of Management
Look at Coverage and BiasHow many persons are receiving

10. services?What proportion of those who might need the
service(s) actually used it/them?Has the service, or network of
services served the intended clients?What were the
demographics and clinical characteristics of clients? Are certain
groups over/under represented? Is there creaming?What
proportion of clients completed treatment (or received full
services) and what were the characteristics of those who
dropped out?Are clients aware of the program?
Coverage deals with whether the program is reaching the
appropriate population, and answers such evaluation questions
as:
*
In general, process evaluations pose questions about coverage
and process
School of Management
Common Coverage Problems“Self-selection” – bias in coverage
due to voluntary over-participation or “creaming”
Under-coverage due to limited funds, poor public education,
poor access, or poor design – e.g., population too broadly
defined, such as all poor mothers
Over-coverage – reaching more than the intended group, e.g.,
Sesame Street, or special education
*
School of Management
Look at Organizational FunctionAre they receiving the proper

11. amount, type, and quality of services? What are waiting
times?Is staffing sufficient in numbers and competencies for the
function?Is the program well-organized? Do staff work well
together?Are resources, facilities, and funding adequate?Is the
program in compliance with applicable professional and legal
standards? Are participants satisfied with services?Is there
appropriate discharge and follow up?
Process questions deal with whether service delivery and
support functions are consistent with program design specs and
standards:
*
School of Management
Common Implementation Failures“Non-programs”, poor quality,
or incomplete intervention, e.g., Title I implementation had no
proof of services for targeted audience of poor children
Wrong or inappropriate intervention, e.g., something that is too
sophisticated to go to scale or resisted
Unstandardized intervention, e.g., Community Action Program
and “maximum feasible participation” on the part of the poor
*
School of Management
Coverage vs. Function Is the program serving clients it was
intended to serve?
How many direct contact hours does each client receive?

12. What is the average age of program participants?
What percent of clients complete the program?
From Workbook 4, Process Evaluations, WHO
http://www.unodc.org/docs/treatment//process_evaluation.pdf
Test your knowledge. Is the following a coverage question or a
function question (1):
*
School of Management
Process Evaluation MethodsQuantitativeQualitativeCombination
School of Management
Qualitative MethodsCase studiesStructured or semi-structured
interviewsFocus groupsDirect observationReviews of program
meeting minutes, progress reports
School of Management
Quantitative MethodsSurveysInformation collected from
program participantsAgency records
School of Management
Summary Points
Program process evaluation tells you if your program operates
as intended;Service utilization issues typically break down into

13. questions about coverage and organizational functionCoverage
problems can occur due to self-selection, under-coverage, and
over-coverage Common implementation failures are incomplete
interventions, delivery of the wrong intervention, and
unstandardized or uncontrolled interventions
*
Design and Measurement Rubric
Objectives
Excellent
Good
Poor
Unacceptable
1. Research Design
Appropriate analytical method/design(s) have been chosen, with
complete justification of the choice(s).
If a process evaluation, describes the standard for comparison.
If an outcome study, identifies the design (experimental, quasi-/
non-experimental).
(3.0-2.9 points)
Appropriate analytical method(s) were chosen, but justification
and explanation could be better developed.

14. (2.8-2.7 pts)
Appropriate analytical methods were chosen, but justification
and explanation is incomplete or poorly developed.
(2.6-2.5 pts)
Inappropriate analytical methods were chosen and/or
justification of those choices is incomplete or weakly
developed.
(2.4 - 0 pts)
2. Measurement and Data Collection Plan
Thorough explanation of data collection and sampling plan,
including (where appropriate), the standards for comparison,
who the treatment and control group are, how large they are and

15. how they are selected
Analytical methods thoroughly and clearly described, including
using established scales where appropriate, discussing when and
where measures will be taken, and how they will be used.
(3.0 – 2.9 points)
Explanation of data collection and sampling plan thorough, but
could be more strongly tied to results and conclusions.
Analysis plan is complete, but minor errors in application of
chosen analysis method(s).
(2.8-2.7 pts)
Explanation of data collection and sampling plan incomplete
and/or weakly tied to results and conclusions.
Analysis plan is complete but significant or numerous errors in
application of chosen analysis method(s) detract from paper.
(2.6-2.5 pts)
Explanation of data collection and sampling plan very weak or
missing.

16. Analysis is inaccurate and/or incomplete.
Does not use correct analytical method(s) and/ or they are
incorrectly applied and interpreted.
(2.4 - 0 pts)
3. Strengths and Limitations of Approach
Lists all the merits and limitations of the design, and steps
taken to minimize any threats.
(2.0-1.9- points)
Merits and limitations of the design and steps taken to minimize
threats are incomplete.
(1.8-1.7 pts)
Missing more than two of limitations of the design and steps to
minimize them.
(1.6-1.5 pts)
Fails to list the merits and limitations of the design, and steps
taken to minimize any threats.
(1.4 pts)
4. Quality of Writing and Grammar.
Scholarly style that is also highly readable.
Grammar, word choice, spelling or punctuation errors are very
rare, no more often than once every 5-6 pages.

17. (2.0 – 1.9 points)
Generally readable but with a few awkward or unclear passages.
Grammar, word choice, spelling and punctuation errors occur,
but are rare and do not detract from the paper.
(1.8 – 1.7 pts)
Awkward or unclear passages occur as often as once per page.
Grammar, word choice, spelling and punctuation errors occur as
often as once per page and detract.
(1.6-1.5 pts)
Awkward or unclear passages occur more often than once per
page.
Grammar, word choice, spelling and punctuation errors occur
more often than once per page and substantially detract from the
paper.
(1.4– 0 pts)
Total Objective = 10 (100%)

18. Program Brief
Selected Program to be Evaluated
The program selected for the proposal paper is the Summer
Youth Employment Program (SYEP) implemented by the
Mosholu Montefiore Community Center (MMCC). MMCC is a
very well-known Center in New York City that receives
millions of dollars’ worth of public funds, every year, to
provide youth, senior and community development programs
throughout the City (MMCC, 2017).
Benefits, Beneficiaries & Scope of the Program
The SYEP program provides young adults, ages 17 through 24,
and youth as young as 14 to 16 years old, with proper working
papers, an opportunity to gain employment for up to eight
weeks during the summer months of July and August. MMCC is
able to provide these services through City, State and Federal
funders including NYC Department of Youth & Community
Development (DYCD) and the NYS Department of Labor
(DOL). The program must meet certain funding criterion to
continuously receive the funding.
One of the goals of MMCC’s SYEP program is to “provide a
valuable work experience at job sites that are safe and well
supervised” (NYC DYCD, 2015), while decreasing youth
unemployment and poverty for at-risk youth. This goal
encompasses assisting youth in distinguishing what their career
interests may be, while exposing them to different industries
within the workforce. The program’s main objective is to
establish good work-habits, professional development, financial
literacy and eventually, long-term employment.
Problems to be Addressed by the Program
While the youth-employee applicant is selected via a lottery,

19. there are specific eligibility requirements in place, to assure
that the youth that come from lower-income, impoverished
homes and neighborhoods, have the opportunity to obtain
gainful employment through this program.
It is pertinent that programs like the SYEP program, continue to
be evaluated, critiqued, then continuously improved. Although
cliché, the statement that “youth is the future”, is a very true
statement. It is important to have programs work to create
opportunities for quality education and gainful employment for
young people, who may not be able to easily access it on their
own. A decrease in quality of these programs could result in at-
risk youth engaging in risky behavior, remaining uneducated
and financially illiterate, resulting in an increase in the
unemployment and poverty rates.
The SYEP program works to address what could be, lack of
examples of professional behavior or steady income, if a youth,
for example, comes from a single-parent home where the parent
may be unemployed. The program partners with the schools and
financial institutions to provide education, training and
knowledge for long-term success for the young candidates,
including professional development and financial management
literacy.
References
NYC Department of Youth & Community Development (2015).
Jobs & Internships. 2014 NYC
SYEP Annual Summary. Retrieved from
http://www.nyc.gov/html/dycd/html/jobs/jobs.shtml
Mosholu Montefiore Community Center (2017). Youth
Employment. Retrieved from
https://www.mmcc.org/youth-employment/
Rationales

20. Assumptions
Employment-based
programs can induce
learning and help close
the “opportunity gap”
between youth
professional
development and
employer demand
Youth employment can
be a critical facilitator
of positive youth
development and
preparation for
adulthood. It can
improve educational
outcomes, minimize
learning loss and
contribute to the
family budget.
Contribution to the
family budget can help
reduce the poverty
rate and contribute to
the local economy
through steady work
and steady pay.
Youth in low income
neighborhoods want to
work in meaningful
jobs while contributing

21. to family income, but
may lack support and
access to opportunities
and ways to display
their skillsets,
Resources
5 Training
Developers/Facilitators
5 Recruiters
1 Program Director
1 Intern
2 Financial Advisors
Activity Groups
Outreach/Recruitment
Professional
Development
Matching
Post-Placement
Education & Support.

22. Outputs
50 Interviews per
week
40 hours/1 week of
Professional
Development.
Onsite, job placement
visits and follow-up
Financial advisors from
local banks provide
financial literacy
workshops, once a
week.
Intermediate-Term Outcomes
Work readiness for
participating youth.
Successful placement
of 70% of participants.
Increased school
attendance for at least
60% of participants.

23. Pre-employment skills.
Participants maintain
steady employment for
at least 6 months.
Short-Term Outcomes
Obtain skill-set and
confidence to interview
for employment.
Recruit applicants,
from targeted, low-
income neighborhoods
Gain temporary
employment.
Learn future job
seeking skills
Learn money
management and
investment techniques
Youth participants have
an increase desire to
go and stay in school.
Problem Statement
There is a high unemployment rate, high poverty, low-school
attendance and barriers to
quality financial education for youth in low income
neighborhoods.

24. Logic Model Diagram:
MMCC's Youth Employment Logic
Model
Goal
Increase youth employment, school attendance
and financial literacy and planning, while
decreasing the poverty rate for youth, through job
placement, training and educating.
Long-Term Outcomes
Increase family income
Decrease of unemployment rate in low-income
neighbhorhoods
Higher high school graduation rates.
Increase in participants confidence resulting in
decrease of fear about not obtaining employment.
Long-term, permanent employment for at least
through the duration of college years.
Please look at attachments first before agreeing. I will need this
in 6 hours.
Attachments are as follows:
· Assignment Instructions (assignment: Data Collection Plan for
future evaluation)
· Lecture Slides
· Brief of Program to be evaluated (for your reference)
· Rubric/Goals for assignment
If you find that a couple more hours are needed, you should let
me know. I prefer QUALITY over speed.