This document discusses process evaluations, which examine how a program or intervention is implemented to understand why certain outcomes result. Process evaluations answer questions about what activities were delivered, to whom, how, and how participants responded. The information is used both formatively, to improve implementation, and summatively, to help interpret outcome evaluation results. Key aspects discussed include stakeholder involvement, developing a logic model, defining objectives and measures, data collection procedures, and prioritizing evaluation questions. Both qualitative and quantitative methods can be used.

1. Process Evaluation: Documenting the
‘How” and Understanding the “Why” of
implementation
Sources:
•Pascale Wortley, Immunization Services Division,
NCIRD, 2008
•Jane Bertrand, JHU, lecture
•Barri Burrus et al, RTI
•Ruth Saunders et al, Health Promotion Practice,
2005

2. Definitions
Process evaluation:
Examines whether program activities been implemented as
intended
Answers questions about what has happened as a result of
an intervention’s implementation
• To whom, what, when, where, how much
intervention has been delivered/received by
participants?
• How have participants reacted to the intervention?

3. 3
Uses of Process Evaluations
Barri Burrus et al, Process Evaluation , RTI, 2008
Formative
• To identify what is working
well and what needs
improvement
• To explore whether
intervention may be
harming participants
Summative
• To use data as mediators
and moderators in analysis
of impact
– Exploration of dosage
treatment effects
• To help ensure intervention
is not falsely rejected
• To document what was
done if outcome warrants
future replication of
intervention

4. ActivitiesInputs Outputs
Intermediate
Effects/
Outcomes
Short-term
Effects/
Outcomes
Long-term
Effects/
Outcomes
Context
Assumptions
Stage of Development
What the program
and its staff
actually do
Results of
activities: who,
what will change

5. ActivitiesInputs Outputs
Intermediate
Effects/
Outcomes
Short-term
Effects/
Outcomes
Long-term
Effects/
Outcomes
Context
Assumptions
Stage of Development
Outcome
evaluationProcess
evaluation

7. What are we trying to learn through
process evaluation?
– Do we have the right mix of activities?
– Are we reaching the intended targets?
– Are the right people involved as partners,
participants, and providers?
– Do the staff/volunteers have the necessary
skills?
– How well do our activities meet with our
priorities

8. Involving stakeholders
• Gain broader perspective, avoid blind spots, try
to ensure utilization of results
• Key stakeholder: Ministry of Health
– Those served or affected by activity
– Those involved in program operations
– Those in a position to make decisions about the
activity (decision factors: cost, acceptability, etc)
• For a manageable process, the list of
stakeholders must be narrowed to primary
intended users

9. Steps in Conducting Process Evaluations (Burrus et al)
• Identify all key components of program
• Create or revise pathway/logic model including process
variables
• Determine objectives of process evaluation with
stakeholders who will use the evidence!
• Ensure that program implementation, dosage, and fidelity
are measured
• Create measures: determine data sources and instruments
• Determine measurement procedures and schedule
• Collect data
• Determine how process measures will fit in the analysis
• Include process data in impact evaluation
9

11. Some useful concepts
• Reach
• Quality of implementation
• Appropriateness
• Satisfaction
• Barriers

12. Reach
• Degree to which intended audience
participates in intervention
– Percent of target population that heard
messages
– Percent of persons attending influenza
vaccination clinic that do not usually get
vaccinated
– Percent of mothers contacted by peer-to-peer
counselors

13. Quality of implementation
• Was activity implemented properly, according to
standards or protocol
– Was feedback session conducted as per guidelines?
– Are staff interacting with right people?
– Did peer counselors interaction with mothers follow
training or protocol?
– Is training of staff for a given activity standardized?
– Was training curriculum delivered in its entirety?

14. Appropriateness/Acceptability
• Interventions or messages that are delivered
may only be effective if judged appropriate by
target population, or if designed in manner to
achieve objective
– Did messages “speak” to target audience?
Note: short term outcomes also related to
appropriateness, e.g.
Did knowledge or skills increase as a result of
training?
Was information provided in training subsequently
used?

15. Satisfaction
• The extent to which participants are satisfied
with training or interaction may influence
subsequent behavior
– Provider satisfaction with session
– Peer counselor satisfaction with training

16. Barriers
• This attempts to understand why something
didn’t happen, and may identify key
environmental variables
– Reasons mothers didn’t bring their children to the
clinic (transportation, clinic schedule, other?)
– Reasons providers didn’t implement
recommendations (resources?)
– Reasons seniors didn’t come for influenza vaccination
clinic (unaware of clinic, concern about vaccine, no
perceived need?)

17. Prioritizing evaluation questions
Is the question:
• Important to your program staff and
stakeholders?
• Does it reflect key goals and objectives of your
program?
• Does it reflect key elements of your
pathway/logic model?
• Will it provide information you can act upon to
make program improvements?
• Can it be answered using available program
resources?
• Are there any available data sources?

18. Process evaluation: Methods
• Quantitative
• Qualitative
• Combination

19. Qualitative methods
• Case studies
• Structured or semi-structured interviews
• Focus groups
• Direct observation
• Reviews of program meeting minutes,
progress reports

20. Quantitative methods
• Surveys
• Information collected from program
participants
• HIS, M&E