The document outlines a process evaluation plan for the ACTION case study program. It includes: 1) Engaging key stakeholders and defining their interests 2) Describing the program's logic model and key process objectives 3) The process evaluation design which uses both qualitative and quantitative methods to assess program reach, fidelity, dose, and satisfaction. 4) Details on data collection, analysis, and recommendations to improve the program based on findings.

1. Group 1: Cheryl Tang, Divya Balasubramanian,
Hsu Yee Wint, Lai Yi Feng, Valerie Koh
CO5222 Program Evaluation
Assignment #3
ACTION Case Study
Process Evaluation Plan

2. Outline
1. Purpose of Evaluation
a. Engaging stakeholders
b. Roles and responsibilities of evaluation team members
2. Description of Program Aspects to be Evaluated
a. Logic model
b. Key process objectives
3. Process Evaluation Design
4. Data Collection and Analysis
5. Recommendations and Dissemination Plan

3. Framework for Program Evaluation in Public Health

4. Engaging Stakeholders
Stakeholder Name Interest or Perspective
MOH ● Interventions leading to improved health outcomes
● “Cost - effectiveness” of the interventions
AIC – Program Owner ● Capabilities of ACTION teams
● Effective partnership with step-down care facilities
● Development and sustainability of the program and
resources involved
CMB & Hospital
Management
● Burden of the workload on involved staff
● Reduction in hospital’s readmission rates / medical
errors upon discharge

5. Engaging Stakeholders
Stakeholder Name Interest or Perspective
Hospital Care Transition
teams (clinicians, nurses and
case managers)
● Accuracy of decision making tool
● Adequacy of recruited patients
● Clear referral mechanism
● Extent of patient’s needs addressed throughout
the program
Patients & Caregivers ● Clarity of information and communication
● Satisfaction level towards care coordinators
Community Providers (GPs,
Polyclinics, FMCs, Social
Support Groups)
● Clear referral mechanism in place to enable
smooth transition

6. Members Roles and responsibilities
Principal evaluator
(Program Director)
● Overall coordinator, ensure evaluation carried out as
planned.
● Data interpretation and final report vetting.
● Communicate and Disseminate findings to key stakeholders
Co-evaluators (Program
Managers)
● Study design.
● Pilot testing of survey questionnaire and topic guides.
● Implementation of evaluation plan.
● Data collection, analyze and interpret findings
● Drafting of report
Statistician (external
evaluator)
● Quantitative Analysis
Sociologist/Psychologist
(external evaluator)
● Draft FGD & IDI structure
● Perform thematic analysis
Roles and Responsibilities of Evaluation Team Members

7. Logic Model

8. Key Process Objectives
Stages of Program Process Objectives
At Admission Increased coverage of target population for enrolment
into ACTION
Admission to Discharge Early initiation of discharge planning for patients
according to their individual needs
Post-discharge Increased utilisation of post-discharge interventions
● Follow-up telephone calls
● Home visits
● Referrals to appropriate community service
providers

9. Evaluation Design
Time frame of evaluation: 4 years over entire implementation of the program.
Evaluation questions will focus on key process objectives and mapped onto the
following areas of evaluation:
1. Reach/Utilization: Coverage analysis from hospital/program records
2. Fidelity/Dose: Adherence to procedures from program records review
3. Satisfaction: Patient/caregivers and staff feedback from surveys, interviews,
focus group discussions.
A combination of qualitative and quantitative methods will be employed for specific
evaluation questions.

10. Timeline for Process Evaluation
Activities Month
Year 1
Y2 Y3 Y4
1 2 3 4 5 6 7 8 9 10 11 12
Coverage analysis
- Participation by target group
Quarterly reports
Review of program records
- Fidelity and dose of services
delivered
Quarterly reports
Engagement of participants
and relevant stakeholders for
satisfaction ratings and
feedback on the program
Yearly reports

12. Evaluation variables
● Dependent variables = outcome variables from evaluation questions
● Independent variables = Factors to stratify evaluation results by
○ Clinical disciplines
○ Individual care coordinators
○ Geographical regions of patients/clients

13. Data Collection and Data Analysis

14. Process Evaluation: Reach /Utilization
Evaluation Question
Data
Collection
Method
Source(s) of Data
Process Indicator(s) and
Standard/Criteria (if applicable)
Data Analysis
How many patients
were enrolled in the
program?
Quantitative Program records Number of participants enrolled per
hospital per quarter.
Total number of patients enrolled in
ACTION
What proportion of the
target population
participated in the
program?
Quantitative Hospital admissions and
program records
Not more than 20% of patients who are
assessed as eligible by the care
coordinator decline to participate in the
program per quarter.
Numerator: # of Patients enrolled in
ACTION program in the quarter
Denominator: # of patients assessed as
eligible for ACTION in the same quarter
Has the program
served the intended
clients?
Quantitative Program records
- Patients’
demographic
characteristics
- Past admissions and
medical history
- Social needs
assessment
At least 80% of participants enrolled
per quarter fulfill AIC’s selection
criteria.
Numerator: # of patients who satisfy the
eligibility criteria of ACTION
Denominator: # of patients enrolled in
ACTION
What was the program
dropout rate and
characteristics of those
who dropped out?
Quantitative Program records
- Exit records
Not more than 10% of participants
drop out prior to being discharged from
the program by the care coordinator, at
the end of the first quarter of the
program.
Numerator: # of patients who
discontinue service before planned
closure/ exit assessment
Denominator: # of patients enrolled in
ACTION

15. Variable
(Patients enrolled in ACTION)
Patients discharged from ACTION Patients dropped out from ACTION
N (%),Mean P-value N(%), Mean P-value
Age
Gender
Ethnicity
Chinese
Indian
Malay
Socio-economic status (Ward Class)
Geographic area
Number of Outpatient visits in previous year
Social Needs Assessment score
Average Length of stay in Hospital
Care Coordinator Incharge
Department Admitted to
Table: Patient Dropout Analysis

16. Process Evaluation: Fidelity
Evaluation
Question
Data
Collection
Method
Source of Data
Process Indicator(s) and
Standard/Criteria (if
applicable)
Data Analysis
Was there
standardized
training provided
for the CCs?
Quantitative Staff training
attendance records
All CCs completed the
prescribed trainings on the
ACTION program.
Numerator: # of recruited CCs who
completed the prescribed trainings
Denominator: # of CCs recruited per year
Did the CCs follow
the protocol in
screening for high-
risk patients?
Quantitative Hospital
admissions and
program records
At least 80% of patients
admitted per quarter are
screened for eligibility by the
CCs within 24 hours of
admission.
Numerator: # of inpatients who were
screened within 24 hours of admission
Denominator: # of patients admitted per
quarter
Was there a proper
needs assessment
conducted for
identified high-risk
patients?
Quantitative Program records At least 80% of patients who are
identified as high-risk are
referred to a CC for a needs
assessment using the InterRai
assessment suite per quarter.
Numerator: # of high-risk patients who
were referred to a CC for a needs
assessment using the InterRai
assessment suite
Denominator: # of patients admitted per
quarter who are eligible for enrolment
based on AIC’s selection criteria

17. Process Evaluation: Dose
Evaluation
Question
Data
Collection
Method
Source(s) of Data
Process Indicator(s) and
Standard/Criteria (if applicable)
Data Analysis
What
interventions,
how much and
how often did
the participants
receive the
interventions?
Quantitative Patients’ case notes
and program records
All participants have a documented,
personalized care plan, prior or
upon their discharge from the
hospital.
Numerator: # of enrolled patients who have a
documented, personalized care plan, prior or
upon their discharge from the hospital
Denominator: # of patients enrolled in
ACTION per quarter
Program records
- Activities log of
telephone calls and
home visits
conducted by the CC
At least 75% of participants receive
a follow-up telephone call from their
assigned CC within 7 days after
discharge.
Numerator: # of enrolled patients who
receive a follow-up telephone call from their
assigned CC within 7 days after discharge
Denominator: # of enrolled patients
discharged per quarter
At least 75% of participants receive
a home visit by their assigned CC
within 1 month after discharge.
Numerator: # of enrolled patients who
receive a home visit from their assigned CC
within 1 month after discharge
Denominator: # of enrolled patients
discharged per quarter

18. Process Evaluation: Dose (cont’d)
Evaluation
Question
Data
Collection
Method
Source(s) of
Data
Process Indicator(s) and
Standard/Criteria (if
applicable)
Data Analysis
What
interventions,
how much
and how often
did the
participants
receive the
interventions?
Quantitative Program
records
- Date of
referrals
- Date patient
contact by
community
service
providers
- Description of
services
provided
At least 75% of participants
with referrals to community
service providers are
contacted by their first service
provider within 7 days after
discharge.
Numerator: # of enrolled
patients with referrals to
community service
providers who are
contacted by their first
service provider within 7
days after discharge
Denominator: # of
enrolled patients
discharged per quarter
with referrals to
community service
providers

19. Patient Data Fields
Hospital,
Program
records
Hospital
records
Program
records
Program
records
Patients’ case
notes,
Program
records
Hospital
records
Program
records
Program
records
Program
records
Patient
identifiers
Date and
time of
admission
Date and
time
screened
by CC
Date of
needs
assessment
by CC
Care plan
documented?
Date of
discharge
Date of
follow-up
telephone
call
Date of
home
visit by
CC
Referral to
community
service
providers
SXXXXXX 11.10.15
(09:00)
11.10.15
(14:00)
11.10.15 Present /
absent
13.10.15 18.10.15 1.11.15 Contacted
by service
provider on
15.10.15

20. Process Evaluation: Quality of Implementation
Evaluation
Question
Data
Collection
Method
Source of Data
Process Indicator(s) and
Standard/Criteria (if
applicable)
Data Analysis
What were the
strengths,
barriers and
challenges in
implementation?
Qualitative
method
Sampling method -
Purposive sampling (all the
healthcare workers from
hospital and care
coordinators)
Sample size - 2 FGD with 4-8
per each FGD
Instrument - open ended
questions
Mode of administration -
2 FGD
(1) FGD with healthcare
workers who communicate
with the care coordinators
(2) all care coordinators
Feedback from care-
coordinators on the
strengths, barriers or
challenges in implementation
and communication with
healthcare workers.
Feedback from healthcare
workers on the strengths,
barriers or challenges in
implementation and
communication with care
coordinators.
Thematic analysis to find
out the strengths, barriers
and challenges during
implementation.

21. Process Evaluation: Quality of Implementation
Evaluation
Question
Data
Collection
Method
Source of Data
Process
Indicator
Data Analysis
What were
participants’
perceptions of
the program
and care
coordinators?
Mixed
method
(Sequential
design)
Qualitative method
Sampling method - Quota sampling (who complete the program
and who drop the program )
Sample size - about 15 participants (until reach the data
situation)
Instrument - Open-ended questions.
Mode of administration - IDIs with patients and their caregivers
Qualitative
feedback
from patients
and their
caregivers
regarding
their
experience
with the
program and
care-
coordinators
Thematic analysis to
find out the patient
satisfaction towards
program and care
coordinators during the
program regards
1)Discharge process
2)Experience with care
coordinator
3)Experience with
Program
Quota sampling of
participants for IDI
participant
complete the
program
participant drop out from the
program
care coordinator 1 3 1
care coordinator 2 3 1
care coordinator 3 3 1
care coordinator 4 3 1
Sample questions for the interview;
How do you feel about the program before, during and discharge
from the program?
How do you feel about the care coordinators?

22. Process Evaluation: Quality of Implementation
Evaluation
Question
Data
Collection
Method
Source of Data
Process Indicator(s)
and
Standard/Criteria (if
applicable)
Data Analysis
What were
participants’
perceptions of
the program
and care
coordinators?
Mixed
method
(Sequential
design)
Quantitative method
Sampling method - convenience sampling
(who complete the program)
Sample size - all the participants who enrolled
in the program and completed the program
Instrument - Patient satisfaction survey
Mode of administration -
Self administered questionnaire upon
discharge from the program
At least 60% of
participants have good
perception towards care
coordinators and
program.
Calculate the % of
participants who rate
the program and care
coordinators as good
at the end of the first
year of the program
5 point likert scale - 1:strongly disagree, 2:
disagree, 3:neutral, 4:agree, 5:strongly agree
Sample questions:
● CC explain details about information on
medication.
● I understand the information provided by
CC.
● I have confidence in CC.

23. Recommendations and
Dissemination Plan

24. Proposing evaluation recommendations
● At the end of process evaluation, findings will be mapped into conclusions
○ Connecting evaluation questions to key process objectives in admission, discharge and post-
discharge activities
○ Considering strengths, value and meaning of the findings
● From the conclusions, recommendations for future directions can be made
○ Developing the next steps for adjustments to ongoing or future programs
● Conclusion and recommendations should be discussed during review meeting
among key program staff and stakeholders (program owner, implementation
site’s management)
○ Deciding and prioritizing recommendations
○ Set target dates for implementation
○ Depending on the nature of the adjustments, different stakeholders may be required to act

25. Linking evaluation questions to decisions
Independent
variables
Purpose Potential Recommendations
Clinical
disciplines
Identify difficulties and challenges faced by
each department
Review allocation of resources across
disciplines
Geographical
regions of
patients/clients
Evaluate performance by patients/clients’
location post-discharge
Identify reason(s) for poor performance
Specific recommendations for program
owner and community partners (e.g.
modifications to administration of outreach
activities, suggestion to expand
geographical coverage by community
providers)
Individual care
coordinators
Evaluate performance by each care
coordinator and correlate quantitative results
from coverage analysis with qualitative data
Identify reasons for poor performance
Specific recommendation for individuals or
group of individuals (e.g. counselling,
refresher training for staff)

26. Implementation of evaluation recommendations
Aspects of findings/recommendations Action by
Reach of the intended target audience ( no. of
patients recruited, no. of referrals - physician
referral to ACTION and patients’ acceptance)
Program owner and key program staff to
enhance buy-in from frontline care providers
such as physicians, and patients/caregivers
Utilization of services ( no.of home visits;
telephone follow-ups)
Program owner and key program staff for re-
allocation or resources and re-prioritization of
objectives
Fidelity of services provided ( proportion of
cases screened within 24 hours after admission
& the proportion of cases screened accordingly
to criteria/screening tools)
Key program staff and institution
management. May include key opinion
leaders of front-line staff it recommendation
requires adjustments to existing workflows
Satisfaction - feedbacks collected and barriers
identified via survey, focus groups or interviews
Frontline care providers if recommendation is
care-specific. Institution/partner management
if recommendation is system-specific

27. Target audience
● Purpose of the final report on process-evaluation:
○ To understand, report and discuss learning points from ACTion implementation in
a hospital over 4 years.
■ Coverage and Utilization
■ Fidelity and Dose
■ Quality of implementation- Satisfaction and Barriers
● The following groups of people will form the target audience for the report
○ Funder-MOH/MOF
○ Program owner: AIC
○ Stakeholders in implementation site: Management and front-line staff
○ Clients/patients and caregivers
○ Community providers and partner institutions

28. Target Audience
Target audience Interest Purpose
Funder- MOH/MOF Coverage and Utilization
Fidelity and Dose
Quality of implementation
Appropriateness
Satisfaction
Barriers
Determine future funding potential, impact to population
health or system/resource utilization
Program owner- AIC Design program adjustments, react to identified problems
and challenges. Determine program success/failure,
potential for expansion or need to scale down
Hospital Management Determine program success/failure, learning points for
other hospital programs/initiatives
Hospital Front-line Staff Satisfaction
Barriers
Determine gaps and problems, providing feedback for
program refinement. Understand changes in care
seeking/delivery experience by providers and end-users.Patients and caregivers
Community partners Quality of implementation
Appropriateness
Satisfaction
Barriers
Determine program success/failure, potential for further
collaboration. Assist in workload projection and future
resource allocation.

29. Dissemination methods
Target
audience
Interest Dissemination methods Frequency of
report
Funder-
MOH/MOF
Coverage and dose
Quality of implementation
Appropriateness
Satisfaction
Barriers
Formal detailed report +/- presentation Annual report at the
end of every year
Program
owner- AIC
Formal detailed report +/- presentation Quarterly reports at
the end of every
quarter
Hospital
Management
Formal detailed report +/- presentation Annual report at the
end of every year
Hospital Front-
line Staff
Satisfaction
Barriers
Presentation of key findings during department
meeting, and feedback gathering
Annual report at the
end of every year
Patients and
caregivers
Key findings and message via roadshows, article in
hospital publication, short video to be screened on
TVs within hospital campus
Annual report at the
end of every year
Community
partners
Quality of implementation
Appropriateness
Satisfaction
Barriers
Summary report, newsletters +/- short presentation Annual report at the
end of every year

30. Implications for stakeholders
Target audience Purpose Implications
Funder- MOH/MOF Determine future funding potential, impact to
population health or system/resource
utilization
Adjustment to funding model, structure
and amount; Adjustment to KPIs or
objectives.
Program owner- AIC Determine program success/failure, potential
for expansion or need to scale down
Refinement of program details and
ongoing/ future implementation plans,
mapping findings into conclusion and
recommendations
Hospital
Management
Determine program success/failure, learning
points for other hospital programs/initiatives
Refinement of program implementation
in consultation with program owner
Hospital Front-line
Staff
Determine gaps and problems, providing
feedback for program refinement. Understand
changes in care seeking/delivery experience
by providers and end-users.
Better understanding of the program;
Assisting in decision on program enrolment
Patients and
caregivers
Community partners Determine program success/failure, potential
for further collaboration. Assist in workload
projection and future resource allocation.
Assisting in decision on continuation in
partnership/collaboration, strengthening
or scaling back of efforts.

31. Thank You!
Questions?