Draft evaluation plan provided to client that contains a summary, evaluation needs, evaluation content, purpose, evaluation model design, and evaluation schedule

1. Evaluation Plan
Ohio Department of Health
Thomas Muldrow
Instructional Design Department, Franklin University
IDPT 670: Capstone
Professor Nicole Hyatt
June 20, 2022

2. Evaluation Plan
Summary
Identify the organizational goals
The Ohio Department of Health (ODH) is a cabinet-level agency, whose mission is to protect
and improve the health of all Ohioans by preventing disease, promoting good health, and
assuring access to quality care. ODH fulfills its mission through collaborative relationships in the
public and private sectors, including with Ohio's 118 local health jurisdictions, each of which is
governed by a Health Commissioner and a locally appointed Board of Health. The department is
responsible for coordinating activities for child and family health services, children with medical
handicaps, early intervention services, nutrition services, and community health services;
ensure the quality of both public health and health care delivery systems; and evaluates health
status, prevents, and controls injuries and diseases (chronic and infectious) and promotes good
health.
Identify the organizational needs and/orgaps
The client is currently working on an online seminar for the new Local Health Department (LHD)
Emergency Response Coordinators (ERC). Over the last two years there has been significant
turnover in the position which has been exacerbated by the COVID-19 pandemic. There is also a
significant gap in performance among existing ERCs that needs to be addressed. The Bureau of
Health Preparedness (BHP) has developed an online seminar for new ERCs and is looking for an
evaluation plan to assess the training and improve overall capacity at the position.
This evaluation should determine how effectively the training is preparing ERCs to perform
assigned duties including interacting with ODH, coordinating with partners, and increasing the
LHDs capability to respond to public health emergencies. The evaluation needs to be able to
assess whether the training results in the desired behavior of the ERCs thereby leading to
increased capacity at the local level.
The learning objectives include:
- Emergency Planning
- Continuity of Operations
- PreparednessCommunicationSystems
- PHEP Capabilities
- CPGs andthe IntegratedPreparednessPlan
- Communicationsw/BHP andGMIS

3. - Grant Administration
The culture and political climate at ODH are such that these interventions will incur little to no
resistance in implementation, support, and evaluation. Further details of the analysis are
described below.
Evaluation needs
Stakeholders
The Bureau of Health Preparedness (BHP) is one area within ODH that is funded primarily by
PHEP and HPP funds to strengthen the capability of public health systems to effectively prepare
for and respond to public health threats and emergencies. Because this is done primarily
through the local health departments, its stakeholders include not only state and local public
health but also the residents of the respective counties. Due to the nature of preparedness
work it is hard to capture the direct impact on the public so while important they are not
considered for the purposes of this evaluation. Additionally, because the evaluation is being
used to assess the training during a pilot with BHP staff the respective unit managers are
included as a primary stakeholder. With that in mind the stakeholders for the evaluation have
been grouped into three categories based on their expectations and priorities. They are:
- Ohio Department of Health - expectation that the LHDs can more effectively partner
with local stakeholders and plan for whole community responses to public health
emergencies equitably addressing the needs their residents
- Bureau of Health Preparedness unit managers – expectation that the content is well
received, comprehensive, and serves as a foundation for new ERCs working at LHDs.
This should enable them to successfully complete annual deliverables aimed at
strengthening the capability of the local jurisdictions
- Local Health Commissioners - expectations that the staff will be able to satisfactorily
complete the assigned deliverables, increasing capacity and providing necessary funding
to further public health preparedness
Evaluation context
Resources
Tools – The majority of this evaluation can be accomplished with the use of computers,
printers, paper, and associated office supplies. The Ohio Department of Health provides all their
employees with laptops, however use of ODH printers may require access to the building. If the
system prevents the use of business laptops, then use of MPD computers would be required

4. and software to manage the project, develop instruments, and analysis of data will need to be
purchased.
Human Resources – As the evaluation consultant, I will complete most of the work. I will
require at least two additional individuals for approximately 1 hour/week to review completed
project work. It would also be beneficial to have access to all subject matter experts via email
and one PMPU staff member periodically to assist with the development of the evaluation
instruments. I will also need access to key stakeholders to review content, validate instruments,
and provide feedback on visualizations.
Financial – As the external consultant, and an employee of the state I do not foresee any
significant expenses because of this project. SME (Subject Matter Expert) support will be
minimal and during business hours (no extra cost). Any incidentals (including all necessary
resources, product licenses, and administration supplies) will be paid for by ODH.
Learners and Context
Learners - The primary learners for this evaluation are all the new ERCs assigned to local health
departments over the last year; many of which have no public health experience. According to
updated attachment #1 (LHD contact information) received over the last year, there are
approximately 20 new ERCs in the state of Ohio. Each session within the PHEP 101 training
should have between 20-30 participants. The newly developed course is beginning in July, and
because of the timing of this project the evaluation will be done internally using BHP staff
members. This will allow for a more thorough evaluation and the ability to implement any
recommendations into the official rollout in July.
The main barrier I see to implementing this training with the ERCs is the ability to obtain usable
data due to the political implications of ODH evaluating local health department employees. The
other challenge I foresee is the difference in competency level between an ERC and BHP staff. It
may not be a one-to-one approximation however the results of the pilot, except for the self-
assessment should be representative of the findings with locals.
Learning Context - The training will be held using Microsoft Teams. The course should take
approximately four weeks to complete with the presenters using PowerPoint slides and
Mentimeter questions to teach the material. There will also be time for questions and answers
following each course within the training. The training is broken into individual courses taught
by the subject matter experts in their respective Bureau of Health Preparedness unit and will be
recorded for those that are unable to attend.
Performance Context - The ERCs all work on a hybrid model with most of their time spent
remotely or within their assigned local jurisdiction. However, the presentation of the material
will take place via an online seminar to allow for maximum participation.

5. Purpose
The purpose of this project is to determine the effectiveness of the online seminar for ERCs
conducted by BHP. This is extremely important to ODH because the ERCs are the main
individuals within the LHD tasked with public health emergency preparedness. This is also
important because it will help raise the overall level of preparedness for the state of Ohio and
will serve as a launching point for additional specialized training in the future.
The evaluation is necessary to assess the intervention developed to teach ERCs the
fundamentals of public health emergency preparedness. The training should result in achieving
standardization among ERCs in the development of local plans and completion of assigned
deliverables aimed at increasing capability and capacity. The evaluation will focus primarily on
the increase in knowledge and understanding of the PHEP fundamentals and whether that
knowledge leads to a change in behavior among ERCs. Additionally, to continually improve the
program there will be an evaluation of the learner's reaction to the training.
Evaluation Questions
1) (Reaction) Is the training going ok and are there any significant issues? If not, what
adjustments need to be made to the scope, the learning environment, or the material?
2) (Learning) How effective is the training in meeting the training objectives and improving
the ERCs understanding of the PHEP fundamentals?
3) (Behavior) Are local health department ERCs better prepared to carry out their assigned
duties based on the information learned? Is there anything missing from the training
that would expedite the transition into the role?
4) (Results) Has there been a decrease in the number of denials and an increase in the
funding received by the LHDs because of the successful completion of the deliverables?
Training Outcomes
The client has already developed a level one evaluation for the overall training, so I am building
a level one evaluation instrument specifically for the pilot that will assess the learner's reaction
to the material and the presenter. The goal is to provide the individual unit managers with
feedback to help improve the unit presentations prior to official roll out in July 2022.

6. The PEHP 101 training was developed based on internal assessment which identified multiple
performance gaps among ERCs. The content has been developed to provide someone new to
public health with the foundational material to perform assigned duties. While it is not
intended to cover the entire scope of the role it is intended to address those areas that have
been observed to present the biggest challenge to inexperienced staff. There are a total of ten
(10) outcomes within the PHEP 101 course.
Outcomes Associated with Question 2:
- Define the fundamental elements of plan writing
- Describe the requirements for an effective COOP plan
- Distinguish the PHEP capabilities and how they should be used to evaluate emergency
preparedness capacity
- Understand an agency's Integrated Preparedness Plan
- Describe the steps involved in submitting a deliverable for grant reimbursement
- Grant Administration
Outcomes Associated with Question 3:
- Identify the proper resources needed for the job
- Able to effectively communicate with ODH
- Able to develop a COOP plan that will ensure the continuation of essential services
- Understand how to assess preparedness and response activities using the 15 PHEP
capabilities
Criteria for Success
In coordination with the stakeholders, the following criteria was determined to indicate that
the training program has been successful:
1. Learners rate the course favorably on the course evaluation with a score of 4.0 or better
on a 5.0 Likert scale
2. Learners achieve an 85% or higher on the post-test upon completion of the course
3. Learners score a 75% or higher on the performance checklist three months post-
completion
4. There is a reduction in interviewing and hiring costs by 15% over the twelve months
preceding the manager training
5. Employee turnover is reduced by 25% annually

7. Evaluation Model Design
Data CollectionMethodsand Instruments
Level BHP pilot group New ERC group
1- Reaction Course evaluationat
completionof training(unit
focused)
Course evaluationdone by
client(overall)
2- Learning Pre-testonday1 of training
Post-testatcompletionof
training
Questionsfromteststobe
spreadacross the sessionsto
testretentionreal-time
3- Behavior Self-assessmentupon
completion of course
Self-assessmentat30- and90-
day point(optional)
4- Results N/A Estimatedchange indeliverable
denialsassociatedwith
completingthe PHEP101
training
Level 1- Reaction Assessment
I will assess learner reaction to the course design, material, and execution by using a course
evaluation upon completion of the training. Because the client has already developed an
assessment for the overall reaction to the course this course evaluation will focus specifically on
the presentation and presenter. This will be able to be used by the respective unit manager to
adjust prior to actual implementation. Each section will also allow for open-ended comments
provided by the learner. The questions will be assessed on a 5-point scale using Strongly agree,
Agree, Neutral, Disagree, and Strongly disagree.
Topic # of questions
Presentation(materials,relevance, understandable) 5
Presenter(prepared,knowledgeable,engaging) 4
Level 2- Knowledge Test
I will evaluate the knowledge of the ERCs by comparing (a) a pre-training knowledge test at the
beginning of the PHEP 101 Training, and (b) a post-training knowledge test at the completion of
training. These will be two separate tests with equivalent test items. This knowledge test will
take place using SurveyMonkey. Ideally the learners would also need to achieve an 85% or
higher on the post-test upon completion of the course to certify completion. The knowledge

8. test will be focused on evaluation question two (2) How effective is the training in meeting the
training objectives and improving the ERCs understanding of the PHEP fundamentals? There are
six training outcomes, found in the table below, that are associated specifically with this
question. I will create or adapt a total of 30 test items each for pre-test and post-test as
outlined below to test the ERCs understanding of the PHEP fundamentals.
Type of
TestItem
Objective
1:
Define the
fundamen
tal
elements
of plan
writing
Objective
2:
Describe
the
requiremen
ts for an
effective
COOPplan
Objective 3:
Distinguish
the PHEP
capabilities
and how they
shouldbe
usedto
evaluate
emergency
preparedness
capacity
Objective 4:
Explainhow
to developan
Integrated
Preparedness
Plan
Objective 5:
Describe the
steps
involvedin
submittinga
deliverable
for grant
reimburseme
nt
Objective 6:
Grant
Administratio
n
True/Fals
e
5 - 5 - - 5
Multiple
Choice
- 5 - 5 5 -
Fill inthe
blank
- - - - - -
Total:30 5 5 5 5 5 5
Level 3 – Behavior Assessment
To assess the ERCs understanding of the material presented by the various units as part of the
PHEP 101 training I will develop a self-assessment to be taken by the Emergency Response
Coordinator 30 (and potentially 90) days after being in the role. We will develop a survey that
can be used to evaluate the usefulness of the information provided on a simple rating score.
The prioritization of questions is focused more on those objectives with a direct correlation to
the completion of assigned deliverables. This has more to deal with the performance gaps
identified previously than the importance of the objective. The table below identifies how many
questions will be included for each course outcome.
Outcome # ofquestions
1. Identify the proper resources needed for the job 3
2. Able to effectively communicate with ODH 2
3. Able to develop plans to prevent, protect against, quickly respond to,
and recover from health emergencies
2
4. Understand how to assess preparedness and response activities using
the 15 PHEP capabilities
3

9. 5. Grant Administration(budget/GMIS) 2
Total: 12
Level 4 – Results Test
The evaluation of results will be gradual over time. The instrument is simply a comparison tool
that tracks the key metrics required to measure total funding being reallocated at the end of
the budget period. In collaboration with the Program Manager, I will develop criteria including
leading and lagging indicators that can be presented to leadership on a quarterly basis. The
instrument will include raw data as well as visualizations that provide leadership with a
“snapshot” of the training results in relation to corporate goals. The impact focuses on the loss
of institutional knowledge at the local level which inhibits the health department’s ability to
respond effectively to a public health emergency.
How Data Will Be Gathered and Analyzed
The table below shows how the data will be gathered and analyzed based on each research
question.
EvaluationQuestion Where we will getthe data How data will be analyzed
Is the traininggoingokand are
there anysignificantissues?If
not,what adjustmentsneedto
be made to the scope,the
learningenvironment,orthe
material?
Course evaluation Differentiate,identifybias,
recognize interrelationshipsand
themes
How effective isthe trainingin
meetingthe trainingobjectives
and improvingthe ERCs
understandingof the PHEP
fundamentals?
Pre-testtopost-testcomparison Mean, median,mode of post-
test
Gain scorescomparingpre-test
to post-test
Are LHD ERCs betterprepared
to carry out theirassigned
dutiesbasedonthe information
learned?Isthere anything
missingfromthe trainingthat
wouldexpedite the transition
intothe role?
Self-assessment Mean, median,mode of scores
Gain scorescomparingmanager
self-assessmentpost
completionand3-monthsafter
the training
Has there beena decrease in
the numberof denialsandan
increase inthe fundingreceived
by the LHDs because of the
successful completionof the
deliverables?
Comparison tool (including
leadingandlaggingindicatorsof
desiredresults)
Mean, median,mode of data
sets
Identifytrendsindepartment
heads’reportsof their
personnel

11. Evaluation Schedule (as of 12 June 2022)
Major activities to be accomplished (Estimate and schedule specific duration of each major
activity)
Activity Description People Duration Completion
Date
Complete Project
Plan
Overall scope of the projectandassociated
deliverables
CT 29 May
Needs
Assessment
Evaluationplanspecifics andclient
requirementsincludingevaluationgoals
CT/PS
/PM
1 week 5 June
Complete
EvaluationPlan
Outlinesdetailsof evaluation(Instruments,
timeline,clientrequirements,expectations
(ANALYSISDOCUMENT DUE-5 JUNE)
CT/PS
/PM
1 week 5 June
Draft Level 1
instrument
Determine specificareasof focus. Provide
draft to PMfor review.
CT 1 week 12 June
Draft Level 2
Instruments
Create testsbasedonlearningobjectives;
revise knowledgetestsbasedonSME feedback
CT 2 weeks 19 June
Conductvalidity
and reliability
tests
Testsurveytool;pilotsurveywithPMPUstaff;
conduct revieww/PM.Refine andfinalizelevel
1 instrument
CT/PM 1 week 19 June
Conductvalidity
and reliability
tests
Conducttestreview w/stakeholdersandPM.
UploadinstrumentintoSurveyMonkey;pilot
testwithPMPU staff;finalize level2
instruments
(LEARNING DESIGN DOC DUE – 26 JUN)
CT/PM
/PS/BT
3 weeks 26 June
Draft level 3
instrument
Developself-evaluationforclientreview CT 2 days 26 June
Gather level 2
data
Email participantstrainingprerequisite (pre-
test) andgather data forlevel 2pre-testbefore
training
CT 1 week 26 June
Online seminar
completed
Trainingwill be overaseriesof days(Assume
conductedbetween28 JUN – 5 JUL)
Gather level 1and
level 2data
Upon completionof training:gatherlevel1
data viaSurveyMonkeyandlevel2via
SurveyMonkey
CT 1 week 11 July
Finalize level3
instrument
Analyze existingperformancecriteria.Revise
and developself-evaluationbasedonfeedback
fromclientand B-team
(ROLLOUT TO ERCS – 11 JULY)
CT 3 weeks 11 July

12. Conductvalidity
and reliability
tests
Conductlevel 3instrumentreview withclient;
pilotself-evaluation w/B-teamand PM for
feedbackandrefinement.Refine andfinalize
survey
(PI PLANDUE- 17 JULY)
CT/PM
/BT
2 weeks 17 July
Gather level 3
data
Sendto participatingBHPstaff forcompletion
(ROLLOUT early July – late Aug)
CT 2 days 11 July
Analyze and
interpret
Conductstatistical analysisof results;create
visuals;interpretwhetherthe evaluation
purpose ismet
CT 10 days 30 July
Reportand make
recommendations
Create evaluationreportsanddocuments;
presentevaluationreportstostakeholders
(CAPSTONEPROJECT DUE – 6 AUG)
CT/PS
/PM
1 week 5 August

13. Reference
Kirkpatrick, D. L., & Kirkpatrick, J. D. (2007). Implementing the four levels: A practical guide for
effective evaluation of training programs. ProQuest Ebook Central https://ebookcentral-
proquest-com.links.franklin.edu