This chapter discusses evaluating occupational safety and health programs for emergency services. It describes conducting both process evaluations to analyze program procedures and outcome evaluations to compare results to goals. Process evaluations ask how program parts affect members and which parts are most/least effective. Outcome evaluations analyze injury rates, knowledge/behavior changes, and policy response over time. The fire chief or safety manager usually oversees evaluations through committees and staff. NFPA 1500 recommends external evaluations every three years to demonstrate improvement efforts. Evaluations should occur before programs begin and be continuous for process and yearly for outcomes, or when serious incidents or changes require reevaluation.

1. Occupational Safety and Health
for Emergency Services
Fourth Edition
Chapter 11 — Safety Program Evaluation

2. Objectives
At the conclusion of this chapter you will be able to:
‣ Describe the purpose of evaluating the safety and
health program.
‣ Compare the two types of evaluations—process and
outcome.
‣ Explain who is responsible for evaluating the safety
and health program.
‣ Describe the recommended frequency of evaluation
and the factors that affect the frequency.

3. Introduction
‣ Chapter 4 of NFPA 1500
‣ Requires risk management plan to be monitored
annually
‣ Evaluated every 3 years by an independent source
‣ New safety and health program
‣ Evaluate its effectiveness
‣ Measuring of results is necessary
‣ Compare where the organization is now to where it
was before

4. The Evaluation Process
‣ Three reasons to evaluate
‣ See if the program is effective
‣ Determine the response from the members’
perspectives
‣ Facilitate program changes
‣ Compare outcomes
‣ Analyze statistics
‣ Changes should be considered

5. Process Evaluation (1 of 2)
‣ Analysis of the procedures of the program
‣ Questions that must be answered:
‣ Who was affected by the program?
‣ To what extent were they affected?
‣ Are improvements occurring as planned?
‣ Which parts of the program appear to be most
effective?
‣ Which parts of the program appear to be least
effective?

6. Process Evaluation (2 of 2)

7. Outcome Evaluation (1 of 2)
‣ Compares the result of the program to the goal
‣ After the program has been in effect for a while
‣ Several areas should be analyzed
‣ Current compared to prior injury rates
‣ Changes in knowledge, behavior, and performance
‣ Changes in the physical environment
‣ Responses to policy changes

8. Outcome Evaluation (2 of 2)

9. Responsibility for Evaluation (1 of 2)
‣ Fire chief, EMS director, or safety and health
manager may oversee evaluation
‣ Evaluation accomplished through
‣ Safety committee
‣ Incident safety officers
‣ Senior staff members
‣ First-line supervisors
‣ Responders

10. Responsibility for Evaluation (2 of 2)
‣ NFPA 1500 recommendations
‣ Externally evaluated
‣ Every three years
‣ External evaluation
‣ Highly recommended
‣ Demonstrates a positive effort
‣ Find areas for improvement

11. Evaluation Frequency (1 of 2)
‣ Program evaluation
‣ Required component
‣ Occurs before the program even starts
‣ Process evaluation can be an almost continuous
process
‣ Outcome evaluation
‣ The end of a specific period of time
‣ Usually one year
‣ Measured time frame

12. Evaluation Frequency (2 of 2)
‣ Frequency should be considered dynamic
‣ Evaluate if one of the following occurs
‣ A serious injury, near miss, or death occurs:
‣ Findings during a postincident analysis require
evaluation
‣ New technology is introduced
‣ Changes are made to applicable regulations or
standards