Kirkpatrick's 4 levels of evaluation assess training programs from different perspectives: Level 1 assesses participant reactions; Level 2 assesses learning; Level 3 evaluates behavior change; Level 4 measures organizational results. Each level requires different data collection methods, with higher levels being more difficult and expensive to measure. The model provides guidelines for evaluation questions, timing, methods and comparing pre- and post-training metrics to determine a program's full impact.

1. Kirkpatrick’s 4 Levels of Evaluation Level 1 - Reaction Level 2 - Learning Level 3 - Behavioral Results (A) Observable (skills) (B) Non-Observable (attitudes) Level 4 - Organizational Results

2. If you want to know … Then you must ask at this level…. Did participants like the program? Did participants learn the content intended? Are they applying skills and behaviors taught? Are they applying non-observable outcomes to the job? Has there been any impact on the organization? Level 1 Level 2 Level 3 (A) Level 3 (B) Level 4

3. What do you want to find out with Level 1 questions? Did the program meet the expectations of trainees? What aspects were most helpful? Interesting? Informative? What aspects were least helpful, interesting, or informative? What were participants’ reactions to the program’s design? Pacing? Materials? Precourse work? Instructor? Do people intend to use what they have learned? How? What barriers, if any, do people believe will inhibit their ability to use what they have learned?

4. Level 2 - Evaluating Learning Measures knowledge, skills, and attitudes Use a control group, if practical Pre-test can serve as a needs assessment Strive for 100% response on questionnaires

5. Knowledge Tests Essay or open-ended answer tests tests Write in or short answer tests Binary true-false tests Multiple choice tests

6. Competency Demonstrations: Testing for Skill Learners are demonstrate competencies while being observed by a trained evaluator. Can use simulations or demonstrations can be part of the learning activity (role plays, etc) Need consistency in observers; multiple observers need to be trained

7. Level 3 Focuses on Transfer: Are They Using What They Learned? Affective outcomes focus on attitudes, values, and beliefs of learners Cognitive outcomes are the concepts, principles, and knowledge used on the job Behavioral or skill outcomes address what learners are able to do that can be observed by others

8. Decisions in Evaluating Level 3 - Behavior When to evaluate How often to evaluate How to evaluate Costs vs. Benefits : When is it worth evaluating at Levels 3 & 4?

9. Guidelines for Level 3 Evaluations Use a control group if practical Allow time for behavior change to take place Evaluate before and after program Survey / interview at least one or more: Trainees Immediate supervisors Trainees’ direct reports

10. Behavior Change without Reinforcement

11. Example of Reinforced Behavior Measured 3 to 6 Months after Training

12. Evaluating Behavior: Questions to Ask Ask if trainees if they are doing anything differently on the job as a result of training? If so, ask them to describe If not, ask why Explore effects of management support, organizational barriers, etc

13. Level 4 Questions: What’s the Organizational Impact? How much did quality improve? How much did productivity increase? How much did did we save or prevent? (accidents, turnover, wasted time, etc) What tangible benefits has the organization seen from the money spent on training?

14. Evaluating Results - Level 4 Look for evidence, not proof Use a control group, if possible, to isolate the effects of training Measure before and after the program Repeat measurement at appropriate times Consider cost vs. benefits Not as expensive as Level 3 to collect Organizational data often available

15. Four Major Categories of Hard Data Output increases Units produced, items sold or assembled Students graduated, patients visited, applications processed Quality improvement Scrap, rework, rejects, error rates, complaints Cost savings Unit costs, overhead, operating, program costs Time savings Cycle time, overtime, equipment downtime

16. Categories of Soft Data Work habits Absenteeism, tardiness, communication breakdowns, first aid treatments Climate Turnover, discrimination charges, job satisfaction, number of grievances New skills Decisions made, problems solved, grievances resolved,intention to use new skills, frequency of use of new skills, importance of new skills

17. Categories of Soft Data, con’t Development Number promotions, pay increases, training programs attended, PA ratings Satisfaction Favorable reactions, attitude changes, increased confidence, customer satisfaction Initiative Successful completion of new projects, number of suggestions implemented, new ideas implemented

18. Sources of Data Organizational Performance Records Participants Supervisors of Participants Direct Reports of Participants Team / Peer Groups Internal or External evaluators as observers