Four Levels Of Evaluation (Kirkpatrick Model) By Ravinder Tulsiani


Published on

Donald Kirkpatrick developed a model of training evaluation in 1959; Arguably the most widely used approach; Simple, Flexible and Complete ...

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Four Levels Of Evaluation (Kirkpatrick Model) By Ravinder Tulsiani

  1. 1. Four Levels of Evaluation Reaction ~ Learning ~ Transfer ~ Impact Ravinder Tulsiani, FCSI, BA-Law Corporate Trainer | Transformational Leader Unleash the E.D.G.E. | Educate ~ Develop ~ Guide ~ Empower1
  2. 2. Objectives  Upon completion of this presentation you will be able to: – State why evaluation of programs is critical to you and your organization. – Apply Kirkpatrick‟s four levels of evaluation to your programs. – Use guidelines for developing evaluations. – Implement various forms and approaches to evaluation2
  3. 3. Why Evaluate?  Determine the effectiveness of the program design How the program was received by the participants How learners fared on assessment of their learning Determine what instructional strategies work presentation mode presentation methods. learning activities desired level of learning  Program improvement3
  4. 4. Why Evaluate?  Should the program be continued?  How do you justify your existence?  How do you determine the return on investment for the program? human capital individual competence social/economic benefit4
  5. 5. Kirkpatrick Foundational Principle The end is the beginning “Trainers must begin with desired results and then determine what behavior is needed to accomplish them. Then trainers must determine the attitudes, knowledge, and skills that are necessary to bring about the desired behavior(s). The final challenge is to present the training program in a way that enables the participants not only to learn what they need to know but also to react favorably to the program.” Source: Effective training evaluation begins before the program even starts. Kirkpatrick says it best on page 26 of Evaluating Training Programs: The Four Levels (1st Edition, Berrett-Koehler, 1993)5
  6. 6. Four Levels of Evaluation Kirkpatrick  During program evaluation – Level One Reaction – Level Two Learning  Post program evaluation – Level Three Behavior – Level Four Results6
  7. 7. Reaction Level A customer satisfaction measure – Were the participants pleased with the program – Perception if they learned anything – Likelihood of applying the content – Effectiveness of particular strategies – Effectiveness of the packaging of the course7
  8. 8. Examples of Level One Your Opinion, Please In a word, how would you describe this workshop? Intent – Solicit feedback about the course. Can also assess whether respondents transposed the numeric scales.8
  9. 9. Example of Level One  Using a number, how would you describe this program? (circle a number) Terrible Average Outstanding 1 2 3 4 5  Intent: Provides quantitative feedback to determine average responses (descriptive data). Watch scale sets!9
  10. 10. Example of Level One  How much did you know about this subject before taking this workshop? Nothing Some A lot 1 2 3 4 5  How much do you know about this subject after participating in this workshop? Nothing Some A lot 1 2 3 4 5  Intent - The question does not assess actual learning, it assesses perceived learning.10
  11. 11. Example Level One  How likely are you to use some or all of the skills taught in this workshop in your work/community/ family? Not Very Likely Likely Likely 1 2 3 4 5  Intent – determine learners perceived relevance of the material. May correlate with the satisfaction learners feel.11
  12. 12. Example of Level One  The best part of this program was…  The one thing that could be improved most ..  Intent Qualitative feedback on the course and help prioritize work in a revision. Develop themes on exercises, pace of course, etc.12
  13. 13. Guidelines for Evaluating Reaction  Decide what you want to find out.  Design a form that will quantify reactions.  Encourage written comments.  Get 100% immediate response.  Get honest responses.  If desirable, get delayed reactions.  Determine acceptable standards.  Measure future reactions against the standard.13
  14. 14. Learning Level  What did the participants learn in the program? – The extent to which participants change attitudes, increase knowledge, and/or increase skill. – What exactly did the participant learn and not learn? – Pretest Posttest14
  15. 15. Learning Level  Requires developing specific learning objectives to be evaluated.  Learning measures should be objective and quantifiable. – Paper pencil tests, performance on skills tests, simulations, role-plays, case study, etc.15
  16. 16. Level Two Examples  Develop a written exam based on the desired learning objectives.  Use the exam as a pretest  Provide participants with a worksheet/activity sheet that will allow for “tracking” during the session.  Emphasize and repeat key learning points during the session.  Use the pretest exam as a posttest exam.  Compute the post-test/pre-test gain on the exam.16
  17. 17. What makes a good test?  The only valid test questions emerge from the objectives.  Consider writing main objectives and supporting objectives.  Test questions usually test supporting objectives.  Ask more than one question on each objective.17
  18. 18. Level Two Strategies  Consider using scenarios, case studies, sample project evaluations, etc, rather than test questions. Develop a rubric of desired responses. – Develop between 3 and 10 questions or scenarios for each main objective.18
  19. 19. Level Two Strategies  Provide instructor feedback during the learning activities. – Requires the instructor to actively monitor participants discussion, practice activities, and engagement. Provide learners feedback. – Ask participants open ended questions (congruent with the learning objectives) during activities to test participant understanding.19
  20. 20. Example Which of the following should be considered when evaluating at the Reaction Level? (more than one answer possible) ___Evaluate only the lesson content ___Obtain both subjective and objective responses ___Get 100% response from participants ___Honest responses are important ___Only the course instructor should review results.20
  21. 21. Example  Match the following to the choices below ___ Reaction Level ___ Learning Level A. Changes in performance at work B. Participant satisfaction C. Organizational Improvement D. What the participant learned in class21
  22. 22. Scenario Example  An instructor would like to know the effectiveness of the course design and how much a participant has learned in a seminar. The instructor would like to achieve at least Level Two evaluation. – What techniques could the instructor use to achieve level two evaluation? – Should the instructor also consider doing a level one evaluation? Why or why not?22
  23. 23. Rubric for Scenario Question Directions to instructor: Use the following topic checklist to determine the completeness of the participants response: ___ Learner demonstrated an accurate understanding of what level two is: learning level. ___ Learner provided at least two specific examples: pretest -posttest, performance rubrics, scenarios, case studies, hands-on practice. ___ Learner demonstrated an accurate understanding of what level one evaluation is: reaction level. ___The learner provided at least three specific examples of why level one is valuable: assess satisfaction, learning activities, course packaging, learning strategies, likelihood of applying learning.23
  24. 24. Behavior Level  How the training affects performance.  The extent to which change in behavior occurred.  Was the learning transferred from the classroom to the real world.  Transfer – Transfer - Transfer24
  25. 25. Conditions Necessary to Change  The person must: – have a desire to change. – know what to do and how to do it. – work in the right climate. – be rewarded for changing.25
  26. 26. Types of Climates  Preventing – forbidden to use the learning.  Discouraging – changes in current way of doing things is not desired.  Neutral – learning is ignored.  Encouraging – receptive to applying new learning.  Requiring – change in behavior is mandated.26
  27. 27. Guidelines for Evaluating Behavior  Measure on a before/after basis  Allow time for behavior change (adaptation) to take place  Survey or interview one or more who are in the best position to see change. – The participant/learner – The supervisor/mentor – Subordinates or peers – Others familiar with the participants actions.27
  28. 28. Guidelines for Evaluating Behavior  Get 100% response or a sample? – Depends on size of group. The more the better.  Repeat at appropriate times – Remember that other factors can influence behavior over time.  Use a control group if practical  Consider cost vs. benefits of the evaluation28
  29. 29. Level Three Examples  Observation  Survey or Interview – Participant and/or others  Performance benchmarks – Before and after – Control group  Evidence or Portfolio29
  30. 30. Survey or Patterned Interview 1. Explain purpose of the survey/interview. 2. Review program objectives and content. 3. Ask the program participant to what extent performance was improved as a result of the program. __ Large extent __ Some extent __ Not at all If “Large extent” or “Some extent”, ask to please explain. 4. If “Not at all”, indicate why not: ___ Program content wasn’t practical ___ No opportunity to use what I learned ___ My supervisor prevented or discouraged me to change ___ Other higher priorities ___ Other reason (please explain) 5. Ask, “In the future, to what extent do you plan to change your behavior?” ___ Large extent ___ Some extent ___ Not at all Ask to please explain:30
  31. 31. Evidence and Portfolio  Thank you for participating. I am very interested in how the evaluation skills you have learned are used in your work. .  Please send me a copy of at least one of the following: – a level three evaluation that you have designed. – a copy of level two evaluations that use more than one method of evaluating participant learning. – a copy of a level one evaluation that you have modified and tell me how it influenced program improvement. – (indicate if you would like my critique on any of the evaluations)  If I do not hear from you before January 30, I will give you a call – no pressure – „just love to learn what you are doing.31
  32. 32. Results Level  Impact of education and training on the organization or community.  The final results that occurred as a result of training.  The ROI for training.32
  33. 33. Examples of Level Four  How did the training save costs  Did work output increase  Was there a change in the quality of work  Did the social condition improve  Did the individual create an impact on the community  Is there evidence that the organization or community has changed.33
  34. 34. Guidelines for Evaluating Results  Measure before and after  Allow time for change to take place  Repeat at appropriate times  Use a control group if practical  Consider cost vs. benefits of doing Level Four  Remember, other factors can affect results  Be satisfied with Evidence if Proof is not possible.34
  35. 35. Thank You! Learn more at: www.TheCorporateTrainer.ca35