• Like
Are we fulfilling our promise
Upcoming SlideShare
Loading in...5

Are we fulfilling our promise

Uploaded on


  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads


Total Views
On Slideshare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide


  • 1. Are we fulfilling our promise?Assessing and reporting training qualityIATEFL BESIG Annual ConferenceStuttgart, GermanyNovember 16, 2012Charles Rei
  • 2. Learning objectivesBy the end of this session we will be able to...1. Identify the benefits of comprehensive training assessment.2. Identify and assess the four levels of training evaluation using quantitative and qualtative metrics.3. Identify external factors which affect assessment.4. Use surveys, performance-based assessment, and client contact to evaluate training.5. Identify the main elements and organization of a comprehensive training evaluation report.
  • 3. Training Perks • A gift to the employees • Knowledge- Education based • Standardized • Performance- based Goals • Personal development
  • 4. Current situation• Surveys / Feedback forms• Observation• Interviews Formative• Testing vs.• Self-assessment Summative• Can do statements assessment• Portfolios and projects• Regular reports• Status meetings• Cancellations and renewals• None whatsoever
  • 5. Benefits of comprehensive assessment• Learner accountability • Their dedication matters• Organizational / professional development • Strengths and areas for improvement • Quality processes • Sharing best practice• Added / demonstrated value • Client communications and relationships • Value-based pricing vs. Cost-plus pricing • Targeted sales leads (added services, target market)
  • 6. THE FOUR LEVELSAssessment Criteria
  • 7. Kirkpatrick’s 4 Levels - Reaction • Enjoyment? • Priority? • Useful? • Satisfied? • Attendance • Hours • Post-lesson contact
  • 8. Kirkpatrick’s 4 Levels - Learning • Remember? • What did you learn? • Project performance • Testing • Performance-based rubrics • Trainer / Self assessment
  • 9. Kirkpatrick’s 4 Levels - Behavior • Did you apply the lessons? • Pre-task reflection • Models and templates • Authentic materials
  • 10. Kirkpatrick’s 4 Levels - Results • Productivity measurements • Better relationships • Reduced stress • Misunderstanding and conflicts • Performance appraisals
  • 11. Holton’s External Elements
  • 12. Holton’s External Elements
  • 13. Holton’s External Elements
  • 14. ASSESSMENT METHODSHow exactly should we get this information?
  • 15. The 5 Ws
  • 16. Surveys / Feedback• All 4 levels• Tips for sucessful surveys • Face-to-face • Concise • Organized / Consistent • Standardized vs. Customized • Most important questions first • Use open questions sparingly • Create realistic baseline expectations
  • 17. Performance-based assessment Presentation Introduction • Learning Performance Step Yes No • Transfer designWelcome and thank the audienceState purpose of the presentationGive an outline of the presentationUse discourse markers in outlineMention admin rules (questions, breaks)Signal the end of the introductionIntroduce the first topic using signpostingMaintain eye contact with audienceSpeak clearlyUse appropriate body language Total
  • 18. Client contact• Externalities• Behavior and results• Targeted trainer-learner conversations • What’s going on in your department right now? • Are there any big projects coming up? • Did that lesson on trade fairs help you in Frankfurt? What helped the most? How was it different than what we practiced? • Have you written any emails in English this week? Can you print it for us?• Client check-ins
  • 20. Ten inputs1. Needs2. Objectives3. Content4. Participants5. Schedule6. Facilities7. Instructor(s)8. Training materials9. Program coordination10. Program evaluation
  • 21. 4 levels• Reaction – customer satisfaction• Learning – improved knowledge and skills• Behavior – application outside the classroom• Results – impacts on the business
  • 22. Externalities• Motivation elements• Ability elements• Environment elements
  • 23. Recommendations• What is impacting our assessment? • 10 Inputs? • External factors?• How do we improve? • Change an input? • Influence an external factor?• What should we share internally?• What should we share externally?
  • 24. INTO PRACTICEWorkshopping the four levels
  • 25. Audience survey - Reactionhttp://vot.rs/5ab67c
  • 26. Focus groups - LearningDiscuss:• Three things you learned from today’s sessions • 4 Levels • Don‘t assume transfer (reality checks) • Learner motivation questions• One thing you would like to hear more about • Operationaizing it. (examples)
  • 27. Focus groups - Behavior• Discuss how you will apply what you have learned from today’s sessions.• Focus group results: • Review the feedback form • Be more proactive about asking for achievements and successes • Look for testing service (before and after) • 3 months later, what has stuck? (before and after rating) • Use the worksheet as basis
  • 28. Focus groups - Results• What impact on your organization do you expect to see as a result of today’s training sessions?
  • 29. TAKING IT FROM HEREConclusion
  • 30. References• Holton III, E. F. (1996, Spring). The Flawed Four-Level Evaluation Model. Human Resource Development Quarterly, 7(1), 5-21.• Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler.• Kirkpatrick, J. (2007, Aug). The Hidden Power of Kirkpatricks Four Levels. T+D Magazine, pp. 34-37.• Kirkpatrick, J. (2011, Spring). The New World Kirkpatrick Four Levels. The Canadian Learning Journal, pp. 19-21.• Maister, D. H. (1993). Managing the Professional Services Firm. New York: Simon & Schuster.• OToole, S. (2009, Aug). Kirkpatrick on Evaluation: Not Crazy After All These Years. Training and Development in Australia, pp. 23-25.• Praslova, L. (2010). Adaptation of Kirkpatricks Four Level Model of Training Criteria to Assessment of Learning Outcomes and Program Evaluation in Higher Education. Educational Assessment, Evaluation & Accountability(22), 215-225.• Steensma, H., & Groeneveld, K. (2010). Evaluating a Training Using the "Four Levels Model". Journal of Workplace Learning, 22(5), 319-331.• Stoel, D. (2004, Jan). The Evaluation Heavy Weight Match. T+D Magazine, pp. 46-48.
  • 31. For more information / questionsContact: Charles Rei charles.rei@reicommunicationtraining.com http://www.reicommunicationtraining.com @Charlesrei1Presentation and handouts at Business English Ideas http://businessenglishideas.blogspot.de/Ongoing conversation on the BESIG Ning http://iatefl-besig.ning.com/
  • 32. Quality processes• If you were the client, would you care about these issues?• Do any of these questions represent an unreasonable client expectation?• How confident do you feel that your organization would outperform the competition on these questions?• How valuable would it be if your organization earned top scores on this consistently?• Can you think of another way to ensure consistent performance in these areas?Thank you!