Are we fulfilling our promise?Assessing and reporting training qualityIATEFL BESIG Annual ConferenceStuttgart, GermanyNovember 16, 2012Charles Rei
Learning objectivesBy the end of this session we will be able to...1. Identify the benefits of comprehensive training assessment.2. Identify and assess the four levels of training evaluation using quantitative and qualtative metrics.3. Identify external factors which affect assessment.4. Use surveys, performance-based assessment, and client contact to evaluate training.5. Identify the main elements and organization of a comprehensive training evaluation report.
Training Perks • A gift to the employees • Knowledge- Education based • Standardized • Performance- based Goals • Personal development
Current situation• Surveys / Feedback forms• Observation• Interviews Formative• Testing vs.• Self-assessment Summative• Can do statements assessment• Portfolios and projects• Regular reports• Status meetings• Cancellations and renewals• None whatsoever
Benefits of comprehensive assessment• Learner accountability • Their dedication matters• Organizational / professional development • Strengths and areas for improvement • Quality processes • Sharing best practice• Added / demonstrated value • Client communications and relationships • Value-based pricing vs. Cost-plus pricing • Targeted sales leads (added services, target market)
Surveys / Feedback• All 4 levels• Tips for sucessful surveys • Face-to-face • Concise • Organized / Consistent • Standardized vs. Customized • Most important questions first • Use open questions sparingly • Create realistic baseline expectations
Performance-based assessment Presentation Introduction • Learning Performance Step Yes No • Transfer designWelcome and thank the audienceState purpose of the presentationGive an outline of the presentationUse discourse markers in outlineMention admin rules (questions, breaks)Signal the end of the introductionIntroduce the first topic using signpostingMaintain eye contact with audienceSpeak clearlyUse appropriate body language Total
Client contact• Externalities• Behavior and results• Targeted trainer-learner conversations • What’s going on in your department right now? • Are there any big projects coming up? • Did that lesson on trade fairs help you in Frankfurt? What helped the most? How was it different than what we practiced? • Have you written any emails in English this week? Can you print it for us?• Client check-ins
Ten inputs1. Needs2. Objectives3. Content4. Participants5. Schedule6. Facilities7. Instructor(s)8. Training materials9. Program coordination10. Program evaluation
4 levels• Reaction – customer satisfaction• Learning – improved knowledge and skills• Behavior – application outside the classroom• Results – impacts on the business
Externalities• Motivation elements• Ability elements• Environment elements
Recommendations• What is impacting our assessment? • 10 Inputs? • External factors?• How do we improve? • Change an input? • Influence an external factor?• What should we share internally?• What should we share externally?
Focus groups - LearningDiscuss:• Three things you learned from today’s sessions • 4 Levels • Don‘t assume transfer (reality checks) • Learner motivation questions• One thing you would like to hear more about • Operationaizing it. (examples)
Focus groups - Behavior• Discuss how you will apply what you have learned from today’s sessions.• Focus group results: • Review the feedback form • Be more proactive about asking for achievements and successes • Look for testing service (before and after) • 3 months later, what has stuck? (before and after rating) • Use the worksheet as basis
Focus groups - Results• What impact on your organization do you expect to see as a result of today’s training sessions?
References• Holton III, E. F. (1996, Spring). The Flawed Four-Level Evaluation Model. Human Resource Development Quarterly, 7(1), 5-21.• Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler.• Kirkpatrick, J. (2007, Aug). The Hidden Power of Kirkpatricks Four Levels. T+D Magazine, pp. 34-37.• Kirkpatrick, J. (2011, Spring). The New World Kirkpatrick Four Levels. The Canadian Learning Journal, pp. 19-21.• Maister, D. H. (1993). Managing the Professional Services Firm. New York: Simon & Schuster.• OToole, S. (2009, Aug). Kirkpatrick on Evaluation: Not Crazy After All These Years. Training and Development in Australia, pp. 23-25.• Praslova, L. (2010). Adaptation of Kirkpatricks Four Level Model of Training Criteria to Assessment of Learning Outcomes and Program Evaluation in Higher Education. Educational Assessment, Evaluation & Accountability(22), 215-225.• Steensma, H., & Groeneveld, K. (2010). Evaluating a Training Using the "Four Levels Model". Journal of Workplace Learning, 22(5), 319-331.• Stoel, D. (2004, Jan). The Evaluation Heavy Weight Match. T+D Magazine, pp. 46-48.
For more information / questionsContact: Charles Rei email@example.com http://www.reicommunicationtraining.com @Charlesrei1Presentation and handouts at Business English Ideas http://businessenglishideas.blogspot.de/Ongoing conversation on the BESIG Ning http://iatefl-besig.ning.com/
Quality processes• If you were the client, would you care about these issues?• Do any of these questions represent an unreasonable client expectation?• How confident do you feel that your organization would outperform the competition on these questions?• How valuable would it be if your organization earned top scores on this consistently?• Can you think of another way to ensure consistent performance in these areas?Thank you!