Are we fulfilling our promise?Assessing and reporting training qualityIATEFL BESIG Annual ConferenceStuttgart, GermanyNove...
Learning objectivesBy the end of this session we will be able to...1. Identify the benefits of comprehensive training   as...
Training           Perks                     • A gift to the                       employees                         • Kno...
Current situation• Surveys / Feedback forms• Observation• Interviews                                 Formative• Testing   ...
Benefits of comprehensive assessment• Learner accountability   • Their dedication matters• Organizational / professional d...
THE FOUR LEVELSAssessment Criteria
Kirkpatrick’s 4 Levels - Reaction                    • Enjoyment?                    • Priority?                    • Usef...
Kirkpatrick’s 4 Levels - Learning                    • Remember?                    • What did you learn?                 ...
Kirkpatrick’s 4 Levels - Behavior                    • Did you apply the                      lessons?                    ...
Kirkpatrick’s 4 Levels - Results                    • Productivity                      measurements                    • ...
Holton’s External Elements
Holton’s External Elements
Holton’s External Elements
ASSESSMENT METHODSHow exactly should we get this information?
The 5 Ws
Surveys / Feedback• All 4 levels• Tips for sucessful surveys   • Face-to-face   • Concise   • Organized / Consistent   • S...
Performance-based assessment            Presentation Introduction                    • Learning            Performance Ste...
Client contact• Externalities• Behavior and results• Targeted trainer-learner conversations   • What’s going on in your de...
BRINGING IT TOGETHERReporting
Ten inputs1.    Needs2.    Objectives3.    Content4.    Participants5.    Schedule6.    Facilities7.    Instructor(s)8.   ...
4 levels• Reaction – customer  satisfaction• Learning – improved  knowledge and skills• Behavior – application  outside th...
Externalities• Motivation elements• Ability elements• Environment elements
Recommendations• What is impacting our assessment?  • 10 Inputs?  • External factors?• How do we improve?  • Change an inp...
INTO PRACTICEWorkshopping the four levels
Audience survey - Reactionhttp://vot.rs/5ab67c
Focus groups - LearningDiscuss:• Three things you learned from today’s sessions  • 4 Levels  • Don‘t assume transfer (real...
Focus groups - Behavior• Discuss how you will apply what you have learned from  today’s sessions.• Focus group results:  •...
Focus groups - Results• What impact on your organization do you expect to see as a result of today’s training sessions?
TAKING IT FROM HEREConclusion
References• Holton III, E. F. (1996, Spring). The Flawed Four-Level Evaluation Model. Human    Resource Development Quarte...
For more information / questionsContact:    Charles Rei            charles.rei@reicommunicationtraining.com            htt...
Quality processes• If you were the client, would you care about these issues?• Do any of these questions represent an unre...
Upcoming SlideShare
Loading in …5
×

Are we fulfilling our promise

2,005 views
1,861 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,005
On SlideShare
0
From Embeds
0
Number of Embeds
1,328
Actions
Shares
0
Downloads
14
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Are we fulfilling our promise

  1. 1. Are we fulfilling our promise?Assessing and reporting training qualityIATEFL BESIG Annual ConferenceStuttgart, GermanyNovember 16, 2012Charles Rei
  2. 2. Learning objectivesBy the end of this session we will be able to...1. Identify the benefits of comprehensive training assessment.2. Identify and assess the four levels of training evaluation using quantitative and qualtative metrics.3. Identify external factors which affect assessment.4. Use surveys, performance-based assessment, and client contact to evaluate training.5. Identify the main elements and organization of a comprehensive training evaluation report.
  3. 3. Training Perks • A gift to the employees • Knowledge- Education based • Standardized • Performance- based Goals • Personal development
  4. 4. Current situation• Surveys / Feedback forms• Observation• Interviews Formative• Testing vs.• Self-assessment Summative• Can do statements assessment• Portfolios and projects• Regular reports• Status meetings• Cancellations and renewals• None whatsoever
  5. 5. Benefits of comprehensive assessment• Learner accountability • Their dedication matters• Organizational / professional development • Strengths and areas for improvement • Quality processes • Sharing best practice• Added / demonstrated value • Client communications and relationships • Value-based pricing vs. Cost-plus pricing • Targeted sales leads (added services, target market)
  6. 6. THE FOUR LEVELSAssessment Criteria
  7. 7. Kirkpatrick’s 4 Levels - Reaction • Enjoyment? • Priority? • Useful? • Satisfied? • Attendance • Hours • Post-lesson contact
  8. 8. Kirkpatrick’s 4 Levels - Learning • Remember? • What did you learn? • Project performance • Testing • Performance-based rubrics • Trainer / Self assessment
  9. 9. Kirkpatrick’s 4 Levels - Behavior • Did you apply the lessons? • Pre-task reflection • Models and templates • Authentic materials
  10. 10. Kirkpatrick’s 4 Levels - Results • Productivity measurements • Better relationships • Reduced stress • Misunderstanding and conflicts • Performance appraisals
  11. 11. Holton’s External Elements
  12. 12. Holton’s External Elements
  13. 13. Holton’s External Elements
  14. 14. ASSESSMENT METHODSHow exactly should we get this information?
  15. 15. The 5 Ws
  16. 16. Surveys / Feedback• All 4 levels• Tips for sucessful surveys • Face-to-face • Concise • Organized / Consistent • Standardized vs. Customized • Most important questions first • Use open questions sparingly • Create realistic baseline expectations
  17. 17. Performance-based assessment Presentation Introduction • Learning Performance Step Yes No • Transfer designWelcome and thank the audienceState purpose of the presentationGive an outline of the presentationUse discourse markers in outlineMention admin rules (questions, breaks)Signal the end of the introductionIntroduce the first topic using signpostingMaintain eye contact with audienceSpeak clearlyUse appropriate body language Total
  18. 18. Client contact• Externalities• Behavior and results• Targeted trainer-learner conversations • What’s going on in your department right now? • Are there any big projects coming up? • Did that lesson on trade fairs help you in Frankfurt? What helped the most? How was it different than what we practiced? • Have you written any emails in English this week? Can you print it for us?• Client check-ins
  19. 19. BRINGING IT TOGETHERReporting
  20. 20. Ten inputs1. Needs2. Objectives3. Content4. Participants5. Schedule6. Facilities7. Instructor(s)8. Training materials9. Program coordination10. Program evaluation
  21. 21. 4 levels• Reaction – customer satisfaction• Learning – improved knowledge and skills• Behavior – application outside the classroom• Results – impacts on the business
  22. 22. Externalities• Motivation elements• Ability elements• Environment elements
  23. 23. Recommendations• What is impacting our assessment? • 10 Inputs? • External factors?• How do we improve? • Change an input? • Influence an external factor?• What should we share internally?• What should we share externally?
  24. 24. INTO PRACTICEWorkshopping the four levels
  25. 25. Audience survey - Reactionhttp://vot.rs/5ab67c
  26. 26. Focus groups - LearningDiscuss:• Three things you learned from today’s sessions • 4 Levels • Don‘t assume transfer (reality checks) • Learner motivation questions• One thing you would like to hear more about • Operationaizing it. (examples)
  27. 27. Focus groups - Behavior• Discuss how you will apply what you have learned from today’s sessions.• Focus group results: • Review the feedback form • Be more proactive about asking for achievements and successes • Look for testing service (before and after) • 3 months later, what has stuck? (before and after rating) • Use the worksheet as basis
  28. 28. Focus groups - Results• What impact on your organization do you expect to see as a result of today’s training sessions?
  29. 29. TAKING IT FROM HEREConclusion
  30. 30. References• Holton III, E. F. (1996, Spring). The Flawed Four-Level Evaluation Model. Human Resource Development Quarterly, 7(1), 5-21.• Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San Francisco: Berrett-Koehler.• Kirkpatrick, J. (2007, Aug). The Hidden Power of Kirkpatricks Four Levels. T+D Magazine, pp. 34-37.• Kirkpatrick, J. (2011, Spring). The New World Kirkpatrick Four Levels. The Canadian Learning Journal, pp. 19-21.• Maister, D. H. (1993). Managing the Professional Services Firm. New York: Simon & Schuster.• OToole, S. (2009, Aug). Kirkpatrick on Evaluation: Not Crazy After All These Years. Training and Development in Australia, pp. 23-25.• Praslova, L. (2010). Adaptation of Kirkpatricks Four Level Model of Training Criteria to Assessment of Learning Outcomes and Program Evaluation in Higher Education. Educational Assessment, Evaluation & Accountability(22), 215-225.• Steensma, H., & Groeneveld, K. (2010). Evaluating a Training Using the "Four Levels Model". Journal of Workplace Learning, 22(5), 319-331.• Stoel, D. (2004, Jan). The Evaluation Heavy Weight Match. T+D Magazine, pp. 46-48.
  31. 31. For more information / questionsContact: Charles Rei charles.rei@reicommunicationtraining.com http://www.reicommunicationtraining.com @Charlesrei1Presentation and handouts at Business English Ideas http://businessenglishideas.blogspot.de/Ongoing conversation on the BESIG Ning http://iatefl-besig.ning.com/
  32. 32. Quality processes• If you were the client, would you care about these issues?• Do any of these questions represent an unreasonable client expectation?• How confident do you feel that your organization would outperform the competition on these questions?• How valuable would it be if your organization earned top scores on this consistently?• Can you think of another way to ensure consistent performance in these areas?Thank you!

×