This document outlines a presentation on assessing and reporting training quality. It discusses: 1. Identifying the benefits of comprehensive training assessment, including learner accountability, organizational development, and demonstrating value. 2. Kirkpatrick's four levels of training evaluation - reaction, learning, behavior, and results - and how to assess each using both quantitative and qualitative metrics. 3. Holton's model of external factors that can influence training assessment. 4. Methods for assessment, including surveys, performance-based assessments, client contact, and focus groups to evaluate different levels. 5. Elements to include in a comprehensive training evaluation report, such as needs, objectives, recommendations, and references.

1. Are we fulfilling our promise?
Assessing and reporting training quality
IATEFL BESIG Annual Conference
Stuttgart, Germany
November 16, 2012
Charles Rei

2. Learning objectives
By the end of this session we will be able to...
1. Identify the benefits of comprehensive training
assessment.
2. Identify and assess the four levels of training evaluation
using quantitative and qualtative metrics.
3. Identify external factors which affect assessment.
4. Use surveys, performance-based assessment, and
client contact to evaluate training.
5. Identify the main elements and organization of a
comprehensive training evaluation report.

3. Training
Perks
• A gift to the
employees
• Knowledge-
Education based
• Standardized
• Performance-
based
Goals
• Personal
development

4. Current situation
• Surveys / Feedback forms
• Observation
• Interviews
Formative
• Testing
vs.
• Self-assessment
Summative
• Can do statements assessment
• Portfolios and projects
• Regular reports
• Status meetings
• Cancellations and renewals
• None whatsoever

5. Benefits of comprehensive assessment
• Learner accountability
• Their dedication matters
• Organizational / professional development
• Strengths and areas for improvement
• Quality processes
• Sharing best practice
• Added / demonstrated value
• Client communications and relationships
• Value-based pricing vs. Cost-plus pricing
• Targeted sales leads (added services, target market)

6. THE FOUR LEVELS
Assessment Criteria

7. Kirkpatrick’s 4 Levels - Reaction
• Enjoyment?
• Priority?
• Useful?
• Satisfied?
• Attendance
• Hours
• Post-lesson contact

8. Kirkpatrick’s 4 Levels - Learning
• Remember?
• What did you learn?
• Project performance
• Testing
• Performance-based
rubrics
• Trainer / Self
assessment

9. Kirkpatrick’s 4 Levels - Behavior
• Did you apply the
lessons?
• Pre-task reflection
• Models and templates
• Authentic materials

10. Kirkpatrick’s 4 Levels - Results
• Productivity
measurements
• Better relationships
• Reduced stress
• Misunderstanding and
conflicts
• Performance
appraisals

11. Holton’s External Elements

12. Holton’s External Elements

13. Holton’s External Elements

14. ASSESSMENT METHODS
How exactly should we get this information?

15. The 5 Ws

16. Surveys / Feedback
• All 4 levels
• Tips for sucessful surveys
• Face-to-face
• Concise
• Organized / Consistent
• Standardized vs. Customized
• Most important questions first
• Use open questions sparingly
• Create realistic baseline expectations

17. Performance-based assessment
Presentation Introduction • Learning
Performance Step Yes No
• Transfer design
Welcome and thank the audience
State purpose of the presentation
Give an outline of the presentation
Use discourse markers in outline
Mention admin rules (questions, breaks)
Signal the end of the introduction
Introduce the first topic using signposting
Maintain eye contact with audience
Speak clearly
Use appropriate body language
Total

18. Client contact
• Externalities
• Behavior and results
• Targeted trainer-learner conversations
• What’s going on in your department right now?
• Are there any big projects coming up?
• Did that lesson on trade fairs help you in Frankfurt? What helped
the most? How was it different than what we practiced?
• Have you written any emails in English this week? Can you print it
for us?
• Client check-ins

19. BRINGING IT TOGETHER
Reporting

20. Ten inputs
1. Needs
2. Objectives
3. Content
4. Participants
5. Schedule
6. Facilities
7. Instructor(s)
8. Training materials
9. Program coordination
10. Program evaluation

21. 4 levels
• Reaction – customer
satisfaction
• Learning – improved
knowledge and skills
• Behavior – application
outside the classroom
• Results – impacts on
the business

22. Externalities
• Motivation elements
• Ability elements
• Environment elements

23. Recommendations
• What is impacting our
assessment?
• 10 Inputs?
• External factors?
• How do we improve?
• Change an input?
• Influence an external
factor?
• What should we share
internally?
• What should we share
externally?

24. INTO PRACTICE
Workshopping the four levels

25. Audience survey - Reaction
http://vot.rs/5ab67c

26. Focus groups - Learning
Discuss:
• Three things you learned from today’s sessions
• 4 Levels
• Don‘t assume transfer (reality checks)
• Learner motivation questions
• One thing you would like to hear more about
• Operationaizing it. (examples)

27. Focus groups - Behavior
• Discuss how you will apply what you have learned from
today’s sessions.
• Focus group results:
• Review the feedback form
• Be more proactive about asking for achievements and successes
• Look for testing service (before and after)
• 3 months later, what has stuck? (before and after rating)
• Use the worksheet as basis

28. Focus groups - Results
• What impact on your organization do you expect to see as
a result of today’s training sessions?

29. TAKING IT FROM HERE
Conclusion

30. References
• Holton III, E. F. (1996, Spring). The Flawed Four-Level Evaluation Model. Human
Resource Development Quarterly, 7(1), 5-21.
• Kirkpatrick, D. L. (1994). Evaluating Training Programs: The Four Levels. San
Francisco: Berrett-Koehler.
• Kirkpatrick, J. (2007, Aug). The Hidden Power of Kirkpatrick's Four Levels. T+D
Magazine, pp. 34-37.
• Kirkpatrick, J. (2011, Spring). The New World Kirkpatrick Four Levels. The
Canadian Learning Journal, pp. 19-21.
• Maister, D. H. (1993). Managing the Professional Services Firm. New York: Simon
& Schuster.
• O'Toole, S. (2009, Aug). Kirkpatrick on Evaluation: Not Crazy After All These
Years. Training and Development in Australia, pp. 23-25.
• Praslova, L. (2010). Adaptation of Kirkpatrick's Four Level Model of Training
Criteria to Assessment of Learning Outcomes and Program Evaluation in Higher
Education. Educational Assessment, Evaluation & Accountability(22), 215-225.
• Steensma, H., & Groeneveld, K. (2010). Evaluating a Training Using the "Four
Levels Model". Journal of Workplace Learning, 22(5), 319-331.
• Stoel, D. (2004, Jan). The Evaluation Heavy Weight Match. T+D Magazine, pp.
46-48.

31. For more information / questions
Contact: Charles Rei
charles.rei@reicommunicationtraining.com
http://www.reicommunicationtraining.com
@Charlesrei1
Presentation and handouts at Business English Ideas
http://businessenglishideas.blogspot.de/
Ongoing conversation on the BESIG Ning
http://iatefl-besig.ning.com/

32. Quality processes
• If you were the client, would you care about these issues?
• Do any of these questions represent an unreasonable
client expectation?
• How confident do you feel that your organization would
outperform the competition on these questions?
• How valuable would it be if your organization earned top
scores on this consistently?
• Can you think of another way to ensure consistent
performance in these areas?
Thank you!