Evaluation of training

25,980 views
25,410 views

Published on

Learning Resource for HRM , HRD professionals and MBA students on how to evaluate training

Published in: Business, Technology
1 Comment
26 Likes
Statistics
Notes
  • A comprehensive yet simple and straightforward coverage of the subject matter -both from an academic and practical perspective - kudos to Mr Jayadeva
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
25,980
On SlideShare
0
From Embeds
0
Number of Embeds
29
Actions
Shares
0
Downloads
1,378
Comments
1
Likes
26
Embeds 0
No embeds

No notes for slide
  • Formative evaluation is also useful in analyzing learning materials, student learning and achievements, and teacher effectiveness.... Formative evaluation is primarily a building process which accumulates a series of components of new materials, skills, and problems into an ultimate meaningful whole. - Wally Guyot (1978)
  • If we refer to Kirkpatrick's four levels of evaluation, levels one and two (reactive and learning) are formative evaluations while levels three and four (performance and impact) are summative evaluations. The reactive evaluation is a tool to help determine if the objectives can be reached, the learning evaluation is a tool to help reach the objectives, the performance evaluation is a tool to see if the objectives have actually been met, while the impact evaluation is a tool to judge the value or worth of the objectives. Thus, there are four major break points. The various instruments used to collect the data are questionnaires, surveys, interviews, observations, and testing. The model or methodology used to gather the data should be a specified step-by-step procedure. It should be carefully designed and executed to ensure the data is accurate and valid.
  • 01/19/12 1. Ask participants, bosses, testing, or ask others who are familiar with the needs or objectives. Some examples are surveys or interviews. 2. a. What are the results that you are trying to do? b. What behaviors do you want the participants to have at the end of the training program. c. What knowledge, skills, and/or attitudes do you want your pupils to demonstrate at the end of the training program. 3. Determine subject content to meet needs and objectives. 4. Four decisions: a. Who is the best suited to receive the training. b. Are the training programs required by law (affirmative action). c. Voluntary or required d. Should hourly and salary be included in the same class or be segregated. 5. Solid week or intermittent days. How often should breaks be taken. Should lunch be brought in or allow participants to leave for a hour.
  • 01/19/12 6. Should be comfortable and convenient and appropriate. 7. a. In-house or outside contractors b. Do instructors need to be ‘tailored’ to the special needs in the training program. 8. Two purposes: a. Maintain interest b. Help communicate ideas and skill transfer. Both of these purposes can be accomplished by using single, special interest video cassettes or some type of packaged program. 9. Two scenarios: a. Frustration, and b. Needs of the instructor. 10. The determining effectiveness of a training program are planning and implementation.
  • 01/19/12
  • 01/19/12 The end results after an evaluation are hopefully positive results for both upper management and the program coordinators.
  • 01/19/12 1. If and when downsizing occurs, this statement shall have more meaning than ever for some unlucky people. HRD departments are regarded by upper management as an overhead and not contributing directly to production.
  • 01/19/12 2. Pilot courses may be implemented to see if the participants have the necessary knowledge, or skills, or behavioral changes to make the program work. 3. Kirkpatrick uses eight factors on how to improve the effectiveness of a training program. These eight factors closely follow the Ten Factors of Developing a Training Program. This is a feedback statement spinning off of the Ten Factors.
  • 01/19/12 These are questions asked by HRD coordinators on training performance and the beginning criteria and the expectations of the resulting training program. Business training operations need quantitative measures as well as qualitative measures. A happy medium between these two criteria is an ideal position to fully understand the training needs and to fulfill its development. Quantitative - the research methodology where the investigator's “values, interpretations, feelings, and musings have no place in the positivist’s view of the scientific inquiry.” (Borg and Gall, 1989) cont.
  • Awareness of the need and value of training to the organization. The necessity of involving the Training Manager (or equivalent) in senior management meetings where decisions are made about future changes when training will be essential. Knowledge of and support of training plans. Active participation in events. Requirement for evaluation to be performed and require regular summary report. Policy and strategic decisions based on results and ROI data. the trainer - training evaluation responsibilities Provision of any necessary pre-programme work etc and programme planning. Identification at the start of the programme of the knowledge and skills level of the trainees/learners. Provision of training and learning resources to enable the learners to learn within the objectives of the programme and the learners' own objectives. Monitoring the learning as the programme progresses. At the end of the programme, assessment of and receipt of reports from the learners of the learning levels achieved. Ensuring the production by the learners of an action plan to reinforce, practise and implement learning.   the line manager - training evaluation responsibilities Work-needs and people identification. Involvement in training programme and evaluation development. Support of pre-event preparation and holding briefing meetings with the learner. Giving ongoing, and practical, support to the training programme. Holding a debriefing meeting with the learner on their return to work to discuss, agree or help to modify and agree action for their action plan. Reviewing the progress of learning implementation. Final review of implementation success and assessment, where possible, of the ROI .   the training manager - training evaluation responsibilities Management of the training department and agreeing the training needs and the programme application Maintenance of interest and support in the planning and implementation of the programmes, including a practical involvement where required The introduction and maintenance of evaluation systems, and production of regular reports for senior management Frequent, relevant contact with senior management Liaison with the learners' line managers and arrangement of learning implementation responsibility learning programmes for the managers Liaison with line managers, where necessary, in the assessment of the training ROI .   the trainee or learner - training evaluation responsibilities Involvement in the planning and design of the training programme where possible Involvement in the planning and design of the evaluation process where possible Obviously, to take interest and an active part in the training programme or activity. To complete a personal action plan during and at the end of the training for implementation on return to work, and to put this into practice, with support from the line manager. Take interest and support the evaluation processes. N.B. Although the principal role of the trainee in the programme is to learn, the learner must be involved in the evaluation process. This is essential, since without their comments much of the evaluation could not occur. Neither would the new knowledge and skills be implemented. For trainees to neglect either responsibility the business wastes its investment in training. Trainees will assist more readily if the process avoids the look and feel of a paper-chase or number-crunching exercise. Instead, make sure trainees understand the importance of their input - exactly what and why they are being asked to do.    
  • 01/19/12
  • 01/19/12 With Reaction and Learning, evaluation should be immediate. But evaluating change in Behavior involves some decision-making.
  • 01/19/12
  • 01/19/12
  • 01/19/12 All of these levels are important. However, in later examples of this model, you shall see where large corporations have taken the Kirkpatrick Model and used all of it, only part of it, and still some reversed the order of the levels.
  • 01/19/12 The employee may - Like the new behavior and continue using it. Not like the new behavior and return to doing things the “old way”. Like the change, but be restrained by outside forces that prevent his continuing to use it.
  • 01/19/12 The employee must want to make the change. The training must provide the what and the how . The employee must return to a work environment that allows and/or encourages the change. There should be rewards - Intrinsic - inner feelings of price and achievement. Extrinsic - such as pay increases or praise.
  • 01/19/12 If you have questions, concerns, or information you would like to share concerning the content of this presentation: Email Professor Pershing – pershin@indiana.edu FAX Professor Pershing – 812.339.8792 Submit notes for discussion section Call Professor Pershing – 812.339.3226 END
  • Evaluation of training

    1. 2. Evaluation of Training by Jayadeva de Silva Raman K. Attri ©
    2. 3. Formative Evaluation <ul><li>Formative evaluation (also known as internal) is a method of judging the worth of a program while the program activities are forming (in progress). This part of the evaluation focuses on the process. </li></ul>
    3. 5. Summative <ul><li>The summative evaluation (also know as external) is a method of judging the worth of a program at the end of the program activities (summation). The focus is on the outcome. </li></ul>
    4. 8. The Ten Factors of Developing a Training Program <ul><li>1. Determine needs </li></ul><ul><li>2. Set objectives </li></ul><ul><li>3. Determine subject content </li></ul><ul><li>4. Select qualified applicants </li></ul><ul><li>5. Determine the best schedule </li></ul>
    5. 9. The Ten Factors of Developing a Training Program <ul><li>6. Select appropriate facilities </li></ul><ul><li>7. Select qualified instructors </li></ul><ul><li>8. Select and prepare audiovisual aids </li></ul><ul><li>9. Co-ordinate the program </li></ul><ul><li>10. Evaluate the program </li></ul>
    6. 10. Donald Kirkpatrick
    7. 11. Kirkpatrick
    8. 12. Evaluating <ul><li>“ The reason for evaluating is to determine the effectiveness of a training program.” (Kirkpatrick, 1994, ) </li></ul>
    9. 13. Reasons for Evaluating <ul><li>Kirkpatrick gives three reasons ‘why’ there is a need to evaluate training: </li></ul>
    10. 14. Reasons for Evaluating <ul><li>1.“To justify the existence of the training department by showing how it contributes to the organizations’ objectives and goals.” </li></ul>
    11. 15. Reasons for Evaluating <ul><li>2. “To decide whether to continue or discontinue training programs.” </li></ul>
    12. 16. Reasons for Evaluating <ul><li>3. “To gain information on how to improve future training programs.” (Kirkpatrick, 1994, ) </li></ul>
    13. 17. Kirkpatrick: Evaluating Training Programs <ul><li>“ What is quality training?” </li></ul><ul><li>“ How do you measure it?” </li></ul><ul><li>“ How do you improve it?” </li></ul>
    14. 18. 'Training Evaluation <ul><li>Senior management </li></ul><ul><li>The trainers </li></ul><ul><li>Line management </li></ul><ul><li>The training manager </li></ul><ul><li>The trainee </li></ul>
    15. 20. 4-Tier Practical Training Value Measurement Model Training Expectation Measurement Trainee feedback scores on initial impression on to what extent training met their expectations of learning, skills and knowledge. Tool: Survey Form Student feedback Tier-1 Expectations
    16. 21. 4-Tier Practical Training Value Measurement Model Training Expectation Measurement Trainee feedback scores on initial impression on to what extent training met their expectations of learning, skills and knowledge. Tool: Survey Form Student feedback Tier-1 Expectations Pre-Training Vs Post-Training Assessment Tier-2 Improvement Training Improvement Measurement Assessment of trainee’s training exposure and expertise gained on same set of tasks before and after training. Includes comparison of in-training & Post-training test performance, If applicable. Tool: Pre-training and Post-training Survey Forms. Post-training performance tests .
    17. 22. 4-Tier Practical Training Value Measurement Model Training Expectation Measurement Trainee feedback scores on initial impression on to what extent training met their expectations of learning, skills and knowledge. Tool: Survey Form Training effectiveness Measurement Post-training normalized feedback scores and its quarterly trends. Feedback from trainee Manager on visible incremental changes in trainee skills, service parameters and on-job behavior. Tool: Quarterly Follow-on Survey from Managers Student feedback Tier-1 Expectations Post-Training On-Job Behavior Survey Tier-3 Effectiveness Pre-Training Vs Post-Training Assessment Tier-2 Improvement Training Improvement Measurement Assessment of trainee’s training exposure and expertise gained on same set of tasks before and after training. Includes comparison of in-training & Post-training test performance, If applicable. Tool: Pre-training and Post-training Survey Forms. Post-training performance tests .
    18. 23. 4-Tier Practical Training Value Measurement Model Training Expectation Measurement Trainee feedback scores on initial impression on to what extent training met their expectations of learning, skills and knowledge. Tool: Survey Form Training effectiveness Measurement Post-training normalized feedback scores and its quarterly trends. Feedback from trainee Manager on visible incremental changes in trainee skills, service parameters and on-job behavior. Tool: Quarterly Follow-on Survey from Managers Student feedback Tier-1 Expectations Post-Training On-Job Behavior Survey Tier-3 Effectiveness Improvement in business indicators, revenue figures or success parameters Tier-4 Impact Pre-Training Vs Post-Training Assessment Tier-2 Improvement Training Improvement Measurement Assessment of trainee’s training exposure and expertise gained on same set of tasks before and after training. Includes comparison of in-training & Post-training test performance, If applicable. Tool: Pre-training and Post-training Survey Forms. Post-training performance tests . Training Impact Measurement Impact of training for improving revenues, enhancing business or other success factors driving the training needs. Tool: Comparison of Baseline data and Quarterly business data, Impact Factor calculation sheets
    19. 24. Tier-1: how closely does Training meet the expectations of trainees? <ul><li>Feedback score collected from each trainee at end of the class </li></ul><ul><li>Record general impression about the Training on a numerical scale of 1-5 or similar </li></ul><ul><li>Average score reflects how closely training met expectations of field and how close it is aligned with business needs identified earlier which drives the training. </li></ul>
    20. 25. Reactions. <ul><li>Reaction may best be defined as how well the trainees liked a particular training program.&quot; </li></ul><ul><li>Reactions are typically measured at the end of training. </li></ul>
    21. 27. FEEDBACK FORM Please take a few minutes to fill out this feedback form. Your feedback will help us strengthen the course delivery. Course Title - Name of the facilitator - You may provide feedback of the program on the following criteria by putting a (  ) mark in the space provided. 4 – Exceeds Expectation; 3 – Meets Expectation; 2 – Needs Improvement; 1 – Unsatisfactory 1. What did you like most in the workshop? 2. What did you dislike in the workshop? 3. What other changes would you suggest in case the same course is conducted in future? Learning & Development Human Resources Overall Evaluation Supporting Materials Relevance Examples, Cases, Simulation & Exercises Communication Presentation Style Subject Knowledge 1 2 3 4
    22. 28. Verification of Reactions <ul><li>Typically 'happy sheets' </li></ul><ul><li>Feedback forms based on subjective personal reaction to the training experience </li></ul><ul><li>Verbal reaction which can be noted and analyzed </li></ul><ul><li>Post-training surveys or questionnaires </li></ul><ul><li>Subsequent verbal or written reports given by delegates to managers back at their jobs </li></ul>
    23. 40. Tier-2: how well Training improved skills of trainees? <ul><li>Collect Pre-training and post-training data on trainee’s skills before and after the training. A well drafted skill based survey can be used. </li></ul><ul><li>Compare improvements in exposure or expertise gained by the trainee as seen on post-training survey by comparing it with pre-training survey. </li></ul><ul><li>Conduct in-training skill tests as part of the qualification criteria to measure the improvement in the learned skills. </li></ul>
    24. 41. Learning. <ul><li>What principles, facts, and techniques were understood and absorbed by the participants?&quot; </li></ul><ul><li>What the trainees know or can do can be measured during and at the end of training </li></ul>
    25. 44. Verification of Learning <ul><li>WHAT? </li></ul><ul><li>What knowledge was acquired? </li></ul><ul><li>What skills were developed or enhanced? </li></ul><ul><li>What attitudes were changed? </li></ul><ul><li>HOW? </li></ul><ul><li>Tests before and after the training </li></ul><ul><li>Interview or observation can be used before and after training. </li></ul><ul><li>Measurement and analysis is possible and easy on a group scale </li></ul><ul><li>Reliable, clear scoring and measurements need to be established </li></ul>
    26. 45. Tier-3: How effective does the training prove in changing On-job-behavior and skills of trainee? <ul><li>Collect quarterly feedback from trainee’s manager on his observations on improvement of trainee’s on-job performance/ behavior or skills after attending training. </li></ul><ul><li>Analyze the trends in key on-job-performance parameters or indicators as seen on survey with respect to previous quarters. </li></ul><ul><li>Convert delta into normalized scores to indicate the value created by training in the work efficiency of the trainee </li></ul><ul><li>Ensure continual measurement of effectiveness after regular intervals to assess long term value of the training. </li></ul>
    27. 46. Behavior. <ul><li>Changes in on-the-job behavior,Behavior changes are acquired in training and they then transfer (or don't transfer) to the work place. </li></ul><ul><li>What skills did the learner develop, that is, what new information is the learner using on the job? </li></ul>
    28. 47. When evaluating change in behavior, decide: <ul><li>When to evaluate </li></ul><ul><li>How often to evaluate </li></ul><ul><li>How to evaluate </li></ul>
    29. 54. Verification of Behaviour <ul><li>What? </li></ul><ul><li>Whether the trainee is able to transfer the learning to the work environment </li></ul><ul><li>New learning is demonstrated </li></ul><ul><li>Whether the trainee is motivated </li></ul><ul><li>How? </li></ul><ul><li>self-assessment can be useful, using carefully designed criteria and measurements </li></ul><ul><li>cooperation and skill of observers, typically line-managers, are important factors, and difficult to control </li></ul><ul><li>Use of focus groups </li></ul>
    30. 55. Tier-4: To what extent training does impact in improving business / revenue or service parameters specific to trainees or group of trainees? <ul><li>Measure quarterly the business indicators of the trainee’s job or service parameters based on nature of job. Business indicators could be collected based on individual job or group responsible for the said function. </li></ul><ul><li>Ideally business indicator data before the training should be used as baseline. </li></ul><ul><li>Record the business parameters or governing service parameters on quarterly basis </li></ul><ul><li>Calculate qualitative or quantitative impact factor based on normalized delta. </li></ul>
    31. 56. Results. <ul><li>Reduction of costs; </li></ul><ul><li>Reduction of turnover and absenteeism; </li></ul><ul><li>Reduction of grievances; </li></ul><ul><li>increase in quality and quantity or production; </li></ul><ul><li>or Improved morale which, it is hoped, will lead to some of the previously stated results. </li></ul><ul><li>These factors are also measurable in the workplace </li></ul>
    32. 61. The Four Levels <ul><li>Reaction </li></ul><ul><li>Learning </li></ul><ul><li>Behavior </li></ul><ul><li>Results </li></ul>
    33. 63. <ul><li>“ The Four Levels represent a sequence of ways to evaluate (training) programs….As you move from one level to the next, the process becomes more difficult and time-consuming, but it also provides more valuable information.” (Kirkpatrick, 1994,) </li></ul>
    34. 64. When all conditions are met, the employee must: <ul><li>Realize an opportunity to use the behavioral changes. </li></ul><ul><li>Make the decision to use the behavioral changes. </li></ul><ul><li>Decide whether or not to continue using the behavioral changes. </li></ul>
    35. 65. Return on Impact (RoI) is a new approach which measures the difference training has created on those governing business/ service or revenue parameters which drives training needs.
    36. 66. The four conditions Kirkpatrick identifies for changes to occur: <ul><li>Desire to change </li></ul><ul><li>Knowledge of what to do and how to do it </li></ul><ul><li>Work in the right climate </li></ul><ul><li>Reward for (positive) change </li></ul>
    37. 68. An individual and group exercise <ul><li>Group 1- As an HR Manager,how will you evaluate:- </li></ul><ul><li>T he output effectiveness of a supervisory training programme? </li></ul><ul><li>Group 2- The Outputs to be Evaluated after a Sales Training programme </li></ul><ul><li>Group 3- The Outputs to be Evaluated after a Customer Relations Training programme </li></ul>
    38. 69. Outputs to be Evaluated after Supervisory and Management Training <ul><li>increased output </li></ul><ul><li>reduced absenteeism and tardiness </li></ul><ul><li>reduced cost of new hires </li></ul><ul><li>reduced turnover </li></ul><ul><li>increased number of employee suggestions </li></ul><ul><li>climate survey data (morale and attitudes) </li></ul>
    39. 70. Outputs to be Evaluated after Sales Training <ul><li>sales volume </li></ul><ul><li>average sale size </li></ul><ul><li>add-on sales </li></ul><ul><li>close-to-call ratio </li></ul><ul><li>ratio of new accounts to old accounts </li></ul><ul><li>number of items per order </li></ul>
    40. 71. Outputs to be Evaluated After Customer Relations Training <ul><li>accuracy of orders </li></ul><ul><li>size of orders </li></ul><ul><li>number of transactions per day </li></ul><ul><li>adherence to credit procedures </li></ul><ul><li>number of lost customers </li></ul><ul><li>amount of repeat business </li></ul><ul><li>number of referrals </li></ul><ul><li>number of complaints </li></ul>
    41. 72. Questions & Concerns

    ×