Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.



Published on

A presentation of a level 4 evaluation

  • Be the first to comment

  • Be the first to like this


  1. 1. The Experience of Conducting a Level 4 Evaluation in a Manufacturing Environment
  2. 2. Training Process Model
  3. 3. Partnership <ul><li>Link training to results </li></ul><ul><li>Level 4 viewed as most important </li></ul><ul><li>(stakeholders) but most difficult (trainers). </li></ul><ul><li>Let stakeholders define their expectations for </li></ul><ul><li>you </li></ul><ul><li>Identify metrics that will best demonstrate & </li></ul><ul><li>deliver on those expectations </li></ul>
  4. 4. Analysis <ul><li>Division Needs Analysis </li></ul><ul><ul><li>Survey of Mill Production Workers </li></ul></ul><ul><ul><li>Survey of Mill Managers </li></ul></ul><ul><ul><li>Review of Results at Mill Managers Yearly Conference </li></ul></ul><ul><li>Review of Division Goals for Year </li></ul><ul><ul><li>Established by Upper Management </li></ul></ul>
  5. 5. Expected Outcomes <ul><li>Identified Needs </li></ul><ul><ul><li>Improved Quality </li></ul></ul><ul><ul><li>Achieved through Improved Operator Training </li></ul></ul><ul><li>Improvement Metrics </li></ul><ul><ul><li>Customer Return Statistics </li></ul></ul><ul><ul><li>Operator-Controllable Errors </li></ul></ul>
  6. 6. Evaluation <ul><li>Level 1 </li></ul><ul><ul><li>Survey of Participants </li></ul></ul><ul><li>Level 2 </li></ul><ul><ul><li>Course Tests Results </li></ul></ul><ul><ul><li>Checklist Completion </li></ul></ul><ul><li>Level 3 </li></ul><ul><ul><li>Observations and Sign-Offs by Supervisors </li></ul></ul><ul><li>Level 4 </li></ul><ul><ul><li>Analysis of Operator-Controllable Error Metrics </li></ul></ul><ul><ul><li>Survey of Training Team to Identify Success Factors </li></ul></ul>
  7. 7. Evaluate <ul><li>Data—borrow or gather yourself. </li></ul><ul><ul><ul><ul><li>Borrowed comes from business itself and can </li></ul></ul></ul></ul><ul><ul><ul><ul><li>provide before/after picture </li></ul></ul></ul></ul><ul><li>Be sure data links training to results </li></ul><ul><ul><ul><ul><li>Use appropriate data to create chain of evidence </li></ul></ul></ul></ul>
  8. 8. Level 4 – 5 Questions <ul><li>What? </li></ul><ul><ul><li>Did training have desired effect on specified quality metrics? </li></ul></ul><ul><ul><li>What factors contributed to success or failure of training? </li></ul></ul><ul><li>Who? </li></ul><ul><ul><li>Specific Operator Positions </li></ul></ul><ul><ul><li>Training Team & Management </li></ul></ul><ul><li>How? </li></ul><ul><ul><li>Training records </li></ul></ul><ul><ul><li>Operator-controllable errors metrics in lbs of paper & $ credit </li></ul></ul><ul><ul><li>Survey </li></ul></ul><ul><li>When? </li></ul><ul><ul><li>Before,during and after training </li></ul></ul><ul><li>Why? </li></ul><ul><ul><li>To determine if training intervention achieved desired results </li></ul></ul><ul><ul><li>To determine most influential factors </li></ul></ul><ul><ul><li>To utilize evaluation to refine program </li></ul></ul>
  9. 9. The Results What the Metrics Revealed
  10. 10. Training/Quality Metrics
  11. 11. The Rest of the Story Look Beyond the Obvious
  12. 12. Training
  13. 13. Quality Metrics
  14. 14. Lessons Learned <ul><li>Be sure to look at entire picture. Do not be satisfied with first picture </li></ul><ul><li>Details can pinpoint specific areas for improvement in program and/or data gathering </li></ul><ul><li>When conducting initial analysis, look at what is working well and determine why </li></ul>
  15. 15. Survey Results What Did the Business Feel Contributed Most to the Process? No surprises here!
  16. 16. Survey Results <ul><li>The respondents to the survey ranked these factors in the following order. </li></ul><ul><li>Support of upper management </li></ul><ul><li>Effectiveness of the Training Coordinators </li></ul><ul><li>Structured training and certification process </li></ul><ul><li>Quality of the training materials </li></ul><ul><li>(tie) Location of training center </li></ul><ul><ul><li> Diversity of training materials </li></ul></ul><ul><ul><li> Focus on safety </li></ul></ul><ul><li>Standardization of training and qualifications </li></ul><ul><li>Hourly Workforce Training Survey and Needs Assessment utilized to determine top training needs </li></ul><ul><li>Effect on employee morale </li></ul>
  17. 17. References <ul><li>Jacobs, Unleashing Employee Expertise </li></ul><ul><li>Kirkpatrick Implementing the Four Levels </li></ul><ul><li>Robinson, Dissertation </li></ul>