Got Results? Evaluating Training Impact

1,324 views
1,161 views

Published on

This presentation was very well received by both the International Personnel Management Association and the Canadian Society for Training and Development's professional events for members in May 2012.

Published in: Education, Business, Technology

Got Results? Evaluating Training Impact

  1. 1. Got Results?Evaluating Training Impact Karen Carleton, MEd, MS Performance Solutions Corp. www.performcorp.caCSTD SymposiumEdmontonMay 31, 2012 Copyright © 2012 Karen Carleton
  2. 2. PollHow certain are you that your Training isgetting the desired results?A) Very certainB) Moderately certainC) Somewhat certainD) Not certain at allE) N/A Copyright © 2012 Karen Carleton
  3. 3. “Intangible assets represent …30-50% of a company’s market value.” -Ernst & Young (in Gayeski, 2005) Human capital Copyright © 2012 Karen Carleton
  4. 4. Copyright © 2012 Karen Carleton 2. Training Evaluation Standard (old & new) 1. Learning Objectives & Performance Objectives3. ROE vs. 4. Transfer ROI & Impact 4
  5. 5. Copyright © 2012 Karen Carleton 2. Training Evaluation Standard (old & new) 1. Learning Objectives & Performance Objectives3. ROE vs. 4. Transfer ROI & Impact 5
  6. 6. TrainSkill-basedResultsfocused,practical Copyright © 2012 Karen Carleton
  7. 7. EducateKnowledge-basedCapacitybuilding 7Copyright © 2012 Karen Carleton
  8. 8. Learn Relevant Immediately meaningful Copyright © 2012 Karen Carleton
  9. 9. KSAs (Competencies) Knowledge – cognitive learning Skills - behavior learning Attitudes – social learning 9 Copyright © 2012 Karen Carleton
  10. 10. Copyright © 2012 Karen Carleton Training Caution the Gap” “Closing
  11. 11. Learning Objectives SWBAT/LWBAT Student/Learner Will Be Able To… e.g. Create Excel spreadsheets Copyright © 2012 Karen Carleton
  12. 12. Copyright © 2012 Karen CarletonPCC MethodPerformanceCriteriaConditionse.g.P – Will typeC – 45 wpm with 2% or less error rateC – after 2 hours of daily practice for 3 weeks. “Will be able to type.” Performance Objectives
  13. 13. Example (L or P)Trainees will be able tooperate the crusher afterpractical training.Learning objectiveA) Learning ObjectiveORB) Performance Objective Copyright © 2012 Karen Carleton
  14. 14. Example (L or P) Employees will correctly demonstrate techniques for resolving conflict, by the end of the 2-day workshop and doing guided practice exercises.A) Learning ObjectiveOR PerformanceB) Performance Objective objective Copyright © 2012 Karen Carleton
  15. 15. Example (L or P) Welding apprentices shall learn how to prepare an edge before welding a plate edge or pipe. Learning objectiveA) Learning ObjectiveORB) Performance Objective Copyright © 2012 Karen Carleton
  16. 16. Example (L or P) Following an in-class activity and video on new sterilization procedures, the nursing aides will accurately demonstrate the proper procedure. Performance A) Learning Objective objective ORCopyright © 2012 Karen Carleton B) Performance Objective
  17. 17. Learning vs. Performance Objectives 1. Most common? A)Learning Objectives B)Performance 2. Preferable? Why? ObjectivesCopyright © 2012 Karen Carleton
  18. 18. Training Design Target Needs Draft Audience Copyright © 2012 Karen Carleton
  19. 19. ADDIE ModelFeedback loop 19 Copyright © 2012 Karen Carleton
  20. 20. Evaluation - determining value or worth;examining, judging; giving an appraisal. Threat OR Learning opportunity Copyright © 2012 Karen Carleton
  21. 21. Copyright © 2012 Karen Carleton 2. Training Evaluation Standard (old & new) 1. Learning Objectives & Performance Objectives3. ROE vs. 4. Transfer ROI & Impact 21
  22. 22. Training Evaluation Gold StandardKirkpatrick Model Kirkpatrick Model Kirkpatrick Model 4 Levels of 50+ year Universally Evaluation old model recognized; CSTD/ASTD Copyright © 2012 Karen Carleton
  23. 23. Kirkpatrick’s 4 Levels 1 • Learner Reaction • Learning (KSAs) 2 • Behavior change (applied 3 learning; on-the-job) • Organizational results ($, 4 time, quality, etc.) Dr. Donald Kirkpatrick Copyright © 2012 Karen Carleton
  24. 24. 4 Levels Stats Level 1 – most companies “Smile sheets”, forms, verbal Level 4 - 5-10% companies Recommend: evaluate all 4 levels Increased complexity & cost with levelsWhich Training Evaluation level(s) does your company use?A) 1 B) 2 C) 3 D) 4 E) Combination of levels Copyright © 2012 Karen Carleton
  25. 25. NEW 4 Levels Model - KBPM “IMPACT METRICS” Level 4 – Results “CONSUMPTIVE For Org. (outcomes) METRICS” Level 3 – Behaviour (applied learning) Level 2 – Learning of knowledge/skills (degree) Level 1 – Reaction favourably to learning event Dr. Jim KirkpatrickCopyright © 2012 Karen Carleton
  26. 26. Desired Results“The Ultimate intent of the Kirkpatrick Four Levels…is to show the value and worth of training” –Kirkpatrick & Kayser Kirkpatrick (2009) “Begin with the end in mind” -Stephen Covey (1989) Copyright © 2012 Karen Carleton
  27. 27. KPBM Model http://www.kirkpatrickpartners.com/
  28. 28. Copyright © 2012 Karen Carleton KBPM Model Don’t assume “Transfer” Consistently reinforce learning & application - e.g. coaching/mentoring, feedback, refreshers, job aids A) Training/L & D is “On Trial” internally B) Training/L & D is “On Trial” as a profession C) Both D) Neitherhttp://www.kirkpatrickpartners.com/ Copyright © 2012 Karen Carleton
  29. 29. Copyright © 2012 Karen Carleton 2. Training Evaluation Standard (old & new) 1. Learning Objectives & Performance Objectives3. ROE vs. 4. Transfer ROI & Impact 29
  30. 30. Kirkpatrick Business Partnership Model Return On Expectations (ROE) Stakeholder value Contributions to “Success” Business Partnership Cooperation Compelling “chain of evidence”http://www.kirkpatrickpartners.com/ Copyright © 2012 Karen Carleton
  31. 31. Return on Investment (ROI) PhD HRD, U Alabama; ~35 books about ROI in ~33 languages ROI Institutehttp://www.roiinstitutecanada.com/ Dr. Jack Phillips Copyright © 2012 Karen Carleton
  32. 32. ROI Methodology1. Reaction, Satisfaction, Planned Action2. Learning3. Application/Implementation4. Business Impact5. Return on Investment (ROI)6. Intangibles $$$ “Level 5” $$$ Copyright © 2012 Karen Carleton
  33. 33. Return on Investment (ROI)(Program Benefits – Program Costs) = ROI Program Costs Net profit/savings; business case Less than 5% organizations do ROI Why? Copyright © 2012 Karen Carleton
  34. 34. Return on Investment (ROI)Bottom-line results Isolation techniquesTraining impact (program vs. other causes) e.g. trends, before/after testROI challenges - Resources, know- how - Linking results back to program (Gayeski, 2005) Copyright © 2012 Karen Carleton
  35. 35. Jack vs. Jim ROI ROE Dr. Jack Phillips Dr. Jim Kirkpatrick http://talentmgt.com/articles/view/beware-vague-learning-jargon/1A) ROEB) ROI 35C) Both Copyright © 2012 Karen Carleton
  36. 36. Copyright © 2012 Karen Carleton 2. Training Evaluation Standard (old & new) 1. Learning Objectives & Performance Objectives3. ROE vs. 4. Transfer ROI & Impact 36
  37. 37. Transfer of Learning Learn ? Apply “That almost magical link betweenclassroom performance and somethingwhich is supposed to happen in the real world.” -J.M. Swinney Copyright © 2012 Karen Carleton
  38. 38. Plan Transfer - avoid wasted resources Training ScrapCopyright © 2012 Karen Carleton
  39. 39. Fostering Connections Simulations, role-plays, guided practice -Realistic practice -High fidelityCopyright © 2012 Karen Carleton
  40. 40. “…training is unlikely to improve job performance without a workenvironment supportive of learning transfer.” -Reid Bates in Improving Learning Transfer in Organizations Copyright © 2012 Karen Carleton
  41. 41. Transfer Supports Discussion Case studies; scenarios Learner goals BEFORE During After Copyright © 2012 Karen Carleton
  42. 42. Transfer Supports Expert guests Examples, analogies Feedback & guidance; reflection Share application tips Before DURING After Copyright © 2012 Karen Carleton
  43. 43. Transfer Supports Others? Checklists, action plans, Q & A, discussion Before During AFTER Copyright © 2012 Karen Carleton
  44. 44. Transfer ExampleTruck driver trainees Read manual Simulator Drive with trainer Drive under observationDrive aloneCopyright © 2012 Karen Carleton
  45. 45. Transfer Example Facilitator course  Study & Icebreakers discussion  Demos  Practice Facilitating with guidanceScaffolding or Support Copyright © 2012 Karen Carleton
  46. 46. Transfer ApproachesTraining Focus Performance FocusTraining Performance needsSome needs analysis Thorough analysis of for training (often) performance needsFocus: Transfer Focus: Results (*may include training) Copyright © 2012 Karen Carleton
  47. 47. Clear Performance StandardsNecessary SupportAppropriate ConsequencesUseful FeedbackIndividual Capacity Copyright © 2012 Karen Carleton*Needed Skills & Knowledge *Adapted from Rummler & Brache, 1995
  48. 48. Top Transfer TipsIdentify measurable Ensure transfer performance resources ($, time, standards people, etc.)Involve Managers & Assess trainee Supervisors perceptions ofCommunicate transfer factors training-PI link to Invite stakeholder Trainees feedback Copyright © 2012 Karen Carleton
  49. 49. Evaluating ImpactEvaluation isessential 1. Post-training 2. On-the-job 3. Program level Copyright © 2012 Karen Carleton
  50. 50. Gauging TransferCopyright © 2012 Karen Carleton
  51. 51. Transfer Supports used internally:A. Post-training SurveyB. Follow-up calls/visitsC. Workplace ObservationsD. Skills demos/TestsE. OtherF. Combination G. Not sure (yet!) Copyright © 2012 Karen Carleton
  52. 52. Summary Quiz1. Three parts to writing performance objectives. Performance1) _______________ Criteria2) _______________ Conditions3) _______________ Knowledge, Skills, & Attitudes2. KSAs= _________________________________.3. The Four Levels of Training Evaluation are: Reaction, Learning, Behavior change, Business Results ______________________________________________.4. Training/Learning and Development professionals should Business Partners be strategic ___________________. Copyright © 2012 Karen Carleton
  53. 53. Summary Quiz5. The main difference between the old and new Kirkpatrick model is:Model is flipped, now begins with business needs__________________________________________.Why?6. Sponsored training should have a Return On xpectations nvestment E__________ AND a Return on I_________.7. Example of a Transfer support?8. Quick way to gauge Training Impact? Copyright © 2012 Karen Carleton
  54. 54. Value-added Training Performance Objectives Plan for Transfer Evaluate at Levels 1 & 2 at least Have an ROE & ROI mindsetCopyright © 2012 Karen Carleton
  55. 55. “Transfer interventions will be most successful when the explicit goal is Performance Improvement.”-Holton & Baldwin in Improving Learning Transfer in Organizations (2003) Copyright © 2012 Karen Carleton
  56. 56. PresenterKaren CarletonPerformance Solutions Corp.Karen@performcorp.cawww.performcorp.ca Back of Your Business Card:(780) 705-7701 #1 – YES – (Send me  Training program evaluation “Three Critical Keys to  Performance needs analysis & Adult Learning”) custom solutions #2 – YES (I’d like a FREE  Seminars & workshops Training Evaluation Strategy Chat for myCopyright © 2012 Karen Carleton company)
  57. 57. References Carliner, S. (2003). Training design basics. Alexandria, VA: ASTD. Designed for Learning. Retrieved January 23, 2012 from, http://tarunagoel.blogspot.com/2009/11/transfer-of-learning- theories-and.html. Gayeski, D. (2005). Chapter 7: Translating Data to Dollars. in Managing Learning and Communication Systems as Business Assets. Upper Saddle River NJ: Pearson Prentice Hall. Kirkpatrick and Partners. Retrieved January 23, 2012 from, http://www.kirkpatrickpartners.com Kirkpatrick, J. D. & Kayser Kirkpatrick, W. (2010). Training on trial: How workplace learning must reinvent itself to remain relevant. New York, NY: AMACOM Books, A Division American Management Association.Copyright © 2012 Karen Carleton
  58. 58. References Bates, R.A. (2003). Chapter twelve -- Managers as Transfer Agents in 243-270. In E. F. Holton III and T. T. Baldwin (Ed.s), Improving Learning Transfer in Organizations (pp. 243-270). San Francisco: Jossey-Bass. ROI Institute Canada, Retrieved May 19, 2012 from, http://www.roiinstitutecanada.com/ Warren, M.W. (1979). Training for Results: A systems approach to the development of the human resources in industry (2nd ed.). Menlo Park, CA: Addison-Wesley Publishing Company.Copyright © 2012 Karen Carleton

×