Start with the story of me finding the binders of program evaluations
1. TRUTHS ABOUTTRAININGEVALUATION David Kelly, CPLP, CRP
2. A LITTLE ABOUT ME Training Director for over 10 Years Director, Center for Learning at ACLD Certified Professional in Learning and Performance (CPLP) Certified ROI Professional (CRP) Board Member of ASTD Long Island Member of ASTD’s National Advisors for Chapters Member of the eLearn Magazine Editorial Board
3. TODAY’S DISCUSSION A Brief History of Measurement in Learning and Development Programs What do Trainers Measure? Does What We Measure Even Matter? The Question of Credibility A Better Definition for Training Success
4. WHAT CAME BEFORE… Why are we talking about ‘evaluation’ of training in the first place?
5. A BRIEF HISTORY OF TRAINING EVALUATIONDon Kirkpatrick – The Four Levels of Evaluation 1. Reaction 2. Learning 3. Behavior 4. ResultsJack Phillips – ROI Methodology 5. ROI (Benefits / Costs)
6. A BRIEF HISTORY OF TRAINING EVALUATION
7. CAPITAL NON-CAPITALINVESTMENTS INVESTMENTS Buildings Marketing Equipment Human Resources Tools Quality Vehicles Some Technology Companies Staff Support Some Technology Processes ROI is usually ROI is not typically measured measuredWHY MEASURE TRAINING?
8. CAPITAL NON-CAPITALINVESTMENTS INVESTMENTS Buildings Marketing Equipment Human Resources Tools Quality Vehicles Some Technology Companies Staff Support Some Technology Processes ~15% of Total ~85% of Total Expenditures ExpendituresWHY MEASURE TRAINING?
9. SATISFACTION WITH MEASURES OF SUCCESSOF LEARNING AND DEVELOPMENTTo what extent are you satisfied with themeasures (value) of learning anddevelopment?1. Very Dissatisfied – 8%2. Dissatisfied – 45%3. Satisfied – 41%4. Very Satisfied – 6%*CEO Survey—Fortune 500 and LargePrivate Companies, ROI Institute
10. WHAT METRICS ARE YOU USING?
11. COMMON LEARNING METRICS Cost per Employee or Hour Do these metrics Course Completions REALLY matter? Learner Satisfaction Attendance Test Scores
12. MEANINGFUL PERFORMANCE METRICS Decreased Time to Performance Reduced Performance Cycle Times Quality Improvements Increased Customer Satisfaction Increased Sales
13. HERE’S MY TWO CENTS… Measurement for measurement’s sake is a waste of time. Ask yourself… “Will I be doing anything with the data we’re collecting?”
14. THE CREDIBILITY ISSUEThe truth is most training evaluation data is either irrelevant or does not hold up to scrutiny *Bozarth-Ferguson Magic Formula used with permission
15. THE CREDIBILITY ISSUETraining: “Sales increased 40% after Training.”Marketing: “That wasn’t training, we ranadvertisements during that period.”Training: “Turnover has decreased 25% as aresult of our new orientation programs.”Human Resources: “That wasn’t training, weupdated hiring criteria.”
16. THE CREDIBILITY ISSUE Source: http://elearningindustry.com/subjects/free-elearning-resources/item/256-free-elearning-roi-
17. THE BOTTOM LINEROI and the metrics used for TrainingEvaluation are driven by companies andconsultants with a vested interest inpeople believing these metrics matter.In most cases, the data collected isn’treally relevant to business outcomes.
18. HERE’S MY TWO CENTS… It’s not about Learning; It’s about Performance It’s not about what people KNOW; it’s about what people DO.
19. HERE’S MY TWO CENTS… Most executives that ask for an ROI, DO NOT WANT AN ACTUAL ROI. They want to know that the efforts had value.
20. MAYBE IT’S NOT ABOUT EVALUATION… What other ways can we determine success besides traditional training evaluation?
21. WHO DECIDES WHAT SUCCESS MEANS?The Program Stakeholders Executives Management Approving Financing of Training Learning and Development Professionals Program Participants Managers of Program Participants
22. HERE’S MY TWO CENTS… The time to think about how you will define success is during the DESIGN of a program, not after it’s completion.
23. WHAT’S YOUR STORY? A strong testimonial from a participant or executive on how training helped workers perform better is often more powerful than any number on a spreadsheet.
24. A STRUCTURED APPROACH
25. THE SUCCESS CASE METHOD Step 1. Identify targeted business goals and impact expectations.
26. THE SUCCESS CASE METHOD Step 2. Survey a large representative sample of all participants in a program to identify high impact and low impact cases.
27. THE SUCCESS CASE METHOD Step 3. Analyze the survey data to identify: a small group of successful participants a small group unsuccessful participants
28. THE SUCCESS CASE METHOD Step 4. Conduct interviews with the two selected groups to: Document the nature and business value of their application of learning Identify the performance factors that supported learning application and obstacles that prevented it.
29. THE SUCCESS CASE METHOD Step 5. Document and disseminate the story: Report impact Applaud successes Use data to educate managers and organization
30. IT’S NOT ABOUT HOW MANY SEATS ARE FILLED It’s not about the Training, it’s about the Performance Improvement. It’s about an ongoing process, not an event. “Evaluating training is like evaluating the wedding instead of the marriage.” - Robert Brinkhoff
31. IT’S NOT ABOUT HOW MANY SEATS ARE FILLED Most learning and performance improvement takes place informally, and on the job. The success of a training program is not determined via a test; it is determined by how effectively trainees use the training when they return to their work.
32. SUMMARY Data is critical to show the value and effectiveness of Learning and Performance Programs Performance is what happens after a Learning Program, so the most important data pertains to how work has changed
33. SUMMARY Success starts with identifying desired impacts during the needs assessment Managers of program participants are the key stakeholders for driving successful learning and performance programs
34. QUESTIONS?David Kelly E-Mail: LnDDave@gmail.com Phone: (516) 474-1852 Twitter: @LnDDave Blog: http://davidkelly.me Also connect via LinkedIn or Facebook