Start with the story of me finding the binders of program evaluations
Truths About Training Evaluation
TRUTHS ABOUTTRAININGEVALUATION David Kelly, CPLP, CRP
A LITTLE ABOUT ME Training Director for over 10 Years Director, Center for Learning at ACLD Certified Professional in Learning and Performance (CPLP) Certified ROI Professional (CRP) Board Member of ASTD Long Island Member of ASTD’s National Advisors for Chapters Member of the eLearn Magazine Editorial Board
TODAY’S DISCUSSION A Brief History of Measurement in Learning and Development Programs What do Trainers Measure? Does What We Measure Even Matter? The Question of Credibility A Better Definition for Training Success
WHAT CAME BEFORE… Why are we talking about ‘evaluation’ of training in the first place?
A BRIEF HISTORY OF TRAINING EVALUATIONDon Kirkpatrick – The Four Levels of Evaluation 1. Reaction 2. Learning 3. Behavior 4. ResultsJack Phillips – ROI Methodology 5. ROI (Benefits / Costs)
CAPITAL NON-CAPITALINVESTMENTS INVESTMENTS Buildings Marketing Equipment Human Resources Tools Quality Vehicles Some Technology Companies Staff Support Some Technology Processes ROI is usually ROI is not typically measured measuredWHY MEASURE TRAINING?
CAPITAL NON-CAPITALINVESTMENTS INVESTMENTS Buildings Marketing Equipment Human Resources Tools Quality Vehicles Some Technology Companies Staff Support Some Technology Processes ~15% of Total ~85% of Total Expenditures ExpendituresWHY MEASURE TRAINING?
SATISFACTION WITH MEASURES OF SUCCESSOF LEARNING AND DEVELOPMENTTo what extent are you satisfied with themeasures (value) of learning anddevelopment?1. Very Dissatisfied – 8%2. Dissatisfied – 45%3. Satisfied – 41%4. Very Satisfied – 6%*CEO Survey—Fortune 500 and LargePrivate Companies, ROI Institute
COMMON LEARNING METRICS Cost per Employee or Hour Do these metrics Course Completions REALLY matter? Learner Satisfaction Attendance Test Scores
MEANINGFUL PERFORMANCE METRICS Decreased Time to Performance Reduced Performance Cycle Times Quality Improvements Increased Customer Satisfaction Increased Sales
HERE’S MY TWO CENTS… Measurement for measurement’s sake is a waste of time. Ask yourself… “Will I be doing anything with the data we’re collecting?”
THE CREDIBILITY ISSUEThe truth is most training evaluation data is either irrelevant or does not hold up to scrutiny *Bozarth-Ferguson Magic Formula used with permission
THE CREDIBILITY ISSUETraining: “Sales increased 40% after Training.”Marketing: “That wasn’t training, we ranadvertisements during that period.”Training: “Turnover has decreased 25% as aresult of our new orientation programs.”Human Resources: “That wasn’t training, weupdated hiring criteria.”
THE CREDIBILITY ISSUE Source: http://elearningindustry.com/subjects/free-elearning-resources/item/256-free-elearning-roi-
THE BOTTOM LINEROI and the metrics used for TrainingEvaluation are driven by companies andconsultants with a vested interest inpeople believing these metrics matter.In most cases, the data collected isn’treally relevant to business outcomes.
HERE’S MY TWO CENTS… It’s not about Learning; It’s about Performance It’s not about what people KNOW; it’s about what people DO.
HERE’S MY TWO CENTS… Most executives that ask for an ROI, DO NOT WANT AN ACTUAL ROI. They want to know that the efforts had value.
MAYBE IT’S NOT ABOUT EVALUATION… What other ways can we determine success besides traditional training evaluation?
WHO DECIDES WHAT SUCCESS MEANS?The Program Stakeholders Executives Management Approving Financing of Training Learning and Development Professionals Program Participants Managers of Program Participants
HERE’S MY TWO CENTS… The time to think about how you will define success is during the DESIGN of a program, not after it’s completion.
WHAT’S YOUR STORY? A strong testimonial from a participant or executive on how training helped workers perform better is often more powerful than any number on a spreadsheet.
THE SUCCESS CASE METHOD Step 1. Identify targeted business goals and impact expectations.
THE SUCCESS CASE METHOD Step 2. Survey a large representative sample of all participants in a program to identify high impact and low impact cases.
THE SUCCESS CASE METHOD Step 3. Analyze the survey data to identify: a small group of successful participants a small group unsuccessful participants
THE SUCCESS CASE METHOD Step 4. Conduct interviews with the two selected groups to: Document the nature and business value of their application of learning Identify the performance factors that supported learning application and obstacles that prevented it.
THE SUCCESS CASE METHOD Step 5. Document and disseminate the story: Report impact Applaud successes Use data to educate managers and organization
IT’S NOT ABOUT HOW MANY SEATS ARE FILLED It’s not about the Training, it’s about the Performance Improvement. It’s about an ongoing process, not an event. “Evaluating training is like evaluating the wedding instead of the marriage.” - Robert Brinkhoff
IT’S NOT ABOUT HOW MANY SEATS ARE FILLED Most learning and performance improvement takes place informally, and on the job. The success of a training program is not determined via a test; it is determined by how effectively trainees use the training when they return to their work.
SUMMARY Data is critical to show the value and effectiveness of Learning and Performance Programs Performance is what happens after a Learning Program, so the most important data pertains to how work has changed
SUMMARY Success starts with identifying desired impacts during the needs assessment Managers of program participants are the key stakeholders for driving successful learning and performance programs
QUESTIONS?David Kelly E-Mail: LnDDave@gmail.com Phone: (516) 474-1852 Twitter: @LnDDave Blog: http://davidkelly.me Also connect via LinkedIn or Facebook