Bottom-line Evaluation: Measuring Results from


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Bottom-line Evaluation: Measuring Results from

  1. 1. Bottom-line Evaluation: Measuring Results from Training and Performance Improvement Tuesday May 19, 2003 8:00-9:30 a.m. ASTD International Conference Donald J. Ford, Ph.D., CPT Training Education Management Redondo Beach, CA 90277
  2. 2. Bio g raphy <ul><li>Donald J. Ford, Ph.D ., C.P.T ., is president and owner of Training Education Management , a training and performance improvement consulting firm in Redondo Beach, California. Dr. Ford also teaches graduate courses in Human Resource Development for Cal State Northridge and Antioch University, L.A. His consulting clients include: ASTD, Toyota, Nissan Motors U.S.A., Rockwell International, U.S. Marines Corps, Orange County Transportation Authority, Glendale Memorial Hospital, and National Education Corp., among others. He has worked in the field of human resource development for eighteen years, including training management positions at Southern California Gas Company, Magnavox, Allied-Signal and Texas Instruments. He possesses another eight years of experience in public education at the secondary, adult and college levels. </li></ul><ul><li>He holds a B.A. and M.A. in history and a Ph.D. in education, all from UCLA. He has published 35 articles and three books on topics in training, education and business. He edited a book entitled, In Action: Designing Training Programs published by ASTD in 1996. His third book entitled, Bottom-Line Training: How to Design and Implement Programs That Boost Profits, was released by Gulf Publishing in July, 1999 and forms the basis for this presentation. Dr. Ford has presented at the ASTD International Conference four times, and at ASTD L.A. conferences and workshops. </li></ul><ul><li>He may be reached at: </li></ul><ul><li>1874 Pacific Coast Highway, Suite 205 </li></ul><ul><li>Redondo Beach, CA 90277 </li></ul><ul><li>Telephone/Fax: 310-316-2240 </li></ul><ul><li>E-mail: </li></ul><ul><li>Web: </li></ul>
  3. 3. Bottom-Line Training Design Model <ul><li>Learning </li></ul><ul><li>Performance </li></ul><ul><li>Financial </li></ul><ul><li>Strategic </li></ul>Results <ul><li>Needs Analysis </li></ul><ul><li>Performance Analysis </li></ul><ul><li>Needs assessment </li></ul><ul><li>Job/Task Analysis </li></ul><ul><li>Learner Analysis </li></ul><ul><li>Context Analysis </li></ul><ul><li>Skill Gap Analysis </li></ul>Analyze <ul><li>Objective </li></ul><ul><li>Deliverables/Strategies </li></ul><ul><li>Budgets/Schedules </li></ul><ul><li>Project management </li></ul><ul><li>Blueprints/Prototypes </li></ul>Design <ul><li>Materials </li></ul><ul><li>Tests/Assessments </li></ul><ul><li>Quality Control </li></ul><ul><li>Production </li></ul>Develop <ul><li>Train the Trainer </li></ul><ul><li>Classroom Delivery </li></ul><ul><li>Non-Classroom Delivery </li></ul>Implement <ul><li>Formative Evaluation </li></ul><ul><li>Reactions/Learning </li></ul><ul><li>Transfer of Training </li></ul><ul><li>Business Results </li></ul>Evaluate
  4. 4. Kirkpatrick Evaluation Model Work Environment <ul><li>Level 4: </li></ul><ul><li>Results </li></ul><ul><ul><li>Performance </li></ul></ul><ul><ul><li>Financial </li></ul></ul><ul><li>Level 3: </li></ul><ul><li>Job Behavior </li></ul><ul><ul><li>Learner </li></ul></ul><ul><ul><li>Organization </li></ul></ul>Training Environment <ul><li>Level 1: Reactions </li></ul><ul><ul><li>Learner </li></ul></ul><ul><ul><li>Client </li></ul></ul><ul><li>Level 2: </li></ul><ul><li>Learning </li></ul><ul><ul><li>Learner </li></ul></ul><ul><ul><li>Organization </li></ul></ul>Learning Event
  5. 5. Typical Evaluation Points Analyze Design/ Develop Implement Evaluate <ul><li>Evaluation Design </li></ul><ul><li>Tests </li></ul><ul><li>Surveys </li></ul><ul><li>Formative Evaluation </li></ul><ul><li>Reactions </li></ul><ul><li>Pre-Test </li></ul><ul><li>Post-Test </li></ul><ul><li>Skills Transfer </li></ul><ul><li>Performance </li></ul><ul><li>Business </li></ul><ul><li>Results </li></ul><ul><li>Data Analysis </li></ul><ul><li>Evaluation Reports </li></ul><ul><li>Baseline Data: </li></ul><ul><li>Learners Performance </li></ul><ul><li>Skill Gaps </li></ul><ul><li>Business Measures </li></ul>
  6. 6. Two Types of Evaluation <ul><li>Formative </li></ul><ul><ul><li>Improve the process & the program </li></ul></ul><ul><ul><li>Focus on process </li></ul></ul><ul><li>Measures: </li></ul><ul><ul><li>Acceptance </li></ul></ul><ul><ul><li>Satisfaction </li></ul></ul><ul><ul><li>Quality </li></ul></ul><ul><li>Summative </li></ul><ul><ul><li>Sum up the impact, Go-No Go </li></ul></ul><ul><ul><li>Focus on results </li></ul></ul><ul><li>Measures: </li></ul><ul><ul><li>Performance change </li></ul></ul><ul><ul><li>Business results </li></ul></ul><ul><ul><li>ROI </li></ul></ul>
  7. 7. HPI Evaluation Model Business Analysis Performance Analysis Cause Analysis Intervention Selection Intervention Implementation Evaluation Evaluation Baseline Formative Evaluation Summative Evaluation Business Goals Performance Gaps Root Causes Solutions Project Management Change Management Business Measures Performance Measures Analysis Evaluation Evaluation Design Customer Satisfaction Performance and ROI HPI Outputs Evaluation Process HPI Process Evaluation Outputs
  8. 8. Figure 21-1: One -Shot Case Study Design T M T = training M = measurement Figure 21-2: One-Group Pre-Post Evaluation Design M = measurement (1 = Pre, 2 = Post) T = training M 1 M 2 T Figure 21-5: Pre-Post Control Group (Classic Experimental Design) M 1 M 2 Group 1: R M 1 M 2 T Group 2: R R = randomize T = training M = measurement (1 = Pre, 2 = Post)
  9. 9. Figure 21-7: Interrupted Time Series Design <ul><li>Examples: </li></ul><ul><li>Accident rates pre and post safety training </li></ul><ul><li>Sales revenues pre and post sales training </li></ul><ul><li>Error rates pre and post quality training </li></ul><ul><li>Turnover rate pre and post management training </li></ul>M 2 M 5 T M 1 M 6 M 3 M 4 M 1- 6 = measures at regular intervals T = training
  10. 10. A Training Parable <ul><li>One day in Training Land, the Senior Vice-President of the company’s largest division came to request training for all of his 2,000 employees. His reason is that the company is about to launch a new line of products and services and he fears employees will not be able to deliver these successfully without being retrained. Training needs to start in six months. </li></ul><ul><li>The content must include: product features and benefits, sales techniques, customer service and a customer information system. </li></ul><ul><li>Your assignment is to design an evaluation plan for this request. </li></ul>
  11. 11. Figure 24-1: Bottom-Line Impact of Training Strategic Growth Revenues Savings Training Cost Avoidance
  12. 12. Sales Training Impact Monthly Sales <ul><li>What was the impact of the sales training on market share? </li></ul><ul><li>If the total annual market for these drugs is $10 million, what was the financial benefit of the sales training? </li></ul>Cancer Drug Sales as % of Market Share 7 8 7 9 10 10 10 7 6 9 7 7 #5 Cp 10 9 10 11 10 9 7 8 10 11 10 8 #4 Cp 14 15 14 14 12 12 13 15 14 13 14 15 #3 Cp 21 20 21 19 20 19 21 20 22 19 21 20 #2 Cp 42 41 39 40 42 43 44 46 44 43 47 46 #1 Cp 6% 6% 8% 7% 6% 7% 5% 3% 4% 5% 3% 4% Us 12/02 11/02 10/02 9/02 8/02 7/02 6/02 5/02 4/02 3/02 2/02 1/02 Product New Sales Training
  13. 13. Figure 23-4: Productivity Gains Caused by Workplace Literacy Training Program <ul><li>What conclusions can you draw from from this chart about the impact of the training? </li></ul><ul><li>If each percentage increase in productivity was worth an average of $5,000 what would be the total value (benefit) of this training? </li></ul>Productivity Level (% of Ideal) Training Group Monthly Productivity Average Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Training Ends = Control Group Productivity Level 0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 Training Begins
  14. 14. Training Return on Investment (ROI) <ul><li>Total benefits ÷ total costs </li></ul><ul><li>Example: </li></ul><ul><ul><li>$ 750,000 total benefits </li></ul></ul><ul><ul><li>$ 187,500 in total costs </li></ul></ul><ul><ul><li>750,000/187,500 = 4:1 </li></ul></ul><ul><ul><li>Company receives $4 in return for each $1 spent or a return of 400% </li></ul></ul>
  15. 15. ROI Example $$ <ul><li>What is the ROI of this training? </li></ul>$750,000 Indirect (lower turnover, absenteeism, higher productivity) $750,000 Direct (sales increase) Benefits $500,000 Indirect (participant salaries) $250,000 Direct (delivery) $250,000 Development Costs Category
  16. 16. Evaluating Behavior, Performance & Skill Transfer <ul><li>Success factors: </li></ul><ul><li>Learned new skills </li></ul><ul><li>Desire to change </li></ul><ul><li>Supportive work climate </li></ul><ul><li>Supportive supervision </li></ul><ul><li>Lack of barriers </li></ul><ul><li>Rewards and incentives </li></ul>
  17. 17. Performance Measurement Methods <ul><li>Action planning </li></ul><ul><li>Pre-post surveys </li></ul><ul><li>Pre-post interviews </li></ul><ul><li>Job observation </li></ul><ul><li>Performance documentation </li></ul><ul><ul><li>Output </li></ul></ul><ul><ul><li>Quality </li></ul></ul><ul><ul><li>Cost </li></ul></ul><ul><ul><li>Time </li></ul></ul><ul><ul><li>Behavior </li></ul></ul>
  18. 18. Figure 23-1: Sample Action Plan for Customer Service Training <ul><li>Summary of Skills Learned </li></ul><ul><li>In this course, I learned the following new customer service skills: </li></ul><ul><ul><li>How to greet customers. </li></ul></ul><ul><ul><li>How to identify customer needs. </li></ul></ul><ul><ul><li>How to locate customer account information in the customer database. </li></ul></ul><ul><ul><li>How to make simple changes to an account, like updating the customer’s name, address, applying payments, waiving late charges and fees. </li></ul></ul><ul><ul><li>How to handle customer complaints and document problems. </li></ul></ul><ul><li>Ways the New Skills Can be Applied </li></ul><ul><li>Once I return to work, I can apply these new skills in the following ways: </li></ul><ul><ul><li>Use the standard customer greeting I learned in class with every customer. </li></ul></ul><ul><ul><li>Ask questions of the customer, following the Inquiry Matrix I learned about in class. </li></ul></ul><ul><ul><li>Listen actively to identify the customer’s real need. </li></ul></ul><ul><ul><li>Use the Customer Database (CD) to access account number in the system. </li></ul></ul><ul><ul><li>Use the system features to update the customer’s account. </li></ul></ul><ul><ul><li>Handle any complaints by using the Customer Complaint Resolution Job Aid. </li></ul></ul>
  19. 19. Sample Action Plan for Customer Service Training, continued <ul><li>Obstacles or Barriers to Skills Application </li></ul><ul><li>Barriers include: </li></ul><ul><ul><li>Lack of time to practice on the job. </li></ul></ul><ul><ul><li>Waiting time for system to access account information. </li></ul></ul><ul><ul><li>Some customers do not speak English well and are hard to understand. </li></ul></ul><ul><ul><li>Some customers are already so upset by the time they call that it is difficult to solve their problem. </li></ul></ul><ul><li>Enablers for Skill Application </li></ul><ul><li>Enablers include: </li></ul><ul><ul><li>My supervisor wants met to succeed and is willing to give me time to learn the new skills. </li></ul></ul><ul><ul><li>My co-workers are willing to help me and answer questions when they have time. </li></ul></ul><ul><ul><li>I like helping people and will try to satisfy upset customers. </li></ul></ul><ul><ul><li>The computer system is being improved to speed up access time. </li></ul></ul>
  20. 20. Sample Action Plan for Customer Service Training, continued <ul><li>Specific Steps To Be Taken and Deadlines </li></ul><ul><li>Step: Deadline: </li></ul><ul><li>Use standard greeting on all calls. Tomorrow </li></ul><ul><li>Begin using the Inquiry Matrix on all calls. Tomorrow </li></ul><ul><li>Practice looking up customer information. Tomorrow </li></ul><ul><li>Become proficient on the Customer Database System. One month from now </li></ul><ul><li>Use the Customer Complaint Resolution Job Aid to One Month from now </li></ul><ul><li>resolve all customer complaints. </li></ul><ul><li>Follow up Actions to be Taken to Review Progress </li></ul><ul><ul><li>Meet with my boss to discuss this action plan within one week. </li></ul></ul><ul><ul><li>Begin using the job aids immediately to help me. </li></ul></ul><ul><ul><li>Ask co-workers for help when I need it. </li></ul></ul><ul><ul><li>Register to attend another customer service class next month. </li></ul></ul>
  21. 21. District Manager Training Pre-Post Survey <ul><li>What conclusions would you draw about the training’s impact on supervisors’ job behavior? </li></ul><ul><li>What recommendations would you make about changes to the program based on this data? </li></ul><ul><li>How could you translate this data into financial results? </li></ul>10% 3.4 3.1 3.7 3.7 2.9 2.5 3.3 3.0 3.8 3.2 Provides coaching and on the job training 29% 3.6 2.8 3.6 3.5 3.9 3.1 2.7 1.9 4.1 2.7 Establishes clear performance expectations and goals 21% 3.4 2.8 3.3 3.0 2.9 2.1 3.0 2.4 4.5 3.5 Communicates effectively with employees Post Pre Post Pre Post Pre Post Pre Post Pre This manager % Gain Average Colleague Employee Manager Self Question
  22. 22. Balanced Scorecard <ul><li>Training Activity </li></ul><ul><ul><li>% Payroll US Avg.=1.9% ‡ </li></ul></ul><ul><ul><li>$/Employee/year US Avg.=$569 ‡ </li></ul></ul><ul><ul><li>Hrs./Empl./year US Avg.= 36 ‡ </li></ul></ul><ul><li>Training Efficiency </li></ul><ul><ul><li>Cost/Hour US Avg.=$28 ‡ </li></ul></ul><ul><ul><li>% Billable Hrs. US Avg.=70% † </li></ul></ul><ul><li>Training Results </li></ul><ul><ul><li>% Positive Course Ratings Avg.=91% † </li></ul></ul><ul><ul><li>% Learning Gain Avg.=45% † </li></ul></ul><ul><ul><li>% Behavior Change Avg.=25% † </li></ul></ul><ul><ul><li>Revenues/Employee Avg.=$100K † </li></ul></ul><ul><ul><li>Market/Book † Avg.=10:1 </li></ul></ul><ul><ul><li>† Source=Author’s Study of High Tech firms </li></ul></ul><ul><li>‡ Source: ASTD Training Data Book , 1996. </li></ul>
  23. 23. Your Scorecard <ul><li>Training Activity </li></ul><ul><ul><li>% Payroll:_____ </li></ul></ul><ul><li>Training budget/Payroll </li></ul><ul><ul><li>Hrs./Empl.____ </li></ul></ul><ul><li>Training hours/Employees </li></ul><ul><ul><li>$/Empl. _____ </li></ul></ul><ul><li>Training budget/Employees </li></ul><ul><li>Training Efficiency </li></ul><ul><ul><li>Cost/Hr. ______ </li></ul></ul><ul><li>Training budget/ training hours </li></ul><ul><ul><li>% Billable ____ </li></ul></ul><ul><li>Billable hours/staff hours </li></ul><ul><li>Training Results </li></ul><ul><ul><li>% Positive Ratings </li></ul></ul><ul><li> Positive ratings/total ratings _________________ </li></ul><ul><ul><li>% Learning Gain </li></ul></ul><ul><li> Post-pre / Pre-test scores </li></ul><ul><li> _________________ </li></ul><ul><ul><li>% Performance Gain </li></ul></ul><ul><li> Post-pre / Pre-performance </li></ul><ul><li> _________________ </li></ul><ul><ul><li>Revenues/Employee </li></ul></ul><ul><li> Total revenues/employees </li></ul><ul><li> _________________ </li></ul><ul><ul><li>Market/Book Ratio </li></ul></ul><ul><li> Total market worth/ total physical assets </li></ul>
  24. 24. Training Design Job Aid <ul><li>Analysis Phase </li></ul><ul><ul><li>Needs Analysis </li></ul></ul><ul><ul><li>Performance Analysis </li></ul></ul><ul><ul><li>Needs Assessment </li></ul></ul><ul><ul><li>Job Analysis </li></ul></ul><ul><ul><li>Task Analysis </li></ul></ul><ul><ul><li>Learner Analysis </li></ul></ul><ul><ul><li>Context Analysis </li></ul></ul><ul><ul><li>Skill Gap Analysis </li></ul></ul><ul><li>Design Phase </li></ul><ul><ul><li>Training Objectives </li></ul></ul><ul><ul><li>Deliverables/Strategies </li></ul></ul><ul><ul><li>Budgets/Schedules </li></ul></ul><ul><ul><li>Project management </li></ul></ul><ul><ul><li>Blueprints/Prototypes </li></ul></ul><ul><li>Development Phase </li></ul><ul><ul><li>Draft Materials </li></ul></ul><ul><ul><li>Tests/Assessment </li></ul></ul><ul><ul><li>Quality Control </li></ul></ul><ul><ul><li>Full-scale Production </li></ul></ul>
  25. 25. Training Design Job Aid, continued <ul><li>Implementation Phase </li></ul><ul><ul><li>Train the Trainer </li></ul></ul><ul><ul><li>Classroom Delivery </li></ul></ul><ul><ul><li>Non-Classroom Delivery </li></ul></ul><ul><li>Evaluation Phase </li></ul><ul><ul><li>Formative </li></ul></ul><ul><ul><li>Reactions </li></ul></ul><ul><ul><li>Learning </li></ul></ul><ul><ul><li>Transfer to Job </li></ul></ul><ul><ul><li>Results/ROI </li></ul></ul>
  26. 26. Closing comments and Q&A Donald J. Ford, Ph.D., CPT Training Education Management Redondo Beach, CA 90277 Telephone/Fax: 310-316-2240 E-mail: Web: